{"thread":{"id":"60908","subject":"[PATCH 00/12] FSMonitor edge cases on case-insensitive file systems","startedAt":"2024-02-13T20:52:24Z","lastAt":"2024-03-06T18:20:10Z","messageCount":91,"participants":["Jeff Hostetler via GitGitGadget","Junio C Hamano","Patrick Steinhardt","Jeff Hostetler","Torsten Bögershausen"],"isPatch":true,"patchVersion":1,"patchTotal":12},"messages":[{"id":"488582","messageId":"pull.1662.git.1707857541.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":null,"subject":"[PATCH 00/12] FSMonitor edge cases on case-insensitive file systems","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:09Z","receivedAt":"2024-02-13T20:52:24Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Fix FSMonitor client code to detect case-incorrect FSEvents and map them to\nthe canonical case expected by the index.\n\nFSEvents are delivered to the FSMonitor daemon using the observed case which\nmay or may not match the expected case stored in the index for tracked files\nand/or directories. This caused index_name_pos() to report a negative index\nposition (defined as the suggested insertion point). Since the value was\nnegative, the FSMonitor refresh lookup would not invalidate the\nCE_FSMONITOR_VALID bit on the \"expected\" (case-insensitive-equivalent)\ncache-entries. Therefore, git status would not report them as modified.\n\nThis was a fairly obscure problem and only happened when the case of a\nsub-directory or a file was artificially changed.\n\nThis first runs the original lookup using the observed case. If that fails,\nit assumes that the observed pathname refers to a file and uses the\ncase-insensitive name-hash hashmap to find an equivalent path (cache-entry)\nin the index. If that fails, it assumes the pathname refers to a directory\nand uses the case-insensitive dir-name-hash to find the equivalent directory\nand then repeats the index_name_pos() lookup to find a directory or\nsuggested insertion point with the expected case.\n\nTwo new test cases were added to t7527 to demonstrate this.\n\nSince this was rather obscure, I also added some additional tracing under\nthe GIT_TRACE_FSMONITOR key.\n\nI also did considerable refactoring of the original code before adding the\nnew lookups.\n\nFinally, I made more explicit the relationship between the FSEvents and the\n(new) sparse-index directory cache-entries, since sparse-index was added\nslightly after the FSMonitor feature.\n\nJeff Hostetler (12):\n sparse-index: pass string length to index_file_exists()\n name-hash: add index_dir_exists2()\n t7527: add case-insensitve test for FSMonitor\n fsmonitor: refactor refresh callback on directory events\n fsmonitor: refactor refresh callback for non-directory events\n fsmonitor: clarify handling of directory events in callback\n fsmonitor: refactor untracked-cache invalidation\n fsmonitor: support case-insensitive directory events\n fsmonitor: refactor non-directory callback\n fsmonitor: support case-insensitive non-directory events\n fsmonitor: refactor bit invalidation in refresh callback\n t7527: update case-insenstive fsmonitor test\n\n fsmonitor.c | 338 +++++++++++++++++++++++++++++------\n name-hash.c | 16 ++\n name-hash.h | 2 +\n sparse-index.c | 4 +-\n t/t7527-builtin-fsmonitor.sh | 220 +++++++++++++++++++++++\n 5 files changed, 522 insertions(+), 58 deletions(-)\n\n\nbase-commit: 3526e67d917bcd03f317a058208fa02737654637\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1662%2Fjeffhostetler%2Ffsmonitor-ignore-case-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1662/jeffhostetler/fsmonitor-ignore-case-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/1662\n-- \ngitgitgadget\n"},{"id":"488583","messageId":"6f81e2e30609c70d4dcdbe9d4f11c4f6b5173c77.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 01/12] sparse-index: pass string length to index_file_exists()","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:10Z","receivedAt":"2024-02-13T20:52:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nThe call to index_file_exists() in the loop in expand_to_path() passes\nthe wrong string length. Let's fix that.\n\nThe loop in expand_to_path() searches the name-hash for each\nsub-directory prefix in the provided pathname. That is, by searching\nfor \"dir1/\" then \"dir1/dir2/\" then \"dir1/dir2/dir3/\" and so on until\nit finds a cache-entry representing a sparse directory.\n\nThe code creates \"strbuf path_mutable\" to contain the working pathname\nand modifies the buffer in-place by temporarily replacing the character\nfollowing each successive \"/\" with NUL for the duration of the call to\nindex_file_exists().\n\nIt does not update the strbuf.len during this substitution.\n\nPass the patched length of the prefix path instead.\n\nSigned-off-by: Jeff Hostetler \n---\n sparse-index.c | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/sparse-index.c b/sparse-index.c\nindex 3578feb2837..e48e40cae71 100644\n--- a/sparse-index.c\n+++ b/sparse-index.c\n@@ -579,8 +579,9 @@ void expand_to_path(struct index_state *istate,\n \t\treplace++;\n \t\ttemp = *replace;\n \t\t*replace = '\\0';\n+\t\tsubstr_len = replace - path_mutable.buf;\n \t\tif (index_file_exists(istate, path_mutable.buf,\n-\t\t\t\t path_mutable.len, icase)) {\n+\t\t\t\t substr_len, icase)) {\n \t\t\t/*\n \t\t\t * We found a parent directory in the name-hash\n \t\t\t * hashtable, because only sparse directory entries\n@@ -593,7 +594,6 @@ void expand_to_path(struct index_state *istate,\n \t\t}\n \n \t\t*replace = temp;\n-\t\tsubstr_len = replace - path_mutable.buf;\n \t}\n \n cleanup:\n-- \ngitgitgadget\n\n"},{"id":"488584","messageId":"3464545fe3feceb08408618c77a70cc95745bfe9.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 02/12] name-hash: add index_dir_exists2()","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:11Z","receivedAt":"2024-02-13T20:52:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a new version of index_dir_exists() to return the canonical\nspelling of the matched directory prefix.\n\nThe existing index_dir_exists() returns a boolean to indicate if\nthere is a case-insensitive match in the directory name-hash, but\nit doesn't tell the caller the exact spelling of that match.\n\nThe new version also copies the matched spelling to a provided strbuf.\nThis lets the caller, for example, then call index_name_pos() with the\ncorrect case to search the cache-entry array for the real insertion\nposition.\n\nSigned-off-by: Jeff Hostetler \n---\n name-hash.c | 16 ++++++++++++++++\n name-hash.h | 2 ++\n 2 files changed, 18 insertions(+)\n\ndiff --git a/name-hash.c b/name-hash.c\nindex 251f036eef6..d735c81acb3 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -694,6 +694,22 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n \tdir = find_dir_entry(istate, name, namelen);\n \treturn dir && dir->nr;\n }\n+int index_dir_exists2(struct index_state *istate, const char *name, int namelen,\n+\t\t struct strbuf *canonical_path)\n+{\n+\tstruct dir_entry *dir;\n+\n+\tstrbuf_init(canonical_path, namelen+1);\n+\n+\tlazy_init_name_hash(istate);\n+\texpand_to_path(istate, name, namelen, 0);\n+\tdir = find_dir_entry(istate, name, namelen);\n+\n+\tif (dir && dir->nr)\n+\t\tstrbuf_add(canonical_path, dir->name, dir->namelen);\n+\n+\treturn dir && dir->nr;\n+}\n \n void adjust_dirname_case(struct index_state *istate, char *name)\n {\ndiff --git a/name-hash.h b/name-hash.h\nindex b1b4b0fb337..2fcac5c4870 100644\n--- a/name-hash.h\n+++ b/name-hash.h\n@@ -5,6 +5,8 @@ struct cache_entry;\n struct index_state;\n \n int index_dir_exists(struct index_state *istate, const char *name, int namelen);\n+int index_dir_exists2(struct index_state *istate, const char *name, int namelen,\n+\t\t struct strbuf *canonical_path);\n void adjust_dirname_case(struct index_state *istate, char *name);\n struct cache_entry *index_file_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n \n-- \ngitgitgadget\n\n"},{"id":"488585","messageId":"272d7805f474411bd30aee8a55f497d6689afb57.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 03/12] t7527: add case-insensitve test for FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:12Z","receivedAt":"2024-02-13T20:52:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nThe FSMonitor client code trusts the spelling of the pathnames in the\nFSEvents received from the FSMonitor daemon. On case-insensitive file\nsystems, these OBSERVED pathnames may be spelled differently than the\nEXPECTED pathnames listed in the .git/index. This causes a miss when\nusing `index_name_pos()` which expects the given case to be correct.\n\nWhen this happens, the FSMonitor client code does not update the state\nof the CE_FSMONITOR_VALID bit when refreshing the index (and before\nstarting to scan the worktree).\n\nThis results in modified files NOT being reported by `git status` when\nthere is a discrepancy in the case-spelling of a tracked file's\npathname.\n\nThis commit contains a (rather contrived) test case to demonstrate\nthis. A later commit in this series will update the FSMonitor client\ncode to recognize these discrepancies and update the CE_ bit accordingly.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 217 +++++++++++++++++++++++++++++++++++\n 1 file changed, 217 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 78503158fd6..5cd68b2ea82 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -1037,4 +1037,221 @@ test_expect_success 'split-index and FSMonitor work well together' '\n \t)\n '\n \n+# The FSMonitor daemon reports the OBSERVED pathname of modified files\n+# and thus contains the OBSERVED spelling on case-insensitive file\n+# systems. The daemon does not (and should not) load the .git/index\n+# file and therefore does not know the expected case-spelling. Since\n+# it is possible for the user to create files/subdirectories with the\n+# incorrect case, a modified file event for a tracked will not have\n+# the EXPECTED case. This can cause `index_name_pos()` to incorrectly\n+# report that the file is untracked. This causes the client to fail to\n+# mark the file as possibly dirty (keeping the CE_FSMONITOR_VALID bit\n+# set) so that `git status` will avoid inspecting it and thus not\n+# present in the status output.\n+#\n+# The setup is a little contrived.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n+\ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n+\n+\tgit init subdir_case_wrong &&\n+\t(\n+\t\tcd subdir_case_wrong &&\n+\t\techo x >AAA &&\n+\t\techo x >BBB &&\n+\n+\t\tmkdir dir1 &&\n+\t\techo x >dir1/file1 &&\n+\t\tmkdir dir1/dir2 &&\n+\t\techo x >dir1/dir2/file2 &&\n+\t\tmkdir dir1/dir2/dir3 &&\n+\t\techo x >dir1/dir2/dir3/file3 &&\n+\n+\t\techo x >yyy &&\n+\t\techo x >zzz &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"data\" &&\n+\n+\t\t# This will cause \"dir1/\" and everything under it\n+\t\t# to be deleted.\n+\t\tgit sparse-checkout set --cone --sparse-index &&\n+\n+\t\t# Create dir2 with the wrong case and then let Git\n+\t\t# repopulate dir3 -- it will not correct the spelling\n+\t\t# of dir2.\n+\t\tmkdir dir1 &&\n+\t\tmkdir dir1/DIR2 &&\n+\t\tgit sparse-checkout add dir1/dir2/dir3\n+\t) &&\n+\n+\tstart_daemon -C subdir_case_wrong --tf \"$PWD/subdir_case_wrong.trace\" &&\n+\n+\t# Enable FSMonitor in the client. Run enough commands for\n+\t# the .git/index to sync up with the daemon with everything\n+\t# marked clean.\n+\tgit -C subdir_case_wrong config core.fsmonitor true &&\n+\tgit -C subdir_case_wrong update-index --fsmonitor &&\n+\tgit -C subdir_case_wrong status &&\n+\n+\t# Make some files dirty so that FSMonitor gets FSEvents for\n+\t# each of them.\n+\techo xx >>subdir_case_wrong/AAA &&\n+\techo xx >>subdir_case_wrong/dir1/DIR2/dir3/file3 &&\n+\techo xx >>subdir_case_wrong/zzz &&\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/subdir_case_wrong.log\" \\\n+\t\tgit -C subdir_case_wrong --no-optional-locks status --short \\\n+\t\t\t>\"$PWD/subdir_case_wrong.out\" &&\n+\n+\t# \"git status\" should have gotten file events for each of\n+\t# the 3 files.\n+\t#\n+\t# \"dir2\" should be in the observed case on disk.\n+\tgrep \"fsmonitor_refresh_callback\" \\\n+\t\t<\"$PWD/subdir_case_wrong.log\" \\\n+\t\t>\"$PWD/subdir_case_wrong.log1\" &&\n+\n+\tgrep -q \"AAA.*pos 0\" \"$PWD/subdir_case_wrong.log1\" &&\n+\tgrep -q \"zzz.*pos 6\" \"$PWD/subdir_case_wrong.log1\" &&\n+\n+\tgrep -q \"dir1/DIR2/dir3/file3.*pos -3\" \"$PWD/subdir_case_wrong.log1\" &&\n+\n+\t# The refresh-callbacks should have caused \"git status\" to clear\n+\t# the CE_FSMONITOR_VALID bit on each of those files and caused\n+\t# the worktree scan to visit them and mark them as modified.\n+\tgrep -q \" M AAA\" \"$PWD/subdir_case_wrong.out\" &&\n+\tgrep -q \" M zzz\" \"$PWD/subdir_case_wrong.out\" &&\n+\n+\t# However, with the fsmonitor client bug, the \"(pos -3)\" causes\n+\t# the client to not update the bit and never rescan the file\n+\t# and therefore not report it as dirty.\n+\t! grep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n+'\n+\n+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n+\ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n+\n+\tgit init file_case_wrong &&\n+\t(\n+\t\tcd file_case_wrong &&\n+\t\techo x >AAA &&\n+\t\techo x >BBB &&\n+\n+\t\tmkdir dir1 &&\n+\t\tmkdir dir1/dir2 &&\n+\t\tmkdir dir1/dir2/dir3 &&\n+\t\techo x >dir1/dir2/dir3/FILE-3-B &&\n+\t\techo x >dir1/dir2/dir3/XXXX-3-X &&\n+\t\techo x >dir1/dir2/dir3/file-3-a &&\n+\t\techo x >dir1/dir2/dir3/yyyy-3-y &&\n+\t\tmkdir dir1/dir2/dir4 &&\n+\t\techo x >dir1/dir2/dir4/FILE-4-A &&\n+\t\techo x >dir1/dir2/dir4/XXXX-4-X &&\n+\t\techo x >dir1/dir2/dir4/file-4-b &&\n+\t\techo x >dir1/dir2/dir4/yyyy-4-y &&\n+\n+\t\techo x >yyy &&\n+\t\techo x >zzz &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"data\"\n+\t) &&\n+\n+\tstart_daemon -C file_case_wrong --tf \"$PWD/file_case_wrong.trace\" &&\n+\n+\t# Enable FSMonitor in the client. Run enough commands for\n+\t# the .git/index to sync up with the daemon with everything\n+\t# marked clean.\n+\tgit -C file_case_wrong config core.fsmonitor true &&\n+\tgit -C file_case_wrong update-index --fsmonitor &&\n+\tgit -C file_case_wrong status &&\n+\n+\t# Make some files dirty so that FSMonitor gets FSEvents for\n+\t# each of them.\n+\techo xx >>file_case_wrong/AAA &&\n+\techo xx >>file_case_wrong/zzz &&\n+\n+\t# Rename some files so that FSMonitor sees a create and delete\n+\t# FSEvent for each. (A simple \"mv foo FOO\" is not portable\n+\t# between macOS and Windows. It works on both platforms, but makes\n+\t# the test messy, since (1) one platform updates \"ctime\" on the\n+\t# moved file and one does not and (2) it causes a directory event\n+\t# on one platform and not on the other which causes additional\n+\t# scanning during \"git status\" which causes a \"H\" vs \"h\" discrepancy\n+\t# in \"git ls-files -f\".) So old-school it and move it out of the\n+\t# way and copy it to the case-incorrect name so that we get fresh\n+\t# \"ctime\" and \"mtime\" values.\n+\n+\tmv file_case_wrong/dir1/dir2/dir3/file-3-a file_case_wrong/dir1/dir2/dir3/ORIG &&\n+\tcp file_case_wrong/dir1/dir2/dir3/ORIG file_case_wrong/dir1/dir2/dir3/FILE-3-A &&\n+\trm file_case_wrong/dir1/dir2/dir3/ORIG &&\n+\tmv file_case_wrong/dir1/dir2/dir4/FILE-4-A file_case_wrong/dir1/dir2/dir4/ORIG &&\n+\tcp file_case_wrong/dir1/dir2/dir4/ORIG file_case_wrong/dir1/dir2/dir4/file-4-a &&\n+\trm file_case_wrong/dir1/dir2/dir4/ORIG &&\n+\n+\t# Run status enough times to fully sync.\n+\t#\n+\t# The first instance should get the create and delete FSEvents\n+\t# for each pair. Status should update the index with a new FSM\n+\t# token (so the next invocation will not see data for these\n+\t# events).\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try1.log\" \\\n+\t\tgit -C file_case_wrong status --short \\\n+\t\t\t>\"$PWD/file_case_wrong-try1.out\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try1.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*file-3-a.*pos 4\" \"$PWD/file_case_wrong-try1.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*FILE-4-A.*pos 6\" \"$PWD/file_case_wrong-try1.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try1.log\" &&\n+\n+\t# FSM refresh will have invalidated the FSM bit and cause a regular\n+\t# (real) scan of these tracked files, so they should have \"H\" status.\n+\t# (We will not see a \"h\" status until the next refresh (on the next\n+\t# command).)\n+\n+\tgit -C file_case_wrong ls-files -f >\"$PWD/file_case_wrong-lsf1.out\" &&\n+\tgrep -q \"H dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-lsf1.out\" &&\n+\tgrep -q \"H dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-lsf1.out\" &&\n+\n+\n+\t# Try the status again. We assume that the above status command\n+\t# advanced the token so that the next one will not see those events.\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try2.log\" \\\n+\t\tgit -C file_case_wrong status --short \\\n+\t\t\t>\"$PWD/file_case_wrong-try2.out\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*file-3-a.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*FILE-4-A.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*file-4-a.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\n+\t# FSM refresh saw nothing, so it will mark all files as valid,\n+\t# so they should now have \"h\" status.\n+\n+\tgit -C file_case_wrong ls-files -f >\"$PWD/file_case_wrong-lsf2.out\" &&\n+\tgrep -q \"h dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-lsf2.out\" &&\n+\tgrep -q \"h dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-lsf2.out\" &&\n+\n+\n+\t# We now have files with clean content, but with case-incorrect\n+\t# file names. Modify them to see if status properly reports\n+\t# them.\n+\n+\techo xx >>file_case_wrong/dir1/dir2/dir3/FILE-3-A &&\n+\techo xx >>file_case_wrong/dir1/dir2/dir4/file-4-a &&\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try3.log\" \\\n+\t\tgit -C file_case_wrong --no-optional-locks status --short \\\n+\t\t\t>\"$PWD/file_case_wrong-try3.out\" &&\n+\t# FSEvents are in observed case.\n+\tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n+\n+\t# Status should say these files are modified, but with the case\n+\t# bug, the \"pos -3\" cause the client to not update the FSM bit\n+\t# and never cause the file to be rescanned and therefore to not\n+\t# report it dirty.\n+\t! grep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n+\t! grep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"488586","messageId":"3fb8e0d0a7c0455cc7a5ba28c12736fd4bbbd44e.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 04/12] fsmonitor: refactor refresh callback on directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:13Z","receivedAt":"2024-02-13T20:52:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 52 ++++++++++++++++++++++++++++++----------------------\n 1 file changed, 30 insertions(+), 22 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex f670c509378..b1ef01bf3cd 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,6 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+static void fsmonitor_refresh_callback_slash(\n+\tstruct index_state *istate, const char *name, int len, int pos)\n+{\n+\tint i;\n+\n+\t/*\n+\t * The daemon can decorate directory events, such as\n+\t * moves or renames, with a trailing slash if the OS\n+\t * FS Event contains sufficient information, such as\n+\t * MacOS.\n+\t *\n+\t * Use this to invalidate the entire cone under that\n+\t * directory.\n+\t *\n+\t * We do not expect an exact match because the index\n+\t * does not normally contain directory entries, so we\n+\t * start at the insertion point and scan.\n+\t */\n+\tif (pos < 0)\n+\t\tpos = -pos - 1;\n+\n+\t/* Mark all entries for the folder invalid */\n+\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\tbreak;\n+\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t}\n+}\n+\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n@@ -193,28 +222,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t\t\t name, pos);\n \n \tif (name[len - 1] == '/') {\n-\t\t/*\n-\t\t * The daemon can decorate directory events, such as\n-\t\t * moves or renames, with a trailing slash if the OS\n-\t\t * FS Event contains sufficient information, such as\n-\t\t * MacOS.\n-\t\t *\n-\t\t * Use this to invalidate the entire cone under that\n-\t\t * directory.\n-\t\t *\n-\t\t * We do not expect an exact match because the index\n-\t\t * does not normally contain directory entries, so we\n-\t\t * start at the insertion point and scan.\n-\t\t */\n-\t\tif (pos < 0)\n-\t\t\tpos = -pos - 1;\n-\n-\t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = pos; i < istate->cache_nr; i++) {\n-\t\t\tif (!starts_with(istate->cache[i]->name, name))\n-\t\t\t\tbreak;\n-\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\t}\n+\t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n \n \t\t/*\n \t\t * We need to remove the traling \"/\" from the path\n-- \ngitgitgadget\n\n"},{"id":"488587","messageId":"0896d4af907d71df29b0c4f5a27d24ea80b3c0e1.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 05/12] fsmonitor: refactor refresh callback for non-directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:14Z","receivedAt":"2024-02-13T20:52:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 66 ++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 38 insertions(+), 28 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex b1ef01bf3cd..614270fa5e8 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,6 +183,42 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+static void fsmonitor_refresh_callback_unqualified(\n+\tstruct index_state *istate, const char *name, int len, int pos)\n+{\n+\tint i;\n+\n+\tif (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t} else {\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t}\n+\t}\n+}\n+\n static void fsmonitor_refresh_callback_slash(\n \tstruct index_state *istate, const char *name, int len, int pos)\n {\n@@ -214,7 +250,7 @@ static void fsmonitor_refresh_callback_slash(\n \n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n-\tint i, len = strlen(name);\n+\tint len = strlen(name);\n \tint pos = index_name_pos(istate, name, len);\n \n \ttrace_printf_key(&trace_fsmonitor,\n@@ -229,34 +265,8 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t\t * for the untracked cache.\n \t\t */\n \t\tname[len - 1] = '\\0';\n-\t} else if (pos >= 0) {\n-\t\t/*\n-\t\t * We have an exact match for this path and can just\n-\t\t * invalidate it.\n-\t\t */\n-\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\t/*\n-\t\t * The path is not a tracked file -or- it is a\n-\t\t * directory event on a platform that cannot\n-\t\t * distinguish between file and directory events in\n-\t\t * the event handler, such as Windows.\n-\t\t *\n-\t\t * Scan as if it is a directory and invalidate the\n-\t\t * cone under it. (But remember to ignore items\n-\t\t * between \"name\" and \"name/\", such as \"name-\" and\n-\t\t * \"name.\".\n-\t\t */\n-\t\tpos = -pos - 1;\n-\n-\t\tfor (i = pos; i < istate->cache_nr; i++) {\n-\t\t\tif (!starts_with(istate->cache[i]->name, name))\n-\t\t\t\tbreak;\n-\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n-\t\t\t\tbreak;\n-\t\t\tif (istate->cache[i]->name[len] == '/')\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\t}\n+\t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n \t}\n \n \t/*\n-- \ngitgitgadget\n\n"},{"id":"488588","messageId":"5b6f8bd1fe7b6c742b25a5a1ed95b528f352215e.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 06/12] fsmonitor: clarify handling of directory events in callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:15Z","receivedAt":"2024-02-13T20:52:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 47 +++++++++++++++++++++++++++++++++--------------\n 1 file changed, 33 insertions(+), 14 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 614270fa5e8..754fe20cfd0 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -219,24 +219,40 @@ static void fsmonitor_refresh_callback_unqualified(\n \t}\n }\n \n-static void fsmonitor_refresh_callback_slash(\n+/*\n+ * The daemon can decorate directory events, such as a move or rename,\n+ * by adding a trailing slash to the observed name. Use this to\n+ * explicitly invalidate the entire cone under that directory.\n+ *\n+ * The daemon can only reliably do that if the OS FSEvent contains\n+ * sufficient information in the event.\n+ *\n+ * macOS FSEvents have enough information.\n+ *\n+ * Other platforms may or may not be able to do it (and it might\n+ * depend on the type of event (for example, a daemon could lstat() an\n+ * observed pathname after a rename, but not after a delete)).\n+ *\n+ * If we find an exact match in the index for a path with a trailing\n+ * slash, it means that we matched a sparse-index directory in a\n+ * cone-mode sparse-checkout (since that's the only time we have\n+ * directories in the index). We should never see this in practice\n+ * (because sparse directories should not be present and therefore\n+ * not generating FS events). Either way, we can treat them in the\n+ * same way and just invalidate the cache-entry and the untracked\n+ * cache (and in this case, the forward cache-entry scan won't find\n+ * anything and it doesn't hurt to let it run).\n+ *\n+ * Return the number of cache-entries that we invalidated. We will\n+ * use this later to determine if we need to attempt a second\n+ * case-insensitive search.\n+ */\n+static int fsmonitor_refresh_callback_slash(\n \tstruct index_state *istate, const char *name, int len, int pos)\n {\n \tint i;\n+\tint nr_in_cone = 0;\n \n-\t/*\n-\t * The daemon can decorate directory events, such as\n-\t * moves or renames, with a trailing slash if the OS\n-\t * FS Event contains sufficient information, such as\n-\t * MacOS.\n-\t *\n-\t * Use this to invalidate the entire cone under that\n-\t * directory.\n-\t *\n-\t * We do not expect an exact match because the index\n-\t * does not normally contain directory entries, so we\n-\t * start at the insertion point and scan.\n-\t */\n \tif (pos < 0)\n \t\tpos = -pos - 1;\n \n@@ -245,7 +261,10 @@ static void fsmonitor_refresh_callback_slash(\n \t\tif (!starts_with(istate->cache[i]->name, name))\n \t\t\tbreak;\n \t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tnr_in_cone++;\n \t}\n+\n+\treturn nr_in_cone;\n }\n \n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n-- \ngitgitgadget\n\n"},{"id":"488589","messageId":"1df4019931c29824b174defb75e09823d604219e.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 07/12] fsmonitor: refactor untracked-cache invalidation","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:16Z","receivedAt":"2024-02-13T20:52:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 38 ++++++++++++++++++++++++++------------\n 1 file changed, 26 insertions(+), 12 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 754fe20cfd0..14585b6c516 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,11 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+/*\n+ * Invalidate the untracked cache for the given pathname. Copy the\n+ * buffer to a proper null-terminated string (since the untracked\n+ * cache code does not use (buf, len) style argument). Also strip any\n+ * trailing slash.\n+ */\n+static void my_invalidate_untracked_cache(\n+\tstruct index_state *istate, const char *name, int len)\n+{\n+\tstruct strbuf work_path = STRBUF_INIT;\n+\n+\tif (!len)\n+\t\treturn;\n+\n+\tif (name[len-1] == '/')\n+\t\tlen--;\n+\n+\tstrbuf_add(&work_path, name, len);\n+\tuntracked_cache_invalidate_path(istate, work_path.buf, 0);\n+\tstrbuf_release(&work_path);\n+}\n+\n static void fsmonitor_refresh_callback_unqualified(\n \tstruct index_state *istate, const char *name, int len, int pos)\n {\n \tint i;\n \n+\tmy_invalidate_untracked_cache(istate, name, len);\n+\n \tif (pos >= 0) {\n \t\t/*\n \t\t * We have an exact match for this path and can just\n@@ -253,6 +277,8 @@ static int fsmonitor_refresh_callback_slash(\n \tint i;\n \tint nr_in_cone = 0;\n \n+\tmy_invalidate_untracked_cache(istate, name, len);\n+\n \tif (pos < 0)\n \t\tpos = -pos - 1;\n \n@@ -278,21 +304,9 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \n \tif (name[len - 1] == '/') {\n \t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n-\n-\t\t/*\n-\t\t * We need to remove the traling \"/\" from the path\n-\t\t * for the untracked cache.\n-\t\t */\n-\t\tname[len - 1] = '\\0';\n \t} else {\n \t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n \t}\n-\n-\t/*\n-\t * Mark the untracked cache dirty even if it wasn't found in the index\n-\t * as it could be a new untracked file.\n-\t */\n-\tuntracked_cache_invalidate_path(istate, name, 0);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"488590","messageId":"e0029a2aad68a60f672e74368a384f68a343e21c.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 08/12] fsmonitor: support case-insensitive directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:17Z","receivedAt":"2024-02-13T20:52:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor_refresh_callback() to handle case-insensitive\nlookups if case-sensitive lookups fail on case-insensitive systems.\nThis can cause 'git status' to report stale status for files if there\nare case issues/errors in the worktree.\n\nThe FSMonitor daemon sends FSEvents using the observed spelling\nof each pathname. On case-insensitive file systems this may be\ndifferent than the expected case spelling.\n\nThe existing code uses index_name_pos() to find the cache-entry for\nthe pathname in the FSEvent and clear the CE_FSMONITOR_VALID bit so\nthat the worktree scan/index refresh will revisit and revalidate the\npath.\n\nOn a case-insensitive file system, the exact match lookup may fail\nto find the associated cache-entry. This causes status to think that\nthe cached CE flags are correct and skip over the file.\n\nUpdate the handling of directory-style FSEvents (ones containing a\npath with a trailing slash) to optionally use the name-hash if the\ncase-correct search does not find a match.\n\n(The FSMonitor daemon can send directory FSEvents if the OS provides\nthat information.)\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 120 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 14585b6c516..73e6ac82af7 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -5,6 +5,7 @@\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"name-hash.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n #include \"trace2.h\"\n@@ -183,6 +184,9 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+static int fsmonitor_refresh_callback_slash(\n+\tstruct index_state *istate, const char *name, int len, int pos);\n+\n /*\n * Invalidate the untracked cache for the given pathname. Copy the\n * buffer to a proper null-terminated string (since the untracked\n@@ -205,6 +209,84 @@ static void my_invalidate_untracked_cache(\n \tstrbuf_release(&work_path);\n }\n \n+/*\n+ * Use the name-hash to lookup the pathname.\n+ *\n+ * Returns the number of cache-entries that we invalidated.\n+ */\n+static int my_callback_name_hash(\n+\tstruct index_state *istate, const char *name, int len)\n+{\n+\tstruct cache_entry *ce = NULL;\n+\n+\tce = index_file_exists(istate, name, len, 1);\n+\tif (!ce)\n+\t\treturn 0;\n+\n+\t/*\n+\t * The index contains a case-insensitive match for the pathname.\n+\t * This could either be a regular file or a sparse-index directory.\n+\t *\n+\t * We should not have seen FSEvents for a sparse-index directory,\n+\t * but we handle it just in case.\n+\t *\n+\t * Either way, we know that there are not any cache-entries for\n+\t * children inside the cone of the directory, so we don't need to\n+\t * do the usual scan.\n+\t */\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback map '%s' '%s'\",\n+\t\t\t name, ce->name);\n+\n+\tmy_invalidate_untracked_cache(istate, ce->name, ce->ce_namelen);\n+\n+\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\treturn 1;\n+}\n+\n+/*\n+ * Use the directory name-hash to find the correct-case spelling\n+ * of the directory. Use the canonical spelling to invalidate all\n+ * of the cache-entries within the matching cone.\n+ *\n+ * The pathname MUST NOT have a trailing slash.\n+ *\n+ * Returns the number of cache-entries that we invalidated.\n+ */\n+static int my_callback_dir_name_hash(\n+\tstruct index_state *istate, const char *name, int len)\n+{\n+\tstruct strbuf canonical_path = STRBUF_INIT;\n+\tint pos;\n+\tint nr_in_cone;\n+\n+\tif (!index_dir_exists2(istate, name, len, &canonical_path))\n+\t\treturn 0; /* name is untracked */\n+\tif (!memcmp(name, canonical_path.buf, len)) {\n+\t\tstrbuf_release(&canonical_path);\n+\t\treturn 0; /* should not happen */\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback map '%s' '%s'\",\n+\t\t\t name, canonical_path.buf);\n+\n+\t/*\n+\t * The directory name-hash only tells us the corrected\n+\t * spelling of the prefix. We have to use this canonical\n+\t * path to do a lookup in the cache-entry array so that we\n+\t * we repeat the original search using the case-corrected\n+\t * spelling.\n+\t */\n+\tstrbuf_addch(&canonical_path, '/');\n+\tpos = index_name_pos(istate, canonical_path.buf,\n+\t\t\t canonical_path.len);\n+\tnr_in_cone = fsmonitor_refresh_callback_slash(\n+\t\tistate, canonical_path.buf, canonical_path.len, pos);\n+\tstrbuf_release(&canonical_path);\n+\treturn nr_in_cone;\n+}\n+\n static void fsmonitor_refresh_callback_unqualified(\n \tstruct index_state *istate, const char *name, int len, int pos)\n {\n@@ -269,7 +351,10 @@ static void fsmonitor_refresh_callback_unqualified(\n *\n * Return the number of cache-entries that we invalidated. We will\n * use this later to determine if we need to attempt a second\n- * case-insensitive search.\n+ * case-insensitive search. That is, if a observed-case search yields\n+ * any results, we assume the prefix is case-correct. If there are\n+ * no matches, we still don't know if the observed path is simply\n+ * untracked or case-incorrect.\n */\n static int fsmonitor_refresh_callback_slash(\n \tstruct index_state *istate, const char *name, int len, int pos)\n@@ -293,17 +378,50 @@ static int fsmonitor_refresh_callback_slash(\n \treturn nr_in_cone;\n }\n \n+/*\n+ * On a case-insensitive FS, use the name-hash and directory name-hash\n+ * to map the case of the observed path to the canonical case expected\n+ * by the index.\n+ *\n+ * The given pathname includes the trailing slash.\n+ *\n+ * Return the number of cache-entries that we invalidated.\n+ */\n+static int fsmonitor_refresh_callback_slash_icase(\n+\tstruct index_state *istate, const char *name, int len)\n+{\n+\tint nr_in_cone;\n+\n+\t/*\n+\t * Look for a case-incorrect sparse-index directory.\n+\t */\n+\tnr_in_cone = my_callback_name_hash(istate, name, len);\n+\tif (nr_in_cone)\n+\t\treturn nr_in_cone;\n+\n+\t/*\n+\t * (len-1) because we do not include the trailing slash in the\n+\t * pathname.\n+\t */\n+\tnr_in_cone = my_callback_dir_name_hash(istate, name, len-1);\n+\treturn nr_in_cone;\n+}\n+\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint len = strlen(name);\n \tint pos = index_name_pos(istate, name, len);\n+\tint nr_in_cone;\n+\n \n \ttrace_printf_key(&trace_fsmonitor,\n \t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n \t\t\t name, pos);\n \n \tif (name[len - 1] == '/') {\n-\t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n+\t\tnr_in_cone = fsmonitor_refresh_callback_slash(istate, name, len, pos);\n+\t\tif (ignore_case && !nr_in_cone)\n+\t\t\tfsmonitor_refresh_callback_slash_icase(istate, name, len);\n \t} else {\n \t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n \t}\n-- \ngitgitgadget\n\n"},{"id":"488591","messageId":"a0cc4c8274c108d231816ac981781d9e3cf8fded.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 09/12] fsmonitor: refactor non-directory callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:18Z","receivedAt":"2024-02-13T20:52:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor the fsmonitor_refresh_callback_unqualified() code\nto try to use the _callback_slash() code and avoid having\na custom filter in the child cache-entry scanner.\n\nOn platforms that DO NOT annotate FS events with a trailing\nslash, if we fail to find an exact match for the pathname\nin the index, we do not know if the pathname represents a\ndirectory or simply an untracked file. Pretend that the pathname\nis a directory and try again before assuming it is an untracked\nfile.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 59 +++++++++++++++++++++++++++++++----------------------\n 1 file changed, 35 insertions(+), 24 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 73e6ac82af7..cb27bae8aa8 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -287,41 +287,52 @@ static int my_callback_dir_name_hash(\n \treturn nr_in_cone;\n }\n \n-static void fsmonitor_refresh_callback_unqualified(\n+/*\n+ * The daemon sent an observed pathname without a trailing slash.\n+ * (This is the normal case.) We do not know if it is a tracked or\n+ * untracked file, a sparse-directory, or a populated directory (on a\n+ * platform such as Windows where FSEvents are not qualified).\n+ *\n+ * The pathname contains the observed case reported by the FS. We\n+ * do not know it is case-correct or -incorrect.\n+ *\n+ * Assume it is case-correct and try an exact match.\n+ *\n+ * Return the number of cache-entries that we invalidated.\n+ */\n+static int fsmonitor_refresh_callback_unqualified(\n \tstruct index_state *istate, const char *name, int len, int pos)\n {\n-\tint i;\n-\n \tmy_invalidate_untracked_cache(istate, name, len);\n \n \tif (pos >= 0) {\n \t\t/*\n-\t\t * We have an exact match for this path and can just\n-\t\t * invalidate it.\n+\t\t * An exact match on a tracked file. We assume that we\n+\t\t * do not need to scan forward for a sparse-directory\n+\t\t * cache-entry with the same pathname, nor for a cone\n+\t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\treturn 1;\n \t} else {\n+\t\tint nr_in_cone;\n+\t\tstruct strbuf work_path = STRBUF_INIT;\n+\n \t\t/*\n-\t\t * The path is not a tracked file -or- it is a\n-\t\t * directory event on a platform that cannot\n-\t\t * distinguish between file and directory events in\n-\t\t * the event handler, such as Windows.\n-\t\t *\n-\t\t * Scan as if it is a directory and invalidate the\n-\t\t * cone under it. (But remember to ignore items\n-\t\t * between \"name\" and \"name/\", such as \"name-\" and\n-\t\t * \"name.\".\n+\t\t * The negative \"pos\" gives us the suggested insertion\n+\t\t * point for the pathname (without the trailing slash).\n+\t\t * We need to see if there is a directory with that\n+\t\t * prefix, but there can be lots of pathnames between\n+\t\t * \"foo\" and \"foo/\" like \"foo-\" or \"foo-bar\", so we\n+\t\t * don't want to do our own scan.\n \t\t */\n-\t\tpos = -pos - 1;\n-\n-\t\tfor (i = pos; i < istate->cache_nr; i++) {\n-\t\t\tif (!starts_with(istate->cache[i]->name, name))\n-\t\t\t\tbreak;\n-\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n-\t\t\t\tbreak;\n-\t\t\tif (istate->cache[i]->name[len] == '/')\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\t}\n+\t\tstrbuf_add(&work_path, name, len);\n+\t\tstrbuf_addch(&work_path, '/');\n+\t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n+\t\tnr_in_cone = fsmonitor_refresh_callback_slash(\n+\t\t\tistate, work_path.buf, work_path.len, pos);\n+\t\tstrbuf_release(&work_path);\n+\t\treturn nr_in_cone;\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"488592","messageId":"bf18401f56c8df0432a1ba52ae79f5e68d81cdb9.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 10/12] fsmonitor: support case-insensitive non-directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:19Z","receivedAt":"2024-02-13T20:52:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 34 +++++++++++++++++++++++++++++++++-\n 1 file changed, 33 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex cb27bae8aa8..a7847f07a40 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -336,6 +336,36 @@ static int fsmonitor_refresh_callback_unqualified(\n \t}\n }\n \n+/*\n+ * On a case-insensitive FS, use the name-hash to map the case of\n+ * the observed path to the canonical case expected by the index.\n+ *\n+ * The given pathname DOES NOT include the trailing slash.\n+ *\n+ * Return the number of cache-entries that we invalidated.\n+ */\n+static int fsmonitor_refresh_callback_unqualified_icase(\n+\tstruct index_state *istate, const char *name, int len)\n+{\n+\tint nr_in_cone;\n+\n+\t/*\n+\t * Look for a case-incorrect match for this non-directory\n+\t * pathname.\n+\t */\n+\tnr_in_cone = my_callback_name_hash(istate, name, len);\n+\tif (nr_in_cone)\n+\t\treturn nr_in_cone;\n+\n+\t/*\n+\t * Try the directory name-hash and see if there is a\n+\t * case-incorrect directory with this pathanme.\n+\t * (len) because we don't have a trailing slash.\n+\t */\n+\tnr_in_cone = my_callback_dir_name_hash(istate, name, len);\n+\treturn nr_in_cone;\n+}\n+\n /*\n * The daemon can decorate directory events, such as a move or rename,\n * by adding a trailing slash to the observed name. Use this to\n@@ -434,7 +464,9 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t\tif (ignore_case && !nr_in_cone)\n \t\t\tfsmonitor_refresh_callback_slash_icase(istate, name, len);\n \t} else {\n-\t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n+\t\tnr_in_cone = fsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n+\t\tif (ignore_case && !nr_in_cone)\n+\t\t\tfsmonitor_refresh_callback_unqualified_icase(istate, name, len);\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"488593","messageId":"7775de735f41bdc601318cd15c3414f12b361a0c.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 11/12] fsmonitor: refactor bit invalidation in refresh callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:20Z","receivedAt":"2024-02-13T20:52:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor code in the fsmonitor_refresh_callback() call chain dealing\nwith invalidating the CE_FSMONITOR_VALID bit and add a trace message.\n\nDuring the refresh, we clear the CE_FSMONITOR_VALID bit in response to\ndata from the FSMonitor daemon (so that a later phase will lstat() and\nverify the true state of the file).\n\nCreate a new function to clear the bit and add some unique tracing for\nit to help debug edge cases.\n\nThis is similar to the existing `mark_fsmonitor_invalid()` function,\nbut we don't need the extra stuff that it does.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 20 +++++++++++++++++---\n 1 file changed, 17 insertions(+), 3 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex a7847f07a40..75c7f73f68d 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -209,6 +209,20 @@ static void my_invalidate_untracked_cache(\n \tstrbuf_release(&work_path);\n }\n \n+/*\n+ * Invalidate the FSM bit on this CE. This is like mark_fsmonitor_invalid()\n+ * but we've already handled the untracked-cache and I want a different\n+ * trace message.\n+ */\n+static void my_invalidate_ce_fsm(struct cache_entry *ce)\n+{\n+\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor_refresh_cb_invalidate '%s'\",\n+\t\t\t\t ce->name);\n+\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+}\n+\n /*\n * Use the name-hash to lookup the pathname.\n *\n@@ -240,7 +254,7 @@ static int my_callback_name_hash(\n \n \tmy_invalidate_untracked_cache(istate, ce->name, ce->ce_namelen);\n \n-\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\tmy_invalidate_ce_fsm(ce);\n \treturn 1;\n }\n \n@@ -312,7 +326,7 @@ static int fsmonitor_refresh_callback_unqualified(\n \t\t * cache-entry with the same pathname, nor for a cone\n \t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n-\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tmy_invalidate_ce_fsm(istate->cache[pos]);\n \t\treturn 1;\n \t} else {\n \t\tint nr_in_cone;\n@@ -412,7 +426,7 @@ static int fsmonitor_refresh_callback_slash(\n \tfor (i = pos; i < istate->cache_nr; i++) {\n \t\tif (!starts_with(istate->cache[i]->name, name))\n \t\t\tbreak;\n-\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tmy_invalidate_ce_fsm(istate->cache[i]);\n \t\tnr_in_cone++;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"488594","messageId":"63edb68303fdd9ddd1c2e31ccf158ef4219cd0b9.1707857541.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH 12/12] t7527: update case-insenstive fsmonitor test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-13T20:52:21Z","receivedAt":"2024-02-13T20:52:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNow that the FSMonitor client has been updated to better\nhandle events on case-insenstive file systems, update the\ntwo tests that demonstrated the bug.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 23 +++++++++++++----------\n 1 file changed, 13 insertions(+), 10 deletions(-)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 5cd68b2ea82..03af8539ca8 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -1116,16 +1116,17 @@ test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n \n \tgrep -q \"dir1/DIR2/dir3/file3.*pos -3\" \"$PWD/subdir_case_wrong.log1\" &&\n \n+\t# Also verify that we get a mapping event to correct the case.\n+\tgrep -q \"map.*dir1/DIR2/dir3/file3.*dir1/dir2/dir3/file3\" \\\n+\t\t\"$PWD/subdir_case_wrong.log1\" &&\n+\n \t# The refresh-callbacks should have caused \"git status\" to clear\n \t# the CE_FSMONITOR_VALID bit on each of those files and caused\n \t# the worktree scan to visit them and mark them as modified.\n \tgrep -q \" M AAA\" \"$PWD/subdir_case_wrong.out\" &&\n \tgrep -q \" M zzz\" \"$PWD/subdir_case_wrong.out\" &&\n \n-\t# However, with the fsmonitor client bug, the \"(pos -3)\" causes\n-\t# the client to not update the bit and never rescan the file\n-\t# and therefore not report it as dirty.\n-\t! grep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n+\tgrep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n '\n \n test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n@@ -1246,12 +1247,14 @@ test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n \tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n \tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n \n-\t# Status should say these files are modified, but with the case\n-\t# bug, the \"pos -3\" cause the client to not update the FSM bit\n-\t# and never cause the file to be rescanned and therefore to not\n-\t# report it dirty.\n-\t! grep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n-\t! grep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n+\t# Also verify that we get a mapping event to correct the case.\n+\tgrep -q \"fsmonitor_refresh_callback map.*dir1/dir2/dir3/FILE-3-A.*dir1/dir2/dir3/file-3-a\" \\\n+\t\t\"$PWD/file_case_wrong-try3.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback map.*dir1/dir2/dir4/file-4-a.*dir1/dir2/dir4/FILE-4-A\" \\\n+\t\t\"$PWD/file_case_wrong-try3.log\" &&\n+\n+\tgrep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n+\tgrep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n '\n \n test_done\n-- \ngitgitgadget\n"},{"id":"488599","messageId":"xmqqeddg2g7j.fsf@gitster.g","threadId":"60908","inReplyTo":"3464545fe3feceb08408618c77a70cc95745bfe9.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 02/12] name-hash: add index_dir_exists2()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-13T21:43:12Z","receivedAt":"2024-02-13T21:43:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Create a new version of index_dir_exists() to return the canonical\n> spelling of the matched directory prefix.\n>\n> The existing index_dir_exists() returns a boolean to indicate if\n> there is a case-insensitive match in the directory name-hash, but\n> it doesn't tell the caller the exact spelling of that match.\n>\n> The new version also copies the matched spelling to a provided strbuf.\n> This lets the caller, for example, then call index_name_pos() with the\n> correct case to search the cache-entry array for the real insertion\n> position.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> name-hash.c | 16 ++++++++++++++++\n> name-hash.h | 2 ++\n> 2 files changed, 18 insertions(+)\n>\n> diff --git a/name-hash.c b/name-hash.c\n> index 251f036eef6..d735c81acb3 100644\n> --- a/name-hash.c\n> +++ b/name-hash.c\n> @@ -694,6 +694,22 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n> \tdir = find_dir_entry(istate, name, namelen);\n> \treturn dir && dir->nr;\n> }\n> +int index_dir_exists2(struct index_state *istate, const char *name, int namelen,\n> +\t\t struct strbuf *canonical_path)\n> +{\n> +\tstruct dir_entry *dir;\n> +\n> +\tstrbuf_init(canonical_path, namelen+1);\n> +\n> +\tlazy_init_name_hash(istate);\n> +\texpand_to_path(istate, name, namelen, 0);\n> +\tdir = find_dir_entry(istate, name, namelen);\n> +\n> +\tif (dir && dir->nr)\n> +\t\tstrbuf_add(canonical_path, dir->name, dir->namelen);\n> +\n> +\treturn dir && dir->nr;\n> +}\n> \n> void adjust_dirname_case(struct index_state *istate, char *name)\n\nMissing inter-function blank line, before the new function.\n\nI wonder if we can avoid such repetition---the body of\nindex_dir_exists() is 100% shared with this new function.\n\nIsn't it extremely unusual to receive \"struct strbuf *\" and call\nstrbuf_init() on it? It means that the caller is expected to have a\nstrbuf and pass a pointer to it, but also it is expected to leave\nthe strbuf uninitialized.\n\nI'd understand if it calls strbuf_reset(), but it may not even be\nnecessary, if we make it responsibility of the caller to pass a\nvalid strbuf to be appended into.\n\n\tint index_dir_find(struct index_state *istate,\n\t\t\t const char *name, int namelen,\n\t\t\t struct strbuf *canonical_path)\n\t{\n struct dir_entry *dir;\n\n lazy_init_name_hash(istate);\n expand_to_path(istate, name, namelen, 0);\n dir = find_dir_entry(istate, name, namelen);\n\n if (canonical_path && dir && dir->nr) {\n\t\t\t// strbuf_reset(canonical_path); ???\n \tstrbuf_add(canonical_path, dir->name, dir->namelen);\n\t\t}\n return dir && dir->nr;\n\t}\n\nThen we can do\n\n\t#define index_dir_exists(i, n, l) index_dir_find((i), (n), (l), NULL)\n\nin the header for existing callers.\n\n> {\n> diff --git a/name-hash.h b/name-hash.h\n> index b1b4b0fb337..2fcac5c4870 100644\n> --- a/name-hash.h\n> +++ b/name-hash.h\n> @@ -5,6 +5,8 @@ struct cache_entry;\n> struct index_state;\n> \n> int index_dir_exists(struct index_state *istate, const char *name, int namelen);\n> +int index_dir_exists2(struct index_state *istate, const char *name, int namelen,\n> +\t\t struct strbuf *canonical_path);\n> void adjust_dirname_case(struct index_state *istate, char *name);\n> struct cache_entry *index_file_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n"},{"id":"488601","messageId":"xmqqzfw410j8.fsf@gitster.g","threadId":"60908","inReplyTo":"6f81e2e30609c70d4dcdbe9d4f11c4f6b5173c77.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 01/12] sparse-index: pass string length to index_file_exists()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-13T22:07:07Z","receivedAt":"2024-02-13T22:07:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> The call to index_file_exists() in the loop in expand_to_path() passes\n> the wrong string length. Let's fix that.\n>\n> The loop in expand_to_path() searches the name-hash for each\n> sub-directory prefix in the provided pathname. That is, by searching\n> for \"dir1/\" then \"dir1/dir2/\" then \"dir1/dir2/dir3/\" and so on until\n> it finds a cache-entry representing a sparse directory.\n>\n> The code creates \"strbuf path_mutable\" to contain the working pathname\n> and modifies the buffer in-place by temporarily replacing the character\n> following each successive \"/\" with NUL for the duration of the call to\n> index_file_exists().\n>\n> It does not update the strbuf.len during this substitution.\n>\n> Pass the patched length of the prefix path instead.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n\nThis looked familiar, and it turns out that\n\nhttps://lore.kernel.org/git/pull.1649.git.1706897095273.gitgitgadget@gmail.com/\n\nhas already been merged to 'master'.\n\n\n> sparse-index.c | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/sparse-index.c b/sparse-index.c\n> index 3578feb2837..e48e40cae71 100644\n> --- a/sparse-index.c\n> +++ b/sparse-index.c\n> @@ -579,8 +579,9 @@ void expand_to_path(struct index_state *istate,\n> \t\treplace++;\n> \t\ttemp = *replace;\n> \t\t*replace = '\\0';\n> +\t\tsubstr_len = replace - path_mutable.buf;\n> \t\tif (index_file_exists(istate, path_mutable.buf,\n> -\t\t\t\t path_mutable.len, icase)) {\n> +\t\t\t\t substr_len, icase)) {\n> \t\t\t/*\n> \t\t\t * We found a parent directory in the name-hash\n> \t\t\t * hashtable, because only sparse directory entries\n> @@ -593,7 +594,6 @@ void expand_to_path(struct index_state *istate,\n> \t\t}\n> \n> \t\t*replace = temp;\n> -\t\tsubstr_len = replace - path_mutable.buf;\n> \t}\n> \n> cleanup:\n"},{"id":"488607","messageId":"xmqq5xyr25hv.fsf@gitster.g","threadId":"60908","inReplyTo":"0896d4af907d71df29b0c4f5a27d24ea80b3c0e1.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 05/12] fsmonitor: refactor refresh callback for non-directory events","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-14T01:34:36Z","receivedAt":"2024-02-14T01:34:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 66 ++++++++++++++++++++++++++++++-----------------------\n> 1 file changed, 38 insertions(+), 28 deletions(-)\n\nUp to this point, I found it a very pleasant read. Nothing\nsurprising or unexpected. Just a simple series of nice clean-ups.\n\nThanks.\n"},{"id":"488628","messageId":"xmqqil2rzdvo.fsf@gitster.g","threadId":"60908","inReplyTo":"5b6f8bd1fe7b6c742b25a5a1ed95b528f352215e.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 06/12] fsmonitor: clarify handling of directory events in callback","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-14T07:47:07Z","receivedAt":"2024-02-14T07:50:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 47 +++++++++++++++++++++++++++++++++--------------\n> 1 file changed, 33 insertions(+), 14 deletions(-)\n>\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 614270fa5e8..754fe20cfd0 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -219,24 +219,40 @@ static void fsmonitor_refresh_callback_unqualified(\n> \t}\n> }\n> \n> -static void fsmonitor_refresh_callback_slash(\n> +/*\n> + * The daemon can decorate directory events, such as a move or rename,\n> + * by adding a trailing slash to the observed name. Use this to\n> + * explicitly invalidate the entire cone under that directory.\n> + *\n> + * The daemon can only reliably do that if the OS FSEvent contains\n> + * sufficient information in the event.\n> + *\n> + * macOS FSEvents have enough information.\n> + *\n> + * Other platforms may or may not be able to do it (and it might\n> + * depend on the type of event (for example, a daemon could lstat() an\n> + * observed pathname after a rename, but not after a delete)).\n> + *\n> + * If we find an exact match in the index for a path with a trailing\n> + * slash, it means that we matched a sparse-index directory in a\n> + * cone-mode sparse-checkout (since that's the only time we have\n> + * directories in the index). We should never see this in practice\n> + * (because sparse directories should not be present and therefore\n> + * not generating FS events). Either way, we can treat them in the\n> + * same way and just invalidate the cache-entry and the untracked\n> + * cache (and in this case, the forward cache-entry scan won't find\n> + * anything and it doesn't hurt to let it run).\n> + *\n> + * Return the number of cache-entries that we invalidated. We will\n> + * use this later to determine if we need to attempt a second\n> + * case-insensitive search.\n> + */\n> +static int fsmonitor_refresh_callback_slash(\n> \tstruct index_state *istate, const char *name, int len, int pos)\n> {\n\nThis was split out a few patches ago, and the caller of course\nignored the return value (void), but now it turns an integer, and\nthis change is without a corresponding update to the caller, which\nleaves readers puzzled.\n\nPerhaps a future patch either updates the existing caller or adds a\nnew caller that utilize the returned value, but then at least the\nproposed commit message for this step should hint how it helps the\ncaller(s) we will see in the future steps if this function returns\nthe number of entries invalidated, iow, how the caller is expected\nto use the returned value from here, no?\n\nAlternatively, this step can limit itself to what the commit title\nclaims to do---to clarify what the helper does with enhanced in-code\ncomments. Then a future step that updates the caller to care about\nthe return value can have both the changes to this callee as well as\nthe caller---which may make it easier to see how the returned info\nhelps the caller. I dunno which is more reasonable.\n\nThanks.\n\n\n\n> \tint i;\n> +\tint nr_in_cone = 0;\n> \n> -\t/*\n> -\t * The daemon can decorate directory events, such as\n> -\t * moves or renames, with a trailing slash if the OS\n> -\t * FS Event contains sufficient information, such as\n> -\t * MacOS.\n> -\t *\n> -\t * Use this to invalidate the entire cone under that\n> -\t * directory.\n> -\t *\n> -\t * We do not expect an exact match because the index\n> -\t * does not normally contain directory entries, so we\n> -\t * start at the insertion point and scan.\n> -\t */\n> \tif (pos < 0)\n> \t\tpos = -pos - 1;\n> \n> @@ -245,7 +261,10 @@ static void fsmonitor_refresh_callback_slash(\n> \t\tif (!starts_with(istate->cache[i]->name, name))\n> \t\t\tbreak;\n> \t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\tnr_in_cone++;\n> \t}\n> +\n> +\treturn nr_in_cone;\n> }\n> \n> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n"},{"id":"488650","messageId":"xmqqo7cjxad3.fsf@gitster.g","threadId":"60908","inReplyTo":"1df4019931c29824b174defb75e09823d604219e.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 07/12] fsmonitor: refactor untracked-cache invalidation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-14T16:46:00Z","receivedAt":"2024-02-14T16:46:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 38 ++++++++++++++++++++++++++------------\n> 1 file changed, 26 insertions(+), 12 deletions(-)\n\nSorry, but the proposed commit log is way lacking for this\nparticular step. Readers have already understood, after reading\nsteps like [04/12] and [05/12], that you use the verb \"refactor\" in\nits usual sense, i.e. reorganize the code around without changing\nbehaviour in order to enhance readability and to make it easier for\ncode reuse in future steps, and these two steps did exactly that:\nhelper functions are split out of larger functions, presumably\neither to allow adding new callers to the helpers, or to make the\nresult of adding more code to the caller easier to follow [*].\n\nHowever, the changes in this step look vastly different, and it is\nnot even clear if this change intends to keep the behaviour before\nand after it the same, or if it does, how they are the same.\n\nI can sort-of see that the original code made a call to\nuntracked_cache_invalidate_path() at the very end of the\nfsmonitor_refresh_callback(), but the updated code no longer does\nso. Why? Is it because it is the root cause of an unstated bug\nthat we don't do so until the end in the current code? Is it\nbecause the order does not matter (how and why?) and the resulting\ncode becomes better (how? simpler to follow? more performant?\navoids duplicated work? something else)?\n\nIt does not help to call a new helper function with a cryptic \"my_\"\nname, either.\n\nPlease try again? Thanks.\n\n\n[Footnote] \n\n * These two are vastly different goals, and there may be other\n reasons why you are doing such refactoring. It would have been\n nicer if such a preliminary refactoring steps had explained what\n the intended course of evolution for the code involved in the\n refactoring is.\n\n\n\n>\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 754fe20cfd0..14585b6c516 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -183,11 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n> \treturn result;\n> }\n> \n> +/*\n> + * Invalidate the untracked cache for the given pathname. Copy the\n> + * buffer to a proper null-terminated string (since the untracked\n> + * cache code does not use (buf, len) style argument). Also strip any\n> + * trailing slash.\n> + */\n> +static void my_invalidate_untracked_cache(\n> +\tstruct index_state *istate, const char *name, int len)\n> +{\n> +\tstruct strbuf work_path = STRBUF_INIT;\n> +\n> +\tif (!len)\n> +\t\treturn;\n> +\n> +\tif (name[len-1] == '/')\n> +\t\tlen--;\n> +\n> +\tstrbuf_add(&work_path, name, len);\n> +\tuntracked_cache_invalidate_path(istate, work_path.buf, 0);\n> +\tstrbuf_release(&work_path);\n> +}\n> +\n> static void fsmonitor_refresh_callback_unqualified(\n> \tstruct index_state *istate, const char *name, int len, int pos)\n> {\n> \tint i;\n> \n> +\tmy_invalidate_untracked_cache(istate, name, len);\n> +\n> \tif (pos >= 0) {\n> \t\t/*\n> \t\t * We have an exact match for this path and can just\n> @@ -253,6 +277,8 @@ static int fsmonitor_refresh_callback_slash(\n> \tint i;\n> \tint nr_in_cone = 0;\n> \n> +\tmy_invalidate_untracked_cache(istate, name, len);\n> +\n> \tif (pos < 0)\n> \t\tpos = -pos - 1;\n> \n> @@ -278,21 +304,9 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \n> \tif (name[len - 1] == '/') {\n> \t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n> -\n> -\t\t/*\n> -\t\t * We need to remove the traling \"/\" from the path\n> -\t\t * for the untracked cache.\n> -\t\t */\n> -\t\tname[len - 1] = '\\0';\n> \t} else {\n> \t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n> \t}\n> -\n> -\t/*\n> -\t * Mark the untracked cache dirty even if it wasn't found in the index\n> -\t * as it could be a new untracked file.\n> -\t */\n> -\tuntracked_cache_invalidate_path(istate, name, 0);\n> }\n> \n> /*\n"},{"id":"488705","messageId":"Zc3aCWwJzawKYE7p@tanuki","threadId":"60908","inReplyTo":"3464545fe3feceb08408618c77a70cc95745bfe9.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 02/12] name-hash: add index_dir_exists2()","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-15T09:31:53Z","receivedAt":"2024-02-15T09:31:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 13, 2024 at 08:52:11PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Create a new version of index_dir_exists() to return the canonical\n> spelling of the matched directory prefix.\n> \n> The existing index_dir_exists() returns a boolean to indicate if\n> there is a case-insensitive match in the directory name-hash, but\n> it doesn't tell the caller the exact spelling of that match.\n> \n> The new version also copies the matched spelling to a provided strbuf.\n> This lets the caller, for example, then call index_name_pos() with the\n> correct case to search the cache-entry array for the real insertion\n> position.\n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> name-hash.c | 16 ++++++++++++++++\n> name-hash.h | 2 ++\n> 2 files changed, 18 insertions(+)\n> \n> diff --git a/name-hash.c b/name-hash.c\n> index 251f036eef6..d735c81acb3 100644\n> --- a/name-hash.c\n> +++ b/name-hash.c\n> @@ -694,6 +694,22 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n> \tdir = find_dir_entry(istate, name, namelen);\n> \treturn dir && dir->nr;\n> }\n> +int index_dir_exists2(struct index_state *istate, const char *name, int namelen,\n> +\t\t struct strbuf *canonical_path)\n> +{\n> +\tstruct dir_entry *dir;\n> +\n> +\tstrbuf_init(canonical_path, namelen+1);\n\nMissing spaces: `namelen + 1`.\n\n> +\tlazy_init_name_hash(istate);\n> +\texpand_to_path(istate, name, namelen, 0);\n> +\tdir = find_dir_entry(istate, name, namelen);\n> +\n> +\tif (dir && dir->nr)\n> +\t\tstrbuf_add(canonical_path, dir->name, dir->namelen);\n> +\n> +\treturn dir && dir->nr;\n> +}\n\nCan we maybe give this function a more descriptive name?\n`index_dir_exists2()` doesn't give the reader any indicator what is\ndifferent about it compared to `index_dir_exists()`. How about\n`index_dir_exists_with_canonical()`?\n\n> void adjust_dirname_case(struct index_state *istate, char *name)\n> {\n> diff --git a/name-hash.h b/name-hash.h\n> index b1b4b0fb337..2fcac5c4870 100644\n> --- a/name-hash.h\n> +++ b/name-hash.h\n> @@ -5,6 +5,8 @@ struct cache_entry;\n> struct index_state;\n> \n> int index_dir_exists(struct index_state *istate, const char *name, int namelen);\n> +int index_dir_exists2(struct index_state *istate, const char *name, int namelen,\n> +\t\t struct strbuf *canonical_path);\n\nIt would also be great to add comments here that explain what those\nfunctions do and what the difference between them is.\n\nPatrick\n\n> void adjust_dirname_case(struct index_state *istate, char *name);\n> struct cache_entry *index_file_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n> \n> -- \n> gitgitgadget\n> \n> \n"},{"id":"488706","messageId":"Zc3aGSbEIFVClRd_@tanuki","threadId":"60908","inReplyTo":"3fb8e0d0a7c0455cc7a5ba28c12736fd4bbbd44e.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 04/12] fsmonitor: refactor refresh callback on directory events","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-15T09:32:09Z","receivedAt":"2024-02-15T09:32:13Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 13, 2024 at 08:52:13PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 52 ++++++++++++++++++++++++++++++----------------------\n> 1 file changed, 30 insertions(+), 22 deletions(-)\n> \n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index f670c509378..b1ef01bf3cd 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -183,6 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n> \treturn result;\n> }\n> \n> +static void fsmonitor_refresh_callback_slash(\n> +\tstruct index_state *istate, const char *name, int len, int pos)\n\n`len` should be `size_t` as it tracks the length of the name. This is\na preexisting issue already because `fsmonitor_refresh_callback()`\nassigns `int len = strlen(name)`, which is wrong.\n\n> +{\n> +\tint i;\n\n`i` is used to iterate through `istate->cache_nr`, which is an `unsigned\nint` and not an `int`. I really wish we would compile the Git code base\nwith `-Wconversion`, but that's a rather big undertaking.\n\nAnyway, none of these issues are new as you merely move the code into a\nnew function.\n\nPatrick\n\n> +\t/*\n> +\t * The daemon can decorate directory events, such as\n> +\t * moves or renames, with a trailing slash if the OS\n> +\t * FS Event contains sufficient information, such as\n> +\t * MacOS.\n> +\t *\n> +\t * Use this to invalidate the entire cone under that\n> +\t * directory.\n> +\t *\n> +\t * We do not expect an exact match because the index\n> +\t * does not normally contain directory entries, so we\n> +\t * start at the insertion point and scan.\n> +\t */\n> +\tif (pos < 0)\n> +\t\tpos = -pos - 1;\n> +\n> +\t/* Mark all entries for the folder invalid */\n> +\tfor (i = pos; i < istate->cache_nr; i++) {\n> +\t\tif (!starts_with(istate->cache[i]->name, name))\n> +\t\t\tbreak;\n> +\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t}\n> +}\n> +\n> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> {\n> \tint i, len = strlen(name);\n> @@ -193,28 +222,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \t\t\t name, pos);\n> \n> \tif (name[len - 1] == '/') {\n> -\t\t/*\n> -\t\t * The daemon can decorate directory events, such as\n> -\t\t * moves or renames, with a trailing slash if the OS\n> -\t\t * FS Event contains sufficient information, such as\n> -\t\t * MacOS.\n> -\t\t *\n> -\t\t * Use this to invalidate the entire cone under that\n> -\t\t * directory.\n> -\t\t *\n> -\t\t * We do not expect an exact match because the index\n> -\t\t * does not normally contain directory entries, so we\n> -\t\t * start at the insertion point and scan.\n> -\t\t */\n> -\t\tif (pos < 0)\n> -\t\t\tpos = -pos - 1;\n> -\n> -\t\t/* Mark all entries for the folder invalid */\n> -\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> -\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> -\t\t\t\tbreak;\n> -\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> -\t\t}\n> +\t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n> \n> \t\t/*\n> \t\t * We need to remove the traling \"/\" from the path\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"488707","messageId":"Zc3aI5FGZdWIUDdi@tanuki","threadId":"60908","inReplyTo":"0896d4af907d71df29b0c4f5a27d24ea80b3c0e1.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 05/12] fsmonitor: refactor refresh callback for non-directory events","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-15T09:32:19Z","receivedAt":"2024-02-15T09:32:23Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 13, 2024 at 08:52:14PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 66 ++++++++++++++++++++++++++++++-----------------------\n> 1 file changed, 38 insertions(+), 28 deletions(-)\n> \n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index b1ef01bf3cd..614270fa5e8 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -183,6 +183,42 @@ static int query_fsmonitor_hook(struct repository *r,\n> \treturn result;\n> }\n> \n> +static void fsmonitor_refresh_callback_unqualified(\n> +\tstruct index_state *istate, const char *name, int len, int pos)\n> +{\n> +\tint i;\n\nSame remarks here regarding the integer types. But again, not a fault of\nyour patch.\n\nPatrick\n\n> +\n> +\tif (pos >= 0) {\n> +\t\t/*\n> +\t\t * We have an exact match for this path and can just\n> +\t\t * invalidate it.\n> +\t\t */\n> +\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t} else {\n> +\t\t/*\n> +\t\t * The path is not a tracked file -or- it is a\n> +\t\t * directory event on a platform that cannot\n> +\t\t * distinguish between file and directory events in\n> +\t\t * the event handler, such as Windows.\n> +\t\t *\n> +\t\t * Scan as if it is a directory and invalidate the\n> +\t\t * cone under it. (But remember to ignore items\n> +\t\t * between \"name\" and \"name/\", such as \"name-\" and\n> +\t\t * \"name.\".\n> +\t\t */\n> +\t\tpos = -pos - 1;\n> +\n> +\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> +\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> +\t\t\t\tbreak;\n> +\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n> +\t\t\t\tbreak;\n> +\t\t\tif (istate->cache[i]->name[len] == '/')\n> +\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\t}\n> +\t}\n> +}\n> +\n> static void fsmonitor_refresh_callback_slash(\n> \tstruct index_state *istate, const char *name, int len, int pos)\n> {\n> @@ -214,7 +250,7 @@ static void fsmonitor_refresh_callback_slash(\n> \n> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> {\n> -\tint i, len = strlen(name);\n> +\tint len = strlen(name);\n> \tint pos = index_name_pos(istate, name, len);\n> \n> \ttrace_printf_key(&trace_fsmonitor,\n> @@ -229,34 +265,8 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \t\t * for the untracked cache.\n> \t\t */\n> \t\tname[len - 1] = '\\0';\n> -\t} else if (pos >= 0) {\n> -\t\t/*\n> -\t\t * We have an exact match for this path and can just\n> -\t\t * invalidate it.\n> -\t\t */\n> -\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> \t} else {\n> -\t\t/*\n> -\t\t * The path is not a tracked file -or- it is a\n> -\t\t * directory event on a platform that cannot\n> -\t\t * distinguish between file and directory events in\n> -\t\t * the event handler, such as Windows.\n> -\t\t *\n> -\t\t * Scan as if it is a directory and invalidate the\n> -\t\t * cone under it. (But remember to ignore items\n> -\t\t * between \"name\" and \"name/\", such as \"name-\" and\n> -\t\t * \"name.\".\n> -\t\t */\n> -\t\tpos = -pos - 1;\n> -\n> -\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> -\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> -\t\t\t\tbreak;\n> -\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n> -\t\t\t\tbreak;\n> -\t\t\tif (istate->cache[i]->name[len] == '/')\n> -\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> -\t\t}\n> +\t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n> \t}\n> \n> \t/*\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"488708","messageId":"Zc3aKSR53Ew4ImMP@tanuki","threadId":"60908","inReplyTo":"5b6f8bd1fe7b6c742b25a5a1ed95b528f352215e.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 06/12] fsmonitor: clarify handling of directory events in callback","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-15T09:32:25Z","receivedAt":"2024-02-15T09:32:29Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 13, 2024 at 08:52:15PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 47 +++++++++++++++++++++++++++++++++--------------\n> 1 file changed, 33 insertions(+), 14 deletions(-)\n> \n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 614270fa5e8..754fe20cfd0 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -219,24 +219,40 @@ static void fsmonitor_refresh_callback_unqualified(\n> \t}\n> }\n> \n> -static void fsmonitor_refresh_callback_slash(\n> +/*\n> + * The daemon can decorate directory events, such as a move or rename,\n> + * by adding a trailing slash to the observed name. Use this to\n> + * explicitly invalidate the entire cone under that directory.\n> + *\n> + * The daemon can only reliably do that if the OS FSEvent contains\n> + * sufficient information in the event.\n> + *\n> + * macOS FSEvents have enough information.\n> + *\n> + * Other platforms may or may not be able to do it (and it might\n> + * depend on the type of event (for example, a daemon could lstat() an\n> + * observed pathname after a rename, but not after a delete)).\n> + *\n> + * If we find an exact match in the index for a path with a trailing\n> + * slash, it means that we matched a sparse-index directory in a\n> + * cone-mode sparse-checkout (since that's the only time we have\n> + * directories in the index). We should never see this in practice\n> + * (because sparse directories should not be present and therefore\n> + * not generating FS events). Either way, we can treat them in the\n> + * same way and just invalidate the cache-entry and the untracked\n> + * cache (and in this case, the forward cache-entry scan won't find\n> + * anything and it doesn't hurt to let it run).\n> + *\n> + * Return the number of cache-entries that we invalidated. We will\n> + * use this later to determine if we need to attempt a second\n> + * case-insensitive search.\n> + */\n> +static int fsmonitor_refresh_callback_slash(\n> \tstruct index_state *istate, const char *name, int len, int pos)\n> {\n> \tint i;\n> +\tint nr_in_cone = 0;\n\nShould we return `size_t` instead of `int`?\n\nPatrick\n\n> -\t/*\n> -\t * The daemon can decorate directory events, such as\n> -\t * moves or renames, with a trailing slash if the OS\n> -\t * FS Event contains sufficient information, such as\n> -\t * MacOS.\n> -\t *\n> -\t * Use this to invalidate the entire cone under that\n> -\t * directory.\n> -\t *\n> -\t * We do not expect an exact match because the index\n> -\t * does not normally contain directory entries, so we\n> -\t * start at the insertion point and scan.\n> -\t */\n> \tif (pos < 0)\n> \t\tpos = -pos - 1;\n> \n> @@ -245,7 +261,10 @@ static void fsmonitor_refresh_callback_slash(\n> \t\tif (!starts_with(istate->cache[i]->name, name))\n> \t\t\tbreak;\n> \t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\tnr_in_cone++;\n> \t}\n> +\n> +\treturn nr_in_cone;\n> }\n> \n> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"488709","messageId":"Zc3aMzBBZBlK3MfI@tanuki","threadId":"60908","inReplyTo":"1df4019931c29824b174defb75e09823d604219e.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 07/12] fsmonitor: refactor untracked-cache invalidation","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-15T09:32:35Z","receivedAt":"2024-02-15T09:32:39Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 13, 2024 at 08:52:16PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Signed-off-by: Jeff Hostetler \n\nJunio already mentioned that this change does more than a mere\nrefactoring, which leaves the reader puzzled a bit.\n\n> ---\n> fsmonitor.c | 38 ++++++++++++++++++++++++++------------\n> 1 file changed, 26 insertions(+), 12 deletions(-)\n> \n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 754fe20cfd0..14585b6c516 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -183,11 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n> \treturn result;\n> }\n> \n> +/*\n> + * Invalidate the untracked cache for the given pathname. Copy the\n> + * buffer to a proper null-terminated string (since the untracked\n> + * cache code does not use (buf, len) style argument). Also strip any\n> + * trailing slash.\n> + */\n> +static void my_invalidate_untracked_cache(\n> +\tstruct index_state *istate, const char *name, int len)\n> +{\n> +\tstruct strbuf work_path = STRBUF_INIT;\n> +\n> +\tif (!len)\n> +\t\treturn;\n> +\n> +\tif (name[len-1] == '/')\n> +\t\tlen--;\n> +\n> +\tstrbuf_add(&work_path, name, len);\n> +\tuntracked_cache_invalidate_path(istate, work_path.buf, 0);\n> +\tstrbuf_release(&work_path);\n> +}\n> +\n> static void fsmonitor_refresh_callback_unqualified(\n> \tstruct index_state *istate, const char *name, int len, int pos)\n> {\n> \tint i;\n> \n> +\tmy_invalidate_untracked_cache(istate, name, len);\n> +\n> \tif (pos >= 0) {\n> \t\t/*\n> \t\t * We have an exact match for this path and can just\n> @@ -253,6 +277,8 @@ static int fsmonitor_refresh_callback_slash(\n> \tint i;\n> \tint nr_in_cone = 0;\n> \n> +\tmy_invalidate_untracked_cache(istate, name, len);\n> +\n> \tif (pos < 0)\n> \t\tpos = -pos - 1;\n> \n> @@ -278,21 +304,9 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \n> \tif (name[len - 1] == '/') {\n> \t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n> -\n> -\t\t/*\n> -\t\t * We need to remove the traling \"/\" from the path\n> -\t\t * for the untracked cache.\n> -\t\t */\n> -\t\tname[len - 1] = '\\0';\n> \t} else {\n> \t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n> \t}\n\nWe can drop the braces here as both branches are now single-line\nstatements.\n\nPatrick\n\n> -\n> -\t/*\n> -\t * Mark the untracked cache dirty even if it wasn't found in the index\n> -\t * as it could be a new untracked file.\n> -\t */\n> -\tuntracked_cache_invalidate_path(istate, name, 0);\n> }\n> \n> /*\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"488710","messageId":"Zc3aO95bvuXXENLj@tanuki","threadId":"60908","inReplyTo":"e0029a2aad68a60f672e74368a384f68a343e21c.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 08/12] fsmonitor: support case-insensitive directory events","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-15T09:32:43Z","receivedAt":"2024-02-15T09:32:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 13, 2024 at 08:52:17PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach fsmonitor_refresh_callback() to handle case-insensitive\n> lookups if case-sensitive lookups fail on case-insensitive systems.\n> This can cause 'git status' to report stale status for files if there\n> are case issues/errors in the worktree.\n> \n> The FSMonitor daemon sends FSEvents using the observed spelling\n> of each pathname. On case-insensitive file systems this may be\n> different than the expected case spelling.\n> \n> The existing code uses index_name_pos() to find the cache-entry for\n> the pathname in the FSEvent and clear the CE_FSMONITOR_VALID bit so\n> that the worktree scan/index refresh will revisit and revalidate the\n> path.\n> \n> On a case-insensitive file system, the exact match lookup may fail\n> to find the associated cache-entry. This causes status to think that\n> the cached CE flags are correct and skip over the file.\n> \n> Update the handling of directory-style FSEvents (ones containing a\n> path with a trailing slash) to optionally use the name-hash if the\n> case-correct search does not find a match.\n> \n> (The FSMonitor daemon can send directory FSEvents if the OS provides\n> that information.)\n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 122 +++++++++++++++++++++++++++++++++++++++++++++++++++-\n> 1 file changed, 120 insertions(+), 2 deletions(-)\n> \n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 14585b6c516..73e6ac82af7 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -5,6 +5,7 @@\n> #include \"ewah/ewok.h\"\n> #include \"fsmonitor.h\"\n> #include \"fsmonitor-ipc.h\"\n> +#include \"name-hash.h\"\n> #include \"run-command.h\"\n> #include \"strbuf.h\"\n> #include \"trace2.h\"\n> @@ -183,6 +184,9 @@ static int query_fsmonitor_hook(struct repository *r,\n> \treturn result;\n> }\n> \n> +static int fsmonitor_refresh_callback_slash(\n> +\tstruct index_state *istate, const char *name, int len, int pos);\n> +\n> /*\n> * Invalidate the untracked cache for the given pathname. Copy the\n> * buffer to a proper null-terminated string (since the untracked\n> @@ -205,6 +209,84 @@ static void my_invalidate_untracked_cache(\n> \tstrbuf_release(&work_path);\n> }\n> \n> +/*\n> + * Use the name-hash to lookup the pathname.\n> + *\n> + * Returns the number of cache-entries that we invalidated.\n> + */\n\nThe function not only looks up the path name, but also invalidates the\ncorresponding cache entry. You imply this with the second sentence, but\nthis could be a bit more explicit.\n\n> +static int my_callback_name_hash(\n> +\tstruct index_state *istate, const char *name, int len)\n\nI find the naming conventions here to be weird with the `my_` prefix.\n\n> +{\n> +\tstruct cache_entry *ce = NULL;\n> +\n> +\tce = index_file_exists(istate, name, len, 1);\n> +\tif (!ce)\n> +\t\treturn 0;\n\nOkay, `index_file_exists()` is called with `icase == 1` here. But is\nthat the correct thing to do on case-sensitive platforms? I would have\nexpected us to honor `core.ignoreCase` here.\n\nTurns out, we only end up calling this function when `ignore_case` is\nset, so we already do. I think this can be clarified both by giving the\nfunction a better name and by documenting this in the comment. Also,\nneither of this or the next function really are callbacks -- they only\nhappen to be called by a callback function.\n\nI'd think something like `lookup_and_invalidate_path_icase()` and\n`lookup_and_invalidate_dir_icase()` could help to clarify intent.\n\n> +\t/*\n> +\t * The index contains a case-insensitive match for the pathname.\n> +\t * This could either be a regular file or a sparse-index directory.\n> +\t *\n> +\t * We should not have seen FSEvents for a sparse-index directory,\n> +\t * but we handle it just in case.\n> +\t *\n> +\t * Either way, we know that there are not any cache-entries for\n> +\t * children inside the cone of the directory, so we don't need to\n> +\t * do the usual scan.\n> +\t */\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"fsmonitor_refresh_callback map '%s' '%s'\",\n> +\t\t\t name, ce->name);\n> +\n> +\tmy_invalidate_untracked_cache(istate, ce->name, ce->ce_namelen);\n> +\n> +\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\treturn 1;\n> +}\n> +\n> +/*\n> + * Use the directory name-hash to find the correct-case spelling\n> + * of the directory. Use the canonical spelling to invalidate all\n> + * of the cache-entries within the matching cone.\n> + *\n> + * The pathname MUST NOT have a trailing slash.\n> + *\n> + * Returns the number of cache-entries that we invalidated.\n> + */\n> +static int my_callback_dir_name_hash(\n> +\tstruct index_state *istate, const char *name, int len)\n> +{\n> +\tstruct strbuf canonical_path = STRBUF_INIT;\n> +\tint pos;\n> +\tint nr_in_cone;\n> +\n> +\tif (!index_dir_exists2(istate, name, len, &canonical_path))\n> +\t\treturn 0; /* name is untracked */\n> +\tif (!memcmp(name, canonical_path.buf, len)) {\n> +\t\tstrbuf_release(&canonical_path);\n> +\t\treturn 0; /* should not happen */\n> +\t}\n\nSo in other words, this function should only be called when we know that\ncasing differs, and thus the passed-in name and the canonical name\nshould never be the same? If this case shouldn't ever happen, shouldn't\nwe report this as an error or use `BUG()` instead of silently ignoring\nthis mismatch of expectations?\n\nPatrick\n\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"fsmonitor_refresh_callback map '%s' '%s'\",\n> +\t\t\t name, canonical_path.buf);\n> +\n> +\t/*\n> +\t * The directory name-hash only tells us the corrected\n> +\t * spelling of the prefix. We have to use this canonical\n> +\t * path to do a lookup in the cache-entry array so that we\n> +\t * we repeat the original search using the case-corrected\n> +\t * spelling.\n> +\t */\n> +\tstrbuf_addch(&canonical_path, '/');\n> +\tpos = index_name_pos(istate, canonical_path.buf,\n> +\t\t\t canonical_path.len);\n> +\tnr_in_cone = fsmonitor_refresh_callback_slash(\n> +\t\tistate, canonical_path.buf, canonical_path.len, pos);\n> +\tstrbuf_release(&canonical_path);\n> +\treturn nr_in_cone;\n> +}\n> +\n> static void fsmonitor_refresh_callback_unqualified(\n> \tstruct index_state *istate, const char *name, int len, int pos)\n> {\n> @@ -269,7 +351,10 @@ static void fsmonitor_refresh_callback_unqualified(\n> *\n> * Return the number of cache-entries that we invalidated. We will\n> * use this later to determine if we need to attempt a second\n> - * case-insensitive search.\n> + * case-insensitive search. That is, if a observed-case search yields\n> + * any results, we assume the prefix is case-correct. If there are\n> + * no matches, we still don't know if the observed path is simply\n> + * untracked or case-incorrect.\n> */\n> static int fsmonitor_refresh_callback_slash(\n> \tstruct index_state *istate, const char *name, int len, int pos)\n> @@ -293,17 +378,50 @@ static int fsmonitor_refresh_callback_slash(\n> \treturn nr_in_cone;\n> }\n> \n> +/*\n> + * On a case-insensitive FS, use the name-hash and directory name-hash\n> + * to map the case of the observed path to the canonical case expected\n> + * by the index.\n> + *\n> + * The given pathname includes the trailing slash.\n> + *\n> + * Return the number of cache-entries that we invalidated.\n> + */\n> +static int fsmonitor_refresh_callback_slash_icase(\n> +\tstruct index_state *istate, const char *name, int len)\n> +{\n> +\tint nr_in_cone;\n> +\n> +\t/*\n> +\t * Look for a case-incorrect sparse-index directory.\n> +\t */\n> +\tnr_in_cone = my_callback_name_hash(istate, name, len);\n> +\tif (nr_in_cone)\n> +\t\treturn nr_in_cone;\n> +\n> +\t/*\n> +\t * (len-1) because we do not include the trailing slash in the\n> +\t * pathname.\n> +\t */\n> +\tnr_in_cone = my_callback_dir_name_hash(istate, name, len-1);\n> +\treturn nr_in_cone;\n> +}\n> +\n> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> {\n> \tint len = strlen(name);\n> \tint pos = index_name_pos(istate, name, len);\n> +\tint nr_in_cone;\n> +\n> \n> \ttrace_printf_key(&trace_fsmonitor,\n> \t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n> \t\t\t name, pos);\n> \n> \tif (name[len - 1] == '/') {\n> -\t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n> +\t\tnr_in_cone = fsmonitor_refresh_callback_slash(istate, name, len, pos);\n> +\t\tif (ignore_case && !nr_in_cone)\n> +\t\t\tfsmonitor_refresh_callback_slash_icase(istate, name, len);\n> \t} else {\n> \t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n> \t}\n> -- \n> gitgitgadget\n> \n> \n"},{"id":"488711","messageId":"Zc3aQ2qoK15w9dkp@tanuki","threadId":"60908","inReplyTo":"7775de735f41bdc601318cd15c3414f12b361a0c.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 11/12] fsmonitor: refactor bit invalidation in refresh callback","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-15T09:32:51Z","receivedAt":"2024-02-15T09:32:55Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 13, 2024 at 08:52:20PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Refactor code in the fsmonitor_refresh_callback() call chain dealing\n> with invalidating the CE_FSMONITOR_VALID bit and add a trace message.\n> \n> During the refresh, we clear the CE_FSMONITOR_VALID bit in response to\n> data from the FSMonitor daemon (so that a later phase will lstat() and\n> verify the true state of the file).\n> \n> Create a new function to clear the bit and add some unique tracing for\n> it to help debug edge cases.\n> \n> This is similar to the existing `mark_fsmonitor_invalid()` function,\n> but we don't need the extra stuff that it does.\n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 20 +++++++++++++++++---\n> 1 file changed, 17 insertions(+), 3 deletions(-)\n> \n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index a7847f07a40..75c7f73f68d 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -209,6 +209,20 @@ static void my_invalidate_untracked_cache(\n> \tstrbuf_release(&work_path);\n> }\n> \n> +/*\n> + * Invalidate the FSM bit on this CE. This is like mark_fsmonitor_invalid()\n> + * but we've already handled the untracked-cache and I want a different\n> + * trace message.\n> + */\n> +static void my_invalidate_ce_fsm(struct cache_entry *ce)\n> +{\n> +\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n> +\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t \"fsmonitor_refresh_cb_invalidate '%s'\",\n> +\t\t\t\t ce->name);\n> +\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n> +}\n\nSame comment here regarding the `my_` prefix.\n\nPatrick\n\n> +\n> /*\n> * Use the name-hash to lookup the pathname.\n> *\n> @@ -240,7 +254,7 @@ static int my_callback_name_hash(\n> \n> \tmy_invalidate_untracked_cache(istate, ce->name, ce->ce_namelen);\n> \n> -\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\tmy_invalidate_ce_fsm(ce);\n> \treturn 1;\n> }\n> \n> @@ -312,7 +326,7 @@ static int fsmonitor_refresh_callback_unqualified(\n> \t\t * cache-entry with the same pathname, nor for a cone\n> \t\t * at that directory. (That is, assume no D/F conflicts.)\n> \t\t */\n> -\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\tmy_invalidate_ce_fsm(istate->cache[pos]);\n> \t\treturn 1;\n> \t} else {\n> \t\tint nr_in_cone;\n> @@ -412,7 +426,7 @@ static int fsmonitor_refresh_callback_slash(\n> \tfor (i = pos; i < istate->cache_nr; i++) {\n> \t\tif (!starts_with(istate->cache[i]->name, name))\n> \t\t\tbreak;\n> -\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\tmy_invalidate_ce_fsm(istate->cache[i]);\n> \t\tnr_in_cone++;\n> \t}\n> \n> -- \n> gitgitgadget\n> \n> \n"},{"id":"488712","messageId":"Zc3aS0Top36mf4wi@tanuki","threadId":"60908","inReplyTo":"a0cc4c8274c108d231816ac981781d9e3cf8fded.1707857541.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 09/12] fsmonitor: refactor non-directory callback","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-15T09:32:59Z","receivedAt":"2024-02-15T09:33:03Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 13, 2024 at 08:52:18PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Refactor the fsmonitor_refresh_callback_unqualified() code\n> to try to use the _callback_slash() code and avoid having\n> a custom filter in the child cache-entry scanner.\n> \n> On platforms that DO NOT annotate FS events with a trailing\n> slash, if we fail to find an exact match for the pathname\n> in the index, we do not know if the pathname represents a\n> directory or simply an untracked file. Pretend that the pathname\n> is a directory and try again before assuming it is an untracked\n> file.\n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 59 +++++++++++++++++++++++++++++++----------------------\n> 1 file changed, 35 insertions(+), 24 deletions(-)\n> \n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 73e6ac82af7..cb27bae8aa8 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -287,41 +287,52 @@ static int my_callback_dir_name_hash(\n> \treturn nr_in_cone;\n> }\n> \n> -static void fsmonitor_refresh_callback_unqualified(\n> +/*\n> + * The daemon sent an observed pathname without a trailing slash.\n> + * (This is the normal case.) We do not know if it is a tracked or\n> + * untracked file, a sparse-directory, or a populated directory (on a\n> + * platform such as Windows where FSEvents are not qualified).\n> + *\n> + * The pathname contains the observed case reported by the FS. We\n> + * do not know it is case-correct or -incorrect.\n> + *\n> + * Assume it is case-correct and try an exact match.\n> + *\n> + * Return the number of cache-entries that we invalidated.\n> + */\n> +static int fsmonitor_refresh_callback_unqualified(\n> \tstruct index_state *istate, const char *name, int len, int pos)\n> {\n> -\tint i;\n> -\n> \tmy_invalidate_untracked_cache(istate, name, len);\n> \n> \tif (pos >= 0) {\n> \t\t/*\n> -\t\t * We have an exact match for this path and can just\n> -\t\t * invalidate it.\n> +\t\t * An exact match on a tracked file. We assume that we\n> +\t\t * do not need to scan forward for a sparse-directory\n> +\t\t * cache-entry with the same pathname, nor for a cone\n> +\t\t * at that directory. (That is, assume no D/F conflicts.)\n> \t\t */\n> \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\treturn 1;\n> \t} else {\n> +\t\tint nr_in_cone;\n> +\t\tstruct strbuf work_path = STRBUF_INIT;\n> +\n> \t\t/*\n> -\t\t * The path is not a tracked file -or- it is a\n> -\t\t * directory event on a platform that cannot\n> -\t\t * distinguish between file and directory events in\n> -\t\t * the event handler, such as Windows.\n> -\t\t *\n> -\t\t * Scan as if it is a directory and invalidate the\n> -\t\t * cone under it. (But remember to ignore items\n> -\t\t * between \"name\" and \"name/\", such as \"name-\" and\n> -\t\t * \"name.\".\n> +\t\t * The negative \"pos\" gives us the suggested insertion\n> +\t\t * point for the pathname (without the trailing slash).\n> +\t\t * We need to see if there is a directory with that\n> +\t\t * prefix, but there can be lots of pathnames between\n> +\t\t * \"foo\" and \"foo/\" like \"foo-\" or \"foo-bar\", so we\n> +\t\t * don't want to do our own scan.\n> \t\t */\n> -\t\tpos = -pos - 1;\n> -\n> -\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> -\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> -\t\t\t\tbreak;\n> -\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n> -\t\t\t\tbreak;\n> -\t\t\tif (istate->cache[i]->name[len] == '/')\n> -\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> -\t\t}\n> +\t\tstrbuf_add(&work_path, name, len);\n> +\t\tstrbuf_addch(&work_path, '/');\n> +\t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n> +\t\tnr_in_cone = fsmonitor_refresh_callback_slash(\n> +\t\t\tistate, work_path.buf, work_path.len, pos);\n> +\t\tstrbuf_release(&work_path);\n> +\t\treturn nr_in_cone;\n\nI didn't spot any users of this return value, and Junio also mentioned\nthis in a preceding patch for a different function. Would it make sense\nto introduce this return value as-needed in a later patch so that the\nreader isn't left wondering why it's changed now?\n\nPatrick\n\n> \t}\n> }\n> \n> -- \n> gitgitgadget\n> \n> \n"},{"id":"488998","messageId":"1913ed1b-a145-e641-6601-d8a55a2a8fec@jeffhostetler.com","threadId":"60908","inReplyTo":"xmqqeddg2g7j.fsf@gitster.g","subject":"Re: [PATCH 02/12] name-hash: add index_dir_exists2()","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-02-20T17:38:26Z","receivedAt":"2024-02-20T17:38:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/13/24 4:43 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> From: Jeff Hostetler \n>>\n>> Create a new version of index_dir_exists() to return the canonical\n>> spelling of the matched directory prefix.\n>>\n>> The existing index_dir_exists() returns a boolean to indicate if\n>> there is a case-insensitive match in the directory name-hash, but\n>> it doesn't tell the caller the exact spelling of that match.\n>>\n>> The new version also copies the matched spelling to a provided strbuf.\n>> This lets the caller, for example, then call index_name_pos() with the\n>> correct case to search the cache-entry array for the real insertion\n>> position.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> name-hash.c | 16 ++++++++++++++++\n>> name-hash.h | 2 ++\n>> 2 files changed, 18 insertions(+)\n>>\n>> diff --git a/name-hash.c b/name-hash.c\n>> index 251f036eef6..d735c81acb3 100644\n>> --- a/name-hash.c\n>> +++ b/name-hash.c\n>> @@ -694,6 +694,22 @@ int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n>> \tdir = find_dir_entry(istate, name, namelen);\n>> \treturn dir && dir->nr;\n>> }\n>> +int index_dir_exists2(struct index_state *istate, const char *name, int namelen,\n>> +\t\t struct strbuf *canonical_path)\n>> +{\n>> +\tstruct dir_entry *dir;\n>> +\n>> +\tstrbuf_init(canonical_path, namelen+1);\n>> +\n>> +\tlazy_init_name_hash(istate);\n>> +\texpand_to_path(istate, name, namelen, 0);\n>> +\tdir = find_dir_entry(istate, name, namelen);\n>> +\n>> +\tif (dir && dir->nr)\n>> +\t\tstrbuf_add(canonical_path, dir->name, dir->namelen);\n>> +\n>> +\treturn dir && dir->nr;\n>> +}\n>> \n>> void adjust_dirname_case(struct index_state *istate, char *name)\n> \n> Missing inter-function blank line, before the new function.\n> \n> I wonder if we can avoid such repetition---the body of\n> index_dir_exists() is 100% shared with this new function.\n> \n> Isn't it extremely unusual to receive \"struct strbuf *\" and call\n> strbuf_init() on it? It means that the caller is expected to have a\n> strbuf and pass a pointer to it, but also it is expected to leave\n> the strbuf uninitialized.\n> \n> I'd understand if it calls strbuf_reset(), but it may not even be\n> necessary, if we make it responsibility of the caller to pass a\n> valid strbuf to be appended into.\n> \n> \tint index_dir_find(struct index_state *istate,\n> \t\t\t const char *name, int namelen,\n> \t\t\t struct strbuf *canonical_path)\n> \t{\n> struct dir_entry *dir;\n> \n> lazy_init_name_hash(istate);\n> expand_to_path(istate, name, namelen, 0);\n> dir = find_dir_entry(istate, name, namelen);\n> \n> if (canonical_path && dir && dir->nr) {\n> \t\t\t// strbuf_reset(canonical_path); ???\n> \tstrbuf_add(canonical_path, dir->name, dir->namelen);\n> \t\t}\n> return dir && dir->nr;\n> \t}\n> \n> Then we can do\n> \n> \t#define index_dir_exists(i, n, l) index_dir_find((i), (n), (l), NULL)\n> \n> in the header for existing callers.\n> \n\nI'm always a little hesitant to change the signature of an existing\nfunction and chasing all of the callers in the middle of another\ntask. It can sometimes be distracting to reviewers.\n\nI like your macro approach here. I'll do that in the next version.\n\nThanks\nJeff\n"},{"id":"488999","messageId":"d40fada8-a3b6-1f41-2ce6-e0813f6ed26e@jeffhostetler.com","threadId":"60908","inReplyTo":"xmqqzfw410j8.fsf@gitster.g","subject":"Re: [PATCH 01/12] sparse-index: pass string length to index_file_exists()","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-02-20T17:34:59Z","receivedAt":"2024-02-20T17:40:18Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/13/24 5:07 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> From: Jeff Hostetler \n>>\n>> The call to index_file_exists() in the loop in expand_to_path() passes\n>> the wrong string length. Let's fix that.\n>>\n>> The loop in expand_to_path() searches the name-hash for each\n>> sub-directory prefix in the provided pathname. That is, by searching\n>> for \"dir1/\" then \"dir1/dir2/\" then \"dir1/dir2/dir3/\" and so on until\n>> it finds a cache-entry representing a sparse directory.\n>>\n>> The code creates \"strbuf path_mutable\" to contain the working pathname\n>> and modifies the buffer in-place by temporarily replacing the character\n>> following each successive \"/\" with NUL for the duration of the call to\n>> index_file_exists().\n>>\n>> It does not update the strbuf.len during this substitution.\n>>\n>> Pass the patched length of the prefix path instead.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n> \n> This looked familiar, and it turns out that\n> \n> https://lore.kernel.org/git/pull.1649.git.1706897095273.gitgitgadget@gmail.com/\n> \n> has already been merged to 'master'.\n> \n> \n\nGreat, thanks! I'll drop from the next version.\n\n"},{"id":"489009","messageId":"3fcb41e4-b17e-48ec-b54c-50452654b28c@jeffhostetler.com","threadId":"60908","inReplyTo":"Zc3aGSbEIFVClRd_@tanuki","subject":"Re: [PATCH 04/12] fsmonitor: refactor refresh callback on directory events","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-02-20T18:54:44Z","receivedAt":"2024-02-20T18:54:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/15/24 4:32 AM, Patrick Steinhardt wrote:\n> On Tue, Feb 13, 2024 at 08:52:13PM +0000, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> fsmonitor.c | 52 ++++++++++++++++++++++++++++++----------------------\n>> 1 file changed, 30 insertions(+), 22 deletions(-)\n>>\n>> diff --git a/fsmonitor.c b/fsmonitor.c\n>> index f670c509378..b1ef01bf3cd 100644\n>> --- a/fsmonitor.c\n>> +++ b/fsmonitor.c\n>> @@ -183,6 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n>> \treturn result;\n>> }\n>> \n>> +static void fsmonitor_refresh_callback_slash(\n>> +\tstruct index_state *istate, const char *name, int len, int pos)\n> \n> `len` should be `size_t` as it tracks the length of the name. This is\n> a preexisting issue already because `fsmonitor_refresh_callback()`\n> assigns `int len = strlen(name)`, which is wrong.\n> \n>> +{\n>> +\tint i;\n> \n> `i` is used to iterate through `istate->cache_nr`, which is an `unsigned\n> int` and not an `int`. I really wish we would compile the Git code base\n> with `-Wconversion`, but that's a rather big undertaking.\n> \n> Anyway, none of these issues are new as you merely move the code into a\n> new function.\n> \n> Patrick\n> \n\nYeah, the int types are bit of a mess, but I'd like to defer that to\nanother series.\n\nThere are several things going on here as you point out.\n\n(1) 'int len' for the length of a pathname buffer. This could be fixed,\nbut the odds of Git correctly working with a >2Gb pathname doesn't make\nthis urgent.\n\n(2) 'int i' is signed, but should be unsigned -- but elsewhere in this\nfunction we use 'int pos' which an index on the same array and has\ndouble duty as the suggested insertion point when negative. So we can\nfix the type of 'i', but that just sets us up to hide errors with 'pos',\nsince they'd have different types. Also, since it is really unlikely\nfor Git to work with >2G cache-entries, I think this one can wait too.\n\nI don't mean to be confrontational, but I think these changes should\nwait until another series that is focused on just those types of issues.\n\nJeff\n\n"},{"id":"489010","messageId":"b379b92f-407a-c4b3-ee65-a7fa6e7dff74@jeffhostetler.com","threadId":"60908","inReplyTo":"xmqqil2rzdvo.fsf@gitster.g","subject":"Re: [PATCH 06/12] fsmonitor: clarify handling of directory events in callback","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-02-20T18:56:01Z","receivedAt":"2024-02-20T18:56:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/14/24 2:47 AM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> From: Jeff Hostetler \n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> fsmonitor.c | 47 +++++++++++++++++++++++++++++++++--------------\n>> 1 file changed, 33 insertions(+), 14 deletions(-)\n>>\n>> diff --git a/fsmonitor.c b/fsmonitor.c\n>> index 614270fa5e8..754fe20cfd0 100644\n>> --- a/fsmonitor.c\n>> +++ b/fsmonitor.c\n>> @@ -219,24 +219,40 @@ static void fsmonitor_refresh_callback_unqualified(\n>> \t}\n>> }\n>> \n>> -static void fsmonitor_refresh_callback_slash(\n...\n>> +static int fsmonitor_refresh_callback_slash(\n>> \tstruct index_state *istate, const char *name, int len, int pos)\n>> {\n> \n> This was split out a few patches ago, and the caller of course\n> ignored the return value (void), but now it turns an integer, and\n> this change is without a corresponding update to the caller, which\n> leaves readers puzzled.\n> \n> Perhaps a future patch either updates the existing caller or adds a\n> new caller that utilize the returned value, but then at least the\n> proposed commit message for this step should hint how it helps the\n> caller(s) we will see in the future steps if this function returns\n> the number of entries invalidated, iow, how the caller is expected\n> to use the returned value from here, no?\n> \n> Alternatively, this step can limit itself to what the commit title\n> claims to do---to clarify what the helper does with enhanced in-code\n> comments. Then a future step that updates the caller to care about\n> the return value can have both the changes to this callee as well as\n> the caller---which may make it easier to see how the returned info\n> helps the caller. I dunno which is more reasonable.\n> \n\nI'll split this into 2 commits. One for the refactor and one for\nthe new return value.\n\nJeff\n\n"},{"id":"489012","messageId":"3ffd638e-d2e1-ab39-e44f-84d2c58d8126@jeffhostetler.com","threadId":"60908","inReplyTo":"Zc3aKSR53Ew4ImMP@tanuki","subject":"Re: [PATCH 06/12] fsmonitor: clarify handling of directory events in callback","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-02-20T19:10:38Z","receivedAt":"2024-02-20T19:10:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/15/24 4:32 AM, Patrick Steinhardt wrote:\n> On Tue, Feb 13, 2024 at 08:52:15PM +0000, Jeff Hostetler via GitGitGadget wrote:\n>> From: Jeff Hostetler \n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> fsmonitor.c | 47 +++++++++++++++++++++++++++++++++--------------\n>> 1 file changed, 33 insertions(+), 14 deletions(-)\n>>\n>> diff --git a/fsmonitor.c b/fsmonitor.c\n>> index 614270fa5e8..754fe20cfd0 100644\n>> --- a/fsmonitor.c\n>> +++ b/fsmonitor.c\n>> @@ -219,24 +219,40 @@ static void fsmonitor_refresh_callback_unqualified(\n...\n>> +static int fsmonitor_refresh_callback_slash(\n>> \tstruct index_state *istate, const char *name, int len, int pos)\n>> {\n>> \tint i;\n>> +\tint nr_in_cone = 0;\n> \n> Should we return `size_t` instead of `int`?\n> \n> Patrick\n\nyeah, I can fix all of the return values to be 'size_t' since\nthat is new functionality and not colliding with the existing\nusages for 'i' and 'pos' that I mentioned in a response on a\nprevious thread.\n\nThanks\nJeff\n\n"},{"id":"489015","messageId":"xmqqjzmzndli.fsf@gitster.g","threadId":"60908","inReplyTo":"b379b92f-407a-c4b3-ee65-a7fa6e7dff74@jeffhostetler.com","subject":"Re: [PATCH 06/12] fsmonitor: clarify handling of directory events in callback","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-20T19:24:25Z","receivedAt":"2024-02-20T19:24:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> I'll split this into 2 commits. One for the refactor and one for\n> the new return value.\n\nAnd the latter one that makes the return value richer contains the\ncaller that makes use of the returned value? That's great. It\nwould make it very much easier to read the resulting commit, as the\npresence of the callers and how they use the returned value would\nmake it self evident why it makes sense to return the number of\nentries invalidated.\n\nThanks.\n"},{"id":"489018","messageId":"xmqqedd7nd4u.fsf@gitster.g","threadId":"60908","inReplyTo":"1913ed1b-a145-e641-6601-d8a55a2a8fec@jeffhostetler.com","subject":"Re: [PATCH 02/12] name-hash: add index_dir_exists2()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-20T19:34:25Z","receivedAt":"2024-02-20T19:34:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> I'm always a little hesitant to change the signature of an existing\n> function and chasing all of the callers in the middle of another\n> task. It can sometimes be distracting to reviewers.\n\nOf course we all should be hesitant. In addition to reviewers,\nthere are topics in flight and topics people are cooking but not\nposted that will be affected. So it is perfectly fine to introduce\nan enhanced version as needed under different name (but let's not\ngive it a meaningless name like \"foo2\" where it is totally unclear\nand unexplained what its difference from \"foo\" is from the name),\nbut if it is meant as an enhanced version, we should aim to share\nthe code and rewrite the original in terms of the enhanced one,\ninstead of simply duplicating to risk unnecessary divergence of the\ntwo functions in the future.\n\nThanks.\n"},{"id":"489077","messageId":"ZdXyi2erhfoW5CMW@tanuki","threadId":"60908","inReplyTo":"3fcb41e4-b17e-48ec-b54c-50452654b28c@jeffhostetler.com","subject":"Re: [PATCH 04/12] fsmonitor: refactor refresh callback on directory events","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-02-21T12:54:35Z","receivedAt":"2024-02-21T12:54:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Feb 20, 2024 at 01:54:44PM -0500, Jeff Hostetler wrote:\n> \n> \n> On 2/15/24 4:32 AM, Patrick Steinhardt wrote:\n> > On Tue, Feb 13, 2024 at 08:52:13PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> > > From: Jeff Hostetler \n> > > \n> > > Signed-off-by: Jeff Hostetler \n> > > ---\n> > > fsmonitor.c | 52 ++++++++++++++++++++++++++++++----------------------\n> > > 1 file changed, 30 insertions(+), 22 deletions(-)\n> > > \n> > > diff --git a/fsmonitor.c b/fsmonitor.c\n> > > index f670c509378..b1ef01bf3cd 100644\n> > > --- a/fsmonitor.c\n> > > +++ b/fsmonitor.c\n> > > @@ -183,6 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n> > > \treturn result;\n> > > }\n> > > +static void fsmonitor_refresh_callback_slash(\n> > > +\tstruct index_state *istate, const char *name, int len, int pos)\n> > \n> > `len` should be `size_t` as it tracks the length of the name. This is\n> > a preexisting issue already because `fsmonitor_refresh_callback()`\n> > assigns `int len = strlen(name)`, which is wrong.\n> > \n> > > +{\n> > > +\tint i;\n> > \n> > `i` is used to iterate through `istate->cache_nr`, which is an `unsigned\n> > int` and not an `int`. I really wish we would compile the Git code base\n> > with `-Wconversion`, but that's a rather big undertaking.\n> > \n> > Anyway, none of these issues are new as you merely move the code into a\n> > new function.\n> > \n> > Patrick\n> > \n> \n> Yeah, the int types are bit of a mess, but I'd like to defer that to\n> another series.\n> \n> There are several things going on here as you point out.\n> \n> (1) 'int len' for the length of a pathname buffer. This could be fixed,\n> but the odds of Git correctly working with a >2Gb pathname doesn't make\n> this urgent.\n> \n> (2) 'int i' is signed, but should be unsigned -- but elsewhere in this\n> function we use 'int pos' which an index on the same array and has\n> double duty as the suggested insertion point when negative. So we can\n> fix the type of 'i', but that just sets us up to hide errors with 'pos',\n> since they'd have different types. Also, since it is really unlikely\n> for Git to work with >2G cache-entries, I think this one can wait too.\n> \n> I don't mean to be confrontational, but I think these changes should\n> wait until another series that is focused on just those types of issues.\n\nNo worries, this doesn't come across as confrontational at all. As I\nsaid myself, the problems aren't really new to your patch series but are\nare preexisting and run deeper than what you can see in the diffs here.\nSo I'm perfectly fine with deferring this.\n\nPatrick\n"},{"id":"489195","messageId":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.git.1707857541.gitgitgadget@gmail.com","subject":"[PATCH v2 00/16] FSMonitor edge cases on case-insensitive file systems","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:04Z","receivedAt":"2024-02-23T03:18:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 2. I think I have addressed all of the comments on the first\nversion and have greatly consolidated/streamlined the \"icase\" code.\n\n============== Fix FSMonitor client code to detect case-incorrect FSEvents\nand map them to the canonical case expected by the index.\n\nFSEvents are delivered to the FSMonitor daemon using the observed case which\nmay or may not match the expected case stored in the index for tracked files\nand/or directories. This caused index_name_pos() to report a negative index\nposition (defined as the suggested insertion point). Since the value was\nnegative, the FSMonitor refresh lookup would not invalidate the\nCE_FSMONITOR_VALID bit on the \"expected\" (case-insensitive-equivalent)\ncache-entries. Therefore, git status would not report them as modified.\n\nThis was a fairly obscure problem and only happened when the case of a\nsub-directory or a file was artificially changed.\n\nThis first runs the original lookup using the observed case. If that fails,\nit assumes that the observed pathname refers to a file and uses the\ncase-insensitive name-hash hashmap to find an equivalent path (cache-entry)\nin the index. If that fails, it assumes the pathname refers to a directory\nand uses the case-insensitive dir-name-hash to find the equivalent directory\nand then repeats the index_name_pos() lookup to find a directory or\nsuggested insertion point with the expected case.\n\nTwo new test cases were added to t7527 to demonstrate this.\n\nSince this was rather obscure, I also added some additional tracing under\nthe GIT_TRACE_FSMONITOR key.\n\nI also did considerable refactoring of the original code before adding the\nnew lookups.\n\nFinally, I made more explicit the relationship between the FSEvents and the\n(new) sparse-index directory cache-entries, since sparse-index was added\nslightly after the FSMonitor feature.\n\nJeff Hostetler (16):\n name-hash: add index_dir_find()\n t7527: add case-insensitve test for FSMonitor\n t7527: temporarily disable case-insensitive tests\n fsmonitor: refactor refresh callback on directory events\n fsmonitor: clarify handling of directory events in callback helper\n fsmonitor: refactor refresh callback for non-directory events\n dir: create untracked_cache_invalidate_trimmed_path()\n fsmonitor: refactor untracked-cache invalidation\n fsmonitor: move untracked invalidation into helper functions\n fsmonitor: return invalidated cache-entry count on directory event\n fsmonitor: remove custom loop from non-directory path handler\n fsmonitor: return invalided cache-entry count on non-directory event\n fsmonitor: trace the new invalidated cache-entry count\n fsmonitor: support case-insensitive events\n fsmonitor: refactor bit invalidation in refresh callback\n t7527: update case-insenstive fsmonitor test\n\n dir.c | 20 +++\n dir.h | 7 +\n fsmonitor.c | 299 ++++++++++++++++++++++++++++-------\n name-hash.c | 9 +-\n name-hash.h | 7 +-\n t/t7527-builtin-fsmonitor.sh | 220 ++++++++++++++++++++++++++\n 6 files changed, 506 insertions(+), 56 deletions(-)\n\n\nbase-commit: f41f85c9ec8d4d46de0fd5fded88db94d3ec8c11\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1662%2Fjeffhostetler%2Ffsmonitor-ignore-case-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1662/jeffhostetler/fsmonitor-ignore-case-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/1662\n\nRange-diff vs v1:\n\n 1: 6f81e2e3060 < -: ----------- sparse-index: pass string length to index_file_exists()\n 2: 3464545fe3f ! 1: 03b07d9c25e name-hash: add index_dir_exists2()\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - name-hash: add index_dir_exists2()\n + name-hash: add index_dir_find()\n \n - Create a new version of index_dir_exists() to return the canonical\n + Replace the index_dir_exists() function with index_dir_find() and\n + change the API to take an optional strbuf to return the canonical\n spelling of the matched directory prefix.\n \n + Create an index_dir_exists() wrapper macro for existing callers.\n +\n The existing index_dir_exists() returns a boolean to indicate if\n there is a case-insensitive match in the directory name-hash, but\n it doesn't tell the caller the exact spelling of that match.\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## name-hash.c ##\n -@@ name-hash.c: int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n - \tdir = find_dir_entry(istate, name, namelen);\n - \treturn dir && dir->nr;\n +@@ name-hash.c: static int same_name(const struct cache_entry *ce, const char *name, int namelen\n + \treturn slow_same_name(name, namelen, ce->name, len);\n }\n -+int index_dir_exists2(struct index_state *istate, const char *name, int namelen,\n -+\t\t struct strbuf *canonical_path)\n -+{\n -+\tstruct dir_entry *dir;\n -+\n -+\tstrbuf_init(canonical_path, namelen+1);\n -+\n -+\tlazy_init_name_hash(istate);\n -+\texpand_to_path(istate, name, namelen, 0);\n -+\tdir = find_dir_entry(istate, name, namelen);\n + \n +-int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n ++int index_dir_find(struct index_state *istate, const char *name, int namelen,\n ++\t\t struct strbuf *canonical_path)\n + {\n + \tstruct dir_entry *dir;\n + \n + \tlazy_init_name_hash(istate);\n + \texpand_to_path(istate, name, namelen, 0);\n + \tdir = find_dir_entry(istate, name, namelen);\n +\n -+\tif (dir && dir->nr)\n ++\tif (canonical_path && dir && dir->nr) {\n ++\t\tstrbuf_reset(canonical_path);\n +\t\tstrbuf_add(canonical_path, dir->name, dir->namelen);\n ++\t}\n +\n -+\treturn dir && dir->nr;\n -+}\n + \treturn dir && dir->nr;\n + }\n \n - void adjust_dirname_case(struct index_state *istate, char *name)\n - {\n \n ## name-hash.h ##\n -@@ name-hash.h: struct cache_entry;\n +@@\n + struct cache_entry;\n struct index_state;\n \n - int index_dir_exists(struct index_state *istate, const char *name, int namelen);\n -+int index_dir_exists2(struct index_state *istate, const char *name, int namelen,\n -+\t\t struct strbuf *canonical_path);\n +-int index_dir_exists(struct index_state *istate, const char *name, int namelen);\n ++\n ++int index_dir_find(struct index_state *istate, const char *name, int namelen,\n ++\t\t struct strbuf *canonical_path);\n ++\n ++#define index_dir_exists(i, n, l) index_dir_find((i), (n), (l), NULL)\n ++\n void adjust_dirname_case(struct index_state *istate, char *name);\n struct cache_entry *index_file_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n \n 3: 272d7805f47 = 2: 7778cee1c10 t7527: add case-insensitve test for FSMonitor\n -: ----------- > 3: dad079ade7f t7527: temporarily disable case-insensitive tests\n 4: 3fb8e0d0a7c ! 4: 5516670e30e fsmonitor: refactor refresh callback on directory events\n @@ Metadata\n ## Commit message ##\n fsmonitor: refactor refresh callback on directory events\n \n + Move the code to handle directory FSEvents (containing pathnames with\n + a trailing slash) into a helper function.\n +\n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n @@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n -+static void fsmonitor_refresh_callback_slash(\n -+\tstruct index_state *istate, const char *name, int len, int pos)\n ++static void handle_path_with_trailing_slash(\n ++\tstruct index_state *istate, const char *name, int pos)\n +{\n +\tint i;\n +\n @@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate,\n -\t\t\t\tbreak;\n -\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n -\t\t}\n -+\t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n ++\t\thandle_path_with_trailing_slash(istate, name, pos);\n \n \t\t/*\n \t\t * We need to remove the traling \"/\" from the path\n 6: 5b6f8bd1fe7 ! 5: c04fd4eae94 fsmonitor: clarify handling of directory events in callback\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: clarify handling of directory events in callback\n + fsmonitor: clarify handling of directory events in callback helper\n +\n + Improve documentation of the refresh callback helper function\n + used for directory FSEvents.\n \n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n -@@ fsmonitor.c: static void fsmonitor_refresh_callback_unqualified(\n - \t}\n +@@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n + \treturn result;\n }\n \n --static void fsmonitor_refresh_callback_slash(\n +/*\n + * The daemon can decorate directory events, such as a move or rename,\n + * by adding a trailing slash to the observed name. Use this to\n @@ fsmonitor.c: static void fsmonitor_refresh_callback_unqualified(\n + * same way and just invalidate the cache-entry and the untracked\n + * cache (and in this case, the forward cache-entry scan won't find\n + * anything and it doesn't hurt to let it run).\n -+ *\n -+ * Return the number of cache-entries that we invalidated. We will\n -+ * use this later to determine if we need to attempt a second\n -+ * case-insensitive search.\n + */\n -+static int fsmonitor_refresh_callback_slash(\n - \tstruct index_state *istate, const char *name, int len, int pos)\n + static void handle_path_with_trailing_slash(\n + \tstruct index_state *istate, const char *name, int pos)\n {\n \tint i;\n -+\tint nr_in_cone = 0;\n \n -\t/*\n -\t * The daemon can decorate directory events, such as\n @@ fsmonitor.c: static void fsmonitor_refresh_callback_unqualified(\n \tif (pos < 0)\n \t\tpos = -pos - 1;\n \n -@@ fsmonitor.c: static void fsmonitor_refresh_callback_slash(\n - \t\tif (!starts_with(istate->cache[i]->name, name))\n - \t\t\tbreak;\n - \t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n -+\t\tnr_in_cone++;\n - \t}\n -+\n -+\treturn nr_in_cone;\n - }\n - \n - static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n 5: 0896d4af907 ! 6: 7ee6ca1aefd fsmonitor: refactor refresh callback for non-directory events\n @@ Metadata\n ## Commit message ##\n fsmonitor: refactor refresh callback for non-directory events\n \n + Move the code handle unqualified FSEvents (without a trailing slash)\n + into a helper function.\n +\n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n @@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n -+static void fsmonitor_refresh_callback_unqualified(\n -+\tstruct index_state *istate, const char *name, int len, int pos)\n ++static void handle_path_without_trailing_slash(\n ++\tstruct index_state *istate, const char *name, int pos)\n +{\n +\tint i;\n +\n @@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n +\t\t * between \"name\" and \"name/\", such as \"name-\" and\n +\t\t * \"name.\".\n +\t\t */\n ++\t\tint len = strlen(name);\n +\t\tpos = -pos - 1;\n +\n +\t\tfor (i = pos; i < istate->cache_nr; i++) {\n @@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n +\t}\n +}\n +\n - static void fsmonitor_refresh_callback_slash(\n - \tstruct index_state *istate, const char *name, int len, int pos)\n - {\n -@@ fsmonitor.c: static void fsmonitor_refresh_callback_slash(\n + /*\n + * The daemon can decorate directory events, such as a move or rename,\n + * by adding a trailing slash to the observed name. Use this to\n +@@ fsmonitor.c: static void handle_path_with_trailing_slash(\n \n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n @@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate,\n -\t\t\tif (istate->cache[i]->name[len] == '/')\n -\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n -\t\t}\n -+\t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n ++\t\thandle_path_without_trailing_slash(istate, name, pos);\n \t}\n \n \t/*\n -: ----------- > 7: 99c0d3e0742 dir: create untracked_cache_invalidate_trimmed_path()\n -: ----------- > 8: f2d6765d84f fsmonitor: refactor untracked-cache invalidation\n 7: 1df4019931c ! 9: af6f57ab3e6 fsmonitor: refactor untracked-cache invalidation\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: refactor untracked-cache invalidation\n + fsmonitor: move untracked invalidation into helper functions\n +\n + Move the call to invalidate the untracked cache for the FSEvent\n + pathname into the two helper functions.\n +\n + In a later commit in this series, we will call these helpers\n + from other contexts and it safer to include the UC invalidation\n + in the helper than to remember to also add it to each helper\n + call-site.\n \n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n -@@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n - \treturn result;\n - }\n - \n -+/*\n -+ * Invalidate the untracked cache for the given pathname. Copy the\n -+ * buffer to a proper null-terminated string (since the untracked\n -+ * cache code does not use (buf, len) style argument). Also strip any\n -+ * trailing slash.\n -+ */\n -+static void my_invalidate_untracked_cache(\n -+\tstruct index_state *istate, const char *name, int len)\n -+{\n -+\tstruct strbuf work_path = STRBUF_INIT;\n -+\n -+\tif (!len)\n -+\t\treturn;\n -+\n -+\tif (name[len-1] == '/')\n -+\t\tlen--;\n -+\n -+\tstrbuf_add(&work_path, name, len);\n -+\tuntracked_cache_invalidate_path(istate, work_path.buf, 0);\n -+\tstrbuf_release(&work_path);\n -+}\n -+\n - static void fsmonitor_refresh_callback_unqualified(\n - \tstruct index_state *istate, const char *name, int len, int pos)\n +@@ fsmonitor.c: static void handle_path_without_trailing_slash(\n {\n \tint i;\n \n -+\tmy_invalidate_untracked_cache(istate, name, len);\n ++\t/*\n ++\t * Mark the untracked cache dirty for this path (regardless of\n ++\t * whether or not we find an exact match for it in the index).\n ++\t * Since the path is unqualified (no trailing slash hint in the\n ++\t * FSEvent), it may refer to a file or directory. So we should\n ++\t * not assume one or the other and should always let the untracked\n ++\t * cache decide what needs to invalidated.\n ++\t */\n ++\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n +\n \tif (pos >= 0) {\n \t\t/*\n \t\t * We have an exact match for this path and can just\n -@@ fsmonitor.c: static int fsmonitor_refresh_callback_slash(\n +@@ fsmonitor.c: static void handle_path_with_trailing_slash(\n + {\n \tint i;\n - \tint nr_in_cone = 0;\n \n -+\tmy_invalidate_untracked_cache(istate, name, len);\n ++\t/*\n ++\t * Mark the untracked cache dirty for this directory path\n ++\t * (regardless of whether or not we find an exact match for it\n ++\t * in the index or find it to be proper prefix of one or more\n ++\t * files in the index), since the FSEvent is hinting that\n ++\t * there may be changes on or within the directory.\n ++\t */\n ++\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n +\n \tif (pos < 0)\n \t\tpos = -pos - 1;\n \n @@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n - \n - \tif (name[len - 1] == '/') {\n - \t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n --\n --\t\t/*\n --\t\t * We need to remove the traling \"/\" from the path\n --\t\t * for the untracked cache.\n --\t\t */\n --\t\tname[len - 1] = '\\0';\n \t} else {\n - \t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n + \t\thandle_path_without_trailing_slash(istate, name, pos);\n \t}\n -\n -\t/*\n -\t * Mark the untracked cache dirty even if it wasn't found in the index\n --\t * as it could be a new untracked file.\n +-\t * as it could be a new untracked file. (Let the untracked cache\n +-\t * layer silently deal with any trailing slash.)\n -\t */\n --\tuntracked_cache_invalidate_path(istate, name, 0);\n +-\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n }\n \n /*\n -: ----------- > 10: 623c6f06e21 fsmonitor: return invalidated cache-entry count on directory event\n 9: a0cc4c8274c ! 11: 1853f77d333 fsmonitor: refactor non-directory callback\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: refactor non-directory callback\n + fsmonitor: remove custom loop from non-directory path handler\n \n - Refactor the fsmonitor_refresh_callback_unqualified() code\n - to try to use the _callback_slash() code and avoid having\n - a custom filter in the child cache-entry scanner.\n + Refactor the code that handles refresh events for pathnames that do\n + not contain a trailing slash. Instead of using a custom loop to try\n + to scan the index and detect if the FSEvent named a file or might be a\n + directory prefix, use the recently created helper function to do that.\n +\n + Also update the comments to describe what and why we are doing this.\n \n On platforms that DO NOT annotate FS events with a trailing\n slash, if we fail to find an exact match for the pathname\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n -@@ fsmonitor.c: static int my_callback_dir_name_hash(\n - \treturn nr_in_cone;\n +@@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n + \treturn result;\n }\n \n --static void fsmonitor_refresh_callback_unqualified(\n ++static size_t handle_path_with_trailing_slash(\n ++\tstruct index_state *istate, const char *name, int pos);\n ++\n +/*\n + * The daemon sent an observed pathname without a trailing slash.\n + * (This is the normal case.) We do not know if it is a tracked or\n @@ fsmonitor.c: static int my_callback_dir_name_hash(\n + * do not know it is case-correct or -incorrect.\n + *\n + * Assume it is case-correct and try an exact match.\n -+ *\n -+ * Return the number of cache-entries that we invalidated.\n + */\n -+static int fsmonitor_refresh_callback_unqualified(\n - \tstruct index_state *istate, const char *name, int len, int pos)\n + static void handle_path_without_trailing_slash(\n + \tstruct index_state *istate, const char *name, int pos)\n {\n -\tint i;\n -\n - \tmy_invalidate_untracked_cache(istate, name, len);\n + \t/*\n + \t * Mark the untracked cache dirty for this path (regardless of\n + \t * whether or not we find an exact match for it in the index).\n +@@ fsmonitor.c: static void handle_path_without_trailing_slash(\n \n \tif (pos >= 0) {\n \t\t/*\n @@ fsmonitor.c: static int my_callback_dir_name_hash(\n +\t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n -+\t\treturn 1;\n \t} else {\n -+\t\tint nr_in_cone;\n +\t\tstruct strbuf work_path = STRBUF_INIT;\n +\n \t\t/*\n @@ fsmonitor.c: static int my_callback_dir_name_hash(\n +\t\t * \"foo\" and \"foo/\" like \"foo-\" or \"foo-bar\", so we\n +\t\t * don't want to do our own scan.\n \t\t */\n +-\t\tint len = strlen(name);\n -\t\tpos = -pos - 1;\n -\n -\t\tfor (i = pos; i < istate->cache_nr; i++) {\n @@ fsmonitor.c: static int my_callback_dir_name_hash(\n -\t\t\tif (istate->cache[i]->name[len] == '/')\n -\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n -\t\t}\n -+\t\tstrbuf_add(&work_path, name, len);\n ++\t\tstrbuf_add(&work_path, name, strlen(name));\n +\t\tstrbuf_addch(&work_path, '/');\n +\t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n -+\t\tnr_in_cone = fsmonitor_refresh_callback_slash(\n -+\t\t\tistate, work_path.buf, work_path.len, pos);\n ++\t\thandle_path_with_trailing_slash(istate, work_path.buf, pos);\n +\t\tstrbuf_release(&work_path);\n -+\t\treturn nr_in_cone;\n \t}\n }\n \n 10: bf18401f56c ! 12: f77d68c78ad fsmonitor: support case-insensitive non-directory events\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: support case-insensitive non-directory events\n + fsmonitor: return invalided cache-entry count on non-directory event\n +\n + Teah the refresh callback helper function for unqualified FSEvents\n + (pathnames without a trailing slash) to return the number of\n + cache-entries that were invalided in response to the event.\n +\n + This will be used in a later commit to help determine if the observed\n + pathname was (possibly) case-incorrect when (on a case-insensitive\n + file system).\n \n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n -@@ fsmonitor.c: static int fsmonitor_refresh_callback_unqualified(\n - \t}\n - }\n - \n -+/*\n -+ * On a case-insensitive FS, use the name-hash to map the case of\n -+ * the observed path to the canonical case expected by the index.\n -+ *\n -+ * The given pathname DOES NOT include the trailing slash.\n +@@ fsmonitor.c: static size_t handle_path_with_trailing_slash(\n + * do not know it is case-correct or -incorrect.\n + *\n + * Assume it is case-correct and try an exact match.\n + *\n + * Return the number of cache-entries that we invalidated.\n -+ */\n -+static int fsmonitor_refresh_callback_unqualified_icase(\n -+\tstruct index_state *istate, const char *name, int len)\n -+{\n -+\tint nr_in_cone;\n -+\n -+\t/*\n -+\t * Look for a case-incorrect match for this non-directory\n -+\t * pathname.\n -+\t */\n -+\tnr_in_cone = my_callback_name_hash(istate, name, len);\n -+\tif (nr_in_cone)\n -+\t\treturn nr_in_cone;\n -+\n -+\t/*\n -+\t * Try the directory name-hash and see if there is a\n -+\t * case-incorrect directory with this pathanme.\n -+\t * (len) because we don't have a trailing slash.\n -+\t */\n -+\tnr_in_cone = my_callback_dir_name_hash(istate, name, len);\n -+\treturn nr_in_cone;\n -+}\n -+\n - /*\n - * The daemon can decorate directory events, such as a move or rename,\n - * by adding a trailing slash to the observed name. Use this to\n -@@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n - \t\tif (ignore_case && !nr_in_cone)\n - \t\t\tfsmonitor_refresh_callback_slash_icase(istate, name, len);\n + */\n +-static void handle_path_without_trailing_slash(\n ++static size_t handle_path_without_trailing_slash(\n + \tstruct index_state *istate, const char *name, int pos)\n + {\n + \t/*\n +@@ fsmonitor.c: static void handle_path_without_trailing_slash(\n + \t\t * at that directory. (That is, assume no D/F conflicts.)\n + \t\t */\n + \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n ++\t\treturn 1;\n \t} else {\n --\t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n -+\t\tnr_in_cone = fsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n -+\t\tif (ignore_case && !nr_in_cone)\n -+\t\t\tfsmonitor_refresh_callback_unqualified_icase(istate, name, len);\n ++\t\tsize_t nr_in_cone;\n + \t\tstruct strbuf work_path = STRBUF_INIT;\n + \n + \t\t/*\n +@@ fsmonitor.c: static void handle_path_without_trailing_slash(\n + \t\tstrbuf_add(&work_path, name, strlen(name));\n + \t\tstrbuf_addch(&work_path, '/');\n + \t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n +-\t\thandle_path_with_trailing_slash(istate, work_path.buf, pos);\n ++\t\tnr_in_cone = handle_path_with_trailing_slash(\n ++\t\t\tistate, work_path.buf, pos);\n + \t\tstrbuf_release(&work_path);\n ++\t\treturn nr_in_cone;\n \t}\n }\n \n -: ----------- > 13: 58b36673e15 fsmonitor: trace the new invalidated cache-entry count\n 8: e0029a2aad6 ! 14: 288f3f4e54e fsmonitor: support case-insensitive directory events\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: support case-insensitive directory events\n + fsmonitor: support case-insensitive events\n \n Teach fsmonitor_refresh_callback() to handle case-insensitive\n lookups if case-sensitive lookups fail on case-insensitive systems.\n @@ Commit message\n to find the associated cache-entry. This causes status to think that\n the cached CE flags are correct and skip over the file.\n \n - Update the handling of directory-style FSEvents (ones containing a\n - path with a trailing slash) to optionally use the name-hash if the\n - case-correct search does not find a match.\n -\n - (The FSMonitor daemon can send directory FSEvents if the OS provides\n - that information.)\n + Update event handling to optionally use the name-hash and dir-name-hash\n + if necessary.\n \n Signed-off-by: Jeff Hostetler \n \n @@ fsmonitor.c\n #include \"strbuf.h\"\n #include \"trace2.h\"\n @@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n - \treturn result;\n - }\n - \n -+static int fsmonitor_refresh_callback_slash(\n -+\tstruct index_state *istate, const char *name, int len, int pos);\n -+\n - /*\n - * Invalidate the untracked cache for the given pathname. Copy the\n - * buffer to a proper null-terminated string (since the untracked\n -@@ fsmonitor.c: static void my_invalidate_untracked_cache(\n - \tstrbuf_release(&work_path);\n - }\n + static size_t handle_path_with_trailing_slash(\n + \tstruct index_state *istate, const char *name, int pos);\n \n +/*\n -+ * Use the name-hash to lookup the pathname.\n ++ * Use the name-hash to do a case-insensitive cache-entry lookup with\n ++ * the pathname and invalidate the cache-entry.\n + *\n + * Returns the number of cache-entries that we invalidated.\n + */\n -+static int my_callback_name_hash(\n -+\tstruct index_state *istate, const char *name, int len)\n ++static size_t handle_using_name_hash_icase(\n ++\tstruct index_state *istate, const char *name)\n +{\n +\tstruct cache_entry *ce = NULL;\n +\n -+\tce = index_file_exists(istate, name, len, 1);\n ++\tce = index_file_exists(istate, name, strlen(name), 1);\n +\tif (!ce)\n +\t\treturn 0;\n +\n +\t/*\n -+\t * The index contains a case-insensitive match for the pathname.\n -+\t * This could either be a regular file or a sparse-index directory.\n ++\t * A case-insensitive search in the name-hash using the\n ++\t * observed pathname found a cache-entry, so the observed path\n ++\t * is case-incorrect. Invalidate the cache-entry and use the\n ++\t * correct spelling from the cache-entry to invalidate the\n ++\t * untracked-cache. Since we now have sparse-directories in\n ++\t * the index, the observed pathname may represent a regular\n ++\t * file or a sparse-index directory.\n +\t *\n -+\t * We should not have seen FSEvents for a sparse-index directory,\n -+\t * but we handle it just in case.\n ++\t * Note that we should not have seen FSEvents for a\n ++\t * sparse-index directory, but we handle it just in case.\n +\t *\n +\t * Either way, we know that there are not any cache-entries for\n +\t * children inside the cone of the directory, so we don't need to\n +\t * do the usual scan.\n +\t */\n +\ttrace_printf_key(&trace_fsmonitor,\n -+\t\t\t \"fsmonitor_refresh_callback map '%s' '%s'\",\n ++\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n +\t\t\t name, ce->name);\n +\n -+\tmy_invalidate_untracked_cache(istate, ce->name, ce->ce_namelen);\n ++\tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n +\n +\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n +\treturn 1;\n +}\n +\n +/*\n -+ * Use the directory name-hash to find the correct-case spelling\n -+ * of the directory. Use the canonical spelling to invalidate all\n -+ * of the cache-entries within the matching cone.\n -+ *\n -+ * The pathname MUST NOT have a trailing slash.\n ++ * Use the dir-name-hash to find the correct-case spelling of the\n ++ * directory. Use the canonical spelling to invalidate all of the\n ++ * cache-entries within the matching cone.\n + *\n + * Returns the number of cache-entries that we invalidated.\n + */\n -+static int my_callback_dir_name_hash(\n -+\tstruct index_state *istate, const char *name, int len)\n ++static size_t handle_using_dir_name_hash_icase(\n ++\tstruct index_state *istate, const char *name)\n +{\n +\tstruct strbuf canonical_path = STRBUF_INIT;\n +\tint pos;\n -+\tint nr_in_cone;\n ++\tsize_t len = strlen(name);\n ++\tsize_t nr_in_cone;\n ++\n ++\tif (name[len - 1] == '/')\n ++\t\tlen--;\n +\n -+\tif (!index_dir_exists2(istate, name, len, &canonical_path))\n ++\tif (!index_dir_find(istate, name, len, &canonical_path))\n +\t\treturn 0; /* name is untracked */\n -+\tif (!memcmp(name, canonical_path.buf, len)) {\n ++\n ++\tif (!memcmp(name, canonical_path.buf, canonical_path.len)) {\n +\t\tstrbuf_release(&canonical_path);\n ++\t\t/*\n ++\t\t * NEEDSWORK: Our caller already tried an exact match\n ++\t\t * and failed to find one. They called us to do an\n ++\t\t * ICASE match, so we should never get an exact match,\n ++\t\t * so we could promote this to a BUG() here if we\n ++\t\t * wanted to. It doesn't hurt anything to just return\n ++\t\t * 0 and go on becaus we should never get here. Or we\n ++\t\t * could just get rid of the memcmp() and this \"if\"\n ++\t\t * clause completely.\n ++\t\t */\n +\t\treturn 0; /* should not happen */\n +\t}\n +\n +\ttrace_printf_key(&trace_fsmonitor,\n -+\t\t\t \"fsmonitor_refresh_callback map '%s' '%s'\",\n ++\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n +\t\t\t name, canonical_path.buf);\n +\n +\t/*\n -+\t * The directory name-hash only tells us the corrected\n -+\t * spelling of the prefix. We have to use this canonical\n -+\t * path to do a lookup in the cache-entry array so that we\n -+\t * we repeat the original search using the case-corrected\n -+\t * spelling.\n ++\t * The dir-name-hash only tells us the corrected spelling of\n ++\t * the prefix. We have to use this canonical path to do a\n ++\t * lookup in the cache-entry array so that we repeat the\n ++\t * original search using the case-corrected spelling.\n +\t */\n +\tstrbuf_addch(&canonical_path, '/');\n +\tpos = index_name_pos(istate, canonical_path.buf,\n +\t\t\t canonical_path.len);\n -+\tnr_in_cone = fsmonitor_refresh_callback_slash(\n -+\t\tistate, canonical_path.buf, canonical_path.len, pos);\n ++\tnr_in_cone = handle_path_with_trailing_slash(\n ++\t\tistate, canonical_path.buf, pos);\n +\tstrbuf_release(&canonical_path);\n +\treturn nr_in_cone;\n +}\n +\n - static void fsmonitor_refresh_callback_unqualified(\n - \tstruct index_state *istate, const char *name, int len, int pos)\n - {\n -@@ fsmonitor.c: static void fsmonitor_refresh_callback_unqualified(\n - *\n - * Return the number of cache-entries that we invalidated. We will\n - * use this later to determine if we need to attempt a second\n -- * case-insensitive search.\n -+ * case-insensitive search. That is, if a observed-case search yields\n -+ * any results, we assume the prefix is case-correct. If there are\n -+ * no matches, we still don't know if the observed path is simply\n -+ * untracked or case-incorrect.\n - */\n - static int fsmonitor_refresh_callback_slash(\n - \tstruct index_state *istate, const char *name, int len, int pos)\n -@@ fsmonitor.c: static int fsmonitor_refresh_callback_slash(\n - \treturn nr_in_cone;\n - }\n + /*\n + * The daemon sent an observed pathname without a trailing slash.\n + * (This is the normal case.) We do not know if it is a tracked or\n +@@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n + \telse\n + \t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n \n -+/*\n -+ * On a case-insensitive FS, use the name-hash and directory name-hash\n -+ * to map the case of the observed path to the canonical case expected\n -+ * by the index.\n -+ *\n -+ * The given pathname includes the trailing slash.\n -+ *\n -+ * Return the number of cache-entries that we invalidated.\n -+ */\n -+static int fsmonitor_refresh_callback_slash_icase(\n -+\tstruct index_state *istate, const char *name, int len)\n -+{\n -+\tint nr_in_cone;\n -+\n -+\t/*\n -+\t * Look for a case-incorrect sparse-index directory.\n -+\t */\n -+\tnr_in_cone = my_callback_name_hash(istate, name, len);\n -+\tif (nr_in_cone)\n -+\t\treturn nr_in_cone;\n -+\n +\t/*\n -+\t * (len-1) because we do not include the trailing slash in the\n -+\t * pathname.\n ++\t * If we did not find an exact match for this pathname or any\n ++\t * cache-entries with this directory prefix and we're on a\n ++\t * case-insensitive file system, try again using the name-hash\n ++\t * and dir-name-hash.\n +\t */\n -+\tnr_in_cone = my_callback_dir_name_hash(istate, name, len-1);\n -+\treturn nr_in_cone;\n -+}\n -+\n - static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n - {\n - \tint len = strlen(name);\n - \tint pos = index_name_pos(istate, name, len);\n -+\tint nr_in_cone;\n ++\tif (!nr_in_cone && ignore_case) {\n ++\t\tnr_in_cone = handle_using_name_hash_icase(istate, name);\n ++\t\tif (!nr_in_cone)\n ++\t\t\tnr_in_cone = handle_using_dir_name_hash_icase(\n ++\t\t\t\tistate, name);\n ++\t}\n +\n - \n - \ttrace_printf_key(&trace_fsmonitor,\n - \t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n - \t\t\t name, pos);\n - \n - \tif (name[len - 1] == '/') {\n --\t\tfsmonitor_refresh_callback_slash(istate, name, len, pos);\n -+\t\tnr_in_cone = fsmonitor_refresh_callback_slash(istate, name, len, pos);\n -+\t\tif (ignore_case && !nr_in_cone)\n -+\t\t\tfsmonitor_refresh_callback_slash_icase(istate, name, len);\n - \t} else {\n - \t\tfsmonitor_refresh_callback_unqualified(istate, name, len, pos);\n - \t}\n + \tif (nr_in_cone)\n + \t\ttrace_printf_key(&trace_fsmonitor,\n + \t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\n 11: 7775de735f4 ! 15: 3a20065dbf8 fsmonitor: refactor bit invalidation in refresh callback\n @@ Commit message\n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n -@@ fsmonitor.c: static void my_invalidate_untracked_cache(\n - \tstrbuf_release(&work_path);\n - }\n +@@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n + static size_t handle_path_with_trailing_slash(\n + \tstruct index_state *istate, const char *name, int pos);\n \n +/*\n + * Invalidate the FSM bit on this CE. This is like mark_fsmonitor_invalid()\n + * but we've already handled the untracked-cache and I want a different\n + * trace message.\n + */\n -+static void my_invalidate_ce_fsm(struct cache_entry *ce)\n ++static void invalidate_ce_fsm(struct cache_entry *ce)\n +{\n +\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n +\t\ttrace_printf_key(&trace_fsmonitor,\n -+\t\t\t\t \"fsmonitor_refresh_cb_invalidate '%s'\",\n ++\t\t\t\t \"fsmonitor_refresh_callback INV: '%s'\",\n +\t\t\t\t ce->name);\n +\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n +}\n +\n /*\n - * Use the name-hash to lookup the pathname.\n - *\n -@@ fsmonitor.c: static int my_callback_name_hash(\n + * Use the name-hash to do a case-insensitive cache-entry lookup with\n + * the pathname and invalidate the cache-entry.\n +@@ fsmonitor.c: static size_t handle_using_name_hash_icase(\n \n - \tmy_invalidate_untracked_cache(istate, ce->name, ce->ce_namelen);\n + \tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n \n -\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n -+\tmy_invalidate_ce_fsm(ce);\n ++\tinvalidate_ce_fsm(ce);\n \treturn 1;\n }\n \n -@@ fsmonitor.c: static int fsmonitor_refresh_callback_unqualified(\n +@@ fsmonitor.c: static size_t handle_path_without_trailing_slash(\n \t\t * cache-entry with the same pathname, nor for a cone\n \t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n -\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n -+\t\tmy_invalidate_ce_fsm(istate->cache[pos]);\n ++\t\tinvalidate_ce_fsm(istate->cache[pos]);\n \t\treturn 1;\n \t} else {\n - \t\tint nr_in_cone;\n -@@ fsmonitor.c: static int fsmonitor_refresh_callback_slash(\n + \t\tsize_t nr_in_cone;\n +@@ fsmonitor.c: static size_t handle_path_with_trailing_slash(\n \tfor (i = pos; i < istate->cache_nr; i++) {\n \t\tif (!starts_with(istate->cache[i]->name, name))\n \t\t\tbreak;\n -\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n -+\t\tmy_invalidate_ce_fsm(istate->cache[i]);\n ++\t\tinvalidate_ce_fsm(istate->cache[i]);\n \t\tnr_in_cone++;\n \t}\n \n 12: 63edb68303f ! 16: 467d3c1fe2c t7527: update case-insenstive fsmonitor test\n @@ Commit message\n \n Now that the FSMonitor client has been updated to better\n handle events on case-insenstive file systems, update the\n - two tests that demonstrated the bug.\n + two tests that demonstrated the bug and remove the temporary\n + SKIPME prereq.\n \n Signed-off-by: Jeff Hostetler \n \n ## t/t7527-builtin-fsmonitor.sh ##\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'split-index and FSMonitor work well together' '\n + #\n + # The setup is a little contrived.\n + #\n +-test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n ++test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n + \ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n + \n + \tgit init subdir_case_wrong &&\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on d\n \n \tgrep -q \"dir1/DIR2/dir3/file3.*pos -3\" \"$PWD/subdir_case_wrong.log1\" &&\n \n +\t# Also verify that we get a mapping event to correct the case.\n -+\tgrep -q \"map.*dir1/DIR2/dir3/file3.*dir1/dir2/dir3/file3\" \\\n ++\tgrep -q \"MAP:.*dir1/DIR2/dir3/file3.*dir1/dir2/dir3/file3\" \\\n +\t\t\"$PWD/subdir_case_wrong.log1\" &&\n +\n \t# The refresh-callbacks should have caused \"git status\" to clear\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'fsmonitor\n +\tgrep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n '\n \n - test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n -@@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n +-test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n ++test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n + \ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n + \n + \tgit init file_case_wrong &&\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor file case wrong on dis\n \tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n \tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n \n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success CASE_INSENSITIVE_FS 'fsmonitor\n -\t! grep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n -\t! grep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n +\t# Also verify that we get a mapping event to correct the case.\n -+\tgrep -q \"fsmonitor_refresh_callback map.*dir1/dir2/dir3/FILE-3-A.*dir1/dir2/dir3/file-3-a\" \\\n ++\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir3/FILE-3-A.*dir1/dir2/dir3/file-3-a\" \\\n +\t\t\"$PWD/file_case_wrong-try3.log\" &&\n -+\tgrep -q \"fsmonitor_refresh_callback map.*dir1/dir2/dir4/file-4-a.*dir1/dir2/dir4/FILE-4-A\" \\\n ++\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir4/file-4-a.*dir1/dir2/dir4/FILE-4-A\" \\\n +\t\t\"$PWD/file_case_wrong-try3.log\" &&\n +\n +\tgrep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n\n-- \ngitgitgadget\n"},{"id":"489196","messageId":"03b07d9c25edb951db4de518212f8a3e9e184e49.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 01/16] name-hash: add index_dir_find()","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:05Z","receivedAt":"2024-02-23T03:18:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nReplace the index_dir_exists() function with index_dir_find() and\nchange the API to take an optional strbuf to return the canonical\nspelling of the matched directory prefix.\n\nCreate an index_dir_exists() wrapper macro for existing callers.\n\nThe existing index_dir_exists() returns a boolean to indicate if\nthere is a case-insensitive match in the directory name-hash, but\nit doesn't tell the caller the exact spelling of that match.\n\nThe new version also copies the matched spelling to a provided strbuf.\nThis lets the caller, for example, then call index_name_pos() with the\ncorrect case to search the cache-entry array for the real insertion\nposition.\n\nSigned-off-by: Jeff Hostetler \n---\n name-hash.c | 9 ++++++++-\n name-hash.h | 7 ++++++-\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/name-hash.c b/name-hash.c\nindex 251f036eef6..3a58ce03d9c 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -685,13 +685,20 @@ static int same_name(const struct cache_entry *ce, const char *name, int namelen\n \treturn slow_same_name(name, namelen, ce->name, len);\n }\n \n-int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n+int index_dir_find(struct index_state *istate, const char *name, int namelen,\n+\t\t struct strbuf *canonical_path)\n {\n \tstruct dir_entry *dir;\n \n \tlazy_init_name_hash(istate);\n \texpand_to_path(istate, name, namelen, 0);\n \tdir = find_dir_entry(istate, name, namelen);\n+\n+\tif (canonical_path && dir && dir->nr) {\n+\t\tstrbuf_reset(canonical_path);\n+\t\tstrbuf_add(canonical_path, dir->name, dir->namelen);\n+\t}\n+\n \treturn dir && dir->nr;\n }\n \ndiff --git a/name-hash.h b/name-hash.h\nindex b1b4b0fb337..0cbfc428631 100644\n--- a/name-hash.h\n+++ b/name-hash.h\n@@ -4,7 +4,12 @@\n struct cache_entry;\n struct index_state;\n \n-int index_dir_exists(struct index_state *istate, const char *name, int namelen);\n+\n+int index_dir_find(struct index_state *istate, const char *name, int namelen,\n+\t\t struct strbuf *canonical_path);\n+\n+#define index_dir_exists(i, n, l) index_dir_find((i), (n), (l), NULL)\n+\n void adjust_dirname_case(struct index_state *istate, char *name);\n struct cache_entry *index_file_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n \n-- \ngitgitgadget\n\n"},{"id":"489197","messageId":"dad079ade7f8817f235d59c5339b82aecf4a17db.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 03/16] t7527: temporarily disable case-insensitive tests","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:07Z","receivedAt":"2024-02-23T03:18:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nAdd non-existent \"SKIPME\" prereq to the case-insensitive tests.\n\nThe previous commit added test cases to demonstrate an error where\nFSMonitor can get confused on a case-insensitive file system when the\non-disk spelling of a file or directory is wrong. Let's disable those\ntests before we incrementally teach Git to properly recognize and\nhandle those types of problems (so that a bisect between here and the\nfinal commit in this patch series won't throw a false alarm).\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 3d21295f789..4acb547819c 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -1051,7 +1051,7 @@ test_expect_success 'split-index and FSMonitor work well together' '\n #\n # The setup is a little contrived.\n #\n-test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n+test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n \ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n \n \tgit init subdir_case_wrong &&\n@@ -1128,7 +1128,7 @@ test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n \t! grep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n '\n \n-test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n+test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n \ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n \n \tgit init file_case_wrong &&\n-- \ngitgitgadget\n\n"},{"id":"489198","messageId":"7778cee1c1056963eaf4c28c667b7194b9eeb450.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 02/16] t7527: add case-insensitve test for FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:06Z","receivedAt":"2024-02-23T03:18:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nThe FSMonitor client code trusts the spelling of the pathnames in the\nFSEvents received from the FSMonitor daemon. On case-insensitive file\nsystems, these OBSERVED pathnames may be spelled differently than the\nEXPECTED pathnames listed in the .git/index. This causes a miss when\nusing `index_name_pos()` which expects the given case to be correct.\n\nWhen this happens, the FSMonitor client code does not update the state\nof the CE_FSMONITOR_VALID bit when refreshing the index (and before\nstarting to scan the worktree).\n\nThis results in modified files NOT being reported by `git status` when\nthere is a discrepancy in the case-spelling of a tracked file's\npathname.\n\nThis commit contains a (rather contrived) test case to demonstrate\nthis. A later commit in this series will update the FSMonitor client\ncode to recognize these discrepancies and update the CE_ bit accordingly.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 217 +++++++++++++++++++++++++++++++++++\n 1 file changed, 217 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 363f9dc0e41..3d21295f789 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -1037,4 +1037,221 @@ test_expect_success 'split-index and FSMonitor work well together' '\n \t)\n '\n \n+# The FSMonitor daemon reports the OBSERVED pathname of modified files\n+# and thus contains the OBSERVED spelling on case-insensitive file\n+# systems. The daemon does not (and should not) load the .git/index\n+# file and therefore does not know the expected case-spelling. Since\n+# it is possible for the user to create files/subdirectories with the\n+# incorrect case, a modified file event for a tracked will not have\n+# the EXPECTED case. This can cause `index_name_pos()` to incorrectly\n+# report that the file is untracked. This causes the client to fail to\n+# mark the file as possibly dirty (keeping the CE_FSMONITOR_VALID bit\n+# set) so that `git status` will avoid inspecting it and thus not\n+# present in the status output.\n+#\n+# The setup is a little contrived.\n+#\n+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n+\ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n+\n+\tgit init subdir_case_wrong &&\n+\t(\n+\t\tcd subdir_case_wrong &&\n+\t\techo x >AAA &&\n+\t\techo x >BBB &&\n+\n+\t\tmkdir dir1 &&\n+\t\techo x >dir1/file1 &&\n+\t\tmkdir dir1/dir2 &&\n+\t\techo x >dir1/dir2/file2 &&\n+\t\tmkdir dir1/dir2/dir3 &&\n+\t\techo x >dir1/dir2/dir3/file3 &&\n+\n+\t\techo x >yyy &&\n+\t\techo x >zzz &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"data\" &&\n+\n+\t\t# This will cause \"dir1/\" and everything under it\n+\t\t# to be deleted.\n+\t\tgit sparse-checkout set --cone --sparse-index &&\n+\n+\t\t# Create dir2 with the wrong case and then let Git\n+\t\t# repopulate dir3 -- it will not correct the spelling\n+\t\t# of dir2.\n+\t\tmkdir dir1 &&\n+\t\tmkdir dir1/DIR2 &&\n+\t\tgit sparse-checkout add dir1/dir2/dir3\n+\t) &&\n+\n+\tstart_daemon -C subdir_case_wrong --tf \"$PWD/subdir_case_wrong.trace\" &&\n+\n+\t# Enable FSMonitor in the client. Run enough commands for\n+\t# the .git/index to sync up with the daemon with everything\n+\t# marked clean.\n+\tgit -C subdir_case_wrong config core.fsmonitor true &&\n+\tgit -C subdir_case_wrong update-index --fsmonitor &&\n+\tgit -C subdir_case_wrong status &&\n+\n+\t# Make some files dirty so that FSMonitor gets FSEvents for\n+\t# each of them.\n+\techo xx >>subdir_case_wrong/AAA &&\n+\techo xx >>subdir_case_wrong/dir1/DIR2/dir3/file3 &&\n+\techo xx >>subdir_case_wrong/zzz &&\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/subdir_case_wrong.log\" \\\n+\t\tgit -C subdir_case_wrong --no-optional-locks status --short \\\n+\t\t\t>\"$PWD/subdir_case_wrong.out\" &&\n+\n+\t# \"git status\" should have gotten file events for each of\n+\t# the 3 files.\n+\t#\n+\t# \"dir2\" should be in the observed case on disk.\n+\tgrep \"fsmonitor_refresh_callback\" \\\n+\t\t<\"$PWD/subdir_case_wrong.log\" \\\n+\t\t>\"$PWD/subdir_case_wrong.log1\" &&\n+\n+\tgrep -q \"AAA.*pos 0\" \"$PWD/subdir_case_wrong.log1\" &&\n+\tgrep -q \"zzz.*pos 6\" \"$PWD/subdir_case_wrong.log1\" &&\n+\n+\tgrep -q \"dir1/DIR2/dir3/file3.*pos -3\" \"$PWD/subdir_case_wrong.log1\" &&\n+\n+\t# The refresh-callbacks should have caused \"git status\" to clear\n+\t# the CE_FSMONITOR_VALID bit on each of those files and caused\n+\t# the worktree scan to visit them and mark them as modified.\n+\tgrep -q \" M AAA\" \"$PWD/subdir_case_wrong.out\" &&\n+\tgrep -q \" M zzz\" \"$PWD/subdir_case_wrong.out\" &&\n+\n+\t# However, with the fsmonitor client bug, the \"(pos -3)\" causes\n+\t# the client to not update the bit and never rescan the file\n+\t# and therefore not report it as dirty.\n+\t! grep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n+'\n+\n+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n+\ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n+\n+\tgit init file_case_wrong &&\n+\t(\n+\t\tcd file_case_wrong &&\n+\t\techo x >AAA &&\n+\t\techo x >BBB &&\n+\n+\t\tmkdir dir1 &&\n+\t\tmkdir dir1/dir2 &&\n+\t\tmkdir dir1/dir2/dir3 &&\n+\t\techo x >dir1/dir2/dir3/FILE-3-B &&\n+\t\techo x >dir1/dir2/dir3/XXXX-3-X &&\n+\t\techo x >dir1/dir2/dir3/file-3-a &&\n+\t\techo x >dir1/dir2/dir3/yyyy-3-y &&\n+\t\tmkdir dir1/dir2/dir4 &&\n+\t\techo x >dir1/dir2/dir4/FILE-4-A &&\n+\t\techo x >dir1/dir2/dir4/XXXX-4-X &&\n+\t\techo x >dir1/dir2/dir4/file-4-b &&\n+\t\techo x >dir1/dir2/dir4/yyyy-4-y &&\n+\n+\t\techo x >yyy &&\n+\t\techo x >zzz &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"data\"\n+\t) &&\n+\n+\tstart_daemon -C file_case_wrong --tf \"$PWD/file_case_wrong.trace\" &&\n+\n+\t# Enable FSMonitor in the client. Run enough commands for\n+\t# the .git/index to sync up with the daemon with everything\n+\t# marked clean.\n+\tgit -C file_case_wrong config core.fsmonitor true &&\n+\tgit -C file_case_wrong update-index --fsmonitor &&\n+\tgit -C file_case_wrong status &&\n+\n+\t# Make some files dirty so that FSMonitor gets FSEvents for\n+\t# each of them.\n+\techo xx >>file_case_wrong/AAA &&\n+\techo xx >>file_case_wrong/zzz &&\n+\n+\t# Rename some files so that FSMonitor sees a create and delete\n+\t# FSEvent for each. (A simple \"mv foo FOO\" is not portable\n+\t# between macOS and Windows. It works on both platforms, but makes\n+\t# the test messy, since (1) one platform updates \"ctime\" on the\n+\t# moved file and one does not and (2) it causes a directory event\n+\t# on one platform and not on the other which causes additional\n+\t# scanning during \"git status\" which causes a \"H\" vs \"h\" discrepancy\n+\t# in \"git ls-files -f\".) So old-school it and move it out of the\n+\t# way and copy it to the case-incorrect name so that we get fresh\n+\t# \"ctime\" and \"mtime\" values.\n+\n+\tmv file_case_wrong/dir1/dir2/dir3/file-3-a file_case_wrong/dir1/dir2/dir3/ORIG &&\n+\tcp file_case_wrong/dir1/dir2/dir3/ORIG file_case_wrong/dir1/dir2/dir3/FILE-3-A &&\n+\trm file_case_wrong/dir1/dir2/dir3/ORIG &&\n+\tmv file_case_wrong/dir1/dir2/dir4/FILE-4-A file_case_wrong/dir1/dir2/dir4/ORIG &&\n+\tcp file_case_wrong/dir1/dir2/dir4/ORIG file_case_wrong/dir1/dir2/dir4/file-4-a &&\n+\trm file_case_wrong/dir1/dir2/dir4/ORIG &&\n+\n+\t# Run status enough times to fully sync.\n+\t#\n+\t# The first instance should get the create and delete FSEvents\n+\t# for each pair. Status should update the index with a new FSM\n+\t# token (so the next invocation will not see data for these\n+\t# events).\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try1.log\" \\\n+\t\tgit -C file_case_wrong status --short \\\n+\t\t\t>\"$PWD/file_case_wrong-try1.out\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try1.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*file-3-a.*pos 4\" \"$PWD/file_case_wrong-try1.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*FILE-4-A.*pos 6\" \"$PWD/file_case_wrong-try1.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try1.log\" &&\n+\n+\t# FSM refresh will have invalidated the FSM bit and cause a regular\n+\t# (real) scan of these tracked files, so they should have \"H\" status.\n+\t# (We will not see a \"h\" status until the next refresh (on the next\n+\t# command).)\n+\n+\tgit -C file_case_wrong ls-files -f >\"$PWD/file_case_wrong-lsf1.out\" &&\n+\tgrep -q \"H dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-lsf1.out\" &&\n+\tgrep -q \"H dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-lsf1.out\" &&\n+\n+\n+\t# Try the status again. We assume that the above status command\n+\t# advanced the token so that the next one will not see those events.\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try2.log\" \\\n+\t\tgit -C file_case_wrong status --short \\\n+\t\t\t>\"$PWD/file_case_wrong-try2.out\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*file-3-a.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*FILE-4-A.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*file-4-a.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\n+\t# FSM refresh saw nothing, so it will mark all files as valid,\n+\t# so they should now have \"h\" status.\n+\n+\tgit -C file_case_wrong ls-files -f >\"$PWD/file_case_wrong-lsf2.out\" &&\n+\tgrep -q \"h dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-lsf2.out\" &&\n+\tgrep -q \"h dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-lsf2.out\" &&\n+\n+\n+\t# We now have files with clean content, but with case-incorrect\n+\t# file names. Modify them to see if status properly reports\n+\t# them.\n+\n+\techo xx >>file_case_wrong/dir1/dir2/dir3/FILE-3-A &&\n+\techo xx >>file_case_wrong/dir1/dir2/dir4/file-4-a &&\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try3.log\" \\\n+\t\tgit -C file_case_wrong --no-optional-locks status --short \\\n+\t\t\t>\"$PWD/file_case_wrong-try3.out\" &&\n+\t# FSEvents are in observed case.\n+\tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n+\n+\t# Status should say these files are modified, but with the case\n+\t# bug, the \"pos -3\" cause the client to not update the FSM bit\n+\t# and never cause the file to be rescanned and therefore to not\n+\t# report it dirty.\n+\t! grep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n+\t! grep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"489199","messageId":"c04fd4eae941e234f4c7364bf035501a8e244db5.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 05/16] fsmonitor: clarify handling of directory events in callback helper","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:09Z","receivedAt":"2024-02-23T03:18:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImprove documentation of the refresh callback helper function\nused for directory FSEvents.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 37 ++++++++++++++++++++++++-------------\n 1 file changed, 24 insertions(+), 13 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 6fecae9aeb2..29cce32d81c 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,24 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+/*\n+ * The daemon can decorate directory events, such as a move or rename,\n+ * by adding a trailing slash to the observed name. Use this to\n+ * explicitly invalidate the entire cone under that directory.\n+ *\n+ * The daemon can only reliably do that if the OS FSEvent contains\n+ * sufficient information in the event.\n+ *\n+ * macOS FSEvents have enough information.\n+ *\n+ * Other platforms may or may not be able to do it (and it might\n+ * depend on the type of event (for example, a daemon could lstat() an\n+ * observed pathname after a rename, but not after a delete)).\n+ *\n+ * If we find an exact match in the index for a path with a trailing\n+ * slash, it means that we matched a sparse-index directory in a\n+ * cone-mode sparse-checkout (since that's the only time we have\n+ * directories in the index). We should never see this in practice\n+ * (because sparse directories should not be present and therefore\n+ * not generating FS events). Either way, we can treat them in the\n+ * same way and just invalidate the cache-entry and the untracked\n+ * cache (and in this case, the forward cache-entry scan won't find\n+ * anything and it doesn't hurt to let it run).\n+ */\n static void handle_path_with_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos)\n {\n \tint i;\n \n-\t/*\n-\t * The daemon can decorate directory events, such as\n-\t * moves or renames, with a trailing slash if the OS\n-\t * FS Event contains sufficient information, such as\n-\t * MacOS.\n-\t *\n-\t * Use this to invalidate the entire cone under that\n-\t * directory.\n-\t *\n-\t * We do not expect an exact match because the index\n-\t * does not normally contain directory entries, so we\n-\t * start at the insertion point and scan.\n-\t */\n \tif (pos < 0)\n \t\tpos = -pos - 1;\n \n-- \ngitgitgadget\n\n"},{"id":"489200","messageId":"5516670e30e26c5b50c67b69e48e3e8a5e0d8990.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 04/16] fsmonitor: refactor refresh callback on directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:08Z","receivedAt":"2024-02-23T03:18:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove the code to handle directory FSEvents (containing pathnames with\na trailing slash) into a helper function.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 52 ++++++++++++++++++++++++++++++----------------------\n 1 file changed, 30 insertions(+), 22 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex f670c509378..6fecae9aeb2 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,6 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+static void handle_path_with_trailing_slash(\n+\tstruct index_state *istate, const char *name, int pos)\n+{\n+\tint i;\n+\n+\t/*\n+\t * The daemon can decorate directory events, such as\n+\t * moves or renames, with a trailing slash if the OS\n+\t * FS Event contains sufficient information, such as\n+\t * MacOS.\n+\t *\n+\t * Use this to invalidate the entire cone under that\n+\t * directory.\n+\t *\n+\t * We do not expect an exact match because the index\n+\t * does not normally contain directory entries, so we\n+\t * start at the insertion point and scan.\n+\t */\n+\tif (pos < 0)\n+\t\tpos = -pos - 1;\n+\n+\t/* Mark all entries for the folder invalid */\n+\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\tbreak;\n+\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t}\n+}\n+\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n@@ -193,28 +222,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t\t\t name, pos);\n \n \tif (name[len - 1] == '/') {\n-\t\t/*\n-\t\t * The daemon can decorate directory events, such as\n-\t\t * moves or renames, with a trailing slash if the OS\n-\t\t * FS Event contains sufficient information, such as\n-\t\t * MacOS.\n-\t\t *\n-\t\t * Use this to invalidate the entire cone under that\n-\t\t * directory.\n-\t\t *\n-\t\t * We do not expect an exact match because the index\n-\t\t * does not normally contain directory entries, so we\n-\t\t * start at the insertion point and scan.\n-\t\t */\n-\t\tif (pos < 0)\n-\t\t\tpos = -pos - 1;\n-\n-\t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = pos; i < istate->cache_nr; i++) {\n-\t\t\tif (!starts_with(istate->cache[i]->name, name))\n-\t\t\t\tbreak;\n-\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\t}\n+\t\thandle_path_with_trailing_slash(istate, name, pos);\n \n \t\t/*\n \t\t * We need to remove the traling \"/\" from the path\n-- \ngitgitgadget\n\n"},{"id":"489201","messageId":"7ee6ca1aefd34a37d749300e317df10d80ef2b29.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 06/16] fsmonitor: refactor refresh callback for non-directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:10Z","receivedAt":"2024-02-23T03:18:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove the code handle unqualified FSEvents (without a trailing slash)\ninto a helper function.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 67 +++++++++++++++++++++++++++++++----------------------\n 1 file changed, 39 insertions(+), 28 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 29cce32d81c..364198d258f 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,6 +183,43 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+static void handle_path_without_trailing_slash(\n+\tstruct index_state *istate, const char *name, int pos)\n+{\n+\tint i;\n+\n+\tif (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t} else {\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tint len = strlen(name);\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t}\n+\t}\n+}\n+\n /*\n * The daemon can decorate directory events, such as a move or rename,\n * by adding a trailing slash to the observed name. Use this to\n@@ -225,7 +262,7 @@ static void handle_path_with_trailing_slash(\n \n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n-\tint i, len = strlen(name);\n+\tint len = strlen(name);\n \tint pos = index_name_pos(istate, name, len);\n \n \ttrace_printf_key(&trace_fsmonitor,\n@@ -240,34 +277,8 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t\t * for the untracked cache.\n \t\t */\n \t\tname[len - 1] = '\\0';\n-\t} else if (pos >= 0) {\n-\t\t/*\n-\t\t * We have an exact match for this path and can just\n-\t\t * invalidate it.\n-\t\t */\n-\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\t/*\n-\t\t * The path is not a tracked file -or- it is a\n-\t\t * directory event on a platform that cannot\n-\t\t * distinguish between file and directory events in\n-\t\t * the event handler, such as Windows.\n-\t\t *\n-\t\t * Scan as if it is a directory and invalidate the\n-\t\t * cone under it. (But remember to ignore items\n-\t\t * between \"name\" and \"name/\", such as \"name-\" and\n-\t\t * \"name.\".\n-\t\t */\n-\t\tpos = -pos - 1;\n-\n-\t\tfor (i = pos; i < istate->cache_nr; i++) {\n-\t\t\tif (!starts_with(istate->cache[i]->name, name))\n-\t\t\t\tbreak;\n-\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n-\t\t\t\tbreak;\n-\t\t\tif (istate->cache[i]->name[len] == '/')\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\t}\n+\t\thandle_path_without_trailing_slash(istate, name, pos);\n \t}\n \n \t/*\n-- \ngitgitgadget\n\n"},{"id":"489202","messageId":"99c0d3e0742c1a7e0f7608707402a772ec112716.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 07/16] dir: create untracked_cache_invalidate_trimmed_path()","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:11Z","receivedAt":"2024-02-23T03:18:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a wrapper function for untracked_cache_invalidate_path()\nthat silently trims a trailing slash, if present, before calling\nthe wrapped function.\n\nThe untracked cache expects to be called with a pathname that\ndoes not contain a trailing slash. This can make it inconvenient\nfor callers that have a directory path. Lets hide this complexity.\n\nThis will be used by a later commit in the FSMonitor code which\nmay receive directory pathnames from an FSEvent.\n\nSigned-off-by: Jeff Hostetler \n---\n dir.c | 20 ++++++++++++++++++++\n dir.h | 7 +++++++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex ac699542302..1157f3e43fa 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -3918,6 +3918,26 @@ void untracked_cache_invalidate_path(struct index_state *istate,\n \t\t\t\t path, strlen(path));\n }\n \n+void untracked_cache_invalidate_trimmed_path(struct index_state *istate,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t int safe_path)\n+{\n+\tsize_t len = strlen(path);\n+\n+\tif (!len)\n+\t\treturn; /* should not happen */\n+\n+\tif (path[len - 1] != '/') {\n+\t\tuntracked_cache_invalidate_path(istate, path, safe_path);\n+\t} else {\n+\t\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t\tstrbuf_add(&tmp, path, len - 1);\n+\t\tuntracked_cache_invalidate_path(istate, tmp.buf, safe_path);\n+\t\tstrbuf_release(&tmp);\n+\t}\n+}\n+\n void untracked_cache_remove_from_index(struct index_state *istate,\n \t\t\t\t const char *path)\n {\ndiff --git a/dir.h b/dir.h\nindex 98aa85fcc0e..45a7b9ec5f2 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -576,6 +576,13 @@ int cmp_dir_entry(const void *p1, const void *p2);\n int check_dir_entry_contains(const struct dir_entry *out, const struct dir_entry *in);\n \n void untracked_cache_invalidate_path(struct index_state *, const char *, int safe_path);\n+/*\n+ * Invalidate the untracked-cache for this path, but first strip\n+ * off a trailing slash, if present.\n+ */\n+void untracked_cache_invalidate_trimmed_path(struct index_state *,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t int safe_path);\n void untracked_cache_remove_from_index(struct index_state *, const char *);\n void untracked_cache_add_to_index(struct index_state *, const char *);\n \n-- \ngitgitgadget\n\n"},{"id":"489203","messageId":"f2d6765d84ff93a729359e7dcfaddd878f9458e2.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 08/16] fsmonitor: refactor untracked-cache invalidation","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:12Z","receivedAt":"2024-02-23T03:18:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUpdate fsmonitor_refresh_callback() to use the new\nuntracked_cache_invalidate_trimmed_path() to invalidate\nthe cache using the observed pathname without needing to\nmodify the caller's buffer.\n\nPreviously, we modified the caller's buffer when the observed pathname\ncontained a trailing slash (and did not restore it). This wasn't a\nproblem for the single use-case caller, but felt dirty nontheless. In\na later commit we will want to invalidate case-corrected versions of\nthe pathname (using possibly borrowed pathnames from the name-hash or\ndir-name-hash) and we may not want to keep the tradition of altering\nthe passed-in pathname.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 11 +++--------\n 1 file changed, 3 insertions(+), 8 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 364198d258f..2787f7ca5d1 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -271,21 +271,16 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \n \tif (name[len - 1] == '/') {\n \t\thandle_path_with_trailing_slash(istate, name, pos);\n-\n-\t\t/*\n-\t\t * We need to remove the traling \"/\" from the path\n-\t\t * for the untracked cache.\n-\t\t */\n-\t\tname[len - 1] = '\\0';\n \t} else {\n \t\thandle_path_without_trailing_slash(istate, name, pos);\n \t}\n \n \t/*\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n-\t * as it could be a new untracked file.\n+\t * as it could be a new untracked file. (Let the untracked cache\n+\t * layer silently deal with any trailing slash.)\n \t */\n-\tuntracked_cache_invalidate_path(istate, name, 0);\n+\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"489204","messageId":"af6f57ab3e6d61036cd969f5fd9256200313aaa9.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 09/16] fsmonitor: move untracked invalidation into helper functions","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:13Z","receivedAt":"2024-02-23T03:18:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove the call to invalidate the untracked cache for the FSEvent\npathname into the two helper functions.\n\nIn a later commit in this series, we will call these helpers\nfrom other contexts and it safer to include the UC invalidation\nin the helper than to remember to also add it to each helper\ncall-site.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 26 +++++++++++++++++++-------\n 1 file changed, 19 insertions(+), 7 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 2787f7ca5d1..2f58ee2fe5a 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -188,6 +188,16 @@ static void handle_path_without_trailing_slash(\n {\n \tint i;\n \n+\t/*\n+\t * Mark the untracked cache dirty for this path (regardless of\n+\t * whether or not we find an exact match for it in the index).\n+\t * Since the path is unqualified (no trailing slash hint in the\n+\t * FSEvent), it may refer to a file or directory. So we should\n+\t * not assume one or the other and should always let the untracked\n+\t * cache decide what needs to invalidated.\n+\t */\n+\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n+\n \tif (pos >= 0) {\n \t\t/*\n \t\t * We have an exact match for this path and can just\n@@ -249,6 +259,15 @@ static void handle_path_with_trailing_slash(\n {\n \tint i;\n \n+\t/*\n+\t * Mark the untracked cache dirty for this directory path\n+\t * (regardless of whether or not we find an exact match for it\n+\t * in the index or find it to be proper prefix of one or more\n+\t * files in the index), since the FSEvent is hinting that\n+\t * there may be changes on or within the directory.\n+\t */\n+\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n+\n \tif (pos < 0)\n \t\tpos = -pos - 1;\n \n@@ -274,13 +293,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t} else {\n \t\thandle_path_without_trailing_slash(istate, name, pos);\n \t}\n-\n-\t/*\n-\t * Mark the untracked cache dirty even if it wasn't found in the index\n-\t * as it could be a new untracked file. (Let the untracked cache\n-\t * layer silently deal with any trailing slash.)\n-\t */\n-\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"489205","messageId":"623c6f06e21c04c55576d1146944b5ff6be66429.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 10/16] fsmonitor: return invalidated cache-entry count on directory event","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:14Z","receivedAt":"2024-02-23T03:18:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the refresh callback helper function for directory FSEvents to\nreturn the number of cache-entries that were invalidated in response\nto a directory event.\n\nThis will be used in a later commit to help determine if the observed\npathname in the FSEvent was a (possibly) case-incorrect directory\nprefix (on a case-insensitive filesystem) of one or more actual\ncache-entries.\n\nIf there exists at least one case-insensitive prefix match, then we\ncan assume that the directory is a (case-incorrect) prefix of at least\none tracked item rather than a completely unknown/untracked file or\ndirectory.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 2f58ee2fe5a..9424bd17230 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -253,11 +253,20 @@ static void handle_path_without_trailing_slash(\n * same way and just invalidate the cache-entry and the untracked\n * cache (and in this case, the forward cache-entry scan won't find\n * anything and it doesn't hurt to let it run).\n+ *\n+ * Return the number of cache-entries that we invalidated. We will\n+ * use this later to determine if we need to attempt a second\n+ * case-insensitive search on case-insensitive file systems. That is,\n+ * if the search using the observed-case in the FSEvent yields any\n+ * results, we assume the prefix is case-correct. If there are no\n+ * matches, we still don't know if the observed path is simply\n+ * untracked or case-incorrect.\n */\n-static void handle_path_with_trailing_slash(\n+static size_t handle_path_with_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos)\n {\n \tint i;\n+\tsize_t nr_in_cone = 0;\n \n \t/*\n \t * Mark the untracked cache dirty for this directory path\n@@ -276,7 +285,10 @@ static void handle_path_with_trailing_slash(\n \t\tif (!starts_with(istate->cache[i]->name, name))\n \t\t\tbreak;\n \t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tnr_in_cone++;\n \t}\n+\n+\treturn nr_in_cone;\n }\n \n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n-- \ngitgitgadget\n\n"},{"id":"489206","messageId":"1853f77d3331f7736139f686ac2efee6d68f9207.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 11/16] fsmonitor: remove custom loop from non-directory path handler","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:15Z","receivedAt":"2024-02-23T03:18:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor the code that handles refresh events for pathnames that do\nnot contain a trailing slash. Instead of using a custom loop to try\nto scan the index and detect if the FSEvent named a file or might be a\ndirectory prefix, use the recently created helper function to do that.\n\nAlso update the comments to describe what and why we are doing this.\n\nOn platforms that DO NOT annotate FS events with a trailing\nslash, if we fail to find an exact match for the pathname\nin the index, we do not know if the pathname represents a\ndirectory or simply an untracked file. Pretend that the pathname\nis a directory and try again before assuming it is an untracked\nfile.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 55 ++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 31 insertions(+), 24 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 9424bd17230..a51c17cda70 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,11 +183,23 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+static size_t handle_path_with_trailing_slash(\n+\tstruct index_state *istate, const char *name, int pos);\n+\n+/*\n+ * The daemon sent an observed pathname without a trailing slash.\n+ * (This is the normal case.) We do not know if it is a tracked or\n+ * untracked file, a sparse-directory, or a populated directory (on a\n+ * platform such as Windows where FSEvents are not qualified).\n+ *\n+ * The pathname contains the observed case reported by the FS. We\n+ * do not know it is case-correct or -incorrect.\n+ *\n+ * Assume it is case-correct and try an exact match.\n+ */\n static void handle_path_without_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos)\n {\n-\tint i;\n-\n \t/*\n \t * Mark the untracked cache dirty for this path (regardless of\n \t * whether or not we find an exact match for it in the index).\n@@ -200,33 +212,28 @@ static void handle_path_without_trailing_slash(\n \n \tif (pos >= 0) {\n \t\t/*\n-\t\t * We have an exact match for this path and can just\n-\t\t * invalidate it.\n+\t\t * An exact match on a tracked file. We assume that we\n+\t\t * do not need to scan forward for a sparse-directory\n+\t\t * cache-entry with the same pathname, nor for a cone\n+\t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n+\t\tstruct strbuf work_path = STRBUF_INIT;\n+\n \t\t/*\n-\t\t * The path is not a tracked file -or- it is a\n-\t\t * directory event on a platform that cannot\n-\t\t * distinguish between file and directory events in\n-\t\t * the event handler, such as Windows.\n-\t\t *\n-\t\t * Scan as if it is a directory and invalidate the\n-\t\t * cone under it. (But remember to ignore items\n-\t\t * between \"name\" and \"name/\", such as \"name-\" and\n-\t\t * \"name.\".\n+\t\t * The negative \"pos\" gives us the suggested insertion\n+\t\t * point for the pathname (without the trailing slash).\n+\t\t * We need to see if there is a directory with that\n+\t\t * prefix, but there can be lots of pathnames between\n+\t\t * \"foo\" and \"foo/\" like \"foo-\" or \"foo-bar\", so we\n+\t\t * don't want to do our own scan.\n \t\t */\n-\t\tint len = strlen(name);\n-\t\tpos = -pos - 1;\n-\n-\t\tfor (i = pos; i < istate->cache_nr; i++) {\n-\t\t\tif (!starts_with(istate->cache[i]->name, name))\n-\t\t\t\tbreak;\n-\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n-\t\t\t\tbreak;\n-\t\t\tif (istate->cache[i]->name[len] == '/')\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\t}\n+\t\tstrbuf_add(&work_path, name, strlen(name));\n+\t\tstrbuf_addch(&work_path, '/');\n+\t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n+\t\thandle_path_with_trailing_slash(istate, work_path.buf, pos);\n+\t\tstrbuf_release(&work_path);\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"489207","messageId":"58b36673e151ad15eb44c9ca1c03cfef51657d11.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 13/16] fsmonitor: trace the new invalidated cache-entry count","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:17Z","receivedAt":"2024-02-23T03:18:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConsolidate the directory/non-directory calls to the refresh handler\ncode. Log the resulting count of invalidated cache-entries.\n\nThe nr_in_cone value will be used in a later commit to decide if\nwe also need to try to do case-insensitive lookups.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex c16ed5d8758..739ddbf7aca 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -308,16 +308,21 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint len = strlen(name);\n \tint pos = index_name_pos(istate, name, len);\n+\tsize_t nr_in_cone;\n \n \ttrace_printf_key(&trace_fsmonitor,\n \t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n \t\t\t name, pos);\n \n-\tif (name[len - 1] == '/') {\n-\t\thandle_path_with_trailing_slash(istate, name, pos);\n-\t} else {\n-\t\thandle_path_without_trailing_slash(istate, name, pos);\n-\t}\n+\tif (name[len - 1] == '/')\n+\t\tnr_in_cone = handle_path_with_trailing_slash(istate, name, pos);\n+\telse\n+\t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n+\n+\tif (nr_in_cone)\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\n+\t\t\t\t (int)nr_in_cone);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"489208","messageId":"f77d68c78ada02bfb4b96759f6ad82ebff00b35b.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 12/16] fsmonitor: return invalided cache-entry count on non-directory event","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:16Z","receivedAt":"2024-02-23T03:18:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeah the refresh callback helper function for unqualified FSEvents\n(pathnames without a trailing slash) to return the number of\ncache-entries that were invalided in response to the event.\n\nThis will be used in a later commit to help determine if the observed\npathname was (possibly) case-incorrect when (on a case-insensitive\nfile system).\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex a51c17cda70..c16ed5d8758 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -196,8 +196,10 @@ static size_t handle_path_with_trailing_slash(\n * do not know it is case-correct or -incorrect.\n *\n * Assume it is case-correct and try an exact match.\n+ *\n+ * Return the number of cache-entries that we invalidated.\n */\n-static void handle_path_without_trailing_slash(\n+static size_t handle_path_without_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos)\n {\n \t/*\n@@ -218,7 +220,9 @@ static void handle_path_without_trailing_slash(\n \t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\treturn 1;\n \t} else {\n+\t\tsize_t nr_in_cone;\n \t\tstruct strbuf work_path = STRBUF_INIT;\n \n \t\t/*\n@@ -232,8 +236,10 @@ static void handle_path_without_trailing_slash(\n \t\tstrbuf_add(&work_path, name, strlen(name));\n \t\tstrbuf_addch(&work_path, '/');\n \t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n-\t\thandle_path_with_trailing_slash(istate, work_path.buf, pos);\n+\t\tnr_in_cone = handle_path_with_trailing_slash(\n+\t\t\tistate, work_path.buf, pos);\n \t\tstrbuf_release(&work_path);\n+\t\treturn nr_in_cone;\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"489209","messageId":"288f3f4e54e98a68d72e97125b1520605c138c3c.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 14/16] fsmonitor: support case-insensitive events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:18Z","receivedAt":"2024-02-23T03:18:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor_refresh_callback() to handle case-insensitive\nlookups if case-sensitive lookups fail on case-insensitive systems.\nThis can cause 'git status' to report stale status for files if there\nare case issues/errors in the worktree.\n\nThe FSMonitor daemon sends FSEvents using the observed spelling\nof each pathname. On case-insensitive file systems this may be\ndifferent than the expected case spelling.\n\nThe existing code uses index_name_pos() to find the cache-entry for\nthe pathname in the FSEvent and clear the CE_FSMONITOR_VALID bit so\nthat the worktree scan/index refresh will revisit and revalidate the\npath.\n\nOn a case-insensitive file system, the exact match lookup may fail\nto find the associated cache-entry. This causes status to think that\nthe cached CE flags are correct and skip over the file.\n\nUpdate event handling to optionally use the name-hash and dir-name-hash\nif necessary.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 110 insertions(+)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 739ddbf7aca..ac638a61c00 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -5,6 +5,7 @@\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"name-hash.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n #include \"trace2.h\"\n@@ -186,6 +187,102 @@ static int query_fsmonitor_hook(struct repository *r,\n static size_t handle_path_with_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos);\n \n+/*\n+ * Use the name-hash to do a case-insensitive cache-entry lookup with\n+ * the pathname and invalidate the cache-entry.\n+ *\n+ * Returns the number of cache-entries that we invalidated.\n+ */\n+static size_t handle_using_name_hash_icase(\n+\tstruct index_state *istate, const char *name)\n+{\n+\tstruct cache_entry *ce = NULL;\n+\n+\tce = index_file_exists(istate, name, strlen(name), 1);\n+\tif (!ce)\n+\t\treturn 0;\n+\n+\t/*\n+\t * A case-insensitive search in the name-hash using the\n+\t * observed pathname found a cache-entry, so the observed path\n+\t * is case-incorrect. Invalidate the cache-entry and use the\n+\t * correct spelling from the cache-entry to invalidate the\n+\t * untracked-cache. Since we now have sparse-directories in\n+\t * the index, the observed pathname may represent a regular\n+\t * file or a sparse-index directory.\n+\t *\n+\t * Note that we should not have seen FSEvents for a\n+\t * sparse-index directory, but we handle it just in case.\n+\t *\n+\t * Either way, we know that there are not any cache-entries for\n+\t * children inside the cone of the directory, so we don't need to\n+\t * do the usual scan.\n+\t */\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n+\t\t\t name, ce->name);\n+\n+\tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n+\n+\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\treturn 1;\n+}\n+\n+/*\n+ * Use the dir-name-hash to find the correct-case spelling of the\n+ * directory. Use the canonical spelling to invalidate all of the\n+ * cache-entries within the matching cone.\n+ *\n+ * Returns the number of cache-entries that we invalidated.\n+ */\n+static size_t handle_using_dir_name_hash_icase(\n+\tstruct index_state *istate, const char *name)\n+{\n+\tstruct strbuf canonical_path = STRBUF_INIT;\n+\tint pos;\n+\tsize_t len = strlen(name);\n+\tsize_t nr_in_cone;\n+\n+\tif (name[len - 1] == '/')\n+\t\tlen--;\n+\n+\tif (!index_dir_find(istate, name, len, &canonical_path))\n+\t\treturn 0; /* name is untracked */\n+\n+\tif (!memcmp(name, canonical_path.buf, canonical_path.len)) {\n+\t\tstrbuf_release(&canonical_path);\n+\t\t/*\n+\t\t * NEEDSWORK: Our caller already tried an exact match\n+\t\t * and failed to find one. They called us to do an\n+\t\t * ICASE match, so we should never get an exact match,\n+\t\t * so we could promote this to a BUG() here if we\n+\t\t * wanted to. It doesn't hurt anything to just return\n+\t\t * 0 and go on becaus we should never get here. Or we\n+\t\t * could just get rid of the memcmp() and this \"if\"\n+\t\t * clause completely.\n+\t\t */\n+\t\treturn 0; /* should not happen */\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n+\t\t\t name, canonical_path.buf);\n+\n+\t/*\n+\t * The dir-name-hash only tells us the corrected spelling of\n+\t * the prefix. We have to use this canonical path to do a\n+\t * lookup in the cache-entry array so that we repeat the\n+\t * original search using the case-corrected spelling.\n+\t */\n+\tstrbuf_addch(&canonical_path, '/');\n+\tpos = index_name_pos(istate, canonical_path.buf,\n+\t\t\t canonical_path.len);\n+\tnr_in_cone = handle_path_with_trailing_slash(\n+\t\tistate, canonical_path.buf, pos);\n+\tstrbuf_release(&canonical_path);\n+\treturn nr_in_cone;\n+}\n+\n /*\n * The daemon sent an observed pathname without a trailing slash.\n * (This is the normal case.) We do not know if it is a tracked or\n@@ -319,6 +416,19 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \telse\n \t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n \n+\t/*\n+\t * If we did not find an exact match for this pathname or any\n+\t * cache-entries with this directory prefix and we're on a\n+\t * case-insensitive file system, try again using the name-hash\n+\t * and dir-name-hash.\n+\t */\n+\tif (!nr_in_cone && ignore_case) {\n+\t\tnr_in_cone = handle_using_name_hash_icase(istate, name);\n+\t\tif (!nr_in_cone)\n+\t\t\tnr_in_cone = handle_using_dir_name_hash_icase(\n+\t\t\t\tistate, name);\n+\t}\n+\n \tif (nr_in_cone)\n \t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\n-- \ngitgitgadget\n\n"},{"id":"489210","messageId":"3a20065dbf80eabfc62c0bdebc16df0b5a4c7b02.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 15/16] fsmonitor: refactor bit invalidation in refresh callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:19Z","receivedAt":"2024-02-23T03:18:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor code in the fsmonitor_refresh_callback() call chain dealing\nwith invalidating the CE_FSMONITOR_VALID bit and add a trace message.\n\nDuring the refresh, we clear the CE_FSMONITOR_VALID bit in response to\ndata from the FSMonitor daemon (so that a later phase will lstat() and\nverify the true state of the file).\n\nCreate a new function to clear the bit and add some unique tracing for\nit to help debug edge cases.\n\nThis is similar to the existing `mark_fsmonitor_invalid()` function,\nbut we don't need the extra stuff that it does.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 20 +++++++++++++++++---\n 1 file changed, 17 insertions(+), 3 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex ac638a61c00..0667a8c297c 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -187,6 +187,20 @@ static int query_fsmonitor_hook(struct repository *r,\n static size_t handle_path_with_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos);\n \n+/*\n+ * Invalidate the FSM bit on this CE. This is like mark_fsmonitor_invalid()\n+ * but we've already handled the untracked-cache and I want a different\n+ * trace message.\n+ */\n+static void invalidate_ce_fsm(struct cache_entry *ce)\n+{\n+\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor_refresh_callback INV: '%s'\",\n+\t\t\t\t ce->name);\n+\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+}\n+\n /*\n * Use the name-hash to do a case-insensitive cache-entry lookup with\n * the pathname and invalidate the cache-entry.\n@@ -224,7 +238,7 @@ static size_t handle_using_name_hash_icase(\n \n \tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n \n-\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\tinvalidate_ce_fsm(ce);\n \treturn 1;\n }\n \n@@ -316,7 +330,7 @@ static size_t handle_path_without_trailing_slash(\n \t\t * cache-entry with the same pathname, nor for a cone\n \t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n-\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tinvalidate_ce_fsm(istate->cache[pos]);\n \t\treturn 1;\n \t} else {\n \t\tsize_t nr_in_cone;\n@@ -394,7 +408,7 @@ static size_t handle_path_with_trailing_slash(\n \tfor (i = pos; i < istate->cache_nr; i++) {\n \t\tif (!starts_with(istate->cache[i]->name, name))\n \t\t\tbreak;\n-\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tinvalidate_ce_fsm(istate->cache[i]);\n \t\tnr_in_cone++;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"489211","messageId":"467d3c1fe2ca86d5ba8dff7e4c6b1f5cc9c5806c.1708658300.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v2 16/16] t7527: update case-insenstive fsmonitor test","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-23T03:18:20Z","receivedAt":"2024-02-23T03:18:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nNow that the FSMonitor client has been updated to better\nhandle events on case-insenstive file systems, update the\ntwo tests that demonstrated the bug and remove the temporary\nSKIPME prereq.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 27 +++++++++++++++------------\n 1 file changed, 15 insertions(+), 12 deletions(-)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 4acb547819c..939521a0fac 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -1051,7 +1051,7 @@ test_expect_success 'split-index and FSMonitor work well together' '\n #\n # The setup is a little contrived.\n #\n-test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n \ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n \n \tgit init subdir_case_wrong &&\n@@ -1116,19 +1116,20 @@ test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on d\n \n \tgrep -q \"dir1/DIR2/dir3/file3.*pos -3\" \"$PWD/subdir_case_wrong.log1\" &&\n \n+\t# Also verify that we get a mapping event to correct the case.\n+\tgrep -q \"MAP:.*dir1/DIR2/dir3/file3.*dir1/dir2/dir3/file3\" \\\n+\t\t\"$PWD/subdir_case_wrong.log1\" &&\n+\n \t# The refresh-callbacks should have caused \"git status\" to clear\n \t# the CE_FSMONITOR_VALID bit on each of those files and caused\n \t# the worktree scan to visit them and mark them as modified.\n \tgrep -q \" M AAA\" \"$PWD/subdir_case_wrong.out\" &&\n \tgrep -q \" M zzz\" \"$PWD/subdir_case_wrong.out\" &&\n \n-\t# However, with the fsmonitor client bug, the \"(pos -3)\" causes\n-\t# the client to not update the bit and never rescan the file\n-\t# and therefore not report it as dirty.\n-\t! grep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n+\tgrep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n '\n \n-test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n \ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n \n \tgit init file_case_wrong &&\n@@ -1246,12 +1247,14 @@ test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor file case wrong on dis\n \tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n \tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n \n-\t# Status should say these files are modified, but with the case\n-\t# bug, the \"pos -3\" cause the client to not update the FSM bit\n-\t# and never cause the file to be rescanned and therefore to not\n-\t# report it dirty.\n-\t! grep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n-\t! grep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n+\t# Also verify that we get a mapping event to correct the case.\n+\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir3/FILE-3-A.*dir1/dir2/dir3/file-3-a\" \\\n+\t\t\"$PWD/file_case_wrong-try3.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir4/file-4-a.*dir1/dir2/dir4/FILE-4-A\" \\\n+\t\t\"$PWD/file_case_wrong-try3.log\" &&\n+\n+\tgrep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n+\tgrep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n '\n \n test_done\n-- \ngitgitgadget\n"},{"id":"489217","messageId":"xmqqy1bb1y9l.fsf@gitster.g","threadId":"60908","inReplyTo":"03b07d9c25edb951db4de518212f8a3e9e184e49.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 01/16] name-hash: add index_dir_find()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T06:37:58Z","receivedAt":"2024-02-23T06:38:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Replace the index_dir_exists() function with index_dir_find() and\n> change the API to take an optional strbuf to return the canonical\n> spelling of the matched directory prefix.\n>\n> Create an index_dir_exists() wrapper macro for existing callers.\n>\n> The existing index_dir_exists() returns a boolean to indicate if\n> there is a case-insensitive match in the directory name-hash, but\n> it doesn't tell the caller the exact spelling of that match.\n>\n> The new version also copies the matched spelling to a provided strbuf.\n> This lets the caller, for example, then call index_name_pos() with the\n> correct case to search the cache-entry array for the real insertion\n> position.\n\nThe usual way to compose a log message of this project is to\n\n - Give an observation on how the current system work in the present\n tense (so no need to say \"Currently X is Y\", just \"X is Y\"), and\n discuss what you perceive as a problem in it.\n\n - Propose a solution (optional---often, problem description\n trivially leads to an obvious solution in reader's minds).\n\n - Give commands to the codebase to \"become like so\".\n\nin this order.\n\nI think the third paragraph you wrote should come at the beginning,\nthen the first (now second) paragraph should describe more clearly\nthat index_dir_find() is a new function and what it does (perhaps by\nreusing what is in the \"The new version also...\" paragraph),\nwithout mentioning index_dir_exists().\n\nThe second (now third) paragraph then can talk about reimplementing\nindex_dir_exists() in terms of index_dir_find().\n\nThe patch text looks good.\n\nThanks.\n\n> -int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n> +int index_dir_find(struct index_state *istate, const char *name, int namelen,\n> +\t\t struct strbuf *canonical_path)\n> {\n> \tstruct dir_entry *dir;\n> \n> \tlazy_init_name_hash(istate);\n> \texpand_to_path(istate, name, namelen, 0);\n> \tdir = find_dir_entry(istate, name, namelen);\n> +\n> +\tif (canonical_path && dir && dir->nr) {\n> +\t\tstrbuf_reset(canonical_path);\n> +\t\tstrbuf_add(canonical_path, dir->name, dir->namelen);\n> +\t}\n> +\n> \treturn dir && dir->nr;\n> }\n> \n> diff --git a/name-hash.h b/name-hash.h\n> index b1b4b0fb337..0cbfc428631 100644\n> --- a/name-hash.h\n> +++ b/name-hash.h\n> @@ -4,7 +4,12 @@\n> struct cache_entry;\n> struct index_state;\n> \n> -int index_dir_exists(struct index_state *istate, const char *name, int namelen);\n> +\n> +int index_dir_find(struct index_state *istate, const char *name, int namelen,\n> +\t\t struct strbuf *canonical_path);\n> +\n> +#define index_dir_exists(i, n, l) index_dir_find((i), (n), (l), NULL)\n> +\n> void adjust_dirname_case(struct index_state *istate, char *name);\n> struct cache_entry *index_file_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n"},{"id":"489221","messageId":"xmqqh6hz1tmx.fsf@gitster.g","threadId":"60908","inReplyTo":"dad079ade7f8817f235d59c5339b82aecf4a17db.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 03/16] t7527: temporarily disable case-insensitive tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T08:17:58Z","receivedAt":"2024-02-23T08:18:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Add non-existent \"SKIPME\" prereq to the case-insensitive tests.\n>\n> The previous commit added test cases to demonstrate an error where\n> FSMonitor can get confused on a case-insensitive file system when the\n> on-disk spelling of a file or directory is wrong. Let's disable those\n> tests before we incrementally teach Git to properly recognize and\n> handle those types of problems (so that a bisect between here and the\n> final commit in this patch series won't throw a false alarm).\n\nYou talk about bisection, but hasn't the previous step already\nbroken bisection without these SKIPME prerequisites? IOW, shouldn't\nthis step squashed into the previous?\n\nAlso, it is much more common to replace \"test_expect_success\" with\n\"test_expect_failure\" to indicate that the steps are broken. Was\nthere a reason why we choose to do it differently?\n\n> Signed-off-by: Jeff Hostetler \n> ---\n> t/t7527-builtin-fsmonitor.sh | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n>\n> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n> index 3d21295f789..4acb547819c 100755\n> --- a/t/t7527-builtin-fsmonitor.sh\n> +++ b/t/t7527-builtin-fsmonitor.sh\n> @@ -1051,7 +1051,7 @@ test_expect_success 'split-index and FSMonitor work well together' '\n> #\n> # The setup is a little contrived.\n> #\n> -test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n> +test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n> \ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n> \n> \tgit init subdir_case_wrong &&\n> @@ -1128,7 +1128,7 @@ test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n> \t! grep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n> '\n> \n> -test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n> +test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n> \ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n> \n> \tgit init file_case_wrong &&\n"},{"id":"489222","messageId":"xmqq5xyf1tmu.fsf@gitster.g","threadId":"60908","inReplyTo":"7ee6ca1aefd34a37d749300e317df10d80ef2b29.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/16] fsmonitor: refactor refresh callback for non-directory events","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T08:18:01Z","receivedAt":"2024-02-23T08:18:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Move the code handle unqualified FSEvents (without a trailing slash)\n> into a helper function.\n\n-ECANNOTPARSE. \"code handle\" -> \"code that handles\"?\n\nIn the patch text itself, there is nothing unexpected. Looking\ngood.\n\nThanks.\n\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 67 +++++++++++++++++++++++++++++++----------------------\n> 1 file changed, 39 insertions(+), 28 deletions(-)\n>\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 29cce32d81c..364198d258f 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -183,6 +183,43 @@ static int query_fsmonitor_hook(struct repository *r,\n> \treturn result;\n> }\n> \n> +static void handle_path_without_trailing_slash(\n> +\tstruct index_state *istate, const char *name, int pos)\n> +{\n> +\tint i;\n> +\n> +\tif (pos >= 0) {\n> +\t\t/*\n> +\t\t * We have an exact match for this path and can just\n> +\t\t * invalidate it.\n> +\t\t */\n> +\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t} else {\n> +\t\t/*\n> +\t\t * The path is not a tracked file -or- it is a\n> +\t\t * directory event on a platform that cannot\n> +\t\t * distinguish between file and directory events in\n> +\t\t * the event handler, such as Windows.\n> +\t\t *\n> +\t\t * Scan as if it is a directory and invalidate the\n> +\t\t * cone under it. (But remember to ignore items\n> +\t\t * between \"name\" and \"name/\", such as \"name-\" and\n> +\t\t * \"name.\".\n> +\t\t */\n> +\t\tint len = strlen(name);\n> +\t\tpos = -pos - 1;\n> +\n> +\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> +\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> +\t\t\t\tbreak;\n> +\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n> +\t\t\t\tbreak;\n> +\t\t\tif (istate->cache[i]->name[len] == '/')\n> +\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\t}\n> +\t}\n> +}\n> +\n> /*\n> * The daemon can decorate directory events, such as a move or rename,\n> * by adding a trailing slash to the observed name. Use this to\n> @@ -225,7 +262,7 @@ static void handle_path_with_trailing_slash(\n> \n> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> {\n> -\tint i, len = strlen(name);\n> +\tint len = strlen(name);\n> \tint pos = index_name_pos(istate, name, len);\n> \n> \ttrace_printf_key(&trace_fsmonitor,\n> @@ -240,34 +277,8 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \t\t * for the untracked cache.\n> \t\t */\n> \t\tname[len - 1] = '\\0';\n> -\t} else if (pos >= 0) {\n> -\t\t/*\n> -\t\t * We have an exact match for this path and can just\n> -\t\t * invalidate it.\n> -\t\t */\n> -\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> \t} else {\n> -\t\t/*\n> -\t\t * The path is not a tracked file -or- it is a\n> -\t\t * directory event on a platform that cannot\n> -\t\t * distinguish between file and directory events in\n> -\t\t * the event handler, such as Windows.\n> -\t\t *\n> -\t\t * Scan as if it is a directory and invalidate the\n> -\t\t * cone under it. (But remember to ignore items\n> -\t\t * between \"name\" and \"name/\", such as \"name-\" and\n> -\t\t * \"name.\".\n> -\t\t */\n> -\t\tpos = -pos - 1;\n> -\n> -\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> -\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> -\t\t\t\tbreak;\n> -\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n> -\t\t\t\tbreak;\n> -\t\t\tif (istate->cache[i]->name[len] == '/')\n> -\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> -\t\t}\n> +\t\thandle_path_without_trailing_slash(istate, name, pos);\n> \t}\n> \n> \t/*\n"},{"id":"489223","messageId":"xmqqbk871tmv.fsf@gitster.g","threadId":"60908","inReplyTo":"5516670e30e26c5b50c67b69e48e3e8a5e0d8990.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 04/16] fsmonitor: refactor refresh callback on directory events","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T08:18:00Z","receivedAt":"2024-02-23T08:18:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Move the code to handle directory FSEvents (containing pathnames with\n> a trailing slash) into a helper function.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 52 ++++++++++++++++++++++++++++++----------------------\n> 1 file changed, 30 insertions(+), 22 deletions(-)\n\nNothing unexpected to see here. Looking good.\n\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index f670c509378..6fecae9aeb2 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -183,6 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n> \treturn result;\n> }\n> \n> +static void handle_path_with_trailing_slash(\n> +\tstruct index_state *istate, const char *name, int pos)\n> +{\n> +\tint i;\n> +\n> +\t/*\n> +\t * The daemon can decorate directory events, such as\n> +\t * moves or renames, with a trailing slash if the OS\n> +\t * FS Event contains sufficient information, such as\n> +\t * MacOS.\n> +\t *\n> +\t * Use this to invalidate the entire cone under that\n> +\t * directory.\n> +\t *\n> +\t * We do not expect an exact match because the index\n> +\t * does not normally contain directory entries, so we\n> +\t * start at the insertion point and scan.\n> +\t */\n> +\tif (pos < 0)\n> +\t\tpos = -pos - 1;\n> +\n> +\t/* Mark all entries for the folder invalid */\n> +\tfor (i = pos; i < istate->cache_nr; i++) {\n> +\t\tif (!starts_with(istate->cache[i]->name, name))\n> +\t\t\tbreak;\n> +\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t}\n> +}\n> +\n> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> {\n> \tint i, len = strlen(name);\n> @@ -193,28 +222,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \t\t\t name, pos);\n> \n> \tif (name[len - 1] == '/') {\n> -\t\t/*\n> -\t\t * The daemon can decorate directory events, such as\n> -\t\t * moves or renames, with a trailing slash if the OS\n> -\t\t * FS Event contains sufficient information, such as\n> -\t\t * MacOS.\n> -\t\t *\n> -\t\t * Use this to invalidate the entire cone under that\n> -\t\t * directory.\n> -\t\t *\n> -\t\t * We do not expect an exact match because the index\n> -\t\t * does not normally contain directory entries, so we\n> -\t\t * start at the insertion point and scan.\n> -\t\t */\n> -\t\tif (pos < 0)\n> -\t\t\tpos = -pos - 1;\n> -\n> -\t\t/* Mark all entries for the folder invalid */\n> -\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> -\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> -\t\t\t\tbreak;\n> -\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> -\t\t}\n> +\t\thandle_path_with_trailing_slash(istate, name, pos);\n> \n> \t\t/*\n> \t\t * We need to remove the traling \"/\" from the path\n"},{"id":"489244","messageId":"xmqqy1bbuloo.fsf@gitster.g","threadId":"60908","inReplyTo":"af6f57ab3e6d61036cd969f5fd9256200313aaa9.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 09/16] fsmonitor: move untracked invalidation into helper functions","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T17:36:55Z","receivedAt":"2024-02-23T17:37:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Move the call to invalidate the untracked cache for the FSEvent\n> pathname into the two helper functions.\n>\n> In a later commit in this series, we will call these helpers\n> from other contexts and it safer to include the UC invalidation\n> in the helper than to remember to also add it to each helper\n> call-site.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 26 +++++++++++++++++++-------\n> 1 file changed, 19 insertions(+), 7 deletions(-)\n\nThanks. The steps in this iteration makes this move much less\nconfusing to me than in the previous one. We used to call one of\n\"handle path with/without trailing slash\" functions and then called\nthe invalidation. Now the invalidation happens in these \"handle path\"\nfunctions.\n\nThe unexplained change in behaviour is that we used to do the rest\nof \"handle path\" and invalidation was done at the end. Now we do it\nupfront. I think the \"rest\" works solely based on what is in the\nmain in-core index array (i.e. the_index.cache[] aka active_cache[])\nand affects only what is in the in-core index array, while\nuntracked_cache_invalidate*() works solely based on what is in the\nuntracked cache extension (i.e. the_index.untracked) and affects\nonly what is in there, so the order of these two does not matter.\n\nAm I correct?\n\nOr does it affect correctness or performance or whatever in any way?\nIOW, is there a reason why it is better to do the invalidation first\nand then doing the \"rest\" after (hence this patch flips the order of\ntwo to _improve_ something)?\n\nThanks.\n"},{"id":"489246","messageId":"xmqqfrxjul7a.fsf@gitster.g","threadId":"60908","inReplyTo":"1853f77d3331f7736139f686ac2efee6d68f9207.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 11/16] fsmonitor: remove custom loop from non-directory path handler","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T17:47:21Z","receivedAt":"2024-02-23T17:47:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> static void handle_path_without_trailing_slash(\n> \tstruct index_state *istate, const char *name, int pos)\n> {\n> -\tint i;\n> -\n> \t/*\n> \t * Mark the untracked cache dirty for this path (regardless of\n> \t * whether or not we find an exact match for it in the index).\n> @@ -200,33 +212,28 @@ static void handle_path_without_trailing_slash(\n> \n> \tif (pos >= 0) {\n> \t\t/*\n> -\t\t * We have an exact match for this path and can just\n> -\t\t * invalidate it.\n> +\t\t * An exact match on a tracked file. We assume that we\n> +\t\t * do not need to scan forward for a sparse-directory\n> +\t\t * cache-entry with the same pathname, nor for a cone\n> +\t\t * at that directory. (That is, assume no D/F conflicts.)\n> \t\t */\n> \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> \t} else {\n> +\t\tstruct strbuf work_path = STRBUF_INIT;\n> +\n> \t\t/*\n> +\t\tstrbuf_add(&work_path, name, strlen(name));\n> +\t\tstrbuf_addch(&work_path, '/');\n> +\t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n> +\t\thandle_path_with_trailing_slash(istate, work_path.buf, pos);\n> +\t\tstrbuf_release(&work_path);\n> \t}\n> }\n\nThe \"with trailing slash\" variant is returning a useful value to\nthis caller that ignores it, but we do not yet return a value from\nthis function, so that is OK. The name being a name that may be in\ndifferent case from what we know in the index is not yet handled in\nthis step (we have \"Assume it is case-correct\" in the comment) and\nthat applies for both the main array of cache entries as well as the\nuntracked cache.\n\nIt will be exciting to see how these are lifted. The main array has\nsome helper functions that uses name-hash features to help icase\nmatches, but I do not offhand recall what we have for the untracked\ncache side.\n"},{"id":"489247","messageId":"xmqqbk87ul15.fsf@gitster.g","threadId":"60908","inReplyTo":"f77d68c78ada02bfb4b96759f6ad82ebff00b35b.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 12/16] fsmonitor: return invalided cache-entry count on non-directory event","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T17:51:02Z","receivedAt":"2024-02-23T17:51:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Teah the refresh callback helper function for unqualified FSEvents\n\nTeach?\n\n> (pathnames without a trailing slash) to return the number of\n> cache-entries that were invalided in response to the event.\n>\n> This will be used in a later commit to help determine if the observed\n> pathname was (possibly) case-incorrect when (on a case-insensitive\n> file system).\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 10 ++++++++--\n> 1 file changed, 8 insertions(+), 2 deletions(-)\n\nI do not see anything unexpected in the change to the code below.\nLooking good.\n\nThanks.\n\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index a51c17cda70..c16ed5d8758 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -196,8 +196,10 @@ static size_t handle_path_with_trailing_slash(\n> * do not know it is case-correct or -incorrect.\n> *\n> * Assume it is case-correct and try an exact match.\n> + *\n> + * Return the number of cache-entries that we invalidated.\n> */\n> -static void handle_path_without_trailing_slash(\n> +static size_t handle_path_without_trailing_slash(\n> \tstruct index_state *istate, const char *name, int pos)\n> {\n> \t/*\n> @@ -218,7 +220,9 @@ static void handle_path_without_trailing_slash(\n> \t\t * at that directory. (That is, assume no D/F conflicts.)\n> \t\t */\n> \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\treturn 1;\n> \t} else {\n> +\t\tsize_t nr_in_cone;\n> \t\tstruct strbuf work_path = STRBUF_INIT;\n> \n> \t\t/*\n> @@ -232,8 +236,10 @@ static void handle_path_without_trailing_slash(\n> \t\tstrbuf_add(&work_path, name, strlen(name));\n> \t\tstrbuf_addch(&work_path, '/');\n> \t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n> -\t\thandle_path_with_trailing_slash(istate, work_path.buf, pos);\n> +\t\tnr_in_cone = handle_path_with_trailing_slash(\n> +\t\t\tistate, work_path.buf, pos);\n> \t\tstrbuf_release(&work_path);\n> +\t\treturn nr_in_cone;\n> \t}\n> }\n"},{"id":"489248","messageId":"xmqq5xyfukx4.fsf@gitster.g","threadId":"60908","inReplyTo":"58b36673e151ad15eb44c9ca1c03cfef51657d11.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 13/16] fsmonitor: trace the new invalidated cache-entry count","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T17:53:27Z","receivedAt":"2024-02-23T17:53:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Consolidate the directory/non-directory calls to the refresh handler\n> code. Log the resulting count of invalidated cache-entries.\n\nOK. Again, there is nothing surprising in the changes in the patch.\nLooking good.\n\n> The nr_in_cone value will be used in a later commit to decide if\n> we also need to try to do case-insensitive lookups.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 15 ++++++++++-----\n> 1 file changed, 10 insertions(+), 5 deletions(-)\n\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index c16ed5d8758..739ddbf7aca 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -308,16 +308,21 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> {\n> \tint len = strlen(name);\n> \tint pos = index_name_pos(istate, name, len);\n> +\tsize_t nr_in_cone;\n> \n> \ttrace_printf_key(&trace_fsmonitor,\n> \t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n> \t\t\t name, pos);\n> \n> -\tif (name[len - 1] == '/') {\n> -\t\thandle_path_with_trailing_slash(istate, name, pos);\n> -\t} else {\n> -\t\thandle_path_without_trailing_slash(istate, name, pos);\n> -\t}\n> +\tif (name[len - 1] == '/')\n> +\t\tnr_in_cone = handle_path_with_trailing_slash(istate, name, pos);\n> +\telse\n> +\t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n> +\n> +\tif (nr_in_cone)\n> +\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\n> +\t\t\t\t (int)nr_in_cone);\n> }\n> \n> /*\n"},{"id":"489249","messageId":"xmqqjzmvt5d3.fsf@gitster.g","threadId":"60908","inReplyTo":"288f3f4e54e98a68d72e97125b1520605c138c3c.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 14/16] fsmonitor: support case-insensitive events","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T18:14:48Z","receivedAt":"2024-02-23T18:14:50Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> +/*\n> + * Use the name-hash to do a case-insensitive cache-entry lookup with\n> + * the pathname and invalidate the cache-entry.\n> + *\n> + * Returns the number of cache-entries that we invalidated.\n> + */\n> +static size_t handle_using_name_hash_icase(\n> +\tstruct index_state *istate, const char *name)\n> +{\n> +\tstruct cache_entry *ce = NULL;\n> +\n> +\tce = index_file_exists(istate, name, strlen(name), 1);\n> +\tif (!ce)\n> +\t\treturn 0;\n> +\n> +\t/*\n> +\t * A case-insensitive search in the name-hash using the\n> +\t * observed pathname found a cache-entry, so the observed path\n> +\t * is case-incorrect. Invalidate the cache-entry and use the\n> +\t * correct spelling from the cache-entry to invalidate the\n> +\t * untracked-cache. Since we now have sparse-directories in\n> +\t * the index, the observed pathname may represent a regular\n> +\t * file or a sparse-index directory.\n> +\t *\n> +\t * Note that we should not have seen FSEvents for a\n> +\t * sparse-index directory, but we handle it just in case.\n> +\t *\n> +\t * Either way, we know that there are not any cache-entries for\n> +\t * children inside the cone of the directory, so we don't need to\n> +\t * do the usual scan.\n> +\t */\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n> +\t\t\t name, ce->name);\n> +\n> +\tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n> +\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\treturn 1;\n> +}\n\nYou first ask the name-hash to turn the incoming \"name\" into the\ncase variant that we know about, i.e. ce->name, and use that to\naccess the untracked cache. Clever and makes sense. But if we have\nce->name, doesn't it mean the name is tracked? Do we find anything\nuseful to do in the untracked cache invalidation codepath in that\ncase?\n\nAn FSmonitor event with case-incorrect pathname for a directory may\nnot be this trivial, I presume, and I expect that is what the\nremainder of this patch is about.\n\n> +\n> +/*\n> + * Use the dir-name-hash to find the correct-case spelling of the\n> + * directory. Use the canonical spelling to invalidate all of the\n> + * cache-entries within the matching cone.\n> + *\n> + * Returns the number of cache-entries that we invalidated.\n> + */\n> +static size_t handle_using_dir_name_hash_icase(\n> +\tstruct index_state *istate, const char *name)\n\nIt is a bit unfortunate that here on the name-hash side we contrast\nthe two helper function variants as \"dir-name\" vs \"name\", while the\noriginal handle_path side use \"without_slash\" vs \"with_slash\".\n\nIf I understand correctly, it is not like there are two distinct\nhashes, \"name-hash\" vs \"dir-name-hash\". Both of these helpers use\nthe same \"name-hash\" mechanism, and this function differs from the\nprevious one in that it is about a directory, which is why it has\n\"dir\" in its name. I wonder if we renamed the other one with\n\"nondir\" in its name, and the other without_slash and with_slash\npair to match, e.g., handle_nondir_path() vs handle_dir_path(), or\nsomething like that, the resulting names for these four functions\nbecome easier to contrast and understand?\n\n> +{\n> +\tstruct strbuf canonical_path = STRBUF_INIT;\n> +\tint pos;\n> +\tsize_t len = strlen(name);\n> +\tsize_t nr_in_cone;\n> +\n> +\tif (name[len - 1] == '/')\n> +\t\tlen--;\n> +\n> +\tif (!index_dir_find(istate, name, len, &canonical_path))\n> +\t\treturn 0; /* name is untracked */\n> +\n> +\tif (!memcmp(name, canonical_path.buf, canonical_path.len)) {\n> +\t\tstrbuf_release(&canonical_path);\n> +\t\t/*\n> +\t\t * NEEDSWORK: Our caller already tried an exact match\n> +\t\t * and failed to find one. They called us to do an\n> +\t\t * ICASE match, so we should never get an exact match,\n> +\t\t * so we could promote this to a BUG() here if we\n> +\t\t * wanted to. It doesn't hurt anything to just return\n> +\t\t * 0 and go on becaus we should never get here. Or we\n> +\t\t * could just get rid of the memcmp() and this \"if\"\n> +\t\t * clause completely.\n> +\t\t */\n> +\t\treturn 0; /* should not happen */\n> +\t}\n\n\"becaus\" -> \"because\".\n\nIf we should never get here, having BUG(\"we should never get here\")\nwould not hurt anything, either. On the other hand, silently\nreturning 0 will hide the bug under the carpet, and I am not sure it\nis fair to call it \"doesn't hurt anything\".\n\n> +\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n> +\t\t\t name, canonical_path.buf);\n> +\n> +\t/*\n> +\t * The dir-name-hash only tells us the corrected spelling of\n> +\t * the prefix. We have to use this canonical path to do a\n> +\t * lookup in the cache-entry array so that we repeat the\n> +\t * original search using the case-corrected spelling.\n> +\t */\n> +\tstrbuf_addch(&canonical_path, '/');\n> +\tpos = index_name_pos(istate, canonical_path.buf,\n> +\t\t\t canonical_path.len);\n> +\tnr_in_cone = handle_path_with_trailing_slash(\n> +\t\tistate, canonical_path.buf, pos);\n> +\tstrbuf_release(&canonical_path);\n> +\treturn nr_in_cone;\n> +}\n\nNice. Do we need to give this corrected name to help untracked\ncache invalidation from the caller that called us?\n\n> @@ -319,6 +416,19 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \telse\n> \t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n> \n> +\t/*\n> +\t * If we did not find an exact match for this pathname or any\n> +\t * cache-entries with this directory prefix and we're on a\n> +\t * case-insensitive file system, try again using the name-hash\n> +\t * and dir-name-hash.\n> +\t */\n> +\tif (!nr_in_cone && ignore_case) {\n> +\t\tnr_in_cone = handle_using_name_hash_icase(istate, name);\n> +\t\tif (!nr_in_cone)\n> +\t\t\tnr_in_cone = handle_using_dir_name_hash_icase(\n> +\t\t\t\tistate, name);\n> +\t}\n\nIt might be interesting to learn how often we go through these\n\"fallback\" code paths by tracing. Maybe it will become too noisy?\nI dunno.\n\n> \tif (nr_in_cone)\n> \t\ttrace_printf_key(&trace_fsmonitor,\n> \t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\n"},{"id":"489250","messageId":"xmqqbk87t57g.fsf@gitster.g","threadId":"60908","inReplyTo":"3a20065dbf80eabfc62c0bdebc16df0b5a4c7b02.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 15/16] fsmonitor: refactor bit invalidation in refresh callback","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-23T18:18:11Z","receivedAt":"2024-02-23T18:18:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> From: Jeff Hostetler \n>\n> Refactor code in the fsmonitor_refresh_callback() call chain dealing\n> with invalidating the CE_FSMONITOR_VALID bit and add a trace message.\n>\n> During the refresh, we clear the CE_FSMONITOR_VALID bit in response to\n> data from the FSMonitor daemon (so that a later phase will lstat() and\n> verify the true state of the file).\n>\n> Create a new function to clear the bit and add some unique tracing for\n> it to help debug edge cases.\n>\n> This is similar to the existing `mark_fsmonitor_invalid()` function,\n> but we don't need the extra stuff that it does.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 20 +++++++++++++++++---\n> 1 file changed, 17 insertions(+), 3 deletions(-)\n>\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index ac638a61c00..0667a8c297c 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -187,6 +187,20 @@ static int query_fsmonitor_hook(struct repository *r,\n> static size_t handle_path_with_trailing_slash(\n> \tstruct index_state *istate, const char *name, int pos);\n> \n> +/*\n> + * Invalidate the FSM bit on this CE. This is like mark_fsmonitor_invalid()\n> + * but we've already handled the untracked-cache and I want a different\n> + * trace message.\n> + */\n\n\"I want\" -> \"want\" perhaps.\n\nMore importantly, when new developers come and want to touch this\nfile in the future, how would they choose which one to call? Would\nit make a better comment if we rewrote the above for such future\ndevelopers as intended audiences?\n\n> +static void invalidate_ce_fsm(struct cache_entry *ce)\n> +{\n> +\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n> +\t\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t\t \"fsmonitor_refresh_callback INV: '%s'\",\n> +\t\t\t\t ce->name);\n> +\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n> +}\n> +\n> /*\n> * Use the name-hash to do a case-insensitive cache-entry lookup with\n> * the pathname and invalidate the cache-entry.\n> @@ -224,7 +238,7 @@ static size_t handle_using_name_hash_icase(\n> \n> \tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n> \n> -\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\tinvalidate_ce_fsm(ce);\n> \treturn 1;\n> }\n> \n> @@ -316,7 +330,7 @@ static size_t handle_path_without_trailing_slash(\n> \t\t * cache-entry with the same pathname, nor for a cone\n> \t\t * at that directory. (That is, assume no D/F conflicts.)\n> \t\t */\n> -\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\tinvalidate_ce_fsm(istate->cache[pos]);\n> \t\treturn 1;\n> \t} else {\n> \t\tsize_t nr_in_cone;\n> @@ -394,7 +408,7 @@ static size_t handle_path_with_trailing_slash(\n> \tfor (i = pos; i < istate->cache_nr; i++) {\n> \t\tif (!starts_with(istate->cache[i]->name, name))\n> \t\t\tbreak;\n> -\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\tinvalidate_ce_fsm(istate->cache[i]);\n> \t\tnr_in_cone++;\n> \t}\n\nNice.\n"},{"id":"489310","messageId":"20240225123050.GA1952@tb-raspi4","threadId":"60908","inReplyTo":"7ee6ca1aefd34a37d749300e317df10d80ef2b29.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/16] fsmonitor: refactor refresh callback for non-directory events","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2024-02-25T12:30:51Z","receivedAt":"2024-02-25T12:30:59Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, Feb 23, 2024 at 03:18:10AM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n>\n> Move the code handle unqualified FSEvents (without a trailing slash)\n> into a helper function.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 67 +++++++++++++++++++++++++++++++----------------------\n> 1 file changed, 39 insertions(+), 28 deletions(-)\n>\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 29cce32d81c..364198d258f 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -183,6 +183,43 @@ static int query_fsmonitor_hook(struct repository *r,\n> \treturn result;\n> }\n>\n> +static void handle_path_without_trailing_slash(\n> +\tstruct index_state *istate, const char *name, int pos)\n> +{\n> +\tint i;\n> +\n> +\tif (pos >= 0) {\n> +\t\t/*\n> +\t\t * We have an exact match for this path and can just\n> +\t\t * invalidate it.\n> +\t\t */\n> +\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t} else {\n> +\t\t/*\n> +\t\t * The path is not a tracked file -or- it is a\n> +\t\t * directory event on a platform that cannot\n> +\t\t * distinguish between file and directory events in\n> +\t\t * the event handler, such as Windows.\n> +\t\t *\n> +\t\t * Scan as if it is a directory and invalidate the\n> +\t\t * cone under it. (But remember to ignore items\n> +\t\t * between \"name\" and \"name/\", such as \"name-\" and\n> +\t\t * \"name.\".\n> +\t\t */\n> +\t\tint len = strlen(name);\n\nshould this be\n\tsize_t len = strlen(name);\n\n> +\t\tpos = -pos - 1;\n> +\n> +\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> +\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> +\t\t\t\tbreak;\n> +\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n> +\t\t\t\tbreak;\n\nHm, this covers all digits, letters, :;<=>?\nbut not e.g. !+-. (and others). What do i miss ?\n\n\n> +\t\t\tif (istate->cache[i]->name[len] == '/')\n> +\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\t}\n> +\t}\n> +}\n> +\n> /*\n> * The daemon can decorate directory events, such as a move or rename,\n> * by adding a trailing slash to the observed name. Use this to\n> @@ -225,7 +262,7 @@ static void handle_path_with_trailing_slash(\n>\n> static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> {\n> -\tint i, len = strlen(name);\n> +\tint len = strlen(name);\n\nSame here: size_t len = strlen() ?\n\n> \tint pos = index_name_pos(istate, name, len);\n>\n> \ttrace_printf_key(&trace_fsmonitor,\n> @@ -240,34 +277,8 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \t\t * for the untracked cache.\n> \t\t */\n> \t\tname[len - 1] = '\\0';\n> -\t} else if (pos >= 0) {\n> -\t\t/*\n> -\t\t * We have an exact match for this path and can just\n> -\t\t * invalidate it.\n> -\t\t */\n> -\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> \t} else {\n> -\t\t/*\n> -\t\t * The path is not a tracked file -or- it is a\n> -\t\t * directory event on a platform that cannot\n> -\t\t * distinguish between file and directory events in\n> -\t\t * the event handler, such as Windows.\n> -\t\t *\n> -\t\t * Scan as if it is a directory and invalidate the\n> -\t\t * cone under it. (But remember to ignore items\n> -\t\t * between \"name\" and \"name/\", such as \"name-\" and\n> -\t\t * \"name.\".\n> -\t\t */\n> -\t\tpos = -pos - 1;\n> -\n> -\t\tfor (i = pos; i < istate->cache_nr; i++) {\n> -\t\t\tif (!starts_with(istate->cache[i]->name, name))\n> -\t\t\t\tbreak;\n> -\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n> -\t\t\t\tbreak;\n> -\t\t\tif (istate->cache[i]->name[len] == '/')\n> -\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n> -\t\t}\n> +\t\thandle_path_without_trailing_slash(istate, name, pos);\n> \t}\n>\n> \t/*\n> --\n> gitgitgadget\n>\n>\n\n"},{"id":"489311","messageId":"20240225123527.GB1952@tb-raspi4","threadId":"60908","inReplyTo":"99c0d3e0742c1a7e0f7608707402a772ec112716.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 07/16] dir: create untracked_cache_invalidate_trimmed_path()","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2024-02-25T12:35:27Z","receivedAt":"2024-02-25T12:35:37Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, Feb 23, 2024 at 03:18:11AM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n>\n> Create a wrapper function for untracked_cache_invalidate_path()\n> that silently trims a trailing slash, if present, before calling\n> the wrapped function.\n>\n> The untracked cache expects to be called with a pathname that\n> does not contain a trailing slash. This can make it inconvenient\n> for callers that have a directory path. Lets hide this complexity.\n>\n> This will be used by a later commit in the FSMonitor code which\n> may receive directory pathnames from an FSEvent.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> dir.c | 20 ++++++++++++++++++++\n> dir.h | 7 +++++++\n> 2 files changed, 27 insertions(+)\n>\n> diff --git a/dir.c b/dir.c\n> index ac699542302..1157f3e43fa 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -3918,6 +3918,26 @@ void untracked_cache_invalidate_path(struct index_state *istate,\n> \t\t\t\t path, strlen(path));\n> }\n>\n> +void untracked_cache_invalidate_trimmed_path(struct index_state *istate,\n> +\t\t\t\t\t const char *path,\n> +\t\t\t\t\t int safe_path)\n> +{\n> +\tsize_t len = strlen(path);\n> +\n> +\tif (!len)\n> +\t\treturn; /* should not happen */\n\nShould a BUG() be used ? Or bug(), for the record:\nPlease see Documentation/technical/api-error-handling.txt\n> +\n> +\tif (path[len - 1] != '/') {\n> +\t\tuntracked_cache_invalidate_path(istate, path, safe_path);\n> +\t} else {\n> +\t\tstruct strbuf tmp = STRBUF_INIT;\n> +\n> +\t\tstrbuf_add(&tmp, path, len - 1);\n> +\t\tuntracked_cache_invalidate_path(istate, tmp.buf, safe_path);\n> +\t\tstrbuf_release(&tmp);\n> +\t}\n> +}\n> +\n> void untracked_cache_remove_from_index(struct index_state *istate,\n> \t\t\t\t const char *path)\n> {\n> diff --git a/dir.h b/dir.h\n> index 98aa85fcc0e..45a7b9ec5f2 100644\n> --- a/dir.h\n> +++ b/dir.h\n> @@ -576,6 +576,13 @@ int cmp_dir_entry(const void *p1, const void *p2);\n> int check_dir_entry_contains(const struct dir_entry *out, const struct dir_entry *in);\n>\n> void untracked_cache_invalidate_path(struct index_state *, const char *, int safe_path);\n> +/*\n> + * Invalidate the untracked-cache for this path, but first strip\n> + * off a trailing slash, if present.\n> + */\n> +void untracked_cache_invalidate_trimmed_path(struct index_state *,\n> +\t\t\t\t\t const char *path,\n> +\t\t\t\t\t int safe_path);\n> void untracked_cache_remove_from_index(struct index_state *, const char *);\n> void untracked_cache_add_to_index(struct index_state *, const char *);\n>\n> --\n> gitgitgadget\n>\n>\n"},{"id":"489312","messageId":"20240225131023.GC1952@tb-raspi4","threadId":"60908","inReplyTo":"288f3f4e54e98a68d72e97125b1520605c138c3c.1708658300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 14/16] fsmonitor: support case-insensitive events","fromName":"Torsten Bögershausen","fromEmail":"tboegi@web.de","sentAt":"2024-02-25T13:10:23Z","receivedAt":"2024-02-25T13:10:32Z","isPatch":true,"sender":{"key":"tboegi@web.de","avatar":"https://avatars.githubusercontent.com/u/7138363?v=4"},"body":"On Fri, Feb 23, 2024 at 03:18:18AM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n>\n> Teach fsmonitor_refresh_callback() to handle case-insensitive\n> lookups if case-sensitive lookups fail on case-insensitive systems.\n> This can cause 'git status' to report stale status for files if there\n> are case issues/errors in the worktree.\n>\n> The FSMonitor daemon sends FSEvents using the observed spelling\n> of each pathname. On case-insensitive file systems this may be\n> different than the expected case spelling.\n>\n> The existing code uses index_name_pos() to find the cache-entry for\n> the pathname in the FSEvent and clear the CE_FSMONITOR_VALID bit so\n> that the worktree scan/index refresh will revisit and revalidate the\n> path.\n>\n> On a case-insensitive file system, the exact match lookup may fail\n> to find the associated cache-entry. This causes status to think that\n> the cached CE flags are correct and skip over the file.\n>\n> Update event handling to optionally use the name-hash and dir-name-hash\n> if necessary.\n>\n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 110 insertions(+)\n>\n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 739ddbf7aca..ac638a61c00 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -5,6 +5,7 @@\n> #include \"ewah/ewok.h\"\n> #include \"fsmonitor.h\"\n> #include \"fsmonitor-ipc.h\"\n> +#include \"name-hash.h\"\n> #include \"run-command.h\"\n> #include \"strbuf.h\"\n> #include \"trace2.h\"\n> @@ -186,6 +187,102 @@ static int query_fsmonitor_hook(struct repository *r,\n> static size_t handle_path_with_trailing_slash(\n> \tstruct index_state *istate, const char *name, int pos);\n>\n> +/*\n> + * Use the name-hash to do a case-insensitive cache-entry lookup with\n> + * the pathname and invalidate the cache-entry.\n> + *\n> + * Returns the number of cache-entries that we invalidated.\n> + */\n> +static size_t handle_using_name_hash_icase(\n> +\tstruct index_state *istate, const char *name)\n> +{\n> +\tstruct cache_entry *ce = NULL;\n> +\n> +\tce = index_file_exists(istate, name, strlen(name), 1);\n> +\tif (!ce)\n> +\t\treturn 0;\n> +\n> +\t/*\n> +\t * A case-insensitive search in the name-hash using the\n> +\t * observed pathname found a cache-entry, so the observed path\n> +\t * is case-incorrect. Invalidate the cache-entry and use the\n> +\t * correct spelling from the cache-entry to invalidate the\n> +\t * untracked-cache. Since we now have sparse-directories in\n> +\t * the index, the observed pathname may represent a regular\n> +\t * file or a sparse-index directory.\n> +\t *\n> +\t * Note that we should not have seen FSEvents for a\n> +\t * sparse-index directory, but we handle it just in case.\n> +\t *\n> +\t * Either way, we know that there are not any cache-entries for\n> +\t * children inside the cone of the directory, so we don't need to\n> +\t * do the usual scan.\n> +\t */\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n> +\t\t\t name, ce->name);\n> +\n> +\tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n> +\n> +\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\treturn 1;\n> +}\n> +\n> +/*\n> + * Use the dir-name-hash to find the correct-case spelling of the\n> + * directory. Use the canonical spelling to invalidate all of the\n> + * cache-entries within the matching cone.\n> + *\n> + * Returns the number of cache-entries that we invalidated.\n> + */\n> +static size_t handle_using_dir_name_hash_icase(\n> +\tstruct index_state *istate, const char *name)\n> +{\n> +\tstruct strbuf canonical_path = STRBUF_INIT;\n> +\tint pos;\n> +\tsize_t len = strlen(name);\n> +\tsize_t nr_in_cone;\n> +\n> +\tif (name[len - 1] == '/')\n> +\t\tlen--;\n> +\n> +\tif (!index_dir_find(istate, name, len, &canonical_path))\n> +\t\treturn 0; /* name is untracked */\n> +\n> +\tif (!memcmp(name, canonical_path.buf, canonical_path.len)) {\n> +\t\tstrbuf_release(&canonical_path);\n> +\t\t/*\n> +\t\t * NEEDSWORK: Our caller already tried an exact match\n> +\t\t * and failed to find one. They called us to do an\n> +\t\t * ICASE match, so we should never get an exact match,\n> +\t\t * so we could promote this to a BUG() here if we\n> +\t\t * wanted to. It doesn't hurt anything to just return\n> +\t\t * 0 and go on becaus we should never get here. Or we\n> +\t\t * could just get rid of the memcmp() and this \"if\"\n> +\t\t * clause completely.\n> +\t\t */\n> +\t\treturn 0; /* should not happen */\n\nIn very very theory, there may be a race-condition,\nwhen a directory is renamed very fast, more than once.\nI don't think, that the \"it did not match exactly, but\nnow it matches\" is a problem.\nQuestion: Does it make sense to just remove this ?\nAnd, may be, find out that the \"corrected spelling (tm)\"\nof \"DIR1\" is not \"dir1\", neither \"Dir1\", but, exactly, \"DIR1\" ?\nWould that be a problem ?\n\n\n> +\t}\n> +\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n> +\t\t\t name, canonical_path.buf);\n> +\n> +\t/*\n> +\t * The dir-name-hash only tells us the corrected spelling of\n> +\t * the prefix. We have to use this canonical path to do a\n> +\t * lookup in the cache-entry array so that we repeat the\n> +\t * original search using the case-corrected spelling.\n> +\t */\n> +\tstrbuf_addch(&canonical_path, '/');\n> +\tpos = index_name_pos(istate, canonical_path.buf,\n> +\t\t\t canonical_path.len);\n> +\tnr_in_cone = handle_path_with_trailing_slash(\n> +\t\tistate, canonical_path.buf, pos);\n> +\tstrbuf_release(&canonical_path);\n> +\treturn nr_in_cone;\n> +}\n> +\n> /*\n> * The daemon sent an observed pathname without a trailing slash.\n> * (This is the normal case.) We do not know if it is a tracked or\n> @@ -319,6 +416,19 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \telse\n> \t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n>\n> +\t/*\n> +\t * If we did not find an exact match for this pathname or any\n> +\t * cache-entries with this directory prefix and we're on a\n> +\t * case-insensitive file system, try again using the name-hash\n> +\t * and dir-name-hash.\n> +\t */\n> +\tif (!nr_in_cone && ignore_case) {\n> +\t\tnr_in_cone = handle_using_name_hash_icase(istate, name);\n> +\t\tif (!nr_in_cone)\n> +\t\t\tnr_in_cone = handle_using_dir_name_hash_icase(\n> +\t\t\t\tistate, name);\n> +\t}\n> +\n> \tif (nr_in_cone)\n> \t\ttrace_printf_key(&trace_fsmonitor,\n> \t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\n> --\n> gitgitgadget\n>\n>\n"},{"id":"489317","messageId":"xmqq4jdwjw2h.fsf@gitster.g","threadId":"60908","inReplyTo":"20240225123050.GA1952@tb-raspi4","subject":"Re: [PATCH v2 06/16] fsmonitor: refactor refresh callback for non-directory events","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-25T17:24:54Z","receivedAt":"2024-02-25T17:24:56Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Torsten Bögershausen writes:\n\n>> +\t\tpos = -pos - 1;\n>> +\n>> +\t\tfor (i = pos; i < istate->cache_nr; i++) {\n>> +\t\t\tif (!starts_with(istate->cache[i]->name, name))\n>> +\t\t\t\tbreak;\n>> +\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n>> +\t\t\t\tbreak;\n>\n> Hm, this covers all digits, letters, :;<=>?\n> but not e.g. !+-. (and others). What do i miss ?\n\nThis is scanning an in-core array of cache entries, which is sorted\nby name in lexicographic order, and the loop knows that files under\nthe directory \"foo\", whose pathnames all share prefix \"foo/\", would\nsort between \"foo.h\" and \"foo00\", because \".\" sorts before \"/\" and\n\"0\" sorts after \"/\".\n\nIt is trying to find where in the array a hypothetical directory\nwould appear, if any of the files in it existed in the array, and\nexiting early, taking advantage of the fact that after seeing\nsomething that sorts after a '/', it will never see an entry that\nshares cache[i]->name[] as a prefix.\n\nIt is not a new code in the patch, of course; merely got moved from\nelsewhere below.\n\n>> -\t\t/*\n>> -\t\t * The path is not a tracked file -or- it is a\n>> -\t\t * directory event on a platform that cannot\n>> -\t\t * distinguish between file and directory events in\n>> -\t\t * the event handler, such as Windows.\n>> -\t\t *\n>> -\t\t * Scan as if it is a directory and invalidate the\n>> -\t\t * cone under it. (But remember to ignore items\n>> -\t\t * between \"name\" and \"name/\", such as \"name-\" and\n>> -\t\t * \"name.\".\n>> -\t\t */\n>> -\t\tpos = -pos - 1;\n>> -\n>> -\t\tfor (i = pos; i < istate->cache_nr; i++) {\n>> -\t\t\tif (!starts_with(istate->cache[i]->name, name))\n>> -\t\t\t\tbreak;\n>> -\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n>> -\t\t\t\tbreak;\n>> -\t\t\tif (istate->cache[i]->name[len] == '/')\n>> -\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n>> -\t\t}\n>> +\t\thandle_path_without_trailing_slash(istate, name, pos);\n>> \t}\n>>\n>> \t/*\n>> --\n>> gitgitgadget\n>>\n>>\n"},{"id":"489388","messageId":"e622183b-23ce-3aad-0918-9690d9ba4d56@jeffhostetler.com","threadId":"60908","inReplyTo":"xmqqh6hz1tmx.fsf@gitster.g","subject":"Re: [PATCH v2 03/16] t7527: temporarily disable case-insensitive tests","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-02-26T17:12:31Z","receivedAt":"2024-02-26T17:12:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/23/24 3:17 AM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> From: Jeff Hostetler \n>>\n>> Add non-existent \"SKIPME\" prereq to the case-insensitive tests.\n>>\n>> The previous commit added test cases to demonstrate an error where\n>> FSMonitor can get confused on a case-insensitive file system when the\n>> on-disk spelling of a file or directory is wrong. Let's disable those\n>> tests before we incrementally teach Git to properly recognize and\n>> handle those types of problems (so that a bisect between here and the\n>> final commit in this patch series won't throw a false alarm).\n> \n> You talk about bisection, but hasn't the previous step already\n> broken bisection without these SKIPME prerequisites? IOW, shouldn't\n> this step squashed into the previous?\n> \n> Also, it is much more common to replace \"test_expect_success\" with\n> \"test_expect_failure\" to indicate that the steps are broken. Was\n> there a reason why we choose to do it differently?\n\nIn step 2 I created test with individual step failures baked into\nthe \"! grep -q\" steps in the bottom of each test. I didn't want a\nfailure in the 50-60 lines of setup code to cause a false alarm.\nSo the step 2 test \"succeeds\" by detecting that the output is\nincomplete/wrong.\n\nI wanted to use a \"test_must_fail\" on those individual grep lines\nrather than a negated grep, but something complained about that\nfunction only worked on \"git\" commands.\n\nI added the SKIPME here in step 3 so that I could fix the series\nin small steps and without worrying about which of the small steps\ncaused the file or directory case to stop being broken (which might\ncause confusion if someone were bisecting in this part of the history.\n\nLet me try again with the normal \"test_expect_failure\" in step 2,\ndrop step 3, and smash step 16 into step 14. With the rearranging\nthat I did in V2, both directories and files should be fixed in the\nsame final step -- rather than in separate steps.\n\nThanks\nJeff\n\n\n> \n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> t/t7527-builtin-fsmonitor.sh | 4 ++--\n>> 1 file changed, 2 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n>> index 3d21295f789..4acb547819c 100755\n>> --- a/t/t7527-builtin-fsmonitor.sh\n>> +++ b/t/t7527-builtin-fsmonitor.sh\n>> @@ -1051,7 +1051,7 @@ test_expect_success 'split-index and FSMonitor work well together' '\n>> #\n>> # The setup is a little contrived.\n>> #\n>> -test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n>> +test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n>> \ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n>> \n>> \tgit init subdir_case_wrong &&\n>> @@ -1128,7 +1128,7 @@ test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n>> \t! grep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n>> '\n>> \n>> -test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n>> +test_expect_success SKIPME,CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n>> \ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n>> \n>> \tgit init file_case_wrong &&\n"},{"id":"489407","messageId":"57e3632f-36db-a9d8-b14c-1a68fea1b68d@jeffhostetler.com","threadId":"60908","inReplyTo":"xmqqy1bbuloo.fsf@gitster.g","subject":"Re: [PATCH v2 09/16] fsmonitor: move untracked invalidation into helper functions","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-02-26T18:45:13Z","receivedAt":"2024-02-26T18:45:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/23/24 12:36 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> From: Jeff Hostetler \n>>\n>> Move the call to invalidate the untracked cache for the FSEvent\n>> pathname into the two helper functions.\n>>\n>> In a later commit in this series, we will call these helpers\n>> from other contexts and it safer to include the UC invalidation\n>> in the helper than to remember to also add it to each helper\n>> call-site.\n>>\n>> Signed-off-by: Jeff Hostetler \n>> ---\n>> fsmonitor.c | 26 +++++++++++++++++++-------\n>> 1 file changed, 19 insertions(+), 7 deletions(-)\n> \n> Thanks. The steps in this iteration makes this move much less\n> confusing to me than in the previous one. We used to call one of\n> \"handle path with/without trailing slash\" functions and then called\n> the invalidation. Now the invalidation happens in these \"handle path\"\n> functions.\n> \n> The unexplained change in behaviour is that we used to do the rest\n> of \"handle path\" and invalidation was done at the end. Now we do it\n> upfront. I think the \"rest\" works solely based on what is in the\n> main in-core index array (i.e. the_index.cache[] aka active_cache[])\n> and affects only what is in the in-core index array, while\n> untracked_cache_invalidate*() works solely based on what is in the\n> untracked cache extension (i.e. the_index.untracked) and affects\n> only what is in there, so the order of these two does not matter.\n> \n> Am I correct?\n> \n> Or does it affect correctness or performance or whatever in any way?\n> IOW, is there a reason why it is better to do the invalidation first\n> and then doing the \"rest\" after (hence this patch flips the order of\n> two to _improve_ something)?\n> \n> Thanks.\n\nThe ce_flags invalidation and the untracked-cache invalidation are\nindependent (as far as I could tell) and it doesn't matter which\norder we do them. Moving the UC to the start of the function was\nan attempt to avoid the usual \"goto the bottom\" or the need to guard\nagainst early \"return\" statements that were present in some of the\noriginal code (or my various refactorings). Moving it to the top\njust let me get it out of the way and not have to contrive things.\n\nI'll update the commit message.\n\nThanks\nJeff\n\n"},{"id":"489416","messageId":"b6cfe94a-adcf-04fa-2ed8-dfd4f0fdc77a@jeffhostetler.com","threadId":"60908","inReplyTo":"xmqqjzmvt5d3.fsf@gitster.g","subject":"Re: [PATCH v2 14/16] fsmonitor: support case-insensitive events","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-02-26T20:41:08Z","receivedAt":"2024-02-26T20:41:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/23/24 1:14 PM, Junio C Hamano wrote:\n> \"Jeff Hostetler via GitGitGadget\" writes:\n> \n>> +/*\n>> + * Use the name-hash to do a case-insensitive cache-entry lookup with\n>> + * the pathname and invalidate the cache-entry.\n>> + *\n>> + * Returns the number of cache-entries that we invalidated.\n>> + */\n>> +static size_t handle_using_name_hash_icase(\n>> +\tstruct index_state *istate, const char *name)\n>> +{\n>> +\tstruct cache_entry *ce = NULL;\n>> +\n>> +\tce = index_file_exists(istate, name, strlen(name), 1);\n>> +\tif (!ce)\n>> +\t\treturn 0;\n>> +\n>> +\t/*\n>> +\t * A case-insensitive search in the name-hash using the\n>> +\t * observed pathname found a cache-entry, so the observed path\n>> +\t * is case-incorrect. Invalidate the cache-entry and use the\n>> +\t * correct spelling from the cache-entry to invalidate the\n>> +\t * untracked-cache. Since we now have sparse-directories in\n>> +\t * the index, the observed pathname may represent a regular\n>> +\t * file or a sparse-index directory.\n>> +\t *\n>> +\t * Note that we should not have seen FSEvents for a\n>> +\t * sparse-index directory, but we handle it just in case.\n>> +\t *\n>> +\t * Either way, we know that there are not any cache-entries for\n>> +\t * children inside the cone of the directory, so we don't need to\n>> +\t * do the usual scan.\n>> +\t */\n>> +\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n>> +\t\t\t name, ce->name);\n>> +\n>> +\tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n>> +\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n>> +\treturn 1;\n>> +}\n> \n> You first ask the name-hash to turn the incoming \"name\" into the\n> case variant that we know about, i.e. ce->name, and use that to\n> access the untracked cache. Clever and makes sense. But if we have\n> ce->name, doesn't it mean the name is tracked? Do we find anything\n> useful to do in the untracked cache invalidation codepath in that\n> case?\n> \n> An FSmonitor event with case-incorrect pathname for a directory may\n> not be this trivial, I presume, and I expect that is what the\n> remainder of this patch is about.\n\nWe're going to use \"handle_using_name_hash_icase()\" to lookup both\nqualified (with trailing slash provided by the daemon) paths and\nunqualified paths (either a file or a directory on a platform that\ncan't tell), so there are 3 cases to worry about.\n\nIf we fail to find a cache-entry in the name-hash, we know nothing\nabout the path and we still have the three cases to worry about and\nwe let the caller deal with that.\n\nIf we DO find a matching cache-entry, then it is either a tracked\nfile or one of the new sparse-directories cache-entries. We now know\nthe correct case-spelling. I don't think it is possible for the UC\nto have an entry for this spelling, so you're right, we may not need\nto explicitly invalidate the UC here. I'll add a comment to the code\nabout this.\n\n\n> \n>> +\n>> +/*\n>> + * Use the dir-name-hash to find the correct-case spelling of the\n>> + * directory. Use the canonical spelling to invalidate all of the\n>> + * cache-entries within the matching cone.\n>> + *\n>> + * Returns the number of cache-entries that we invalidated.\n>> + */\n>> +static size_t handle_using_dir_name_hash_icase(\n>> +\tstruct index_state *istate, const char *name)\n> \n> It is a bit unfortunate that here on the name-hash side we contrast\n> the two helper function variants as \"dir-name\" vs \"name\", while the\n> original handle_path side use \"without_slash\" vs \"with_slash\".\n> \n> If I understand correctly, it is not like there are two distinct\n> hashes, \"name-hash\" vs \"dir-name-hash\". Both of these helpers use\n> the same \"name-hash\" mechanism, and this function differs from the\n> previous one in that it is about a directory, which is why it has\n> \"dir\" in its name. I wonder if we renamed the other one with\n> \"nondir\" in its name, and the other without_slash and with_slash\n> pair to match, e.g., handle_nondir_path() vs handle_dir_path(), or\n> something like that, the resulting names for these four functions\n> become easier to contrast and understand?\n\nname-hash.[ch] has 2 distinct hash-maps inside it. The \"name-hash\"\nthat we typically think about. And a well-hidden \"dir-name-hash\"\nin the same source file.\n\nThe \"name-hash\" maps each cache-entry's pathname to its ce* in\nthe cache-entry[] (case-insensitively).\n\nThe \"dir-name-hash\" maps each unique directory prefix over all\nof the cache-entries to the case-correct prefix. That is, if the\nindex contains \"dir1/Dir2/DIR3/file1\" and \"dir1/dir4/file2\", the\ndir hash will have 4 entries\n { \"dir1\", \"dir1/Dir2\", \"dir1/Dir2/DIR3\", \"dir1/dir4\" }.\nThis lets us do lookups without having to do a linear search on\nthe entire cache-entry[] every time.\n\nThese 2 hashes are demand-loaded only when needed (and usually only\nwhen ignore_case is set IIRC).\n\nWhen \"handle_using_dir_name_hash_icase()\" is called we still don't\nknow if the pathname is actually a file or directory, all we know\nis that we did not find a case-sensitive exact match nor a\ncase-insensitive match against the cache-entry[] using the name-hash.\nThe pathname could be a (unqualified) directory or just a plain\nuntracked file. So here, if we find it in the dir-name-hash, we now\nknow that it is a directory and that there was a case-error and we now\nknow the directory's correct case-spelling.\n\nSo we use that discovered case-correct spelling to invalidate the\nuntracked-cache.\n\n> \n>> +{\n>> +\tstruct strbuf canonical_path = STRBUF_INIT;\n>> +\tint pos;\n>> +\tsize_t len = strlen(name);\n>> +\tsize_t nr_in_cone;\n>> +\n>> +\tif (name[len - 1] == '/')\n>> +\t\tlen--;\n>> +\n>> +\tif (!index_dir_find(istate, name, len, &canonical_path))\n>> +\t\treturn 0; /* name is untracked */\n>> +\n>> +\tif (!memcmp(name, canonical_path.buf, canonical_path.len)) {\n>> +\t\tstrbuf_release(&canonical_path);\n>> +\t\t/*\n>> +\t\t * NEEDSWORK: Our caller already tried an exact match\n>> +\t\t * and failed to find one. They called us to do an\n>> +\t\t * ICASE match, so we should never get an exact match,\n>> +\t\t * so we could promote this to a BUG() here if we\n>> +\t\t * wanted to. It doesn't hurt anything to just return\n>> +\t\t * 0 and go on becaus we should never get here. Or we\n>> +\t\t * could just get rid of the memcmp() and this \"if\"\n>> +\t\t * clause completely.\n>> +\t\t */\n>> +\t\treturn 0; /* should not happen */\n>> +\t}\n> \n> \"becaus\" -> \"because\".\n> \n> If we should never get here, having BUG(\"we should never get here\")\n> would not hurt anything, either. On the other hand, silently\n> returning 0 will hide the bug under the carpet, and I am not sure it\n> is fair to call it \"doesn't hurt anything\".\n\nI'll make it a BUG().\n\n> \n>> +\n>> +\ttrace_printf_key(&trace_fsmonitor,\n>> +\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n>> +\t\t\t name, canonical_path.buf);\n>> +\n>> +\t/*\n>> +\t * The dir-name-hash only tells us the corrected spelling of\n>> +\t * the prefix. We have to use this canonical path to do a\n>> +\t * lookup in the cache-entry array so that we repeat the\n>> +\t * original search using the case-corrected spelling.\n>> +\t */\n>> +\tstrbuf_addch(&canonical_path, '/');\n>> +\tpos = index_name_pos(istate, canonical_path.buf,\n>> +\t\t\t canonical_path.len);\n>> +\tnr_in_cone = handle_path_with_trailing_slash(\n>> +\t\tistate, canonical_path.buf, pos);\n>> +\tstrbuf_release(&canonical_path);\n>> +\treturn nr_in_cone;\n>> +}\n> \n> Nice. Do we need to give this corrected name to help untracked\n> cache invalidation from the caller that called us?\n\nIn an earlier commit, I moved the call to invalidate the untracked-cache\ninto the two handle_path_with[out]_trailing_slash() functions so that\nwe wouldn't have to worry about it here.\n\n> \n>> @@ -319,6 +416,19 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n>> \telse\n>> \t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n>> \n>> +\t/*\n>> +\t * If we did not find an exact match for this pathname or any\n>> +\t * cache-entries with this directory prefix and we're on a\n>> +\t * case-insensitive file system, try again using the name-hash\n>> +\t * and dir-name-hash.\n>> +\t */\n>> +\tif (!nr_in_cone && ignore_case) {\n>> +\t\tnr_in_cone = handle_using_name_hash_icase(istate, name);\n>> +\t\tif (!nr_in_cone)\n>> +\t\t\tnr_in_cone = handle_using_dir_name_hash_icase(\n>> +\t\t\t\tistate, name);\n>> +\t}\n> \n> It might be interesting to learn how often we go through these\n> \"fallback\" code paths by tracing. Maybe it will become too noisy?\n> I dunno.\n\nI'm afraid it will be very noisy. On Windows and Mac we'll probably end\nup falling back for anything that is untracked, unfortunately. That is,\nif there is no cache-entry for \"foo.obj\", then we'll look for a\ncase-error (maybe there is a tracked \"FOO.OBJ\" file in the index or a\n\"Foo.Obj\" directory), before we can say it is untracked.\n\nI'm not happy about this (and no, I haven't had time to measure the\nperf hit we'll take), but right now I'm just worried about the\ncorrectness -- I've had several reports of stale/incomplete status\nwhen IDE tools change file/directory case in unexpected ways....\n\nThanks\nJeff\n"},{"id":"489417","messageId":"aaf53a10-2965-0284-120d-c59685c5090c@jeffhostetler.com","threadId":"60908","inReplyTo":"20240225131023.GC1952@tb-raspi4","subject":"Re: [PATCH v2 14/16] fsmonitor: support case-insensitive events","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-02-26T20:47:41Z","receivedAt":"2024-02-26T20:47:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 2/25/24 8:10 AM, Torsten Bögershausen wrote:\n>> +\tif (!memcmp(name, canonical_path.buf, canonical_path.len)) {\n>> +\t\tstrbuf_release(&canonical_path);\n>> +\t\t/*\n>> +\t\t * NEEDSWORK: Our caller already tried an exact match\n>> +\t\t * and failed to find one. They called us to do an\n>> +\t\t * ICASE match, so we should never get an exact match,\n>> +\t\t * so we could promote this to a BUG() here if we\n>> +\t\t * wanted to. It doesn't hurt anything to just return\n>> +\t\t * 0 and go on becaus we should never get here. Or we\n>> +\t\t * could just get rid of the memcmp() and this \"if\"\n>> +\t\t * clause completely.\n>> +\t\t */\n>> +\t\treturn 0; /* should not happen */\n> \n> In very very theory, there may be a race-condition,\n> when a directory is renamed very fast, more than once.\n> I don't think, that the \"it did not match exactly, but\n> now it matches\" is a problem.\n> Question: Does it make sense to just remove this ?\n> And, may be, find out that the \"corrected spelling (tm)\"\n> of \"DIR1\" is not \"dir1\", neither \"Dir1\", but, exactly, \"DIR1\" ?\n> Would that be a problem ?\n> \n\nI just meant that the dir-name-hash that we computed when\nwe loaded the index found an exact-case match here that\nwasn't found when called index_name_pos() and the negative\n\"pos\" didn't point to this exact-case prefix. This should\nnot happen.\n\nYeah I didn't think it should be a fatal condition, but\nsince it shouldn't happen, we can make it a BUG() and see.\n\nJeff\n"},{"id":"489418","messageId":"xmqqy1b6kjqc.fsf@gitster.g","threadId":"60908","inReplyTo":"b6cfe94a-adcf-04fa-2ed8-dfd4f0fdc77a@jeffhostetler.com","subject":"Re: [PATCH v2 14/16] fsmonitor: support case-insensitive events","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-26T21:18:19Z","receivedAt":"2024-02-26T21:18:22Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> I'm not happy about this (and no, I haven't had time to measure the\n> perf hit we'll take), but right now I'm just worried about the\n> correctness -- I've had several reports of stale/incomplete status\n> when IDE tools change file/directory case in unexpected ways....\n\nOf course, it is a good discipline, and I fully support the\ndirection, to focus on the correctness first.\n"},{"id":"489419","messageId":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v2.git.1708658300.gitgitgadget@gmail.com","subject":"[PATCH v3 00/14] FSMonitor edge cases on case-insensitive file systems","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:11Z","receivedAt":"2024-02-26T21:39:29Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"Here is version 3. I think I have addressed the remaining comments.\n\nI cleaned up the test code to use the test_expect_failure at the beginning\nand squashed in the test_expect_success version of tests into the final\ncommit in the series.\n\nI moved the invalidate_ce_fsm() commit earlier in the series, so that the\nfinal commit actually uses all of the up-to-this-point changes to fix the\nproblem.\n\nI converted a few \"should not happens\" to BUG()s.\n\nThanks to everyone for their time and attention reviewing this. Jeff\n\nJeff Hostetler (14):\n name-hash: add index_dir_find()\n t7527: add case-insensitve test for FSMonitor\n fsmonitor: refactor refresh callback on directory events\n fsmonitor: clarify handling of directory events in callback helper\n fsmonitor: refactor refresh callback for non-directory events\n dir: create untracked_cache_invalidate_trimmed_path()\n fsmonitor: refactor untracked-cache invalidation\n fsmonitor: move untracked-cache invalidation into helper functions\n fsmonitor: return invalidated cache-entry count on directory event\n fsmonitor: remove custom loop from non-directory path handler\n fsmonitor: return invalided cache-entry count on non-directory event\n fsmonitor: trace the new invalidated cache-entry count\n fsmonitor: refactor bit invalidation in refresh callback\n fsmonitor: support case-insensitive events\n\n dir.c | 20 +++\n dir.h | 7 +\n fsmonitor.c | 312 +++++++++++++++++++++++++++++------\n name-hash.c | 9 +-\n name-hash.h | 7 +-\n t/t7527-builtin-fsmonitor.sh | 223 +++++++++++++++++++++++++\n 6 files changed, 522 insertions(+), 56 deletions(-)\n\n\nbase-commit: f41f85c9ec8d4d46de0fd5fded88db94d3ec8c11\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-1662%2Fjeffhostetler%2Ffsmonitor-ignore-case-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-1662/jeffhostetler/fsmonitor-ignore-case-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/1662\n\nRange-diff vs v2:\n\n 1: 03b07d9c25e ! 1: 64ae07aaeaa name-hash: add index_dir_find()\n @@ Metadata\n ## Commit message ##\n name-hash: add index_dir_find()\n \n - Replace the index_dir_exists() function with index_dir_find() and\n - change the API to take an optional strbuf to return the canonical\n - spelling of the matched directory prefix.\n + index_dir_exists() returns a boolean to indicate if there is a\n + case-insensitive match in the directory name-hash, but does not\n + provide the caller with the exact spelling of that match.\n \n - Create an index_dir_exists() wrapper macro for existing callers.\n + Create index_dir_find() to do the case-insensitive search *and*\n + optionally return the spelling of the matched directory prefix in a\n + provided strbuf.\n \n - The existing index_dir_exists() returns a boolean to indicate if\n - there is a case-insensitive match in the directory name-hash, but\n - it doesn't tell the caller the exact spelling of that match.\n -\n - The new version also copies the matched spelling to a provided strbuf.\n - This lets the caller, for example, then call index_name_pos() with the\n - correct case to search the cache-entry array for the real insertion\n - position.\n + To avoid code duplication, convert index_dir_exists() to be a trivial\n + wrapper around the new index_dir_find().\n \n Signed-off-by: Jeff Hostetler \n \n 2: 7778cee1c10 ! 2: beeebf55963 t7527: add case-insensitve test for FSMonitor\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'split-index and FSMonitor wor\n +#\n +# The setup is a little contrived.\n +#\n -+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n ++test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n +\ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n +\n +\tgit init subdir_case_wrong &&\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'split-index and FSMonitor wor\n +\tgrep -q \" M AAA\" \"$PWD/subdir_case_wrong.out\" &&\n +\tgrep -q \" M zzz\" \"$PWD/subdir_case_wrong.out\" &&\n +\n -+\t# However, with the fsmonitor client bug, the \"(pos -3)\" causes\n -+\t# the client to not update the bit and never rescan the file\n -+\t# and therefore not report it as dirty.\n -+\t! grep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n ++\t# Expect Breakage: with the case confusion, the \"(pos -3)\" causes\n ++\t# the client to not clear the CE_FSMONITOR_VALID bit and therefore\n ++\t# status will not rescan the file and therefore not report it as dirty.\n ++\tgrep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n +'\n +\n -+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n ++test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n +\ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n +\n +\tgit init file_case_wrong &&\n @@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'split-index and FSMonitor wor\n +\tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n +\tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n +\n -+\t# Status should say these files are modified, but with the case\n -+\t# bug, the \"pos -3\" cause the client to not update the FSM bit\n -+\t# and never cause the file to be rescanned and therefore to not\n -+\t# report it dirty.\n -+\t! grep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n -+\t! grep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n ++\t# Expect Breakage: with the case confusion, the \"(pos-3)\" and\n ++\t# \"(pos -9)\" causes the client to not clear the CE_FSMONITOR_VALID\n ++\t# bit and therefore status will not rescan the files and therefore\n ++\t# not report them as dirty.\n ++\tgrep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n ++\tgrep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n +'\n +\n test_done\n 3: dad079ade7f < -: ----------- t7527: temporarily disable case-insensitive tests\n 4: 5516670e30e = 3: 518cb4dd5df fsmonitor: refactor refresh callback on directory events\n 5: c04fd4eae94 = 4: 9a4b5bf990b fsmonitor: clarify handling of directory events in callback helper\n 6: 7ee6ca1aefd ! 5: 348b9b0c94e fsmonitor: refactor refresh callback for non-directory events\n @@ Metadata\n ## Commit message ##\n fsmonitor: refactor refresh callback for non-directory events\n \n - Move the code handle unqualified FSEvents (without a trailing slash)\n - into a helper function.\n + Move the code that handles unqualified FSEvents (without a trailing\n + slash) into a helper function.\n \n Signed-off-by: Jeff Hostetler \n \n 7: 99c0d3e0742 ! 6: ed735e3f1cb dir: create untracked_cache_invalidate_trimmed_path()\n @@ dir.c: void untracked_cache_invalidate_path(struct index_state *istate,\n +\tsize_t len = strlen(path);\n +\n +\tif (!len)\n -+\t\treturn; /* should not happen */\n ++\t\tBUG(\"untracked_cache_invalidate_trimmed_path given zero length path\");\n +\n +\tif (path[len - 1] != '/') {\n +\t\tuntracked_cache_invalidate_path(istate, path, safe_path);\n 8: f2d6765d84f = 7: 2a43c6cbe0d fsmonitor: refactor untracked-cache invalidation\n 9: af6f57ab3e6 ! 8: 6e87ea6deaf fsmonitor: move untracked invalidation into helper functions\n @@ Metadata\n Author: Jeff Hostetler \n \n ## Commit message ##\n - fsmonitor: move untracked invalidation into helper functions\n + fsmonitor: move untracked-cache invalidation into helper functions\n \n - Move the call to invalidate the untracked cache for the FSEvent\n + Move the call to invalidate the untracked-cache for the FSEvent\n pathname into the two helper functions.\n \n In a later commit in this series, we will call these helpers\n from other contexts and it safer to include the UC invalidation\n - in the helper than to remember to also add it to each helper\n + in the helpers than to remember to also add it to each helper\n call-site.\n \n + This has the side-effect of invalidating the UC *before* we\n + invalidate the ce_flags in the cache-entry. These activities\n + are independent and do not affect each other. Also, by doing\n + the UC work first, we can avoid worrying about \"early returns\"\n + or the need for the usual \"goto the end\" in each of the\n + handler functions.\n +\n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n 10: 623c6f06e21 = 9: 5fea8b9476e fsmonitor: return invalidated cache-entry count on directory event\n 11: 1853f77d333 = 10: 3fa7536cf80 fsmonitor: remove custom loop from non-directory path handler\n 12: f77d68c78ad ! 11: 53f73c1515d fsmonitor: return invalided cache-entry count on non-directory event\n @@ Metadata\n ## Commit message ##\n fsmonitor: return invalided cache-entry count on non-directory event\n \n - Teah the refresh callback helper function for unqualified FSEvents\n + Teach the refresh callback helper function for unqualified FSEvents\n (pathnames without a trailing slash) to return the number of\n cache-entries that were invalided in response to the event.\n \n 13: 58b36673e15 = 12: 0148319aea5 fsmonitor: trace the new invalidated cache-entry count\n 15: 3a20065dbf8 ! 13: 04867eccfcd fsmonitor: refactor bit invalidation in refresh callback\n @@ Commit message\n it to help debug edge cases.\n \n This is similar to the existing `mark_fsmonitor_invalid()` function,\n - but we don't need the extra stuff that it does.\n + but it also does untracked-cache invalidation and we've already\n + handled that in the refresh-callback handlers, so but we don't need\n + to repeat that.\n \n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n @@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n - static size_t handle_path_with_trailing_slash(\n - \tstruct index_state *istate, const char *name, int pos);\n + \treturn result;\n + }\n \n +/*\n + * Invalidate the FSM bit on this CE. This is like mark_fsmonitor_invalid()\n -+ * but we've already handled the untracked-cache and I want a different\n -+ * trace message.\n ++ * but we've already handled the untracked-cache, so let's not repeat that\n ++ * work. This also lets us have a different trace message so that we can\n ++ * see everything that was done as part of the refresh-callback.\n + */\n +static void invalidate_ce_fsm(struct cache_entry *ce)\n +{\n -+\tif (ce->ce_flags & CE_FSMONITOR_VALID)\n ++\tif (ce->ce_flags & CE_FSMONITOR_VALID) {\n +\t\ttrace_printf_key(&trace_fsmonitor,\n +\t\t\t\t \"fsmonitor_refresh_callback INV: '%s'\",\n +\t\t\t\t ce->name);\n -+\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n ++\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n ++\t}\n +}\n +\n - /*\n - * Use the name-hash to do a case-insensitive cache-entry lookup with\n - * the pathname and invalidate the cache-entry.\n -@@ fsmonitor.c: static size_t handle_using_name_hash_icase(\n - \n - \tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n - \n --\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n -+\tinvalidate_ce_fsm(ce);\n - \treturn 1;\n - }\n + static size_t handle_path_with_trailing_slash(\n + \tstruct index_state *istate, const char *name, int pos);\n \n @@ fsmonitor.c: static size_t handle_path_without_trailing_slash(\n \t\t * cache-entry with the same pathname, nor for a cone\n 14: 288f3f4e54e ! 14: ec036c04d1b fsmonitor: support case-insensitive events\n @@ Commit message\n Update event handling to optionally use the name-hash and dir-name-hash\n if necessary.\n \n + Also update t7527 to convert the \"test_expect_failure\" to \"_success\"\n + now that we have fixed the bug.\n +\n Signed-off-by: Jeff Hostetler \n \n ## fsmonitor.c ##\n @@ fsmonitor.c\n #include \"run-command.h\"\n #include \"strbuf.h\"\n #include \"trace2.h\"\n -@@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n +@@ fsmonitor.c: static void invalidate_ce_fsm(struct cache_entry *ce)\n static size_t handle_path_with_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos);\n \n @@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n +\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n +\t\t\t name, ce->name);\n +\n ++\t/*\n ++\t * NEEDSWORK: We used the name-hash to find the correct\n ++\t * case-spelling of the pathname in the cache-entry[], so\n ++\t * technically this is a tracked file or a sparse-directory.\n ++\t * It should not have any entries in the untracked-cache, so\n ++\t * we should not need to use the case-corrected spelling to\n ++\t * invalidate the the untracked-cache. So we may not need to\n ++\t * do this. For now, I'm going to be conservative and always\n ++\t * do it; we can revisit this later.\n ++\t */\n +\tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n +\n -+\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n ++\tinvalidate_ce_fsm(ce);\n +\treturn 1;\n +}\n +\n @@ fsmonitor.c: static int query_fsmonitor_hook(struct repository *r,\n +\t\t * ICASE match, so we should never get an exact match,\n +\t\t * so we could promote this to a BUG() here if we\n +\t\t * wanted to. It doesn't hurt anything to just return\n -+\t\t * 0 and go on becaus we should never get here. Or we\n ++\t\t * 0 and go on because we should never get here. Or we\n +\t\t * could just get rid of the memcmp() and this \"if\"\n +\t\t * clause completely.\n +\t\t */\n -+\t\treturn 0; /* should not happen */\n ++\t\tBUG(\"handle_using_dir_name_hash_icase(%s) did not exact match\",\n ++\t\t name);\n +\t}\n +\n +\ttrace_printf_key(&trace_fsmonitor,\n @@ fsmonitor.c: static void fsmonitor_refresh_callback(struct index_state *istate,\n \tif (nr_in_cone)\n \t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\n +\n + ## t/t7527-builtin-fsmonitor.sh ##\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_success 'split-index and FSMonitor work well together' '\n + #\n + # The setup is a little contrived.\n + #\n +-test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n ++test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n + \ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n + \n + \tgit init subdir_case_wrong &&\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n + \n + \tgrep -q \"dir1/DIR2/dir3/file3.*pos -3\" \"$PWD/subdir_case_wrong.log1\" &&\n + \n ++\t# Verify that we get a mapping event to correct the case.\n ++\tgrep -q \"MAP:.*dir1/DIR2/dir3/file3.*dir1/dir2/dir3/file3\" \\\n ++\t\t\"$PWD/subdir_case_wrong.log1\" &&\n ++\n + \t# The refresh-callbacks should have caused \"git status\" to clear\n + \t# the CE_FSMONITOR_VALID bit on each of those files and caused\n + \t# the worktree scan to visit them and mark them as modified.\n + \tgrep -q \" M AAA\" \"$PWD/subdir_case_wrong.out\" &&\n + \tgrep -q \" M zzz\" \"$PWD/subdir_case_wrong.out\" &&\n +-\n +-\t# Expect Breakage: with the case confusion, the \"(pos -3)\" causes\n +-\t# the client to not clear the CE_FSMONITOR_VALID bit and therefore\n +-\t# status will not rescan the file and therefore not report it as dirty.\n + \tgrep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n + '\n + \n +-test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n ++test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n + \ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n + \n + \tgit init file_case_wrong &&\n +@@ t/t7527-builtin-fsmonitor.sh: test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n + \tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try3.log\" \\\n + \t\tgit -C file_case_wrong --no-optional-locks status --short \\\n + \t\t\t>\"$PWD/file_case_wrong-try3.out\" &&\n ++\n ++\t# Verify that we get a mapping event to correct the case.\n ++\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir3/FILE-3-A.*dir1/dir2/dir3/file-3-a\" \\\n ++\t\t\"$PWD/file_case_wrong-try3.log\" &&\n ++\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir4/file-4-a.*dir1/dir2/dir4/FILE-4-A\" \\\n ++\t\t\"$PWD/file_case_wrong-try3.log\" &&\n ++\n + \t# FSEvents are in observed case.\n + \tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n + \tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n + \n +-\t# Expect Breakage: with the case confusion, the \"(pos-3)\" and\n +-\t# \"(pos -9)\" causes the client to not clear the CE_FSMONITOR_VALID\n +-\t# bit and therefore status will not rescan the files and therefore\n +-\t# not report them as dirty.\n ++\t# The refresh-callbacks should have caused \"git status\" to clear\n ++\t# the CE_FSMONITOR_VALID bit on each of those files and caused\n ++\t# the worktree scan to visit them and mark them as modified.\n + \tgrep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n + \tgrep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n + '\n 16: 467d3c1fe2c < -: ----------- t7527: update case-insenstive fsmonitor test\n\n-- \ngitgitgadget\n"},{"id":"489420","messageId":"64ae07aaeaa2f9e91d153ddd1aacefb62188456f.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 01/14] name-hash: add index_dir_find()","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:12Z","receivedAt":"2024-02-26T21:39:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nindex_dir_exists() returns a boolean to indicate if there is a\ncase-insensitive match in the directory name-hash, but does not\nprovide the caller with the exact spelling of that match.\n\nCreate index_dir_find() to do the case-insensitive search *and*\noptionally return the spelling of the matched directory prefix in a\nprovided strbuf.\n\nTo avoid code duplication, convert index_dir_exists() to be a trivial\nwrapper around the new index_dir_find().\n\nSigned-off-by: Jeff Hostetler \n---\n name-hash.c | 9 ++++++++-\n name-hash.h | 7 ++++++-\n 2 files changed, 14 insertions(+), 2 deletions(-)\n\ndiff --git a/name-hash.c b/name-hash.c\nindex 251f036eef6..3a58ce03d9c 100644\n--- a/name-hash.c\n+++ b/name-hash.c\n@@ -685,13 +685,20 @@ static int same_name(const struct cache_entry *ce, const char *name, int namelen\n \treturn slow_same_name(name, namelen, ce->name, len);\n }\n \n-int index_dir_exists(struct index_state *istate, const char *name, int namelen)\n+int index_dir_find(struct index_state *istate, const char *name, int namelen,\n+\t\t struct strbuf *canonical_path)\n {\n \tstruct dir_entry *dir;\n \n \tlazy_init_name_hash(istate);\n \texpand_to_path(istate, name, namelen, 0);\n \tdir = find_dir_entry(istate, name, namelen);\n+\n+\tif (canonical_path && dir && dir->nr) {\n+\t\tstrbuf_reset(canonical_path);\n+\t\tstrbuf_add(canonical_path, dir->name, dir->namelen);\n+\t}\n+\n \treturn dir && dir->nr;\n }\n \ndiff --git a/name-hash.h b/name-hash.h\nindex b1b4b0fb337..0cbfc428631 100644\n--- a/name-hash.h\n+++ b/name-hash.h\n@@ -4,7 +4,12 @@\n struct cache_entry;\n struct index_state;\n \n-int index_dir_exists(struct index_state *istate, const char *name, int namelen);\n+\n+int index_dir_find(struct index_state *istate, const char *name, int namelen,\n+\t\t struct strbuf *canonical_path);\n+\n+#define index_dir_exists(i, n, l) index_dir_find((i), (n), (l), NULL)\n+\n void adjust_dirname_case(struct index_state *istate, char *name);\n struct cache_entry *index_file_exists(struct index_state *istate, const char *name, int namelen, int igncase);\n \n-- \ngitgitgadget\n\n"},{"id":"489421","messageId":"beeebf5596385cf72f7aa566fd277da738e627fd.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 02/14] t7527: add case-insensitve test for FSMonitor","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:13Z","receivedAt":"2024-02-26T21:39:31Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nThe FSMonitor client code trusts the spelling of the pathnames in the\nFSEvents received from the FSMonitor daemon. On case-insensitive file\nsystems, these OBSERVED pathnames may be spelled differently than the\nEXPECTED pathnames listed in the .git/index. This causes a miss when\nusing `index_name_pos()` which expects the given case to be correct.\n\nWhen this happens, the FSMonitor client code does not update the state\nof the CE_FSMONITOR_VALID bit when refreshing the index (and before\nstarting to scan the worktree).\n\nThis results in modified files NOT being reported by `git status` when\nthere is a discrepancy in the case-spelling of a tracked file's\npathname.\n\nThis commit contains a (rather contrived) test case to demonstrate\nthis. A later commit in this series will update the FSMonitor client\ncode to recognize these discrepancies and update the CE_ bit accordingly.\n\nSigned-off-by: Jeff Hostetler \n---\n t/t7527-builtin-fsmonitor.sh | 217 +++++++++++++++++++++++++++++++++++\n 1 file changed, 217 insertions(+)\n\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 363f9dc0e41..830f2d9de33 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -1037,4 +1037,221 @@ test_expect_success 'split-index and FSMonitor work well together' '\n \t)\n '\n \n+# The FSMonitor daemon reports the OBSERVED pathname of modified files\n+# and thus contains the OBSERVED spelling on case-insensitive file\n+# systems. The daemon does not (and should not) load the .git/index\n+# file and therefore does not know the expected case-spelling. Since\n+# it is possible for the user to create files/subdirectories with the\n+# incorrect case, a modified file event for a tracked will not have\n+# the EXPECTED case. This can cause `index_name_pos()` to incorrectly\n+# report that the file is untracked. This causes the client to fail to\n+# mark the file as possibly dirty (keeping the CE_FSMONITOR_VALID bit\n+# set) so that `git status` will avoid inspecting it and thus not\n+# present in the status output.\n+#\n+# The setup is a little contrived.\n+#\n+test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n+\ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n+\n+\tgit init subdir_case_wrong &&\n+\t(\n+\t\tcd subdir_case_wrong &&\n+\t\techo x >AAA &&\n+\t\techo x >BBB &&\n+\n+\t\tmkdir dir1 &&\n+\t\techo x >dir1/file1 &&\n+\t\tmkdir dir1/dir2 &&\n+\t\techo x >dir1/dir2/file2 &&\n+\t\tmkdir dir1/dir2/dir3 &&\n+\t\techo x >dir1/dir2/dir3/file3 &&\n+\n+\t\techo x >yyy &&\n+\t\techo x >zzz &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"data\" &&\n+\n+\t\t# This will cause \"dir1/\" and everything under it\n+\t\t# to be deleted.\n+\t\tgit sparse-checkout set --cone --sparse-index &&\n+\n+\t\t# Create dir2 with the wrong case and then let Git\n+\t\t# repopulate dir3 -- it will not correct the spelling\n+\t\t# of dir2.\n+\t\tmkdir dir1 &&\n+\t\tmkdir dir1/DIR2 &&\n+\t\tgit sparse-checkout add dir1/dir2/dir3\n+\t) &&\n+\n+\tstart_daemon -C subdir_case_wrong --tf \"$PWD/subdir_case_wrong.trace\" &&\n+\n+\t# Enable FSMonitor in the client. Run enough commands for\n+\t# the .git/index to sync up with the daemon with everything\n+\t# marked clean.\n+\tgit -C subdir_case_wrong config core.fsmonitor true &&\n+\tgit -C subdir_case_wrong update-index --fsmonitor &&\n+\tgit -C subdir_case_wrong status &&\n+\n+\t# Make some files dirty so that FSMonitor gets FSEvents for\n+\t# each of them.\n+\techo xx >>subdir_case_wrong/AAA &&\n+\techo xx >>subdir_case_wrong/dir1/DIR2/dir3/file3 &&\n+\techo xx >>subdir_case_wrong/zzz &&\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/subdir_case_wrong.log\" \\\n+\t\tgit -C subdir_case_wrong --no-optional-locks status --short \\\n+\t\t\t>\"$PWD/subdir_case_wrong.out\" &&\n+\n+\t# \"git status\" should have gotten file events for each of\n+\t# the 3 files.\n+\t#\n+\t# \"dir2\" should be in the observed case on disk.\n+\tgrep \"fsmonitor_refresh_callback\" \\\n+\t\t<\"$PWD/subdir_case_wrong.log\" \\\n+\t\t>\"$PWD/subdir_case_wrong.log1\" &&\n+\n+\tgrep -q \"AAA.*pos 0\" \"$PWD/subdir_case_wrong.log1\" &&\n+\tgrep -q \"zzz.*pos 6\" \"$PWD/subdir_case_wrong.log1\" &&\n+\n+\tgrep -q \"dir1/DIR2/dir3/file3.*pos -3\" \"$PWD/subdir_case_wrong.log1\" &&\n+\n+\t# The refresh-callbacks should have caused \"git status\" to clear\n+\t# the CE_FSMONITOR_VALID bit on each of those files and caused\n+\t# the worktree scan to visit them and mark them as modified.\n+\tgrep -q \" M AAA\" \"$PWD/subdir_case_wrong.out\" &&\n+\tgrep -q \" M zzz\" \"$PWD/subdir_case_wrong.out\" &&\n+\n+\t# Expect Breakage: with the case confusion, the \"(pos -3)\" causes\n+\t# the client to not clear the CE_FSMONITOR_VALID bit and therefore\n+\t# status will not rescan the file and therefore not report it as dirty.\n+\tgrep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n+'\n+\n+test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n+\ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n+\n+\tgit init file_case_wrong &&\n+\t(\n+\t\tcd file_case_wrong &&\n+\t\techo x >AAA &&\n+\t\techo x >BBB &&\n+\n+\t\tmkdir dir1 &&\n+\t\tmkdir dir1/dir2 &&\n+\t\tmkdir dir1/dir2/dir3 &&\n+\t\techo x >dir1/dir2/dir3/FILE-3-B &&\n+\t\techo x >dir1/dir2/dir3/XXXX-3-X &&\n+\t\techo x >dir1/dir2/dir3/file-3-a &&\n+\t\techo x >dir1/dir2/dir3/yyyy-3-y &&\n+\t\tmkdir dir1/dir2/dir4 &&\n+\t\techo x >dir1/dir2/dir4/FILE-4-A &&\n+\t\techo x >dir1/dir2/dir4/XXXX-4-X &&\n+\t\techo x >dir1/dir2/dir4/file-4-b &&\n+\t\techo x >dir1/dir2/dir4/yyyy-4-y &&\n+\n+\t\techo x >yyy &&\n+\t\techo x >zzz &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"data\"\n+\t) &&\n+\n+\tstart_daemon -C file_case_wrong --tf \"$PWD/file_case_wrong.trace\" &&\n+\n+\t# Enable FSMonitor in the client. Run enough commands for\n+\t# the .git/index to sync up with the daemon with everything\n+\t# marked clean.\n+\tgit -C file_case_wrong config core.fsmonitor true &&\n+\tgit -C file_case_wrong update-index --fsmonitor &&\n+\tgit -C file_case_wrong status &&\n+\n+\t# Make some files dirty so that FSMonitor gets FSEvents for\n+\t# each of them.\n+\techo xx >>file_case_wrong/AAA &&\n+\techo xx >>file_case_wrong/zzz &&\n+\n+\t# Rename some files so that FSMonitor sees a create and delete\n+\t# FSEvent for each. (A simple \"mv foo FOO\" is not portable\n+\t# between macOS and Windows. It works on both platforms, but makes\n+\t# the test messy, since (1) one platform updates \"ctime\" on the\n+\t# moved file and one does not and (2) it causes a directory event\n+\t# on one platform and not on the other which causes additional\n+\t# scanning during \"git status\" which causes a \"H\" vs \"h\" discrepancy\n+\t# in \"git ls-files -f\".) So old-school it and move it out of the\n+\t# way and copy it to the case-incorrect name so that we get fresh\n+\t# \"ctime\" and \"mtime\" values.\n+\n+\tmv file_case_wrong/dir1/dir2/dir3/file-3-a file_case_wrong/dir1/dir2/dir3/ORIG &&\n+\tcp file_case_wrong/dir1/dir2/dir3/ORIG file_case_wrong/dir1/dir2/dir3/FILE-3-A &&\n+\trm file_case_wrong/dir1/dir2/dir3/ORIG &&\n+\tmv file_case_wrong/dir1/dir2/dir4/FILE-4-A file_case_wrong/dir1/dir2/dir4/ORIG &&\n+\tcp file_case_wrong/dir1/dir2/dir4/ORIG file_case_wrong/dir1/dir2/dir4/file-4-a &&\n+\trm file_case_wrong/dir1/dir2/dir4/ORIG &&\n+\n+\t# Run status enough times to fully sync.\n+\t#\n+\t# The first instance should get the create and delete FSEvents\n+\t# for each pair. Status should update the index with a new FSM\n+\t# token (so the next invocation will not see data for these\n+\t# events).\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try1.log\" \\\n+\t\tgit -C file_case_wrong status --short \\\n+\t\t\t>\"$PWD/file_case_wrong-try1.out\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try1.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*file-3-a.*pos 4\" \"$PWD/file_case_wrong-try1.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*FILE-4-A.*pos 6\" \"$PWD/file_case_wrong-try1.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try1.log\" &&\n+\n+\t# FSM refresh will have invalidated the FSM bit and cause a regular\n+\t# (real) scan of these tracked files, so they should have \"H\" status.\n+\t# (We will not see a \"h\" status until the next refresh (on the next\n+\t# command).)\n+\n+\tgit -C file_case_wrong ls-files -f >\"$PWD/file_case_wrong-lsf1.out\" &&\n+\tgrep -q \"H dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-lsf1.out\" &&\n+\tgrep -q \"H dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-lsf1.out\" &&\n+\n+\n+\t# Try the status again. We assume that the above status command\n+\t# advanced the token so that the next one will not see those events.\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try2.log\" \\\n+\t\tgit -C file_case_wrong status --short \\\n+\t\t\t>\"$PWD/file_case_wrong-try2.out\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*file-3-a.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*FILE-4-A.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\t! grep -q \"fsmonitor_refresh_callback.*file-4-a.*pos\" \"$PWD/file_case_wrong-try2.log\" &&\n+\n+\t# FSM refresh saw nothing, so it will mark all files as valid,\n+\t# so they should now have \"h\" status.\n+\n+\tgit -C file_case_wrong ls-files -f >\"$PWD/file_case_wrong-lsf2.out\" &&\n+\tgrep -q \"h dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-lsf2.out\" &&\n+\tgrep -q \"h dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-lsf2.out\" &&\n+\n+\n+\t# We now have files with clean content, but with case-incorrect\n+\t# file names. Modify them to see if status properly reports\n+\t# them.\n+\n+\techo xx >>file_case_wrong/dir1/dir2/dir3/FILE-3-A &&\n+\techo xx >>file_case_wrong/dir1/dir2/dir4/file-4-a &&\n+\n+\tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try3.log\" \\\n+\t\tgit -C file_case_wrong --no-optional-locks status --short \\\n+\t\t\t>\"$PWD/file_case_wrong-try3.out\" &&\n+\t# FSEvents are in observed case.\n+\tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n+\n+\t# Expect Breakage: with the case confusion, the \"(pos-3)\" and\n+\t# \"(pos -9)\" causes the client to not clear the CE_FSMONITOR_VALID\n+\t# bit and therefore status will not rescan the files and therefore\n+\t# not report them as dirty.\n+\tgrep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n+\tgrep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"489422","messageId":"518cb4dd5df0c71ef43078129fbf78a8b232a749.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 03/14] fsmonitor: refactor refresh callback on directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:14Z","receivedAt":"2024-02-26T21:39:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove the code to handle directory FSEvents (containing pathnames with\na trailing slash) into a helper function.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 52 ++++++++++++++++++++++++++++++----------------------\n 1 file changed, 30 insertions(+), 22 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex f670c509378..6fecae9aeb2 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,6 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+static void handle_path_with_trailing_slash(\n+\tstruct index_state *istate, const char *name, int pos)\n+{\n+\tint i;\n+\n+\t/*\n+\t * The daemon can decorate directory events, such as\n+\t * moves or renames, with a trailing slash if the OS\n+\t * FS Event contains sufficient information, such as\n+\t * MacOS.\n+\t *\n+\t * Use this to invalidate the entire cone under that\n+\t * directory.\n+\t *\n+\t * We do not expect an exact match because the index\n+\t * does not normally contain directory entries, so we\n+\t * start at the insertion point and scan.\n+\t */\n+\tif (pos < 0)\n+\t\tpos = -pos - 1;\n+\n+\t/* Mark all entries for the folder invalid */\n+\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\tbreak;\n+\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t}\n+}\n+\n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint i, len = strlen(name);\n@@ -193,28 +222,7 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t\t\t name, pos);\n \n \tif (name[len - 1] == '/') {\n-\t\t/*\n-\t\t * The daemon can decorate directory events, such as\n-\t\t * moves or renames, with a trailing slash if the OS\n-\t\t * FS Event contains sufficient information, such as\n-\t\t * MacOS.\n-\t\t *\n-\t\t * Use this to invalidate the entire cone under that\n-\t\t * directory.\n-\t\t *\n-\t\t * We do not expect an exact match because the index\n-\t\t * does not normally contain directory entries, so we\n-\t\t * start at the insertion point and scan.\n-\t\t */\n-\t\tif (pos < 0)\n-\t\t\tpos = -pos - 1;\n-\n-\t\t/* Mark all entries for the folder invalid */\n-\t\tfor (i = pos; i < istate->cache_nr; i++) {\n-\t\t\tif (!starts_with(istate->cache[i]->name, name))\n-\t\t\t\tbreak;\n-\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\t}\n+\t\thandle_path_with_trailing_slash(istate, name, pos);\n \n \t\t/*\n \t\t * We need to remove the traling \"/\" from the path\n-- \ngitgitgadget\n\n"},{"id":"489423","messageId":"9a4b5bf990bcddb2707207b7a523f0b9945d84a0.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 04/14] fsmonitor: clarify handling of directory events in callback helper","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:15Z","receivedAt":"2024-02-26T21:39:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nImprove documentation of the refresh callback helper function\nused for directory FSEvents.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 37 ++++++++++++++++++++++++-------------\n 1 file changed, 24 insertions(+), 13 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 6fecae9aeb2..29cce32d81c 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,24 +183,35 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+/*\n+ * The daemon can decorate directory events, such as a move or rename,\n+ * by adding a trailing slash to the observed name. Use this to\n+ * explicitly invalidate the entire cone under that directory.\n+ *\n+ * The daemon can only reliably do that if the OS FSEvent contains\n+ * sufficient information in the event.\n+ *\n+ * macOS FSEvents have enough information.\n+ *\n+ * Other platforms may or may not be able to do it (and it might\n+ * depend on the type of event (for example, a daemon could lstat() an\n+ * observed pathname after a rename, but not after a delete)).\n+ *\n+ * If we find an exact match in the index for a path with a trailing\n+ * slash, it means that we matched a sparse-index directory in a\n+ * cone-mode sparse-checkout (since that's the only time we have\n+ * directories in the index). We should never see this in practice\n+ * (because sparse directories should not be present and therefore\n+ * not generating FS events). Either way, we can treat them in the\n+ * same way and just invalidate the cache-entry and the untracked\n+ * cache (and in this case, the forward cache-entry scan won't find\n+ * anything and it doesn't hurt to let it run).\n+ */\n static void handle_path_with_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos)\n {\n \tint i;\n \n-\t/*\n-\t * The daemon can decorate directory events, such as\n-\t * moves or renames, with a trailing slash if the OS\n-\t * FS Event contains sufficient information, such as\n-\t * MacOS.\n-\t *\n-\t * Use this to invalidate the entire cone under that\n-\t * directory.\n-\t *\n-\t * We do not expect an exact match because the index\n-\t * does not normally contain directory entries, so we\n-\t * start at the insertion point and scan.\n-\t */\n \tif (pos < 0)\n \t\tpos = -pos - 1;\n \n-- \ngitgitgadget\n\n"},{"id":"489424","messageId":"348b9b0c94e23d82f1df69b0b24b70d6cfb48a19.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 05/14] fsmonitor: refactor refresh callback for non-directory events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:16Z","receivedAt":"2024-02-26T21:39:33Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove the code that handles unqualified FSEvents (without a trailing\nslash) into a helper function.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 67 +++++++++++++++++++++++++++++++----------------------\n 1 file changed, 39 insertions(+), 28 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 29cce32d81c..364198d258f 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,6 +183,43 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+static void handle_path_without_trailing_slash(\n+\tstruct index_state *istate, const char *name, int pos)\n+{\n+\tint i;\n+\n+\tif (pos >= 0) {\n+\t\t/*\n+\t\t * We have an exact match for this path and can just\n+\t\t * invalidate it.\n+\t\t */\n+\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t} else {\n+\t\t/*\n+\t\t * The path is not a tracked file -or- it is a\n+\t\t * directory event on a platform that cannot\n+\t\t * distinguish between file and directory events in\n+\t\t * the event handler, such as Windows.\n+\t\t *\n+\t\t * Scan as if it is a directory and invalidate the\n+\t\t * cone under it. (But remember to ignore items\n+\t\t * between \"name\" and \"name/\", such as \"name-\" and\n+\t\t * \"name.\".\n+\t\t */\n+\t\tint len = strlen(name);\n+\t\tpos = -pos - 1;\n+\n+\t\tfor (i = pos; i < istate->cache_nr; i++) {\n+\t\t\tif (!starts_with(istate->cache[i]->name, name))\n+\t\t\t\tbreak;\n+\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n+\t\t\t\tbreak;\n+\t\t\tif (istate->cache[i]->name[len] == '/')\n+\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\t}\n+\t}\n+}\n+\n /*\n * The daemon can decorate directory events, such as a move or rename,\n * by adding a trailing slash to the observed name. Use this to\n@@ -225,7 +262,7 @@ static void handle_path_with_trailing_slash(\n \n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n-\tint i, len = strlen(name);\n+\tint len = strlen(name);\n \tint pos = index_name_pos(istate, name, len);\n \n \ttrace_printf_key(&trace_fsmonitor,\n@@ -240,34 +277,8 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t\t * for the untracked cache.\n \t\t */\n \t\tname[len - 1] = '\\0';\n-\t} else if (pos >= 0) {\n-\t\t/*\n-\t\t * We have an exact match for this path and can just\n-\t\t * invalidate it.\n-\t\t */\n-\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n-\t\t/*\n-\t\t * The path is not a tracked file -or- it is a\n-\t\t * directory event on a platform that cannot\n-\t\t * distinguish between file and directory events in\n-\t\t * the event handler, such as Windows.\n-\t\t *\n-\t\t * Scan as if it is a directory and invalidate the\n-\t\t * cone under it. (But remember to ignore items\n-\t\t * between \"name\" and \"name/\", such as \"name-\" and\n-\t\t * \"name.\".\n-\t\t */\n-\t\tpos = -pos - 1;\n-\n-\t\tfor (i = pos; i < istate->cache_nr; i++) {\n-\t\t\tif (!starts_with(istate->cache[i]->name, name))\n-\t\t\t\tbreak;\n-\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n-\t\t\t\tbreak;\n-\t\t\tif (istate->cache[i]->name[len] == '/')\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\t}\n+\t\thandle_path_without_trailing_slash(istate, name, pos);\n \t}\n \n \t/*\n-- \ngitgitgadget\n\n"},{"id":"489425","messageId":"ed735e3f1cb90ef887a211a5e208ebe9f13810a3.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 06/14] dir: create untracked_cache_invalidate_trimmed_path()","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:17Z","receivedAt":"2024-02-26T21:39:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nCreate a wrapper function for untracked_cache_invalidate_path()\nthat silently trims a trailing slash, if present, before calling\nthe wrapped function.\n\nThe untracked cache expects to be called with a pathname that\ndoes not contain a trailing slash. This can make it inconvenient\nfor callers that have a directory path. Lets hide this complexity.\n\nThis will be used by a later commit in the FSMonitor code which\nmay receive directory pathnames from an FSEvent.\n\nSigned-off-by: Jeff Hostetler \n---\n dir.c | 20 ++++++++++++++++++++\n dir.h | 7 +++++++\n 2 files changed, 27 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex ac699542302..20ebe4cba26 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -3918,6 +3918,26 @@ void untracked_cache_invalidate_path(struct index_state *istate,\n \t\t\t\t path, strlen(path));\n }\n \n+void untracked_cache_invalidate_trimmed_path(struct index_state *istate,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t int safe_path)\n+{\n+\tsize_t len = strlen(path);\n+\n+\tif (!len)\n+\t\tBUG(\"untracked_cache_invalidate_trimmed_path given zero length path\");\n+\n+\tif (path[len - 1] != '/') {\n+\t\tuntracked_cache_invalidate_path(istate, path, safe_path);\n+\t} else {\n+\t\tstruct strbuf tmp = STRBUF_INIT;\n+\n+\t\tstrbuf_add(&tmp, path, len - 1);\n+\t\tuntracked_cache_invalidate_path(istate, tmp.buf, safe_path);\n+\t\tstrbuf_release(&tmp);\n+\t}\n+}\n+\n void untracked_cache_remove_from_index(struct index_state *istate,\n \t\t\t\t const char *path)\n {\ndiff --git a/dir.h b/dir.h\nindex 98aa85fcc0e..45a7b9ec5f2 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -576,6 +576,13 @@ int cmp_dir_entry(const void *p1, const void *p2);\n int check_dir_entry_contains(const struct dir_entry *out, const struct dir_entry *in);\n \n void untracked_cache_invalidate_path(struct index_state *, const char *, int safe_path);\n+/*\n+ * Invalidate the untracked-cache for this path, but first strip\n+ * off a trailing slash, if present.\n+ */\n+void untracked_cache_invalidate_trimmed_path(struct index_state *,\n+\t\t\t\t\t const char *path,\n+\t\t\t\t\t int safe_path);\n void untracked_cache_remove_from_index(struct index_state *, const char *);\n void untracked_cache_add_to_index(struct index_state *, const char *);\n \n-- \ngitgitgadget\n\n"},{"id":"489426","messageId":"2a43c6cbe0dd5e3d343cb40ec040b7269379e5a4.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 07/14] fsmonitor: refactor untracked-cache invalidation","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:18Z","receivedAt":"2024-02-26T21:39:34Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nUpdate fsmonitor_refresh_callback() to use the new\nuntracked_cache_invalidate_trimmed_path() to invalidate\nthe cache using the observed pathname without needing to\nmodify the caller's buffer.\n\nPreviously, we modified the caller's buffer when the observed pathname\ncontained a trailing slash (and did not restore it). This wasn't a\nproblem for the single use-case caller, but felt dirty nontheless. In\na later commit we will want to invalidate case-corrected versions of\nthe pathname (using possibly borrowed pathnames from the name-hash or\ndir-name-hash) and we may not want to keep the tradition of altering\nthe passed-in pathname.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 11 +++--------\n 1 file changed, 3 insertions(+), 8 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 364198d258f..2787f7ca5d1 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -271,21 +271,16 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \n \tif (name[len - 1] == '/') {\n \t\thandle_path_with_trailing_slash(istate, name, pos);\n-\n-\t\t/*\n-\t\t * We need to remove the traling \"/\" from the path\n-\t\t * for the untracked cache.\n-\t\t */\n-\t\tname[len - 1] = '\\0';\n \t} else {\n \t\thandle_path_without_trailing_slash(istate, name, pos);\n \t}\n \n \t/*\n \t * Mark the untracked cache dirty even if it wasn't found in the index\n-\t * as it could be a new untracked file.\n+\t * as it could be a new untracked file. (Let the untracked cache\n+\t * layer silently deal with any trailing slash.)\n \t */\n-\tuntracked_cache_invalidate_path(istate, name, 0);\n+\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"489427","messageId":"6e87ea6deafdb5f26e66925c5869d78c967af2dc.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 08/14] fsmonitor: move untracked-cache invalidation into helper functions","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:19Z","receivedAt":"2024-02-26T21:39:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nMove the call to invalidate the untracked-cache for the FSEvent\npathname into the two helper functions.\n\nIn a later commit in this series, we will call these helpers\nfrom other contexts and it safer to include the UC invalidation\nin the helpers than to remember to also add it to each helper\ncall-site.\n\nThis has the side-effect of invalidating the UC *before* we\ninvalidate the ce_flags in the cache-entry. These activities\nare independent and do not affect each other. Also, by doing\nthe UC work first, we can avoid worrying about \"early returns\"\nor the need for the usual \"goto the end\" in each of the\nhandler functions.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 26 +++++++++++++++++++-------\n 1 file changed, 19 insertions(+), 7 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 2787f7ca5d1..2f58ee2fe5a 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -188,6 +188,16 @@ static void handle_path_without_trailing_slash(\n {\n \tint i;\n \n+\t/*\n+\t * Mark the untracked cache dirty for this path (regardless of\n+\t * whether or not we find an exact match for it in the index).\n+\t * Since the path is unqualified (no trailing slash hint in the\n+\t * FSEvent), it may refer to a file or directory. So we should\n+\t * not assume one or the other and should always let the untracked\n+\t * cache decide what needs to invalidated.\n+\t */\n+\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n+\n \tif (pos >= 0) {\n \t\t/*\n \t\t * We have an exact match for this path and can just\n@@ -249,6 +259,15 @@ static void handle_path_with_trailing_slash(\n {\n \tint i;\n \n+\t/*\n+\t * Mark the untracked cache dirty for this directory path\n+\t * (regardless of whether or not we find an exact match for it\n+\t * in the index or find it to be proper prefix of one or more\n+\t * files in the index), since the FSEvent is hinting that\n+\t * there may be changes on or within the directory.\n+\t */\n+\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n+\n \tif (pos < 0)\n \t\tpos = -pos - 1;\n \n@@ -274,13 +293,6 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \t} else {\n \t\thandle_path_without_trailing_slash(istate, name, pos);\n \t}\n-\n-\t/*\n-\t * Mark the untracked cache dirty even if it wasn't found in the index\n-\t * as it could be a new untracked file. (Let the untracked cache\n-\t * layer silently deal with any trailing slash.)\n-\t */\n-\tuntracked_cache_invalidate_trimmed_path(istate, name, 0);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"489428","messageId":"5fea8b9476e6bf0e0635e886364f0d718deac97c.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 09/14] fsmonitor: return invalidated cache-entry count on directory event","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:20Z","receivedAt":"2024-02-26T21:39:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the refresh callback helper function for directory FSEvents to\nreturn the number of cache-entries that were invalidated in response\nto a directory event.\n\nThis will be used in a later commit to help determine if the observed\npathname in the FSEvent was a (possibly) case-incorrect directory\nprefix (on a case-insensitive filesystem) of one or more actual\ncache-entries.\n\nIf there exists at least one case-insensitive prefix match, then we\ncan assume that the directory is a (case-incorrect) prefix of at least\none tracked item rather than a completely unknown/untracked file or\ndirectory.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 2f58ee2fe5a..9424bd17230 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -253,11 +253,20 @@ static void handle_path_without_trailing_slash(\n * same way and just invalidate the cache-entry and the untracked\n * cache (and in this case, the forward cache-entry scan won't find\n * anything and it doesn't hurt to let it run).\n+ *\n+ * Return the number of cache-entries that we invalidated. We will\n+ * use this later to determine if we need to attempt a second\n+ * case-insensitive search on case-insensitive file systems. That is,\n+ * if the search using the observed-case in the FSEvent yields any\n+ * results, we assume the prefix is case-correct. If there are no\n+ * matches, we still don't know if the observed path is simply\n+ * untracked or case-incorrect.\n */\n-static void handle_path_with_trailing_slash(\n+static size_t handle_path_with_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos)\n {\n \tint i;\n+\tsize_t nr_in_cone = 0;\n \n \t/*\n \t * Mark the untracked cache dirty for this directory path\n@@ -276,7 +285,10 @@ static void handle_path_with_trailing_slash(\n \t\tif (!starts_with(istate->cache[i]->name, name))\n \t\t\tbreak;\n \t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tnr_in_cone++;\n \t}\n+\n+\treturn nr_in_cone;\n }\n \n static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n-- \ngitgitgadget\n\n"},{"id":"489429","messageId":"3fa7536cf8039c4e9324d9ea6d2b0e9ae4bbb10b.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 10/14] fsmonitor: remove custom loop from non-directory path handler","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:21Z","receivedAt":"2024-02-26T21:39:37Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor the code that handles refresh events for pathnames that do\nnot contain a trailing slash. Instead of using a custom loop to try\nto scan the index and detect if the FSEvent named a file or might be a\ndirectory prefix, use the recently created helper function to do that.\n\nAlso update the comments to describe what and why we are doing this.\n\nOn platforms that DO NOT annotate FS events with a trailing\nslash, if we fail to find an exact match for the pathname\nin the index, we do not know if the pathname represents a\ndirectory or simply an untracked file. Pretend that the pathname\nis a directory and try again before assuming it is an untracked\nfile.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 55 ++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 31 insertions(+), 24 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 9424bd17230..a51c17cda70 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,11 +183,23 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+static size_t handle_path_with_trailing_slash(\n+\tstruct index_state *istate, const char *name, int pos);\n+\n+/*\n+ * The daemon sent an observed pathname without a trailing slash.\n+ * (This is the normal case.) We do not know if it is a tracked or\n+ * untracked file, a sparse-directory, or a populated directory (on a\n+ * platform such as Windows where FSEvents are not qualified).\n+ *\n+ * The pathname contains the observed case reported by the FS. We\n+ * do not know it is case-correct or -incorrect.\n+ *\n+ * Assume it is case-correct and try an exact match.\n+ */\n static void handle_path_without_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos)\n {\n-\tint i;\n-\n \t/*\n \t * Mark the untracked cache dirty for this path (regardless of\n \t * whether or not we find an exact match for it in the index).\n@@ -200,33 +212,28 @@ static void handle_path_without_trailing_slash(\n \n \tif (pos >= 0) {\n \t\t/*\n-\t\t * We have an exact match for this path and can just\n-\t\t * invalidate it.\n+\t\t * An exact match on a tracked file. We assume that we\n+\t\t * do not need to scan forward for a sparse-directory\n+\t\t * cache-entry with the same pathname, nor for a cone\n+\t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n \t} else {\n+\t\tstruct strbuf work_path = STRBUF_INIT;\n+\n \t\t/*\n-\t\t * The path is not a tracked file -or- it is a\n-\t\t * directory event on a platform that cannot\n-\t\t * distinguish between file and directory events in\n-\t\t * the event handler, such as Windows.\n-\t\t *\n-\t\t * Scan as if it is a directory and invalidate the\n-\t\t * cone under it. (But remember to ignore items\n-\t\t * between \"name\" and \"name/\", such as \"name-\" and\n-\t\t * \"name.\".\n+\t\t * The negative \"pos\" gives us the suggested insertion\n+\t\t * point for the pathname (without the trailing slash).\n+\t\t * We need to see if there is a directory with that\n+\t\t * prefix, but there can be lots of pathnames between\n+\t\t * \"foo\" and \"foo/\" like \"foo-\" or \"foo-bar\", so we\n+\t\t * don't want to do our own scan.\n \t\t */\n-\t\tint len = strlen(name);\n-\t\tpos = -pos - 1;\n-\n-\t\tfor (i = pos; i < istate->cache_nr; i++) {\n-\t\t\tif (!starts_with(istate->cache[i]->name, name))\n-\t\t\t\tbreak;\n-\t\t\tif ((unsigned char)istate->cache[i]->name[len] > '/')\n-\t\t\t\tbreak;\n-\t\t\tif (istate->cache[i]->name[len] == '/')\n-\t\t\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n-\t\t}\n+\t\tstrbuf_add(&work_path, name, strlen(name));\n+\t\tstrbuf_addch(&work_path, '/');\n+\t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n+\t\thandle_path_with_trailing_slash(istate, work_path.buf, pos);\n+\t\tstrbuf_release(&work_path);\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"489430","messageId":"53f73c1515d0f64bf13f0b30e9039dedbf441f2c.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 11/14] fsmonitor: return invalided cache-entry count on non-directory event","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:22Z","receivedAt":"2024-02-26T21:39:38Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach the refresh callback helper function for unqualified FSEvents\n(pathnames without a trailing slash) to return the number of\ncache-entries that were invalided in response to the event.\n\nThis will be used in a later commit to help determine if the observed\npathname was (possibly) case-incorrect when (on a case-insensitive\nfile system).\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 10 ++++++++--\n 1 file changed, 8 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex a51c17cda70..c16ed5d8758 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -196,8 +196,10 @@ static size_t handle_path_with_trailing_slash(\n * do not know it is case-correct or -incorrect.\n *\n * Assume it is case-correct and try an exact match.\n+ *\n+ * Return the number of cache-entries that we invalidated.\n */\n-static void handle_path_without_trailing_slash(\n+static size_t handle_path_without_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos)\n {\n \t/*\n@@ -218,7 +220,9 @@ static void handle_path_without_trailing_slash(\n \t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\treturn 1;\n \t} else {\n+\t\tsize_t nr_in_cone;\n \t\tstruct strbuf work_path = STRBUF_INIT;\n \n \t\t/*\n@@ -232,8 +236,10 @@ static void handle_path_without_trailing_slash(\n \t\tstrbuf_add(&work_path, name, strlen(name));\n \t\tstrbuf_addch(&work_path, '/');\n \t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n-\t\thandle_path_with_trailing_slash(istate, work_path.buf, pos);\n+\t\tnr_in_cone = handle_path_with_trailing_slash(\n+\t\t\tistate, work_path.buf, pos);\n \t\tstrbuf_release(&work_path);\n+\t\treturn nr_in_cone;\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"489431","messageId":"0148319aea5cf32c454be9d1a0efc7934580d7a7.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 12/14] fsmonitor: trace the new invalidated cache-entry count","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:23Z","receivedAt":"2024-02-26T21:39:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nConsolidate the directory/non-directory calls to the refresh handler\ncode. Log the resulting count of invalidated cache-entries.\n\nThe nr_in_cone value will be used in a later commit to decide if\nwe also need to try to do case-insensitive lookups.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 15 ++++++++++-----\n 1 file changed, 10 insertions(+), 5 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex c16ed5d8758..739ddbf7aca 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -308,16 +308,21 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n {\n \tint len = strlen(name);\n \tint pos = index_name_pos(istate, name, len);\n+\tsize_t nr_in_cone;\n \n \ttrace_printf_key(&trace_fsmonitor,\n \t\t\t \"fsmonitor_refresh_callback '%s' (pos %d)\",\n \t\t\t name, pos);\n \n-\tif (name[len - 1] == '/') {\n-\t\thandle_path_with_trailing_slash(istate, name, pos);\n-\t} else {\n-\t\thandle_path_without_trailing_slash(istate, name, pos);\n-\t}\n+\tif (name[len - 1] == '/')\n+\t\tnr_in_cone = handle_path_with_trailing_slash(istate, name, pos);\n+\telse\n+\t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n+\n+\tif (nr_in_cone)\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\n+\t\t\t\t (int)nr_in_cone);\n }\n \n /*\n-- \ngitgitgadget\n\n"},{"id":"489432","messageId":"04867eccfcd89e4a905899f6ee5f595396a4a9ea.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 13/14] fsmonitor: refactor bit invalidation in refresh callback","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:24Z","receivedAt":"2024-02-26T21:39:40Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nRefactor code in the fsmonitor_refresh_callback() call chain dealing\nwith invalidating the CE_FSMONITOR_VALID bit and add a trace message.\n\nDuring the refresh, we clear the CE_FSMONITOR_VALID bit in response to\ndata from the FSMonitor daemon (so that a later phase will lstat() and\nverify the true state of the file).\n\nCreate a new function to clear the bit and add some unique tracing for\nit to help debug edge cases.\n\nThis is similar to the existing `mark_fsmonitor_invalid()` function,\nbut it also does untracked-cache invalidation and we've already\nhandled that in the refresh-callback handlers, so but we don't need\nto repeat that.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 20 ++++++++++++++++++--\n 1 file changed, 18 insertions(+), 2 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 739ddbf7aca..3c87449be87 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -183,6 +183,22 @@ static int query_fsmonitor_hook(struct repository *r,\n \treturn result;\n }\n \n+/*\n+ * Invalidate the FSM bit on this CE. This is like mark_fsmonitor_invalid()\n+ * but we've already handled the untracked-cache, so let's not repeat that\n+ * work. This also lets us have a different trace message so that we can\n+ * see everything that was done as part of the refresh-callback.\n+ */\n+static void invalidate_ce_fsm(struct cache_entry *ce)\n+{\n+\tif (ce->ce_flags & CE_FSMONITOR_VALID) {\n+\t\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t\t \"fsmonitor_refresh_callback INV: '%s'\",\n+\t\t\t\t ce->name);\n+\t\tce->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t}\n+}\n+\n static size_t handle_path_with_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos);\n \n@@ -219,7 +235,7 @@ static size_t handle_path_without_trailing_slash(\n \t\t * cache-entry with the same pathname, nor for a cone\n \t\t * at that directory. (That is, assume no D/F conflicts.)\n \t\t */\n-\t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tinvalidate_ce_fsm(istate->cache[pos]);\n \t\treturn 1;\n \t} else {\n \t\tsize_t nr_in_cone;\n@@ -297,7 +313,7 @@ static size_t handle_path_with_trailing_slash(\n \tfor (i = pos; i < istate->cache_nr; i++) {\n \t\tif (!starts_with(istate->cache[i]->name, name))\n \t\t\tbreak;\n-\t\tistate->cache[i]->ce_flags &= ~CE_FSMONITOR_VALID;\n+\t\tinvalidate_ce_fsm(istate->cache[i]);\n \t\tnr_in_cone++;\n \t}\n \n-- \ngitgitgadget\n\n"},{"id":"489433","messageId":"ec036c04d1b003385683f0ed1dec80cc3bbfb813.1708983566.git.gitgitgadget@gmail.com","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"[PATCH v3 14/14] fsmonitor: support case-insensitive events","fromName":"Jeff Hostetler via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2024-02-26T21:39:25Z","receivedAt":"2024-02-26T21:39:41Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fsmonitor_refresh_callback() to handle case-insensitive\nlookups if case-sensitive lookups fail on case-insensitive systems.\nThis can cause 'git status' to report stale status for files if there\nare case issues/errors in the worktree.\n\nThe FSMonitor daemon sends FSEvents using the observed spelling\nof each pathname. On case-insensitive file systems this may be\ndifferent than the expected case spelling.\n\nThe existing code uses index_name_pos() to find the cache-entry for\nthe pathname in the FSEvent and clear the CE_FSMONITOR_VALID bit so\nthat the worktree scan/index refresh will revisit and revalidate the\npath.\n\nOn a case-insensitive file system, the exact match lookup may fail\nto find the associated cache-entry. This causes status to think that\nthe cached CE flags are correct and skip over the file.\n\nUpdate event handling to optionally use the name-hash and dir-name-hash\nif necessary.\n\nAlso update t7527 to convert the \"test_expect_failure\" to \"_success\"\nnow that we have fixed the bug.\n\nSigned-off-by: Jeff Hostetler \n---\n fsmonitor.c | 121 +++++++++++++++++++++++++++++++++++\n t/t7527-builtin-fsmonitor.sh | 26 +++++---\n 2 files changed, 137 insertions(+), 10 deletions(-)\n\ndiff --git a/fsmonitor.c b/fsmonitor.c\nindex 3c87449be87..2b17d60bbbe 100644\n--- a/fsmonitor.c\n+++ b/fsmonitor.c\n@@ -5,6 +5,7 @@\n #include \"ewah/ewok.h\"\n #include \"fsmonitor.h\"\n #include \"fsmonitor-ipc.h\"\n+#include \"name-hash.h\"\n #include \"run-command.h\"\n #include \"strbuf.h\"\n #include \"trace2.h\"\n@@ -202,6 +203,113 @@ static void invalidate_ce_fsm(struct cache_entry *ce)\n static size_t handle_path_with_trailing_slash(\n \tstruct index_state *istate, const char *name, int pos);\n \n+/*\n+ * Use the name-hash to do a case-insensitive cache-entry lookup with\n+ * the pathname and invalidate the cache-entry.\n+ *\n+ * Returns the number of cache-entries that we invalidated.\n+ */\n+static size_t handle_using_name_hash_icase(\n+\tstruct index_state *istate, const char *name)\n+{\n+\tstruct cache_entry *ce = NULL;\n+\n+\tce = index_file_exists(istate, name, strlen(name), 1);\n+\tif (!ce)\n+\t\treturn 0;\n+\n+\t/*\n+\t * A case-insensitive search in the name-hash using the\n+\t * observed pathname found a cache-entry, so the observed path\n+\t * is case-incorrect. Invalidate the cache-entry and use the\n+\t * correct spelling from the cache-entry to invalidate the\n+\t * untracked-cache. Since we now have sparse-directories in\n+\t * the index, the observed pathname may represent a regular\n+\t * file or a sparse-index directory.\n+\t *\n+\t * Note that we should not have seen FSEvents for a\n+\t * sparse-index directory, but we handle it just in case.\n+\t *\n+\t * Either way, we know that there are not any cache-entries for\n+\t * children inside the cone of the directory, so we don't need to\n+\t * do the usual scan.\n+\t */\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n+\t\t\t name, ce->name);\n+\n+\t/*\n+\t * NEEDSWORK: We used the name-hash to find the correct\n+\t * case-spelling of the pathname in the cache-entry[], so\n+\t * technically this is a tracked file or a sparse-directory.\n+\t * It should not have any entries in the untracked-cache, so\n+\t * we should not need to use the case-corrected spelling to\n+\t * invalidate the the untracked-cache. So we may not need to\n+\t * do this. For now, I'm going to be conservative and always\n+\t * do it; we can revisit this later.\n+\t */\n+\tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n+\n+\tinvalidate_ce_fsm(ce);\n+\treturn 1;\n+}\n+\n+/*\n+ * Use the dir-name-hash to find the correct-case spelling of the\n+ * directory. Use the canonical spelling to invalidate all of the\n+ * cache-entries within the matching cone.\n+ *\n+ * Returns the number of cache-entries that we invalidated.\n+ */\n+static size_t handle_using_dir_name_hash_icase(\n+\tstruct index_state *istate, const char *name)\n+{\n+\tstruct strbuf canonical_path = STRBUF_INIT;\n+\tint pos;\n+\tsize_t len = strlen(name);\n+\tsize_t nr_in_cone;\n+\n+\tif (name[len - 1] == '/')\n+\t\tlen--;\n+\n+\tif (!index_dir_find(istate, name, len, &canonical_path))\n+\t\treturn 0; /* name is untracked */\n+\n+\tif (!memcmp(name, canonical_path.buf, canonical_path.len)) {\n+\t\tstrbuf_release(&canonical_path);\n+\t\t/*\n+\t\t * NEEDSWORK: Our caller already tried an exact match\n+\t\t * and failed to find one. They called us to do an\n+\t\t * ICASE match, so we should never get an exact match,\n+\t\t * so we could promote this to a BUG() here if we\n+\t\t * wanted to. It doesn't hurt anything to just return\n+\t\t * 0 and go on because we should never get here. Or we\n+\t\t * could just get rid of the memcmp() and this \"if\"\n+\t\t * clause completely.\n+\t\t */\n+\t\tBUG(\"handle_using_dir_name_hash_icase(%s) did not exact match\",\n+\t\t name);\n+\t}\n+\n+\ttrace_printf_key(&trace_fsmonitor,\n+\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n+\t\t\t name, canonical_path.buf);\n+\n+\t/*\n+\t * The dir-name-hash only tells us the corrected spelling of\n+\t * the prefix. We have to use this canonical path to do a\n+\t * lookup in the cache-entry array so that we repeat the\n+\t * original search using the case-corrected spelling.\n+\t */\n+\tstrbuf_addch(&canonical_path, '/');\n+\tpos = index_name_pos(istate, canonical_path.buf,\n+\t\t\t canonical_path.len);\n+\tnr_in_cone = handle_path_with_trailing_slash(\n+\t\tistate, canonical_path.buf, pos);\n+\tstrbuf_release(&canonical_path);\n+\treturn nr_in_cone;\n+}\n+\n /*\n * The daemon sent an observed pathname without a trailing slash.\n * (This is the normal case.) We do not know if it is a tracked or\n@@ -335,6 +443,19 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n \telse\n \t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n \n+\t/*\n+\t * If we did not find an exact match for this pathname or any\n+\t * cache-entries with this directory prefix and we're on a\n+\t * case-insensitive file system, try again using the name-hash\n+\t * and dir-name-hash.\n+\t */\n+\tif (!nr_in_cone && ignore_case) {\n+\t\tnr_in_cone = handle_using_name_hash_icase(istate, name);\n+\t\tif (!nr_in_cone)\n+\t\t\tnr_in_cone = handle_using_dir_name_hash_icase(\n+\t\t\t\tistate, name);\n+\t}\n+\n \tif (nr_in_cone)\n \t\ttrace_printf_key(&trace_fsmonitor,\n \t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\ndiff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\nindex 830f2d9de33..730f3c7f810 100755\n--- a/t/t7527-builtin-fsmonitor.sh\n+++ b/t/t7527-builtin-fsmonitor.sh\n@@ -1051,7 +1051,7 @@ test_expect_success 'split-index and FSMonitor work well together' '\n #\n # The setup is a little contrived.\n #\n-test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n \ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n \n \tgit init subdir_case_wrong &&\n@@ -1116,19 +1116,19 @@ test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n \n \tgrep -q \"dir1/DIR2/dir3/file3.*pos -3\" \"$PWD/subdir_case_wrong.log1\" &&\n \n+\t# Verify that we get a mapping event to correct the case.\n+\tgrep -q \"MAP:.*dir1/DIR2/dir3/file3.*dir1/dir2/dir3/file3\" \\\n+\t\t\"$PWD/subdir_case_wrong.log1\" &&\n+\n \t# The refresh-callbacks should have caused \"git status\" to clear\n \t# the CE_FSMONITOR_VALID bit on each of those files and caused\n \t# the worktree scan to visit them and mark them as modified.\n \tgrep -q \" M AAA\" \"$PWD/subdir_case_wrong.out\" &&\n \tgrep -q \" M zzz\" \"$PWD/subdir_case_wrong.out\" &&\n-\n-\t# Expect Breakage: with the case confusion, the \"(pos -3)\" causes\n-\t# the client to not clear the CE_FSMONITOR_VALID bit and therefore\n-\t# status will not rescan the file and therefore not report it as dirty.\n \tgrep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n '\n \n-test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n+test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n \ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n \n \tgit init file_case_wrong &&\n@@ -1242,14 +1242,20 @@ test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n \tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try3.log\" \\\n \t\tgit -C file_case_wrong --no-optional-locks status --short \\\n \t\t\t>\"$PWD/file_case_wrong-try3.out\" &&\n+\n+\t# Verify that we get a mapping event to correct the case.\n+\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir3/FILE-3-A.*dir1/dir2/dir3/file-3-a\" \\\n+\t\t\"$PWD/file_case_wrong-try3.log\" &&\n+\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir4/file-4-a.*dir1/dir2/dir4/FILE-4-A\" \\\n+\t\t\"$PWD/file_case_wrong-try3.log\" &&\n+\n \t# FSEvents are in observed case.\n \tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n \tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n \n-\t# Expect Breakage: with the case confusion, the \"(pos-3)\" and\n-\t# \"(pos -9)\" causes the client to not clear the CE_FSMONITOR_VALID\n-\t# bit and therefore status will not rescan the files and therefore\n-\t# not report them as dirty.\n+\t# The refresh-callbacks should have caused \"git status\" to clear\n+\t# the CE_FSMONITOR_VALID bit on each of those files and caused\n+\t# the worktree scan to visit them and mark them as modified.\n \tgrep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n \tgrep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n '\n-- \ngitgitgadget\n"},{"id":"489455","messageId":"xmqqfrxeit1k.fsf@gitster.g","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/14] FSMonitor edge cases on case-insensitive file systems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-02-27T01:40:07Z","receivedAt":"2024-02-27T01:40:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Jeff Hostetler via GitGitGadget\" writes:\n\n> Here is version 3. I think I have addressed the remaining comments.\n>\n> I cleaned up the test code to use the test_expect_failure at the beginning\n> and squashed in the test_expect_success version of tests into the final\n> commit in the series.\n>\n> I moved the invalidate_ce_fsm() commit earlier in the series, so that the\n> final commit actually uses all of the up-to-this-point changes to fix the\n> problem.\n>\n> I converted a few \"should not happens\" to BUG()s.\n>\n> Thanks to everyone for their time and attention reviewing this. Jeff\n>\n> Jeff Hostetler (14):\n> name-hash: add index_dir_find()\n> t7527: add case-insensitve test for FSMonitor\n> fsmonitor: refactor refresh callback on directory events\n> fsmonitor: clarify handling of directory events in callback helper\n> fsmonitor: refactor refresh callback for non-directory events\n> dir: create untracked_cache_invalidate_trimmed_path()\n> fsmonitor: refactor untracked-cache invalidation\n> fsmonitor: move untracked-cache invalidation into helper functions\n> fsmonitor: return invalidated cache-entry count on directory event\n> fsmonitor: remove custom loop from non-directory path handler\n> fsmonitor: return invalided cache-entry count on non-directory event\n> fsmonitor: trace the new invalidated cache-entry count\n> fsmonitor: refactor bit invalidation in refresh callback\n> fsmonitor: support case-insensitive events\n>\n> dir.c | 20 +++\n> dir.h | 7 +\n> fsmonitor.c | 312 +++++++++++++++++++++++++++++------\n> name-hash.c | 9 +-\n> name-hash.h | 7 +-\n> t/t7527-builtin-fsmonitor.sh | 223 +++++++++++++++++++++++++\n> 6 files changed, 522 insertions(+), 56 deletions(-)\n\nMuch nicer. Will queue. Thanks.\n"},{"id":"490083","messageId":"ZehofMaSZyUq8S1N@tanuki","threadId":"60908","inReplyTo":"pull.1662.v3.git.1708983565.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 00/14] FSMonitor edge cases on case-insensitive file systems","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-06T12:58:36Z","receivedAt":"2024-03-06T12:58:43Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 26, 2024 at 09:39:11PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> Here is version 3. I think I have addressed the remaining comments.\n> \n> I cleaned up the test code to use the test_expect_failure at the beginning\n> and squashed in the test_expect_success version of tests into the final\n> commit in the series.\n> \n> I moved the invalidate_ce_fsm() commit earlier in the series, so that the\n> final commit actually uses all of the up-to-this-point changes to fix the\n> problem.\n> \n> I converted a few \"should not happens\" to BUG()s.\n> \n> Thanks to everyone for their time and attention reviewing this. Jeff\n\nI gave this whole patch series a read and didn't much to complain about.\nThere are a couple of nits, but none of them really require a reroll in\nmy opinion.\n\nThanks!\n\nPatrick\n"},{"id":"490084","messageId":"ZehogvB9hhh4ENdR@tanuki","threadId":"60908","inReplyTo":"53f73c1515d0f64bf13f0b30e9039dedbf441f2c.1708983566.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 11/14] fsmonitor: return invalided cache-entry count on non-directory event","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-06T12:58:42Z","receivedAt":"2024-03-06T12:58:47Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"Nit, not worth a reroll: the subject says \"invalided\" instead of\n\"invalidated\".\n\nPatrick\n\nOn Mon, Feb 26, 2024 at 09:39:22PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach the refresh callback helper function for unqualified FSEvents\n> (pathnames without a trailing slash) to return the number of\n> cache-entries that were invalided in response to the event.\n> \n> This will be used in a later commit to help determine if the observed\n> pathname was (possibly) case-incorrect when (on a case-insensitive\n> file system).\n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 10 ++++++++--\n> 1 file changed, 8 insertions(+), 2 deletions(-)\n> \n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index a51c17cda70..c16ed5d8758 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -196,8 +196,10 @@ static size_t handle_path_with_trailing_slash(\n> * do not know it is case-correct or -incorrect.\n> *\n> * Assume it is case-correct and try an exact match.\n> + *\n> + * Return the number of cache-entries that we invalidated.\n> */\n> -static void handle_path_without_trailing_slash(\n> +static size_t handle_path_without_trailing_slash(\n> \tstruct index_state *istate, const char *name, int pos)\n> {\n> \t/*\n> @@ -218,7 +220,9 @@ static void handle_path_without_trailing_slash(\n> \t\t * at that directory. (That is, assume no D/F conflicts.)\n> \t\t */\n> \t\tistate->cache[pos]->ce_flags &= ~CE_FSMONITOR_VALID;\n> +\t\treturn 1;\n> \t} else {\n> +\t\tsize_t nr_in_cone;\n> \t\tstruct strbuf work_path = STRBUF_INIT;\n> \n> \t\t/*\n> @@ -232,8 +236,10 @@ static void handle_path_without_trailing_slash(\n> \t\tstrbuf_add(&work_path, name, strlen(name));\n> \t\tstrbuf_addch(&work_path, '/');\n> \t\tpos = index_name_pos(istate, work_path.buf, work_path.len);\n> -\t\thandle_path_with_trailing_slash(istate, work_path.buf, pos);\n> +\t\tnr_in_cone = handle_path_with_trailing_slash(\n> +\t\t\tistate, work_path.buf, pos);\n> \t\tstrbuf_release(&work_path);\n> +\t\treturn nr_in_cone;\n> \t}\n> }\n> \n> -- \n> gitgitgadget\n> \n"},{"id":"490085","messageId":"ZehojvoI1UOJmcD3@tanuki","threadId":"60908","inReplyTo":"ec036c04d1b003385683f0ed1dec80cc3bbfb813.1708983566.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 14/14] fsmonitor: support case-insensitive events","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2024-03-06T12:58:54Z","receivedAt":"2024-03-06T12:58:59Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Mon, Feb 26, 2024 at 09:39:25PM +0000, Jeff Hostetler via GitGitGadget wrote:\n> From: Jeff Hostetler \n> \n> Teach fsmonitor_refresh_callback() to handle case-insensitive\n> lookups if case-sensitive lookups fail on case-insensitive systems.\n> This can cause 'git status' to report stale status for files if there\n> are case issues/errors in the worktree.\n> \n> The FSMonitor daemon sends FSEvents using the observed spelling\n> of each pathname. On case-insensitive file systems this may be\n> different than the expected case spelling.\n> \n> The existing code uses index_name_pos() to find the cache-entry for\n> the pathname in the FSEvent and clear the CE_FSMONITOR_VALID bit so\n> that the worktree scan/index refresh will revisit and revalidate the\n> path.\n> \n> On a case-insensitive file system, the exact match lookup may fail\n> to find the associated cache-entry. This causes status to think that\n> the cached CE flags are correct and skip over the file.\n> \n> Update event handling to optionally use the name-hash and dir-name-hash\n> if necessary.\n> \n> Also update t7527 to convert the \"test_expect_failure\" to \"_success\"\n> now that we have fixed the bug.\n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> fsmonitor.c | 121 +++++++++++++++++++++++++++++++++++\n> t/t7527-builtin-fsmonitor.sh | 26 +++++---\n> 2 files changed, 137 insertions(+), 10 deletions(-)\n> \n> diff --git a/fsmonitor.c b/fsmonitor.c\n> index 3c87449be87..2b17d60bbbe 100644\n> --- a/fsmonitor.c\n> +++ b/fsmonitor.c\n> @@ -5,6 +5,7 @@\n> #include \"ewah/ewok.h\"\n> #include \"fsmonitor.h\"\n> #include \"fsmonitor-ipc.h\"\n> +#include \"name-hash.h\"\n> #include \"run-command.h\"\n> #include \"strbuf.h\"\n> #include \"trace2.h\"\n> @@ -202,6 +203,113 @@ static void invalidate_ce_fsm(struct cache_entry *ce)\n> static size_t handle_path_with_trailing_slash(\n> \tstruct index_state *istate, const char *name, int pos);\n> \n> +/*\n> + * Use the name-hash to do a case-insensitive cache-entry lookup with\n> + * the pathname and invalidate the cache-entry.\n> + *\n> + * Returns the number of cache-entries that we invalidated.\n> + */\n> +static size_t handle_using_name_hash_icase(\n> +\tstruct index_state *istate, const char *name)\n> +{\n> +\tstruct cache_entry *ce = NULL;\n> +\n> +\tce = index_file_exists(istate, name, strlen(name), 1);\n> +\tif (!ce)\n> +\t\treturn 0;\n> +\n> +\t/*\n> +\t * A case-insensitive search in the name-hash using the\n> +\t * observed pathname found a cache-entry, so the observed path\n> +\t * is case-incorrect. Invalidate the cache-entry and use the\n> +\t * correct spelling from the cache-entry to invalidate the\n> +\t * untracked-cache. Since we now have sparse-directories in\n> +\t * the index, the observed pathname may represent a regular\n> +\t * file or a sparse-index directory.\n> +\t *\n> +\t * Note that we should not have seen FSEvents for a\n> +\t * sparse-index directory, but we handle it just in case.\n> +\t *\n> +\t * Either way, we know that there are not any cache-entries for\n> +\t * children inside the cone of the directory, so we don't need to\n> +\t * do the usual scan.\n> +\t */\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n> +\t\t\t name, ce->name);\n> +\n> +\t/*\n> +\t * NEEDSWORK: We used the name-hash to find the correct\n> +\t * case-spelling of the pathname in the cache-entry[], so\n> +\t * technically this is a tracked file or a sparse-directory.\n> +\t * It should not have any entries in the untracked-cache, so\n> +\t * we should not need to use the case-corrected spelling to\n> +\t * invalidate the the untracked-cache. So we may not need to\n> +\t * do this. For now, I'm going to be conservative and always\n> +\t * do it; we can revisit this later.\n> +\t */\n> +\tuntracked_cache_invalidate_trimmed_path(istate, ce->name, 0);\n> +\n> +\tinvalidate_ce_fsm(ce);\n> +\treturn 1;\n> +}\n> +\n> +/*\n> + * Use the dir-name-hash to find the correct-case spelling of the\n> + * directory. Use the canonical spelling to invalidate all of the\n> + * cache-entries within the matching cone.\n> + *\n> + * Returns the number of cache-entries that we invalidated.\n> + */\n> +static size_t handle_using_dir_name_hash_icase(\n> +\tstruct index_state *istate, const char *name)\n> +{\n> +\tstruct strbuf canonical_path = STRBUF_INIT;\n> +\tint pos;\n> +\tsize_t len = strlen(name);\n> +\tsize_t nr_in_cone;\n> +\n> +\tif (name[len - 1] == '/')\n> +\t\tlen--;\n\nNit: this could use `strip_suffix()`.\n\n> +\n> +\tif (!index_dir_find(istate, name, len, &canonical_path))\n> +\t\treturn 0; /* name is untracked */\n> +\n> +\tif (!memcmp(name, canonical_path.buf, canonical_path.len)) {\n> +\t\tstrbuf_release(&canonical_path);\n> +\t\t/*\n> +\t\t * NEEDSWORK: Our caller already tried an exact match\n> +\t\t * and failed to find one. They called us to do an\n> +\t\t * ICASE match, so we should never get an exact match,\n> +\t\t * so we could promote this to a BUG() here if we\n> +\t\t * wanted to. It doesn't hurt anything to just return\n\nNit: this comment is stale as this has been promoted to a BUG already.\n\nPatrick\n\n> +\t\t * 0 and go on because we should never get here. Or we\n> +\t\t * could just get rid of the memcmp() and this \"if\"\n> +\t\t * clause completely.\n> +\t\t */\n> +\t\tBUG(\"handle_using_dir_name_hash_icase(%s) did not exact match\",\n> +\t\t name);\n> +\t}\n> +\n> +\ttrace_printf_key(&trace_fsmonitor,\n> +\t\t\t \"fsmonitor_refresh_callback MAP: '%s' '%s'\",\n> +\t\t\t name, canonical_path.buf);\n> +\n> +\t/*\n> +\t * The dir-name-hash only tells us the corrected spelling of\n> +\t * the prefix. We have to use this canonical path to do a\n> +\t * lookup in the cache-entry array so that we repeat the\n> +\t * original search using the case-corrected spelling.\n> +\t */\n> +\tstrbuf_addch(&canonical_path, '/');\n> +\tpos = index_name_pos(istate, canonical_path.buf,\n> +\t\t\t canonical_path.len);\n> +\tnr_in_cone = handle_path_with_trailing_slash(\n> +\t\tistate, canonical_path.buf, pos);\n> +\tstrbuf_release(&canonical_path);\n> +\treturn nr_in_cone;\n> +}\n> +\n> /*\n> * The daemon sent an observed pathname without a trailing slash.\n> * (This is the normal case.) We do not know if it is a tracked or\n> @@ -335,6 +443,19 @@ static void fsmonitor_refresh_callback(struct index_state *istate, char *name)\n> \telse\n> \t\tnr_in_cone = handle_path_without_trailing_slash(istate, name, pos);\n> \n> +\t/*\n> +\t * If we did not find an exact match for this pathname or any\n> +\t * cache-entries with this directory prefix and we're on a\n> +\t * case-insensitive file system, try again using the name-hash\n> +\t * and dir-name-hash.\n> +\t */\n> +\tif (!nr_in_cone && ignore_case) {\n> +\t\tnr_in_cone = handle_using_name_hash_icase(istate, name);\n> +\t\tif (!nr_in_cone)\n> +\t\t\tnr_in_cone = handle_using_dir_name_hash_icase(\n> +\t\t\t\tistate, name);\n> +\t}\n> +\n> \tif (nr_in_cone)\n> \t\ttrace_printf_key(&trace_fsmonitor,\n> \t\t\t\t \"fsmonitor_refresh_callback CNT: %d\",\n> diff --git a/t/t7527-builtin-fsmonitor.sh b/t/t7527-builtin-fsmonitor.sh\n> index 830f2d9de33..730f3c7f810 100755\n> --- a/t/t7527-builtin-fsmonitor.sh\n> +++ b/t/t7527-builtin-fsmonitor.sh\n> @@ -1051,7 +1051,7 @@ test_expect_success 'split-index and FSMonitor work well together' '\n> #\n> # The setup is a little contrived.\n> #\n> -test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n> +test_expect_success CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n> \ttest_when_finished \"stop_daemon_delete_repo subdir_case_wrong\" &&\n> \n> \tgit init subdir_case_wrong &&\n> @@ -1116,19 +1116,19 @@ test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor subdir case wrong on disk' '\n> \n> \tgrep -q \"dir1/DIR2/dir3/file3.*pos -3\" \"$PWD/subdir_case_wrong.log1\" &&\n> \n> +\t# Verify that we get a mapping event to correct the case.\n> +\tgrep -q \"MAP:.*dir1/DIR2/dir3/file3.*dir1/dir2/dir3/file3\" \\\n> +\t\t\"$PWD/subdir_case_wrong.log1\" &&\n> +\n> \t# The refresh-callbacks should have caused \"git status\" to clear\n> \t# the CE_FSMONITOR_VALID bit on each of those files and caused\n> \t# the worktree scan to visit them and mark them as modified.\n> \tgrep -q \" M AAA\" \"$PWD/subdir_case_wrong.out\" &&\n> \tgrep -q \" M zzz\" \"$PWD/subdir_case_wrong.out\" &&\n> -\n> -\t# Expect Breakage: with the case confusion, the \"(pos -3)\" causes\n> -\t# the client to not clear the CE_FSMONITOR_VALID bit and therefore\n> -\t# status will not rescan the file and therefore not report it as dirty.\n> \tgrep -q \" M dir1/dir2/dir3/file3\" \"$PWD/subdir_case_wrong.out\"\n> '\n> \n> -test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n> +test_expect_success CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n> \ttest_when_finished \"stop_daemon_delete_repo file_case_wrong\" &&\n> \n> \tgit init file_case_wrong &&\n> @@ -1242,14 +1242,20 @@ test_expect_failure CASE_INSENSITIVE_FS 'fsmonitor file case wrong on disk' '\n> \tGIT_TRACE_FSMONITOR=\"$PWD/file_case_wrong-try3.log\" \\\n> \t\tgit -C file_case_wrong --no-optional-locks status --short \\\n> \t\t\t>\"$PWD/file_case_wrong-try3.out\" &&\n> +\n> +\t# Verify that we get a mapping event to correct the case.\n> +\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir3/FILE-3-A.*dir1/dir2/dir3/file-3-a\" \\\n> +\t\t\"$PWD/file_case_wrong-try3.log\" &&\n> +\tgrep -q \"fsmonitor_refresh_callback MAP:.*dir1/dir2/dir4/file-4-a.*dir1/dir2/dir4/FILE-4-A\" \\\n> +\t\t\"$PWD/file_case_wrong-try3.log\" &&\n> +\n> \t# FSEvents are in observed case.\n> \tgrep -q \"fsmonitor_refresh_callback.*FILE-3-A.*pos -3\" \"$PWD/file_case_wrong-try3.log\" &&\n> \tgrep -q \"fsmonitor_refresh_callback.*file-4-a.*pos -9\" \"$PWD/file_case_wrong-try3.log\" &&\n> \n> -\t# Expect Breakage: with the case confusion, the \"(pos-3)\" and\n> -\t# \"(pos -9)\" causes the client to not clear the CE_FSMONITOR_VALID\n> -\t# bit and therefore status will not rescan the files and therefore\n> -\t# not report them as dirty.\n> +\t# The refresh-callbacks should have caused \"git status\" to clear\n> +\t# the CE_FSMONITOR_VALID bit on each of those files and caused\n> +\t# the worktree scan to visit them and mark them as modified.\n> \tgrep -q \" M dir1/dir2/dir3/file-3-a\" \"$PWD/file_case_wrong-try3.out\" &&\n> \tgrep -q \" M dir1/dir2/dir4/FILE-4-A\" \"$PWD/file_case_wrong-try3.out\"\n> '\n> -- \n> gitgitgadget\n"},{"id":"490103","messageId":"xmqqa5nbs2wu.fsf@gitster.g","threadId":"60908","inReplyTo":"ZehofMaSZyUq8S1N@tanuki","subject":"Re: [PATCH v3 00/14] FSMonitor edge cases on case-insensitive file systems","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2024-03-06T17:09:21Z","receivedAt":"2024-03-06T17:09:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Patrick Steinhardt writes:\n\n> On Mon, Feb 26, 2024 at 09:39:11PM +0000, Jeff Hostetler via GitGitGadget wrote:\n>> Here is version 3. I think I have addressed the remaining comments.\n>> \n>> I cleaned up the test code to use the test_expect_failure at the beginning\n>> and squashed in the test_expect_success version of tests into the final\n>> commit in the series.\n>> \n>> I moved the invalidate_ce_fsm() commit earlier in the series, so that the\n>> final commit actually uses all of the up-to-this-point changes to fix the\n>> problem.\n>> \n>> I converted a few \"should not happens\" to BUG()s.\n>> \n>> Thanks to everyone for their time and attention reviewing this. Jeff\n>\n> I gave this whole patch series a read and didn't much to complain about.\n> There are a couple of nits, but none of them really require a reroll in\n> my opinion.\n>\n> Thanks!\n>\n> Patrick\n\nThanks.\n"},{"id":"490108","messageId":"feb7725d-877b-f738-5914-84e5cf26aa80@jeffhostetler.com","threadId":"60908","inReplyTo":"ZehofMaSZyUq8S1N@tanuki","subject":"Re: [PATCH v3 00/14] FSMonitor edge cases on case-insensitive file systems","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2024-03-06T18:10:41Z","receivedAt":"2024-03-06T18:20:10Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 3/6/24 7:58 AM, Patrick Steinhardt wrote:\n> On Mon, Feb 26, 2024 at 09:39:11PM +0000, Jeff Hostetler via GitGitGadget wrote:\n>> Here is version 3. I think I have addressed the remaining comments.\n>>\n>> I cleaned up the test code to use the test_expect_failure at the beginning\n>> and squashed in the test_expect_success version of tests into the final\n>> commit in the series.\n>>\n>> I moved the invalidate_ce_fsm() commit earlier in the series, so that the\n>> final commit actually uses all of the up-to-this-point changes to fix the\n>> problem.\n>>\n>> I converted a few \"should not happens\" to BUG()s.\n>>\n>> Thanks to everyone for their time and attention reviewing this. Jeff\n> \n> I gave this whole patch series a read and didn't much to complain about.\n> There are a couple of nits, but none of them really require a reroll in\n> my opinion.\n> \n> Thanks!\n> \n> Patrick\n\nThanks!!!\nJeff\n"}]}