{"thread":{"id":"55639","subject":"[PATCH 0/5] Directory traversal fixes","startedAt":"2021-05-07T04:05:07Z","lastAt":"2021-05-19T01:44:36Z","messageCount":90,"participants":["Elijah Newren via GitGitGadget","Eric Sunshine","Elijah Newren","Derrick Stolee","Jeff King","Derrick Stolee via GitGitGadget","Junio C Hamano","Philip Oakley","Jeff Hostetler"],"isPatch":true,"patchVersion":1,"patchTotal":5},"messages":[{"id":"423845","messageId":"a3bd253fa8e8ae47d19beb35327d8283ffa49289.1620360300.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.git.git.1620360300.gitgitgadget@gmail.com","subject":"[PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-07T04:04:56Z","receivedAt":"2021-05-07T04:05:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nPNPM is apparently creating deeply nested (but ignored) directory\nstructures; traversing them is costly performance-wise, unnecessary, and\nin some cases is even throwing warnings/errors because the paths are too\nlong to handle on various platforms. Add a testcase that demonstrates\nthis problem.\n\nInitial-test-by: Jason Gore \nHelped-by: brian m. carlson \nSigned-off-by: Elijah Newren \n---\n t/t7300-clean.sh | 40 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 40 insertions(+)\n\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex a74816ca8b46..5f1dc397c11e 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,4 +746,44 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_failure 'avoid traversing into ignored directories' '\n+\ttest_when_finished rm -f output error &&\n+\ttest_create_repo avoid-traversing-deep-hierarchy &&\n+\t(\n+\t\tcd avoid-traversing-deep-hierarchy &&\n+\n+\t\t>directory-random-file.txt &&\n+\t\t# Put this file under directory400/directory399/.../directory1/\n+\t\tdepth=400 &&\n+\t\tfor x in $(test_seq 1 $depth); do\n+\t\t\tmkdir \"tmpdirectory$x\" &&\n+\t\t\tmv directory* \"tmpdirectory$x\" &&\n+\t\t\tmv \"tmpdirectory$x\" \"directory$x\"\n+\t\tdone &&\n+\n+\t\tgit clean -ffdxn -e directory$depth >../output 2>../error &&\n+\n+\t\ttest_must_be_empty ../output &&\n+\t\t# We especially do not want things like\n+\t\t# \"warning: could not open directory \"\n+\t\t# appearing in the error output. It is true that directories\n+\t\t# that are too long cannot be opened, but we should not be\n+\t\t# recursing into those directories anyway since the very first\n+\t\t# level is ignored.\n+\t\ttest_must_be_empty ../error &&\n+\n+\t\t# alpine-linux-musl fails to \"rm -rf\" a directory with such\n+\t\t# a deeply nested hierarchy. Help it out by deleting the\n+\t\t# leading directories ourselves. Super slow, but, what else\n+\t\t# can we do? Without this, we will hit a\n+\t\t# error: Tests passed but test cleanup failed; aborting\n+\t\t# so do this ugly manual cleanup...\n+\t\twhile test ! -f directory-random-file.txt; do\n+\t\t\tname=$(ls -d directory*) &&\n+\t\t\tmv $name/* . &&\n+\t\t\trmdir $name\n+\t\tdone\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"423844","messageId":"pull.1020.git.git.1620360300.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":null,"subject":"[PATCH 0/5] Directory traversal fixes","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-07T04:04:55Z","receivedAt":"2021-05-07T04:05:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patchset fixes a few directory traversal issues, where fill_directory()\nwould traverse into directories that it shouldn't and not traverse into\ndirectories that it should. One of these issues was reported recently on\nthis list[1], another was found at $DAYJOB.\n\nThe fifth patch might have backward compatibility implications, but is easy\nto review. Even if the logic in dir.c makes your eyes glaze over, at least\ntake a look at the fifth patch.\n\nAlso, if anyone has any ideas about a better place to put the \"Some\nsidenotes\" from the third commit message rather than keeping them in a\nrandom commit message, that might be helpful too.\n\n[1] See\nhttps://lore.kernel.org/git/DM6PR00MB06829EC5B85E0C5AC595004E894E9@DM6PR00MB0682.namprd00.prod.outlook.com/\nor alternatively https://github.com/git-for-windows/git/issues/2732.\n\nElijah Newren (5):\n t7300: add testcase showing unnecessary traversal into ignored\n directory\n t3001, t7300: add testcase showcasing missed directory traversal\n dir: avoid unnecessary traversal into ignored directory\n dir: traverse into untracked directories if they may have ignored\n subfiles\n [RFC] ls-files: error out on -i unless -o or -c are specified\n\n builtin/ls-files.c | 3 ++\n dir.c | 50 ++++++++++++++++---------\n t/t1306-xdg-files.sh | 2 +-\n t/t3001-ls-files-others-exclude.sh | 5 +++\n t/t3003-ls-files-exclude.sh | 4 +-\n t/t7300-clean.sh | 59 ++++++++++++++++++++++++++++++\n 6 files changed, 103 insertions(+), 20 deletions(-)\n\n\nbase-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1020%2Fnewren%2Fdirectory-traversal-fixes-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1020/newren/directory-traversal-fixes-v1\nPull-Request: https://github.com/git/git/pull/1020\n-- \ngitgitgadget\n"},{"id":"423846","messageId":"aa3a41e26eca0d3c9668b91ea9f1cc8f8faccdea.1620360300.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.git.git.1620360300.gitgitgadget@gmail.com","subject":"[PATCH 2/5] t3001, t7300: add testcase showcasing missed directory traversal","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-07T04:04:57Z","receivedAt":"2021-05-07T04:05:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn the last commit, we added a testcase showing that the directory\ntraversal machinery sometimes traverses into directories unnecessarily.\nHere we show that there are cases where it does the opposite: it does\nnot traverse into directories, despite those directories having\nimportant files that need to be flagged.\n\nAdd a testcase showing that `git ls-files -o -i --directory` can omit\nsome of the files it should be listing, and another showing that `git\nclean -fX` can fail to clean out some of the expected files.\n\nSigned-off-by: Elijah Newren \n---\n t/t3001-ls-files-others-exclude.sh | 5 +++++\n t/t7300-clean.sh | 19 +++++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex 1ec7cb57c7a8..ac05d1a17931 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,6 +292,11 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+\t# Expectation same as previous test\n+\tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n+\ttest_cmp expect actual\n+'\n \n test_expect_success 'ls-files with \"**\" patterns and no slashes' '\n \tgit ls-files -o -i --exclude \"one**a.1\" >actual &&\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 5f1dc397c11e..337f9af1d74b 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -786,4 +786,23 @@ test_expect_failure 'avoid traversing into ignored directories' '\n \t)\n '\n \n+test_expect_failure 'traverse into directories that may have ignored entries' '\n+\ttest_when_finished rm -f output &&\n+\ttest_create_repo need-to-traverse-into-hierarchy &&\n+\t(\n+\t\tcd need-to-traverse-into-hierarchy &&\n+\t\tmkdir -p modules/foobar/src/generated &&\n+\t\t> modules/foobar/src/generated/code.c &&\n+\t\t> modules/foobar/Makefile &&\n+\t\techo \"/modules/**/src/generated/\" >.gitignore &&\n+\n+\t\tgit clean -fX modules/foobar >../output &&\n+\n+\t\tgrep Removing ../output &&\n+\n+\t\ttest_path_is_missing modules/foobar/src/generated/code.c &&\n+\t\ttest_path_is_file modules/foobar/Makefile\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"423847","messageId":"3c3f6111da13c885b341053a477d857bd897708f.1620360300.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.git.git.1620360300.gitgitgadget@gmail.com","subject":"[PATCH 3/5] dir: avoid unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-07T04:04:58Z","receivedAt":"2021-05-07T04:05:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe show_other_directories case in treat_directory() tried to handle\nboth excludes and untracked files with the same logic, and mishandled\nboth the excludes and the untracked files in the process, in different\nways. Split that logic apart, and then focus on the logic for the\nexcludes; a subsequent commit will address the logic for untracked\nfiles.\n\nFor show_other_directories, an excluded directory means that\nevery path underneath that directory will also be excluded. Given that\nthe calling code requested to just show directories when everything\nunder a directory had the same state (that's what the\n\"DIR_SHOW_OTHER_DIRECTORIES\" flag means), we generally do not need to\ntraverse into such directories and can just immediately mark them as\nignored (i.e. as path_excluded). The only reason we cannot just\nimmediately return path_excluded is the DIR_HIDE_EMPTY_DIRECTORIES flag\nand the possibility that the ignored directory is an empty directory.\nThe code previously treated DIR_SHOW_IGNORED_TOO in most cases as an\nexception as well, which was wrong. It can sometimes reduce the number\nof cases where we need to recurse (namely if\nDIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set), but should not be able\nto increase the number of cases where we need to recurse. Fix the logic\naccordingly.\n\nSome sidenotes about possible confusion with dir.c:\n\n* \"ignored\" often refers to an untracked ignore\", i.e. a file which is\n not tracked which matches one of the ignore/exclusion rules. But you\n can also have a \"tracked ignore\", a tracked file that happens to match\n one of the ignore/exclusion rules and which dir.c has to worry about\n since \"git ls-files -c -i\" is supposed to list them.\n\n* The dir code often uses \"ignored\" and \"excluded\" interchangeably,\n which you need to keep in mind while reading the code. Sadly, though,\n it can get very confusing since ignore rules can have exclusions, as\n in the last of the following .gitignore rules:\n .gitignore\n *~\n *.log\n !settings.log\n In the last entry above, (pathspec->items[3].magic & PATHSPEC_EXCLUDE)\n will be true due the the '!' negating the rule. Someone might refer\n to this as \"excluded\". That means the file 'settings.log' will not\n match, and thus not be ignored. So we won't return path_excluded for\n it. So it's an exclude rule that prevents the file from being an\n exclude. The non-excluded rules are the ones that result in files\n being excludes. Great fun, eh?\n\nSometimes it feels like dir.c needs its own glossary with its many\ndefinitions, including the multiply-defined terms.\n\nReported-by: Jason Gore \nSigned-off-by: Elijah Newren \n---\n dir.c | 44 +++++++++++++++++++++++++++++---------------\n t/t7300-clean.sh | 2 +-\n 2 files changed, 30 insertions(+), 16 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3474e67e8f3c..4b183749843e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1844,6 +1844,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t}\n \n \t/* This is the \"show_other_directories\" case */\n+\tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n \t * If we have a pathspec which could match something _below_ this\n@@ -1854,27 +1855,40 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tif (matches_how == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n \t\treturn path_recurse;\n \n+\t/* Special cases for where this directory is excluded/ignored */\n+\tif (excluded) {\n+\t\t/*\n+\t\t * In the show_other_directories case, if we're not\n+\t\t * hiding empty directories, there is no need to\n+\t\t * recurse into an ignored directory.\n+\t\t */\n+\t\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\t\treturn path_excluded;\n+\n+\t\t/*\n+\t\t * Even if we are hiding empty directories, we can still avoid\n+\t\t * recursing into ignored directories for DIR_SHOW_IGNORED_TOO\n+\t\t * if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n+\t\t */\n+\t\tif ((dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n+\t\t\treturn path_excluded;\n+\t}\n+\n \t/*\n-\t * Other than the path_recurse case immediately above, we only need\n-\t * to recurse into untracked/ignored directories if either of the\n-\t * following bits is set:\n+\t * Other than the path_recurse case above, we only need to\n+\t * recurse into untracked directories if either of the following\n+\t * bits is set:\n \t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n \t * there are ignored entries below)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n-\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n-\t\treturn excluded ? path_excluded : path_untracked;\n-\n-\t/*\n-\t * ...and even if DIR_SHOW_IGNORED_TOO is set, we can still avoid\n-\t * recursing into ignored directories if the path is excluded and\n-\t * DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n-\t */\n-\tif (excluded &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n-\t\treturn path_excluded;\n+\tif (!excluded &&\n+\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n+\t\treturn path_untracked;\n+\t}\n \n \t/*\n \t * Even if we don't want to know all the paths under an untracked or\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 337f9af1d74b..00e5fa35dae3 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,7 +746,7 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_failure 'avoid traversing into ignored directories' '\n+test_expect_success 'avoid traversing into ignored directories' '\n \ttest_when_finished rm -f output error &&\n \ttest_create_repo avoid-traversing-deep-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"423848","messageId":"fad048339b8131391f9f8952e215849a21145ea8.1620360300.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.git.git.1620360300.gitgitgadget@gmail.com","subject":"[PATCH 4/5] dir: traverse into untracked directories if they may have ignored subfiles","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-07T04:04:59Z","receivedAt":"2021-05-07T04:05:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA directory that is untracked does not imply that all files under it\nshould be categorized as untracked; in particular, if the caller is\ninterested in ignored files, many files or directories underneath the\nuntracked directory may be ignored. We previously partially handled\nthis right with DIR_SHOW_IGNORED_TOO, but missed DIR_SHOW_IGNORED. It\nwas not obvious, though, because the logic for untracked and excluded\nfiles had been fused together making it harder to reason about. The\nprevious commit split that logic out, making it easier to notice that\nDIR_SHOW_IGNORED was missing. Add it.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 10 ++++++----\n t/t3001-ls-files-others-exclude.sh | 2 +-\n t/t7300-clean.sh | 2 +-\n 3 files changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 4b183749843e..3beb8e17a839 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1877,15 +1877,17 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/*\n \t * Other than the path_recurse case above, we only need to\n-\t * recurse into untracked directories if either of the following\n+\t * recurse into untracked directories if any of the following\n \t * bits is set:\n-\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n-\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED (because then we need to determine if\n+\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED_TOO (same as above)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n \tif (!excluded &&\n-\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t !(dir->flags & (DIR_SHOW_IGNORED |\n+\t\t\t DIR_SHOW_IGNORED_TOO |\n \t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n \t\treturn path_untracked;\n \t}\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex ac05d1a17931..516c95ea0e82 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,7 +292,7 @@ EOF\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+test_expect_success 'ls-files with \"**\" patterns and --directory' '\n \t# Expectation same as previous test\n \tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 00e5fa35dae3..c2a3b7b6a52b 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -786,7 +786,7 @@ test_expect_success 'avoid traversing into ignored directories' '\n \t)\n '\n \n-test_expect_failure 'traverse into directories that may have ignored entries' '\n+test_expect_success 'traverse into directories that may have ignored entries' '\n \ttest_when_finished rm -f output &&\n \ttest_create_repo need-to-traverse-into-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"423849","messageId":"3d8dd00ccd10d6b7ce9f928f31f8a04e885730fc.1620360300.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.git.git.1620360300.gitgitgadget@gmail.com","subject":"[PATCH 5/5] [RFC] ls-files: error out on -i unless -o or -c are specified","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-07T04:05:00Z","receivedAt":"2021-05-07T04:05:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nls-files --ignored can be used together with either --others or\n--cached. After being perplexed for a bit and digging in to the code, I\nassumed that ls-files -i was just broken and not printing anything and\nhad a nice patch ready to submit when I finally realized that -i can be\nused with --cached to find tracked ignores.\n\nWhile that was a mistake on my part, and a careful reading of the\ndocumentation could have made this more clear, I suspect this is an\nerror others are likely to make as well. In fact, of two uses in our\ntestsuite, I believe one of the two did make this error. In t1306.13,\nthere are NO tracked files, and all the excludes built up and used in\nthat test and in previous tests thus have to be about untracked files.\nHowever, since they were looking for an empty result, the mistake went\nunnoticed as their erroneous command also just happened to give an empty\nanswer.\n\n-i will most the time be used with -o, which would suggest we could just\nmake -i imply -o in the absence of either a -o or -c, but that would be\na backward incompatible break. Instead, let's just flag -i without\neither a -o or -c as an error, and update the two relevant testcases to\nspecify their intent.\n\nSigned-off-by: Elijah Newren \n---\n builtin/ls-files.c | 3 +++\n t/t1306-xdg-files.sh | 2 +-\n t/t3003-ls-files-exclude.sh | 4 ++--\n 3 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 60a2913a01e9..9f74b1ab2e69 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -748,6 +748,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tif (pathspec.nr && error_unmatch)\n \t\tps_matched = xcalloc(pathspec.nr, 1);\n \n+\tif ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached)\n+\t\tdie(\"ls-files --ignored is usually used with --others, but --cached is the default. Please specify which you want.\");\n+\n \tif ((dir.flags & DIR_SHOW_IGNORED) && !exc_given)\n \t\tdie(\"ls-files --ignored needs some exclude pattern\");\n \ndiff --git a/t/t1306-xdg-files.sh b/t/t1306-xdg-files.sh\nindex dd87b43be1a6..40d3c42618c0 100755\n--- a/t/t1306-xdg-files.sh\n+++ b/t/t1306-xdg-files.sh\n@@ -116,7 +116,7 @@ test_expect_success 'Exclusion in a non-XDG global ignore file' '\n test_expect_success 'Checking XDG ignore file when HOME is unset' '\n \t(sane_unset HOME &&\n \t git config --unset core.excludesfile &&\n-\t git ls-files --exclude-standard --ignored >actual) &&\n+\t git ls-files --exclude-standard --ignored --others >actual) &&\n \ttest_must_be_empty actual\n '\n \ndiff --git a/t/t3003-ls-files-exclude.sh b/t/t3003-ls-files-exclude.sh\nindex d5ec333131f9..c41c4f046abf 100755\n--- a/t/t3003-ls-files-exclude.sh\n+++ b/t/t3003-ls-files-exclude.sh\n@@ -29,11 +29,11 @@ test_expect_success 'add file to gitignore' '\n '\n check_all_output\n \n-test_expect_success 'ls-files -i lists only tracked-but-ignored files' '\n+test_expect_success 'ls-files -i -c lists only tracked-but-ignored files' '\n \techo content >other-file &&\n \tgit add other-file &&\n \techo file >expect &&\n-\tgit ls-files -i --exclude-standard >output &&\n+\tgit ls-files -i -c --exclude-standard >output &&\n \ttest_cmp expect output\n '\n \n-- \ngitgitgadget\n"},{"id":"423852","messageId":"CAPig+cTzvKhUaNEXwRAVwV1Vkb7jpHNkcOytEPq0Gg33TrJXNg@mail.gmail.com","threadId":"55639","inReplyTo":"a3bd253fa8e8ae47d19beb35327d8283ffa49289.1620360300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-05-07T04:27:39Z","receivedAt":"2021-05-07T04:32:45Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, May 7, 2021 at 12:05 AM Elijah Newren via GitGitGadget\n wrote:\n> PNPM is apparently creating deeply nested (but ignored) directory\n> structures; traversing them is costly performance-wise, unnecessary, and\n> in some cases is even throwing warnings/errors because the paths are too\n> long to handle on various platforms. Add a testcase that demonstrates\n> this problem.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\n> @@ -746,4 +746,44 @@ test_expect_success 'clean untracked paths by pathspec' '\n> +test_expect_failure 'avoid traversing into ignored directories' '\n> + test_when_finished rm -f output error &&\n> + test_create_repo avoid-traversing-deep-hierarchy &&\n> + (\n> + cd avoid-traversing-deep-hierarchy &&\n> +\n> + >directory-random-file.txt &&\n> + # Put this file under directory400/directory399/.../directory1/\n> + depth=400 &&\n> + for x in $(test_seq 1 $depth); do\n> + mkdir \"tmpdirectory$x\" &&\n> + mv directory* \"tmpdirectory$x\" &&\n> + mv \"tmpdirectory$x\" \"directory$x\"\n> + done &&\n\nIs this expensive/slow loop needed because you'd otherwise run afoul\nof command-line length limits on some platforms if you tried creating\nthe entire mess of directories with a single `mkdir -p`?\n\n> + git clean -ffdxn -e directory$depth >../output 2>../error &&\n> +\n> + test_must_be_empty ../output &&\n> + # We especially do not want things like\n> + # \"warning: could not open directory \"\n> + # appearing in the error output. It is true that directories\n> + # that are too long cannot be opened, but we should not be\n> + # recursing into those directories anyway since the very first\n> + # level is ignored.\n> + test_must_be_empty ../error &&\n> +\n> + # alpine-linux-musl fails to \"rm -rf\" a directory with such\n> + # a deeply nested hierarchy. Help it out by deleting the\n> + # leading directories ourselves. Super slow, but, what else\n> + # can we do? Without this, we will hit a\n> + # error: Tests passed but test cleanup failed; aborting\n> + # so do this ugly manual cleanup...\n> + while test ! -f directory-random-file.txt; do\n> + name=$(ls -d directory*) &&\n> + mv $name/* . &&\n> + rmdir $name\n> + done\n\nShouldn't this cleanup loop be under the control of\ntest_when_finished() to ensure it is invoked regardless of how the\ntest exits?\n\n> + )\n> +'\n"},{"id":"423855","messageId":"CABPp-BGzUBF8S8t3nixi3TCkiBe7oS1fQ5cNCozicb20iZxshA@mail.gmail.com","threadId":"55639","inReplyTo":"CAPig+cTzvKhUaNEXwRAVwV1Vkb7jpHNkcOytEPq0Gg33TrJXNg@mail.gmail.com","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-07T05:00:49Z","receivedAt":"2021-05-07T05:01:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, May 6, 2021 at 9:27 PM Eric Sunshine wrote:\n>\n> On Fri, May 7, 2021 at 12:05 AM Elijah Newren via GitGitGadget\n> wrote:\n> > PNPM is apparently creating deeply nested (but ignored) directory\n> > structures; traversing them is costly performance-wise, unnecessary, and\n> > in some cases is even throwing warnings/errors because the paths are too\n> > long to handle on various platforms. Add a testcase that demonstrates\n> > this problem.\n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\n> > @@ -746,4 +746,44 @@ test_expect_success 'clean untracked paths by pathspec' '\n> > +test_expect_failure 'avoid traversing into ignored directories' '\n> > + test_when_finished rm -f output error &&\n> > + test_create_repo avoid-traversing-deep-hierarchy &&\n> > + (\n> > + cd avoid-traversing-deep-hierarchy &&\n> > +\n> > + >directory-random-file.txt &&\n> > + # Put this file under directory400/directory399/.../directory1/\n> > + depth=400 &&\n> > + for x in $(test_seq 1 $depth); do\n> > + mkdir \"tmpdirectory$x\" &&\n> > + mv directory* \"tmpdirectory$x\" &&\n> > + mv \"tmpdirectory$x\" \"directory$x\"\n> > + done &&\n>\n> Is this expensive/slow loop needed because you'd otherwise run afoul\n> of command-line length limits on some platforms if you tried creating\n> the entire mess of directories with a single `mkdir -p`?\n\nThe whole point is creating a path long enough that it runs afoul of\nlimits, yes.\n\nIf we had an alternative way to check whether dir.c actually recursed\ninto a directory, then I could dispense with this and just have a\nsingle directory (and it could be named a single character long for\nthat matter too), but I don't know of a good way to do that. (Some\npossiibilities I considered along that route are mentioned at\nhttps://lore.kernel.org/git/CABPp-BF3e+MWQAGb6ER7d5jqjcV=kYqQ2stM_oDyaqvonPPPSw@mail.gmail.com/)\n\n> > + git clean -ffdxn -e directory$depth >../output 2>../error &&\n> > +\n> > + test_must_be_empty ../output &&\n> > + # We especially do not want things like\n> > + # \"warning: could not open directory \"\n> > + # appearing in the error output. It is true that directories\n> > + # that are too long cannot be opened, but we should not be\n> > + # recursing into those directories anyway since the very first\n> > + # level is ignored.\n> > + test_must_be_empty ../error &&\n> > +\n> > + # alpine-linux-musl fails to \"rm -rf\" a directory with such\n> > + # a deeply nested hierarchy. Help it out by deleting the\n> > + # leading directories ourselves. Super slow, but, what else\n> > + # can we do? Without this, we will hit a\n> > + # error: Tests passed but test cleanup failed; aborting\n> > + # so do this ugly manual cleanup...\n> > + while test ! -f directory-random-file.txt; do\n> > + name=$(ls -d directory*) &&\n> > + mv $name/* . &&\n> > + rmdir $name\n> > + done\n>\n> Shouldn't this cleanup loop be under the control of\n> test_when_finished() to ensure it is invoked regardless of how the\n> test exits?\n\nI thought about that, but if the test fails, it seems nicer to leave\neverything behind so it can be inspected. It's similar to test_done,\nwhich will only delete the $TRASH_DIRECTORY if all the tests passed.\nSo no, I don't think this should be under the control of\ntest_when_finished.\n"},{"id":"423858","messageId":"CAPig+cR3Z1dy8ibLEr+64C57dgrivPbgTveirnmSt-5D1wo59Q@mail.gmail.com","threadId":"55639","inReplyTo":"CABPp-BGzUBF8S8t3nixi3TCkiBe7oS1fQ5cNCozicb20iZxshA@mail.gmail.com","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-05-07T05:31:57Z","receivedAt":"2021-05-07T05:32:11Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, May 7, 2021 at 1:01 AM Elijah Newren wrote:\n> On Thu, May 6, 2021 at 9:27 PM Eric Sunshine wrote:\n> > Is this expensive/slow loop needed because you'd otherwise run afoul\n> > of command-line length limits on some platforms if you tried creating\n> > the entire mess of directories with a single `mkdir -p`?\n>\n> The whole point is creating a path long enough that it runs afoul of\n> limits, yes.\n>\n> If we had an alternative way to check whether dir.c actually recursed\n> into a directory, then I could dispense with this and just have a\n> single directory (and it could be named a single character long for\n> that matter too), but I don't know of a good way to do that. (Some\n> possiibilities I considered along that route are mentioned at\n> https://lore.kernel.org/git/CABPp-BF3e+MWQAGb6ER7d5jqjcV=kYqQ2stM_oDyaqvonPPPSw@mail.gmail.com/)\n\nThanks, I read that exchange (of course) immediately after sending the\nabove question.\n\n> > > + while test ! -f directory-random-file.txt; do\n> > > + name=$(ls -d directory*) &&\n> > > + mv $name/* . &&\n> > > + rmdir $name\n> > > + done\n> >\n> > Shouldn't this cleanup loop be under the control of\n> > test_when_finished() to ensure it is invoked regardless of how the\n> > test exits?\n>\n> I thought about that, but if the test fails, it seems nicer to leave\n> everything behind so it can be inspected. It's similar to test_done,\n> which will only delete the $TRASH_DIRECTORY if all the tests passed.\n> So no, I don't think this should be under the control of\n> test_when_finished.\n\nI may be confused, but I'm not following this reasoning. If you're\nusing `-i` to debug a failure within the test, then the\ntest_when_finished() cleanup actions won't be triggered anyhow\n(they're suppressed by `-i`), so everything will be left behind as\ndesired.\n\nThe problem with not placing this under control of\ntest_when_finished() is that, if something in the test proper does\nbreak, after the \"test failed\" message, you'll get the undesirable\nalpine-linux-musl behavior you explained in your earlier email where\ntest_done() bombs out.\n"},{"id":"423859","messageId":"CABPp-BFYpGYvy1wNdZBGObcRuVHMP=MxZtmikugA2hOUEHb-=w@mail.gmail.com","threadId":"55639","inReplyTo":"CAPig+cR3Z1dy8ibLEr+64C57dgrivPbgTveirnmSt-5D1wo59Q@mail.gmail.com","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-07T05:42:34Z","receivedAt":"2021-05-07T05:42:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, May 6, 2021 at 10:32 PM Eric Sunshine wrote:\n>\n> On Fri, May 7, 2021 at 1:01 AM Elijah Newren wrote:\n> > On Thu, May 6, 2021 at 9:27 PM Eric Sunshine wrote:\n> > > Is this expensive/slow loop needed because you'd otherwise run afoul\n> > > of command-line length limits on some platforms if you tried creating\n> > > the entire mess of directories with a single `mkdir -p`?\n> >\n> > The whole point is creating a path long enough that it runs afoul of\n> > limits, yes.\n> >\n> > If we had an alternative way to check whether dir.c actually recursed\n> > into a directory, then I could dispense with this and just have a\n> > single directory (and it could be named a single character long for\n> > that matter too), but I don't know of a good way to do that. (Some\n> > possiibilities I considered along that route are mentioned at\n> > https://lore.kernel.org/git/CABPp-BF3e+MWQAGb6ER7d5jqjcV=kYqQ2stM_oDyaqvonPPPSw@mail.gmail.com/)\n>\n> Thanks, I read that exchange (of course) immediately after sending the\n> above question.\n>\n> > > > + while test ! -f directory-random-file.txt; do\n> > > > + name=$(ls -d directory*) &&\n> > > > + mv $name/* . &&\n> > > > + rmdir $name\n> > > > + done\n> > >\n> > > Shouldn't this cleanup loop be under the control of\n> > > test_when_finished() to ensure it is invoked regardless of how the\n> > > test exits?\n> >\n> > I thought about that, but if the test fails, it seems nicer to leave\n> > everything behind so it can be inspected. It's similar to test_done,\n> > which will only delete the $TRASH_DIRECTORY if all the tests passed.\n> > So no, I don't think this should be under the control of\n> > test_when_finished.\n>\n> I may be confused, but I'm not following this reasoning. If you're\n> using `-i` to debug a failure within the test, then the\n> test_when_finished() cleanup actions won't be triggered anyhow\n> (they're suppressed by `-i`), so everything will be left behind as\n> desired.\n\nI didn't know that about --immediate. It's good to know. However,\nnot all debugging is done with -i; someone can also just run the\ntestsuite expecting everything to pass, see a failure, and then decide\nto go look around (and then maybe re-run with -i if the initial\nlooking around isn't clear). I do that every once in a while.\n\n> The problem with not placing this under control of\n> test_when_finished() is that, if something in the test proper does\n> break, after the \"test failed\" message, you'll get the undesirable\n> alpine-linux-musl behavior you explained in your earlier email where\n> test_done() bombs out.\n\nUnless I'm misunderstanding the test_done() code (I'm looking at\ntest-lib.sh, lines 1149-1183), test_done() only bombs out when it\ntries to \"rm -rf $TRASH_DIRECTORY\", and it only runs that command if\nthere are 0 test failures (see test-lib.sh, lines 1149-1183). So, if\nsomething in the test proper does break, that by itself will prevent\ntest_done() from bombing out.\n"},{"id":"423860","messageId":"CAPig+cTrjcPjv8jva_ORbuSxm6XBumA1CZNcaDYYqqzTH7JJzw@mail.gmail.com","threadId":"55639","inReplyTo":"CABPp-BFYpGYvy1wNdZBGObcRuVHMP=MxZtmikugA2hOUEHb-=w@mail.gmail.com","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-05-07T05:56:34Z","receivedAt":"2021-05-07T05:56:50Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, May 7, 2021 at 1:42 AM Elijah Newren wrote:\n> On Thu, May 6, 2021 at 10:32 PM Eric Sunshine wrote:\n> > I may be confused, but I'm not following this reasoning. If you're\n> > using `-i` to debug a failure within the test, then the\n> > test_when_finished() cleanup actions won't be triggered anyhow\n> > (they're suppressed by `-i`), so everything will be left behind as\n> > desired.\n>\n> I didn't know that about --immediate. It's good to know. However,\n> not all debugging is done with -i; someone can also just run the\n> testsuite expecting everything to pass, see a failure, and then decide\n> to go look around (and then maybe re-run with -i if the initial\n> looking around isn't clear). I do that every once in a while.\n\nThat's certainly an approach, and it's made easier when each test\ncreates its own repo (as the tests you write typically do).\n\nIn general. though, the majority of Git test scripts run all their\ntests in a single repo (per test script), with the result that state\nfrom a failed test is very frequently clobbered by subsequent tests,\nwhich is why --immediate is so useful (it stops the script as soon as\none test fails, so the test state is preserved as well as it can be).\nDue to the \"clobbering\" problem, I don't think I've ever tried\ndebugging a failed test without using --immediate.\n\n> > The problem with not placing this under control of\n> > test_when_finished() is that, if something in the test proper does\n> > break, after the \"test failed\" message, you'll get the undesirable\n> > alpine-linux-musl behavior you explained in your earlier email where\n> > test_done() bombs out.\n>\n> Unless I'm misunderstanding the test_done() code (I'm looking at\n> test-lib.sh, lines 1149-1183), test_done() only bombs out when it\n> tries to \"rm -rf $TRASH_DIRECTORY\", and it only runs that command if\n> there are 0 test failures (see test-lib.sh, lines 1149-1183). So, if\n> something in the test proper does break, that by itself will prevent\n> test_done() from bombing out.\n\nI see what you're saying. Okay.\n"},{"id":"423875","messageId":"3343338e-9262-dfd1-90ef-36908c91dad4@gmail.com","threadId":"55639","inReplyTo":"pull.1020.git.git.1620360300.gitgitgadget@gmail.com","subject":"[PATCH 6/5] dir: update stale description of treat_directory()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-07T16:22:50Z","receivedAt":"2021-05-07T16:22:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/7/2021 12:04 AM, Elijah Newren via GitGitGadget wrote:\n> This patchset fixes a few directory traversal issues, where fill_directory()\n> would traverse into directories that it shouldn't and not traverse into\n> directories that it should. One of these issues was reported recently on\n> this list[1], another was found at $DAYJOB.\n> \n> The fifth patch might have backward compatibility implications, but is easy\n> to review. Even if the logic in dir.c makes your eyes glaze over, at least\n> take a look at the fifth patch.\n\nMy eyes were glazing over, so I went to read the whole treat_directory()\nmethod and its related documentation comment. I found it to be a bit\nconfusing that it was referencing names that were deprecated 12 years ago.\n\nHere is a patch that you could add to this series to improve these\ncomments.\n\nThanks,\n-Stolee\n\n-- >8 --\n\nFrom 587a94ac396c969b6e7734ee46afeac20e87ccb9 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Fri, 7 May 2021 12:14:13 -0400\nSubject: [PATCH] dir: update stale description of treat_directory()\n\nThe documentation comment for treat_directory() was originally written\nin 095952 (Teach directory traversal about subprojects, 2007-04-11)\nwhich was before the 'struct dir_struct' split its bitfield of named\noptions into a 'flags' enum in 7c4c97c0 (Turn the flags in struct\ndir_struct into a single variable, 2009-02-16). When those flags\nchanged, the comment became stale, since members like\n'show_other_directories' transitioned into flags like\nDIR_SHOW_OTHER_DIRECTORIES.\n\nUpdate the comments for treat_directory() to use these flag names rather\nthan the old member names.\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3beb8e17a83..0a0138bc1aa 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1749,13 +1749,13 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n * Case 3: if we didn't have it in the index previously, we\n * have a few sub-cases:\n *\n- * (a) if \"show_other_directories\" is true, we show it as\n- * just a directory, unless \"hide_empty_directories\" is\n+ * (a) if DIR_SHOW_OTHER_DIRECTORIES flag is set, we show it as\n+ * just a directory, unless DIR_HIDE_EMPTY_DIRECTORIES is\n * also true, in which case we need to check if it contains any\n * untracked and / or ignored files.\n- * (b) if it looks like a git directory, and we don't have\n- * 'no_gitlinks' set we treat it as a gitlink, and show it\n- * as a directory.\n+ * (b) if it looks like a git directory and we don't have the\n+ * DIR_NO_GITLINKS flag, then we treat it as a gitlink, and\n+ * show it as a directory.\n * (c) otherwise, we recurse into it.\n */\n static enum path_treatment treat_directory(struct dir_struct *dir,\n@@ -1843,7 +1843,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_recurse;\n \t}\n \n-\t/* This is the \"show_other_directories\" case */\n \tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n@@ -1858,7 +1857,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* Special cases for where this directory is excluded/ignored */\n \tif (excluded) {\n \t\t/*\n-\t\t * In the show_other_directories case, if we're not\n+\t\t * If DIR_SHOW_OTHER_DIRECTORIES is set and we're not\n \t\t * hiding empty directories, there is no need to\n \t\t * recurse into an ignored directory.\n \t\t */\n-- \n2.31.1.vfs.0.0.80.gb082c853c0e\n\n\n"},{"id":"423876","messageId":"f99d7cd9-5e27-b5de-12fc-0a42be2d14ec@gmail.com","threadId":"55639","inReplyTo":"pull.1020.git.git.1620360300.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/5] Directory traversal fixes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-07T16:27:08Z","receivedAt":"2021-05-07T16:27:16Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/7/2021 12:04 AM, Elijah Newren via GitGitGadget wrote:\n> This patchset fixes a few directory traversal issues, where fill_directory()\n> would traverse into directories that it shouldn't and not traverse into\n> directories that it should. One of these issues was reported recently on\n> this list[1], another was found at $DAYJOB.\n> \n> The fifth patch might have backward compatibility implications, but is easy\n> to review. Even if the logic in dir.c makes your eyes glaze over, at least\n> take a look at the fifth patch.\n> \n> Also, if anyone has any ideas about a better place to put the \"Some\n> sidenotes\" from the third commit message rather than keeping them in a\n> random commit message, that might be helpful too.\n\nAs for your patches themselves, I can't claim to understand all the\ncomplicated details about how treat_directory() is working, but your\npatches are well organized and the new tests are the real proof that\nthis is working as intended.\n\nThanks for the attention to detail here.\n\n-Stolee\n"},{"id":"423878","messageId":"CABPp-BFudyBP8c_ROmxDEeuu5AfdoyVW8hNPYqVdFPFNofJnCQ@mail.gmail.com","threadId":"55639","inReplyTo":"3343338e-9262-dfd1-90ef-36908c91dad4@gmail.com","subject":"Re: [PATCH 6/5] dir: update stale description of treat_directory()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-07T17:57:41Z","receivedAt":"2021-05-07T17:57:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, May 7, 2021 at 9:22 AM Derrick Stolee wrote:\n>\n> On 5/7/2021 12:04 AM, Elijah Newren via GitGitGadget wrote:\n> > This patchset fixes a few directory traversal issues, where fill_directory()\n> > would traverse into directories that it shouldn't and not traverse into\n> > directories that it should. One of these issues was reported recently on\n> > this list[1], another was found at $DAYJOB.\n> >\n> > The fifth patch might have backward compatibility implications, but is easy\n> > to review. Even if the logic in dir.c makes your eyes glaze over, at least\n> > take a look at the fifth patch.\n>\n> My eyes were glazing over, so I went to read the whole treat_directory()\n> method and its related documentation comment. I found it to be a bit\n> confusing that it was referencing names that were deprecated 12 years ago.\n>\n> Here is a patch that you could add to this series to improve these\n> comments.\n>\n> Thanks,\n> -Stolee\n>\n> -- >8 --\n>\n> From 587a94ac396c969b6e7734ee46afeac20e87ccb9 Mon Sep 17 00:00:00 2001\n> From: Derrick Stolee \n> Date: Fri, 7 May 2021 12:14:13 -0400\n> Subject: [PATCH] dir: update stale description of treat_directory()\n>\n> The documentation comment for treat_directory() was originally written\n> in 095952 (Teach directory traversal about subprojects, 2007-04-11)\n> which was before the 'struct dir_struct' split its bitfield of named\n> options into a 'flags' enum in 7c4c97c0 (Turn the flags in struct\n> dir_struct into a single variable, 2009-02-16). When those flags\n> changed, the comment became stale, since members like\n> 'show_other_directories' transitioned into flags like\n> DIR_SHOW_OTHER_DIRECTORIES.\n>\n> Update the comments for treat_directory() to use these flag names rather\n> than the old member names.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> dir.c | 13 ++++++-------\n> 1 file changed, 6 insertions(+), 7 deletions(-)\n>\n> diff --git a/dir.c b/dir.c\n> index 3beb8e17a83..0a0138bc1aa 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1749,13 +1749,13 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n> * Case 3: if we didn't have it in the index previously, we\n> * have a few sub-cases:\n> *\n> - * (a) if \"show_other_directories\" is true, we show it as\n> - * just a directory, unless \"hide_empty_directories\" is\n> + * (a) if DIR_SHOW_OTHER_DIRECTORIES flag is set, we show it as\n> + * just a directory, unless DIR_HIDE_EMPTY_DIRECTORIES is\n> * also true, in which case we need to check if it contains any\n> * untracked and / or ignored files.\n> - * (b) if it looks like a git directory, and we don't have\n> - * 'no_gitlinks' set we treat it as a gitlink, and show it\n> - * as a directory.\n> + * (b) if it looks like a git directory and we don't have the\n> + * DIR_NO_GITLINKS flag, then we treat it as a gitlink, and\n> + * show it as a directory.\n> * (c) otherwise, we recurse into it.\n> */\n> static enum path_treatment treat_directory(struct dir_struct *dir,\n> @@ -1843,7 +1843,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> return path_recurse;\n> }\n>\n> - /* This is the \"show_other_directories\" case */\n> assert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n>\n> /*\n> @@ -1858,7 +1857,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> /* Special cases for where this directory is excluded/ignored */\n> if (excluded) {\n> /*\n> - * In the show_other_directories case, if we're not\n> + * If DIR_SHOW_OTHER_DIRECTORIES is set and we're not\n> * hiding empty directories, there is no need to\n> * recurse into an ignored directory.\n> */\n> --\n> 2.31.1.vfs.0.0.80.gb082c853c0e\n\nLooks good to me; I'll give it some more time for other comments to\ncome in, but when I re-roll, I'll include this patch of yours.\n"},{"id":"423889","messageId":"YJXHrXWVDSgg5TZw@coredump.intra.peff.net","threadId":"55639","inReplyTo":"CABPp-BGzUBF8S8t3nixi3TCkiBe7oS1fQ5cNCozicb20iZxshA@mail.gmail.com","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2021-05-07T23:05:17Z","receivedAt":"2021-05-07T23:05:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, May 06, 2021 at 10:00:49PM -0700, Elijah Newren wrote:\n\n> > > + >directory-random-file.txt &&\n> > > + # Put this file under directory400/directory399/.../directory1/\n> > > + depth=400 &&\n> > > + for x in $(test_seq 1 $depth); do\n> > > + mkdir \"tmpdirectory$x\" &&\n> > > + mv directory* \"tmpdirectory$x\" &&\n> > > + mv \"tmpdirectory$x\" \"directory$x\"\n> > > + done &&\n> >\n> > Is this expensive/slow loop needed because you'd otherwise run afoul\n> > of command-line length limits on some platforms if you tried creating\n> > the entire mess of directories with a single `mkdir -p`?\n> \n> The whole point is creating a path long enough that it runs afoul of\n> limits, yes.\n> \n> If we had an alternative way to check whether dir.c actually recursed\n> into a directory, then I could dispense with this and just have a\n> single directory (and it could be named a single character long for\n> that matter too), but I don't know of a good way to do that. (Some\n> possiibilities I considered along that route are mentioned at\n> https://lore.kernel.org/git/CABPp-BF3e+MWQAGb6ER7d5jqjcV=kYqQ2stM_oDyaqvonPPPSw@mail.gmail.com/)\n\nI don't have a better way of checking the dir.c behavior. But I think\nthe other half of Eric's question was: why can't we do this setup way\nmore efficiently with \"mkdir -p\"?\n\nI'd be suspicious that it would work portably because of the long path.\nBut I think the perl I showed earlier would create it in much less time:\n\n $ touch directory-file\n $ time sh -c '\n for x in $(seq 1 400)\n do\n mkdir tmpdirectory$x &&\n\tmv directory* tmpdirectory$x &&\n\tmv tmpdirectory$x directory$x\n done\n '\n real\t0m2.222s\n user\t0m1.481s\n sys\t\t0m0.816s\n\n $ time perl -e '\n for (reverse 1..400) {\n my $d = \"directory$_\";\n\tmkdir($d) and chdir($d) or die \"mkdir($d): $!\";\n }\n open(my $fh, \">\", \"some-file\");\n '\n real\t0m0.010s\n user\t0m0.001s\n sys\t\t0m0.009s\n\n-Peff\n"},{"id":"423890","messageId":"CAPig+cSP-5mvK0z=A21y4DQ_u07S+xHV3L+ADgaxzaK51yjX_w@mail.gmail.com","threadId":"55639","inReplyTo":"YJXHrXWVDSgg5TZw@coredump.intra.peff.net","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-05-07T23:15:38Z","receivedAt":"2021-05-07T23:15:59Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, May 7, 2021 at 7:05 PM Jeff King wrote:\n> I don't have a better way of checking the dir.c behavior. But I think\n> the other half of Eric's question was: why can't we do this setup way\n> more efficiently with \"mkdir -p\"?\n\nI didn't really have that other half-question, as I understood the\nportability ramifications. Rather, I just wanted to make sure the\nreason I thought the code was doing the for-loop-plus-mv dance was\nindeed correct, and that I wasn't overlooking something non-obvious. I\nwas also indirectly hinting that that bit of code might deserve an\nin-code comment explaining why the for-loop is there so that someone\ndoesn't come along in the future and try replacing it with `mkdir -p`.\n\n> I'd be suspicious that it would work portably because of the long path.\n> But I think the perl I showed earlier would create it in much less time:\n>\n> $ time perl -e '\n> for (reverse 1..400) {\n> my $d = \"directory$_\";\n> mkdir($d) and chdir($d) or die \"mkdir($d): $!\";\n> }\n> open(my $fh, \">\", \"some-file\");\n> '\n\nYep, this and your other Perl code snippet for removing the directory\nseemed much nicer than the far more expensive shell for-loop-plus-mv\n(especially for Windows folk).\n"},{"id":"423893","messageId":"CABPp-BEyUX8j-_rhNTyR5ZiqeLUWbBS-9q5+gEUWOEHpchambQ@mail.gmail.com","threadId":"55639","inReplyTo":"YJXHrXWVDSgg5TZw@coredump.intra.peff.net","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-08T00:04:24Z","receivedAt":"2021-05-08T00:04:39Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, May 7, 2021 at 4:05 PM Jeff King wrote:\n>\n> On Thu, May 06, 2021 at 10:00:49PM -0700, Elijah Newren wrote:\n>\n> > > > + >directory-random-file.txt &&\n> > > > + # Put this file under directory400/directory399/.../directory1/\n> > > > + depth=400 &&\n> > > > + for x in $(test_seq 1 $depth); do\n> > > > + mkdir \"tmpdirectory$x\" &&\n> > > > + mv directory* \"tmpdirectory$x\" &&\n> > > > + mv \"tmpdirectory$x\" \"directory$x\"\n> > > > + done &&\n> > >\n> > > Is this expensive/slow loop needed because you'd otherwise run afoul\n> > > of command-line length limits on some platforms if you tried creating\n> > > the entire mess of directories with a single `mkdir -p`?\n> >\n> > The whole point is creating a path long enough that it runs afoul of\n> > limits, yes.\n> >\n> > If we had an alternative way to check whether dir.c actually recursed\n> > into a directory, then I could dispense with this and just have a\n> > single directory (and it could be named a single character long for\n> > that matter too), but I don't know of a good way to do that. (Some\n> > possiibilities I considered along that route are mentioned at\n> > https://lore.kernel.org/git/CABPp-BF3e+MWQAGb6ER7d5jqjcV=kYqQ2stM_oDyaqvonPPPSw@mail.gmail.com/)\n>\n> I don't have a better way of checking the dir.c behavior. But I think\n> the other half of Eric's question was: why can't we do this setup way\n> more efficiently with \"mkdir -p\"?\n\nI think I figured it out. I now have the test simplified down to just:\n\ntest_expect_success 'avoid traversing into ignored directories' '\n test_when_finished rm -f output error trace.* &&\n test_create_repo avoid-traversing-deep-hierarchy &&\n (\n mkdir -p untracked/subdir/with/a &&\n >untracked/subdir/with/a/random-file.txt &&\n\n GIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n git clean -ffdxn -e untracked &&\n\n grep data.*read_directo.*visited ../trace.output \\\n | cut -d \"|\" -f 9 >../trace.relevant &&\n cat >../trace.expect <<-EOF &&\n directories-visited:1\n paths-visited:4\n EOF\n test_cmp ../trace.expect ../trace.relevant\n )\n'\n\nThis relies on a few extra changes to the code: (1) switching the\nexisting trace calls in dir.c over to using trace2 variants, and (2)\nadding two new counters (visited_directories and visited_paths) that\nare output using the trace2 framework. I'm a little unsure if I\nshould check the paths-visited counter (will some platform have\nadditional files in every directory besides '.' and '..'? Or not have\none of those?), but it is good to have it check that the code in this\ncase visits no directories other than the toplevel one (i.e. that\ndirectories-visited is 1).\n\nNew patches incoming shortly...\n"},{"id":"423894","messageId":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.git.git.1620360300.gitgitgadget@gmail.com","subject":"[PATCH v2 0/8] Directory traversal fixes","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T00:08:12Z","receivedAt":"2021-05-08T00:08:24Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patchset fixes a few directory traversal issues, where fill_directory()\nwould traverse into directories that it shouldn't and not traverse into\ndirectories that it should.\n\nChanges since v2:\n\n * Added a patch from Stolee to clean up some nearby comments that were made\n out-of-date 12 years ago\n * Added a new RFC patch that switches dir.c from using trace1 to trace2\n * Added a new RFC patch that adds directories-visited and paths-visited\n statistics using the trace2 output, and use that to vastly simplify (and\n accelerate) the t7300 testcase\n\nI'm curious what others think of the backward compatibility ramifications of\nthe RFC patches, patch 5 & patch 6. And whether my use of trace2 is clean,\nidiomatic, correct, etc. I've not used it before for things other than\nregion_enter & region_leave.\n\nAlso, if anyone has any ideas about a better place to put the \"Some\nsidenotes\" from the third commit message rather than keeping them in a\nrandom commit message, that might be helpful too.\n\n[1] See\nhttps://lore.kernel.org/git/DM6PR00MB06829EC5B85E0C5AC595004E894E9@DM6PR00MB0682.namprd00.prod.outlook.com/\nor alternatively https://github.com/git-for-windows/git/issues/2732.\n\nDerrick Stolee (1):\n dir: update stale description of treat_directory()\n\nElijah Newren (7):\n t7300: add testcase showing unnecessary traversal into ignored\n directory\n t3001, t7300: add testcase showcasing missed directory traversal\n dir: avoid unnecessary traversal into ignored directory\n dir: traverse into untracked directories if they may have ignored\n subfiles\n [RFC] ls-files: error out on -i unless -o or -c are specified\n [RFC] dir: convert trace calls to trace2 equivalents\n [RFC] dir: reported number of visited directories and paths with\n trace2\n\n builtin/ls-files.c | 3 +\n dir.c | 103 +++++++++------\n dir.h | 4 +\n t/t1306-xdg-files.sh | 2 +-\n t/t3001-ls-files-others-exclude.sh | 5 +\n t/t3003-ls-files-exclude.sh | 4 +-\n t/t7063-status-untracked-cache.sh | 194 ++++++++++++++++-------------\n t/t7300-clean.sh | 41 ++++++\n t/t7519-status-fsmonitor.sh | 8 +-\n 9 files changed, 238 insertions(+), 126 deletions(-)\n\n\nbase-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1020%2Fnewren%2Fdirectory-traversal-fixes-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1020/newren/directory-traversal-fixes-v2\nPull-Request: https://github.com/git/git/pull/1020\n\nRange-diff vs v1:\n\n 1: a3bd253fa8e8 = 1: a3bd253fa8e8 t7300: add testcase showing unnecessary traversal into ignored directory\n 2: aa3a41e26eca = 2: aa3a41e26eca t3001, t7300: add testcase showcasing missed directory traversal\n 3: 3c3f6111da13 = 3: 3c3f6111da13 dir: avoid unnecessary traversal into ignored directory\n 4: fad048339b81 = 4: fad048339b81 dir: traverse into untracked directories if they may have ignored subfiles\n 5: 3d8dd00ccd10 = 5: 3d8dd00ccd10 [RFC] ls-files: error out on -i unless -o or -c are specified\n -: ------------ > 6: 1d825dfdc70b dir: update stale description of treat_directory()\n -: ------------ > 7: 3a2394506a53 [RFC] dir: convert trace calls to trace2 equivalents\n -: ------------ > 8: fba4d65b78c7 [RFC] dir: reported number of visited directories and paths with trace2\n\n-- \ngitgitgadget\n"},{"id":"423895","messageId":"a3bd253fa8e8ae47d19beb35327d8283ffa49289.1620432500.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","subject":"[PATCH v2 1/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T00:08:13Z","receivedAt":"2021-05-08T00:08:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nPNPM is apparently creating deeply nested (but ignored) directory\nstructures; traversing them is costly performance-wise, unnecessary, and\nin some cases is even throwing warnings/errors because the paths are too\nlong to handle on various platforms. Add a testcase that demonstrates\nthis problem.\n\nInitial-test-by: Jason Gore \nHelped-by: brian m. carlson \nSigned-off-by: Elijah Newren \n---\n t/t7300-clean.sh | 40 ++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 40 insertions(+)\n\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex a74816ca8b46..5f1dc397c11e 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,4 +746,44 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_failure 'avoid traversing into ignored directories' '\n+\ttest_when_finished rm -f output error &&\n+\ttest_create_repo avoid-traversing-deep-hierarchy &&\n+\t(\n+\t\tcd avoid-traversing-deep-hierarchy &&\n+\n+\t\t>directory-random-file.txt &&\n+\t\t# Put this file under directory400/directory399/.../directory1/\n+\t\tdepth=400 &&\n+\t\tfor x in $(test_seq 1 $depth); do\n+\t\t\tmkdir \"tmpdirectory$x\" &&\n+\t\t\tmv directory* \"tmpdirectory$x\" &&\n+\t\t\tmv \"tmpdirectory$x\" \"directory$x\"\n+\t\tdone &&\n+\n+\t\tgit clean -ffdxn -e directory$depth >../output 2>../error &&\n+\n+\t\ttest_must_be_empty ../output &&\n+\t\t# We especially do not want things like\n+\t\t# \"warning: could not open directory \"\n+\t\t# appearing in the error output. It is true that directories\n+\t\t# that are too long cannot be opened, but we should not be\n+\t\t# recursing into those directories anyway since the very first\n+\t\t# level is ignored.\n+\t\ttest_must_be_empty ../error &&\n+\n+\t\t# alpine-linux-musl fails to \"rm -rf\" a directory with such\n+\t\t# a deeply nested hierarchy. Help it out by deleting the\n+\t\t# leading directories ourselves. Super slow, but, what else\n+\t\t# can we do? Without this, we will hit a\n+\t\t# error: Tests passed but test cleanup failed; aborting\n+\t\t# so do this ugly manual cleanup...\n+\t\twhile test ! -f directory-random-file.txt; do\n+\t\t\tname=$(ls -d directory*) &&\n+\t\t\tmv $name/* . &&\n+\t\t\trmdir $name\n+\t\tdone\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"423896","messageId":"3c3f6111da13c885b341053a477d857bd897708f.1620432500.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","subject":"[PATCH v2 3/8] dir: avoid unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T00:08:15Z","receivedAt":"2021-05-08T00:08:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe show_other_directories case in treat_directory() tried to handle\nboth excludes and untracked files with the same logic, and mishandled\nboth the excludes and the untracked files in the process, in different\nways. Split that logic apart, and then focus on the logic for the\nexcludes; a subsequent commit will address the logic for untracked\nfiles.\n\nFor show_other_directories, an excluded directory means that\nevery path underneath that directory will also be excluded. Given that\nthe calling code requested to just show directories when everything\nunder a directory had the same state (that's what the\n\"DIR_SHOW_OTHER_DIRECTORIES\" flag means), we generally do not need to\ntraverse into such directories and can just immediately mark them as\nignored (i.e. as path_excluded). The only reason we cannot just\nimmediately return path_excluded is the DIR_HIDE_EMPTY_DIRECTORIES flag\nand the possibility that the ignored directory is an empty directory.\nThe code previously treated DIR_SHOW_IGNORED_TOO in most cases as an\nexception as well, which was wrong. It can sometimes reduce the number\nof cases where we need to recurse (namely if\nDIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set), but should not be able\nto increase the number of cases where we need to recurse. Fix the logic\naccordingly.\n\nSome sidenotes about possible confusion with dir.c:\n\n* \"ignored\" often refers to an untracked ignore\", i.e. a file which is\n not tracked which matches one of the ignore/exclusion rules. But you\n can also have a \"tracked ignore\", a tracked file that happens to match\n one of the ignore/exclusion rules and which dir.c has to worry about\n since \"git ls-files -c -i\" is supposed to list them.\n\n* The dir code often uses \"ignored\" and \"excluded\" interchangeably,\n which you need to keep in mind while reading the code. Sadly, though,\n it can get very confusing since ignore rules can have exclusions, as\n in the last of the following .gitignore rules:\n .gitignore\n *~\n *.log\n !settings.log\n In the last entry above, (pathspec->items[3].magic & PATHSPEC_EXCLUDE)\n will be true due the the '!' negating the rule. Someone might refer\n to this as \"excluded\". That means the file 'settings.log' will not\n match, and thus not be ignored. So we won't return path_excluded for\n it. So it's an exclude rule that prevents the file from being an\n exclude. The non-excluded rules are the ones that result in files\n being excludes. Great fun, eh?\n\nSometimes it feels like dir.c needs its own glossary with its many\ndefinitions, including the multiply-defined terms.\n\nReported-by: Jason Gore \nSigned-off-by: Elijah Newren \n---\n dir.c | 44 +++++++++++++++++++++++++++++---------------\n t/t7300-clean.sh | 2 +-\n 2 files changed, 30 insertions(+), 16 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3474e67e8f3c..4b183749843e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1844,6 +1844,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t}\n \n \t/* This is the \"show_other_directories\" case */\n+\tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n \t * If we have a pathspec which could match something _below_ this\n@@ -1854,27 +1855,40 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tif (matches_how == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n \t\treturn path_recurse;\n \n+\t/* Special cases for where this directory is excluded/ignored */\n+\tif (excluded) {\n+\t\t/*\n+\t\t * In the show_other_directories case, if we're not\n+\t\t * hiding empty directories, there is no need to\n+\t\t * recurse into an ignored directory.\n+\t\t */\n+\t\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\t\treturn path_excluded;\n+\n+\t\t/*\n+\t\t * Even if we are hiding empty directories, we can still avoid\n+\t\t * recursing into ignored directories for DIR_SHOW_IGNORED_TOO\n+\t\t * if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n+\t\t */\n+\t\tif ((dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n+\t\t\treturn path_excluded;\n+\t}\n+\n \t/*\n-\t * Other than the path_recurse case immediately above, we only need\n-\t * to recurse into untracked/ignored directories if either of the\n-\t * following bits is set:\n+\t * Other than the path_recurse case above, we only need to\n+\t * recurse into untracked directories if either of the following\n+\t * bits is set:\n \t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n \t * there are ignored entries below)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n-\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n-\t\treturn excluded ? path_excluded : path_untracked;\n-\n-\t/*\n-\t * ...and even if DIR_SHOW_IGNORED_TOO is set, we can still avoid\n-\t * recursing into ignored directories if the path is excluded and\n-\t * DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n-\t */\n-\tif (excluded &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n-\t\treturn path_excluded;\n+\tif (!excluded &&\n+\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n+\t\treturn path_untracked;\n+\t}\n \n \t/*\n \t * Even if we don't want to know all the paths under an untracked or\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 337f9af1d74b..00e5fa35dae3 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,7 +746,7 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_failure 'avoid traversing into ignored directories' '\n+test_expect_success 'avoid traversing into ignored directories' '\n \ttest_when_finished rm -f output error &&\n \ttest_create_repo avoid-traversing-deep-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"423897","messageId":"aa3a41e26eca0d3c9668b91ea9f1cc8f8faccdea.1620432500.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","subject":"[PATCH v2 2/8] t3001, t7300: add testcase showcasing missed directory traversal","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T00:08:14Z","receivedAt":"2021-05-08T00:08:29Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn the last commit, we added a testcase showing that the directory\ntraversal machinery sometimes traverses into directories unnecessarily.\nHere we show that there are cases where it does the opposite: it does\nnot traverse into directories, despite those directories having\nimportant files that need to be flagged.\n\nAdd a testcase showing that `git ls-files -o -i --directory` can omit\nsome of the files it should be listing, and another showing that `git\nclean -fX` can fail to clean out some of the expected files.\n\nSigned-off-by: Elijah Newren \n---\n t/t3001-ls-files-others-exclude.sh | 5 +++++\n t/t7300-clean.sh | 19 +++++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex 1ec7cb57c7a8..ac05d1a17931 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,6 +292,11 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+\t# Expectation same as previous test\n+\tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n+\ttest_cmp expect actual\n+'\n \n test_expect_success 'ls-files with \"**\" patterns and no slashes' '\n \tgit ls-files -o -i --exclude \"one**a.1\" >actual &&\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 5f1dc397c11e..337f9af1d74b 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -786,4 +786,23 @@ test_expect_failure 'avoid traversing into ignored directories' '\n \t)\n '\n \n+test_expect_failure 'traverse into directories that may have ignored entries' '\n+\ttest_when_finished rm -f output &&\n+\ttest_create_repo need-to-traverse-into-hierarchy &&\n+\t(\n+\t\tcd need-to-traverse-into-hierarchy &&\n+\t\tmkdir -p modules/foobar/src/generated &&\n+\t\t> modules/foobar/src/generated/code.c &&\n+\t\t> modules/foobar/Makefile &&\n+\t\techo \"/modules/**/src/generated/\" >.gitignore &&\n+\n+\t\tgit clean -fX modules/foobar >../output &&\n+\n+\t\tgrep Removing ../output &&\n+\n+\t\ttest_path_is_missing modules/foobar/src/generated/code.c &&\n+\t\ttest_path_is_file modules/foobar/Makefile\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"423898","messageId":"fad048339b8131391f9f8952e215849a21145ea8.1620432500.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","subject":"[PATCH v2 4/8] dir: traverse into untracked directories if they may have ignored subfiles","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T00:08:16Z","receivedAt":"2021-05-08T00:08:30Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA directory that is untracked does not imply that all files under it\nshould be categorized as untracked; in particular, if the caller is\ninterested in ignored files, many files or directories underneath the\nuntracked directory may be ignored. We previously partially handled\nthis right with DIR_SHOW_IGNORED_TOO, but missed DIR_SHOW_IGNORED. It\nwas not obvious, though, because the logic for untracked and excluded\nfiles had been fused together making it harder to reason about. The\nprevious commit split that logic out, making it easier to notice that\nDIR_SHOW_IGNORED was missing. Add it.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 10 ++++++----\n t/t3001-ls-files-others-exclude.sh | 2 +-\n t/t7300-clean.sh | 2 +-\n 3 files changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 4b183749843e..3beb8e17a839 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1877,15 +1877,17 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/*\n \t * Other than the path_recurse case above, we only need to\n-\t * recurse into untracked directories if either of the following\n+\t * recurse into untracked directories if any of the following\n \t * bits is set:\n-\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n-\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED (because then we need to determine if\n+\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED_TOO (same as above)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n \tif (!excluded &&\n-\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t !(dir->flags & (DIR_SHOW_IGNORED |\n+\t\t\t DIR_SHOW_IGNORED_TOO |\n \t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n \t\treturn path_untracked;\n \t}\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex ac05d1a17931..516c95ea0e82 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,7 +292,7 @@ EOF\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+test_expect_success 'ls-files with \"**\" patterns and --directory' '\n \t# Expectation same as previous test\n \tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 00e5fa35dae3..c2a3b7b6a52b 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -786,7 +786,7 @@ test_expect_success 'avoid traversing into ignored directories' '\n \t)\n '\n \n-test_expect_failure 'traverse into directories that may have ignored entries' '\n+test_expect_success 'traverse into directories that may have ignored entries' '\n \ttest_when_finished rm -f output &&\n \ttest_create_repo need-to-traverse-into-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"423899","messageId":"3d8dd00ccd10d6b7ce9f928f31f8a04e885730fc.1620432501.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","subject":"[PATCH v2 5/8] [RFC] ls-files: error out on -i unless -o or -c are specified","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T00:08:17Z","receivedAt":"2021-05-08T00:08:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nls-files --ignored can be used together with either --others or\n--cached. After being perplexed for a bit and digging in to the code, I\nassumed that ls-files -i was just broken and not printing anything and\nhad a nice patch ready to submit when I finally realized that -i can be\nused with --cached to find tracked ignores.\n\nWhile that was a mistake on my part, and a careful reading of the\ndocumentation could have made this more clear, I suspect this is an\nerror others are likely to make as well. In fact, of two uses in our\ntestsuite, I believe one of the two did make this error. In t1306.13,\nthere are NO tracked files, and all the excludes built up and used in\nthat test and in previous tests thus have to be about untracked files.\nHowever, since they were looking for an empty result, the mistake went\nunnoticed as their erroneous command also just happened to give an empty\nanswer.\n\n-i will most the time be used with -o, which would suggest we could just\nmake -i imply -o in the absence of either a -o or -c, but that would be\na backward incompatible break. Instead, let's just flag -i without\neither a -o or -c as an error, and update the two relevant testcases to\nspecify their intent.\n\nSigned-off-by: Elijah Newren \n---\n builtin/ls-files.c | 3 +++\n t/t1306-xdg-files.sh | 2 +-\n t/t3003-ls-files-exclude.sh | 4 ++--\n 3 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 60a2913a01e9..9f74b1ab2e69 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -748,6 +748,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tif (pathspec.nr && error_unmatch)\n \t\tps_matched = xcalloc(pathspec.nr, 1);\n \n+\tif ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached)\n+\t\tdie(\"ls-files --ignored is usually used with --others, but --cached is the default. Please specify which you want.\");\n+\n \tif ((dir.flags & DIR_SHOW_IGNORED) && !exc_given)\n \t\tdie(\"ls-files --ignored needs some exclude pattern\");\n \ndiff --git a/t/t1306-xdg-files.sh b/t/t1306-xdg-files.sh\nindex dd87b43be1a6..40d3c42618c0 100755\n--- a/t/t1306-xdg-files.sh\n+++ b/t/t1306-xdg-files.sh\n@@ -116,7 +116,7 @@ test_expect_success 'Exclusion in a non-XDG global ignore file' '\n test_expect_success 'Checking XDG ignore file when HOME is unset' '\n \t(sane_unset HOME &&\n \t git config --unset core.excludesfile &&\n-\t git ls-files --exclude-standard --ignored >actual) &&\n+\t git ls-files --exclude-standard --ignored --others >actual) &&\n \ttest_must_be_empty actual\n '\n \ndiff --git a/t/t3003-ls-files-exclude.sh b/t/t3003-ls-files-exclude.sh\nindex d5ec333131f9..c41c4f046abf 100755\n--- a/t/t3003-ls-files-exclude.sh\n+++ b/t/t3003-ls-files-exclude.sh\n@@ -29,11 +29,11 @@ test_expect_success 'add file to gitignore' '\n '\n check_all_output\n \n-test_expect_success 'ls-files -i lists only tracked-but-ignored files' '\n+test_expect_success 'ls-files -i -c lists only tracked-but-ignored files' '\n \techo content >other-file &&\n \tgit add other-file &&\n \techo file >expect &&\n-\tgit ls-files -i --exclude-standard >output &&\n+\tgit ls-files -i -c --exclude-standard >output &&\n \ttest_cmp expect output\n '\n \n-- \ngitgitgadget\n\n"},{"id":"423900","messageId":"1d825dfdc70b8c658c4b6317310706bb6386f468.1620432501.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","subject":"[PATCH v2 6/8] dir: update stale description of treat_directory()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T00:08:18Z","receivedAt":"2021-05-08T00:08:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe documentation comment for treat_directory() was originally written\nin 095952 (Teach directory traversal about subprojects, 2007-04-11)\nwhich was before the 'struct dir_struct' split its bitfield of named\noptions into a 'flags' enum in 7c4c97c0 (Turn the flags in struct\ndir_struct into a single variable, 2009-02-16). When those flags\nchanged, the comment became stale, since members like\n'show_other_directories' transitioned into flags like\nDIR_SHOW_OTHER_DIRECTORIES.\n\nUpdate the comments for treat_directory() to use these flag names rather\nthan the old member names.\n\nSigned-off-by: Derrick Stolee \nReviewed-by: Elijah Newren \n---\n dir.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3beb8e17a839..0a0138bc1aa6 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1749,13 +1749,13 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n * Case 3: if we didn't have it in the index previously, we\n * have a few sub-cases:\n *\n- * (a) if \"show_other_directories\" is true, we show it as\n- * just a directory, unless \"hide_empty_directories\" is\n+ * (a) if DIR_SHOW_OTHER_DIRECTORIES flag is set, we show it as\n+ * just a directory, unless DIR_HIDE_EMPTY_DIRECTORIES is\n * also true, in which case we need to check if it contains any\n * untracked and / or ignored files.\n- * (b) if it looks like a git directory, and we don't have\n- * 'no_gitlinks' set we treat it as a gitlink, and show it\n- * as a directory.\n+ * (b) if it looks like a git directory and we don't have the\n+ * DIR_NO_GITLINKS flag, then we treat it as a gitlink, and\n+ * show it as a directory.\n * (c) otherwise, we recurse into it.\n */\n static enum path_treatment treat_directory(struct dir_struct *dir,\n@@ -1843,7 +1843,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_recurse;\n \t}\n \n-\t/* This is the \"show_other_directories\" case */\n \tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n@@ -1858,7 +1857,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* Special cases for where this directory is excluded/ignored */\n \tif (excluded) {\n \t\t/*\n-\t\t * In the show_other_directories case, if we're not\n+\t\t * If DIR_SHOW_OTHER_DIRECTORIES is set and we're not\n \t\t * hiding empty directories, there is no need to\n \t\t * recurse into an ignored directory.\n \t\t */\n-- \ngitgitgadget\n\n"},{"id":"423901","messageId":"3a2394506a536707788744a8b082077a017f8c8c.1620432501.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","subject":"[PATCH v2 7/8] [RFC] dir: convert trace calls to trace2 equivalents","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T00:08:19Z","receivedAt":"2021-05-08T00:08:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 34 ++++--\n t/t7063-status-untracked-cache.sh | 193 +++++++++++++++++-------------\n t/t7519-status-fsmonitor.sh | 8 +-\n 3 files changed, 135 insertions(+), 100 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 0a0138bc1aa6..23c71ab7e9a1 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2775,12 +2775,29 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \treturn root;\n }\n \n+static void trace2_read_directory_statistics(struct dir_struct *dir,\n+\t\t\t\t\t struct repository *repo)\n+{\n+\tif (!dir->untracked)\n+\t\treturn;\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"node-creation\", dir->untracked->dir_created);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"gitignore-invalidation\",\n+\t\t\t dir->untracked->gitignore_invalidated);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"directory-invalidation\",\n+\t\t\t dir->untracked->dir_invalidated);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"opendir\", dir->untracked->dir_opened);\n+}\n+\n int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t const char *path, int len, const struct pathspec *pathspec)\n {\n \tstruct untracked_cache_dir *untracked;\n \n-\ttrace_performance_enter();\n+\ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n \n \tif (has_symlink_leading_path(path, len)) {\n \t\ttrace_performance_leave(\"read directory %.*s\", len, path);\n@@ -2799,23 +2816,13 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tQSORT(dir->entries, dir->nr, cmp_dir_entry);\n \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n \n-\ttrace_performance_leave(\"read directory %.*s\", len, path);\n+\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n \tif (dir->untracked) {\n \t\tstatic int force_untracked_cache = -1;\n-\t\tstatic struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n \n \t\tif (force_untracked_cache < 0)\n \t\t\tforce_untracked_cache =\n \t\t\t\tgit_env_bool(\"GIT_FORCE_UNTRACKED_CACHE\", 0);\n-\t\ttrace_printf_key(&trace_untracked_stats,\n-\t\t\t\t \"node creation: %u\\n\"\n-\t\t\t\t \"gitignore invalidation: %u\\n\"\n-\t\t\t\t \"directory invalidation: %u\\n\"\n-\t\t\t\t \"opendir: %u\\n\",\n-\t\t\t\t dir->untracked->dir_created,\n-\t\t\t\t dir->untracked->gitignore_invalidated,\n-\t\t\t\t dir->untracked->dir_invalidated,\n-\t\t\t\t dir->untracked->dir_opened);\n \t\tif (force_untracked_cache &&\n \t\t\tdir->untracked == istate->untracked &&\n \t\t (dir->untracked->dir_opened ||\n@@ -2826,6 +2833,9 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t\tFREE_AND_NULL(dir->untracked);\n \t\t}\n \t}\n+\n+\tif (trace2_is_enabled())\n+\t\ttrace2_read_directory_statistics(dir, istate->repo);\n \treturn dir->nr;\n }\n \ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex accefde72fb1..6bce65b439e3 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -57,6 +57,19 @@ iuc () {\n \treturn $ret\n }\n \n+get_relevant_traces() {\n+\t# From the GIT_TRACE2_PERF data of the form\n+\t# $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT\n+\t# extract the $RELEVANT_STAT fields. We don't care about region_enter\n+\t# or region_leave, or stats for things outside read_directory.\n+\tINPUT_FILE=$1\n+\tOUTPUT_FILE=$2\n+\tgrep data.*read_directo $INPUT_FILE \\\n+\t | cut -d \"|\" -f 9 \\\n+\t >$OUTPUT_FILE\n+}\n+\n+\n test_lazy_prereq UNTRACKED_CACHE '\n \t{ git update-index --test-untracked-cache; ret=$?; } &&\n \ttest $ret -ne 1\n@@ -129,19 +142,20 @@ EOF\n \n test_expect_success 'status first time (empty cache)' '\n \tavoid_racy &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <four &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <.gitignore &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <>.git/info/exclude &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n+\t: >../trace.output &&\n \tavoid_racy &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../status.actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <../before\n \t) &&\n@@ -346,12 +346,12 @@ test_expect_success UNTRACKED_CACHE 'ignore .git changes when invalidating UNTR'\n \tEOF\n \t(\n \t\tcd dot-git &&\n-\t\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace-after\" \\\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace-after\" \\\n \t\tgit status &&\n \t\ttest-tool dump-untracked-cache >../after\n \t) &&\n-\tgrep \"directory invalidation\" trace-before >>before &&\n-\tgrep \"directory invalidation\" trace-after >>after &&\n+\tgrep \"directory-invalidation\" trace-before | cut -d\"|\" -f 9 >>before &&\n+\tgrep \"directory-invalidation\" trace-after | cut -d\"|\" -f 9 >>after &&\n \t# UNTR extension unchanged, dir invalidation count unchanged\n \ttest_cmp before after\n '\n-- \ngitgitgadget\n\n"},{"id":"423902","messageId":"fba4d65b78c724eba7ae1fbc286ac29a210d6ac8.1620432501.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","subject":"[PATCH v2 8/8] [RFC] dir: reported number of visited directories and paths with trace2","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T00:08:20Z","receivedAt":"2021-05-08T00:08:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nPreviously, tests that wanted to verify that we don't traverse into a\ndeep directory hierarchy that is ignored had no easy way to verify and\nenforce that behavior. Record information about the number of\ndirectories and paths we inspect while traversing the directory\nhierarchy in read_directory(), and when trace2 is enabled, print these\nstatistics.\n\nMake use of these statistics in t7300 to simplify (and vastly improve\nthe performance of) the \"avoid traversing into ignored directories\"\ntest.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 8 ++++++\n dir.h | 4 +++\n t/t7063-status-untracked-cache.sh | 1 +\n t/t7300-clean.sh | 46 ++++++++++---------------------\n 4 files changed, 27 insertions(+), 32 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 23c71ab7e9a1..896a9a62b2c7 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2455,6 +2455,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \n \tif (open_cached_dir(&cdir, dir, untracked, istate, &path, check_only))\n \t\tgoto out;\n+\tdir->visited_directories++;\n \n \tif (untracked)\n \t\tuntracked->check_only = !!check_only;\n@@ -2463,6 +2464,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t/* check how the file or directory should be treated */\n \t\tstate = treat_path(dir, untracked, &cdir, istate, &path,\n \t\t\t\t baselen, pathspec);\n+\t\tdir->visited_paths++;\n \n \t\tif (state > dir_state)\n \t\t\tdir_state = state;\n@@ -2778,6 +2780,10 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n static void trace2_read_directory_statistics(struct dir_struct *dir,\n \t\t\t\t\t struct repository *repo)\n {\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"directories-visited\", dir->visited_directories);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"paths-visited\", dir->visited_paths);\n \tif (!dir->untracked)\n \t\treturn;\n \ttrace2_data_intmax(\"read_directory\", repo,\n@@ -2798,6 +2804,8 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tstruct untracked_cache_dir *untracked;\n \n \ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n+\tdir->visited_paths = 0;\n+\tdir->visited_directories = 0;\n \n \tif (has_symlink_leading_path(path, len)) {\n \t\ttrace_performance_leave(\"read directory %.*s\", len, path);\ndiff --git a/dir.h b/dir.h\nindex 04d886cfce75..22c67907f689 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -336,6 +336,10 @@ struct dir_struct {\n \tstruct oid_stat ss_info_exclude;\n \tstruct oid_stat ss_excludes_file;\n \tunsigned unmanaged_exclude_files;\n+\n+\t/* Stats about the traversal */\n+\tunsigned visited_paths;\n+\tunsigned visited_directories;\n };\n \n /*Count the number of slashes for string s*/\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 6bce65b439e3..1517c316892f 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -65,6 +65,7 @@ get_relevant_traces() {\n \tINPUT_FILE=$1\n \tOUTPUT_FILE=$2\n \tgrep data.*read_directo $INPUT_FILE \\\n+\t | grep -v visited \\\n \t | cut -d \"|\" -f 9 \\\n \t >$OUTPUT_FILE\n }\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex c2a3b7b6a52b..2c10a7b64f11 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -747,42 +747,24 @@ test_expect_success 'clean untracked paths by pathspec' '\n '\n \n test_expect_success 'avoid traversing into ignored directories' '\n-\ttest_when_finished rm -f output error &&\n+\ttest_when_finished rm -f output error trace.* &&\n \ttest_create_repo avoid-traversing-deep-hierarchy &&\n \t(\n \t\tcd avoid-traversing-deep-hierarchy &&\n \n-\t\t>directory-random-file.txt &&\n-\t\t# Put this file under directory400/directory399/.../directory1/\n-\t\tdepth=400 &&\n-\t\tfor x in $(test_seq 1 $depth); do\n-\t\t\tmkdir \"tmpdirectory$x\" &&\n-\t\t\tmv directory* \"tmpdirectory$x\" &&\n-\t\t\tmv \"tmpdirectory$x\" \"directory$x\"\n-\t\tdone &&\n-\n-\t\tgit clean -ffdxn -e directory$depth >../output 2>../error &&\n-\n-\t\ttest_must_be_empty ../output &&\n-\t\t# We especially do not want things like\n-\t\t# \"warning: could not open directory \"\n-\t\t# appearing in the error output. It is true that directories\n-\t\t# that are too long cannot be opened, but we should not be\n-\t\t# recursing into those directories anyway since the very first\n-\t\t# level is ignored.\n-\t\ttest_must_be_empty ../error &&\n-\n-\t\t# alpine-linux-musl fails to \"rm -rf\" a directory with such\n-\t\t# a deeply nested hierarchy. Help it out by deleting the\n-\t\t# leading directories ourselves. Super slow, but, what else\n-\t\t# can we do? Without this, we will hit a\n-\t\t# error: Tests passed but test cleanup failed; aborting\n-\t\t# so do this ugly manual cleanup...\n-\t\twhile test ! -f directory-random-file.txt; do\n-\t\t\tname=$(ls -d directory*) &&\n-\t\t\tmv $name/* . &&\n-\t\t\trmdir $name\n-\t\tdone\n+\t\tmkdir -p untracked/subdir/with/a &&\n+\t\t>untracked/subdir/with/a/random-file.txt &&\n+\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n+\t\tgit clean -ffdxn -e untracked &&\n+\n+\t\tgrep data.*read_directo.*visited ../trace.output \\\n+\t\t\t| cut -d \"|\" -f 9 >../trace.relevant &&\n+\t\tcat >../trace.expect <<-EOF &&\n+\t\t directories-visited:1\n+\t\t paths-visited:4\n+\t\tEOF\n+\t\ttest_cmp ../trace.expect ../trace.relevant\n \t)\n '\n \n-- \ngitgitgadget\n"},{"id":"423903","messageId":"CAPig+cQw5xBK00rtMVw4b6e_21vgp33yYDfdRape202dpWQjbA@mail.gmail.com","threadId":"55639","inReplyTo":"CABPp-BEyUX8j-_rhNTyR5ZiqeLUWbBS-9q5+gEUWOEHpchambQ@mail.gmail.com","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2021-05-08T00:10:49Z","receivedAt":"2021-05-08T00:11:02Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Fri, May 7, 2021 at 8:04 PM Elijah Newren wrote:\n> I think I figured it out. I now have the test simplified down to just:\n>\n> test_expect_success 'avoid traversing into ignored directories' '\n> test_when_finished rm -f output error trace.* &&\n> test_create_repo avoid-traversing-deep-hierarchy &&\n> (\n> mkdir -p untracked/subdir/with/a &&\n> >untracked/subdir/with/a/random-file.txt &&\n>\n> GIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n> git clean -ffdxn -e untracked &&\n>\n> grep data.*read_directo.*visited ../trace.output \\\n> | cut -d \"|\" -f 9 >../trace.relevant &&\n> cat >../trace.expect <<-EOF &&\n> directories-visited:1\n> paths-visited:4\n> EOF\n> test_cmp ../trace.expect ../trace.relevant\n> )\n> '\n\nI believe that you can close the subshell immediately after `git\nclean`, which would allow you to drop all the \"../\" prefixes on\npathnames.\n\n> This relies on a few extra changes to the code: (1) switching the\n> existing trace calls in dir.c over to using trace2 variants, and (2)\n> adding two new counters (visited_directories and visited_paths) that\n> are output using the trace2 framework. I'm a little unsure if I\n> should check the paths-visited counter (will some platform have\n> additional files in every directory besides '.' and '..'? Or not have\n> one of those?), but it is good to have it check that the code in this\n> case visits no directories other than the toplevel one (i.e. that\n> directories-visited is 1).\n\nI can't find the reference, but I recall a reply by jrneider (to some\nproposed patch) that not all platforms are guaranteed to have \".\" and\n\"..\" entries (but I'm not sure we need to worry about that presently).\n"},{"id":"423923","messageId":"xmqqczu1v8i0.fsf@gitster.g","threadId":"55639","inReplyTo":"a3bd253fa8e8ae47d19beb35327d8283ffa49289.1620432500.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-08T10:13:11Z","receivedAt":"2021-05-08T10:13:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> From: Elijah Newren \n>\n> PNPM is apparently creating deeply nested (but ignored) directory\n\nSorry, but what's PNPM?\n\n> structures; traversing them is costly performance-wise, unnecessary, and\n> in some cases is even throwing warnings/errors because the paths are too\n> long to handle on various platforms. Add a testcase that demonstrates\n> this problem.\n>\n> Initial-test-by: Jason Gore \n> Helped-by: brian m. carlson \n> Signed-off-by: Elijah Newren \n> ---\n> t/t7300-clean.sh | 40 ++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 40 insertions(+)\n>\n> diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\n> index a74816ca8b46..5f1dc397c11e 100755\n> --- a/t/t7300-clean.sh\n> +++ b/t/t7300-clean.sh\n> @@ -746,4 +746,44 @@ test_expect_success 'clean untracked paths by pathspec' '\n> \ttest_must_be_empty actual\n> '\n> \n> +test_expect_failure 'avoid traversing into ignored directories' '\n> +\ttest_when_finished rm -f output error &&\n> +\ttest_create_repo avoid-traversing-deep-hierarchy &&\n> +\t(\n> +\t\tcd avoid-traversing-deep-hierarchy &&\n> +\n> +\t\t>directory-random-file.txt &&\n> +\t\t# Put this file under directory400/directory399/.../directory1/\n> +\t\tdepth=400 &&\n> +\t\tfor x in $(test_seq 1 $depth); do\n\nStyle. Lose semicolon, have \"do\" on the next line on its own,\naligned with \"for\". Tip: you shouldn't need any semicolon other\nthan the doubled ones in case/esac in your shell script.\n\n> +\t\t\tmkdir \"tmpdirectory$x\" &&\n> +\t\t\tmv directory* \"tmpdirectory$x\" &&\n> +\t\t\tmv \"tmpdirectory$x\" \"directory$x\"\n> +\t\tdone &&\n> +\n> +\t\tgit clean -ffdxn -e directory$depth >../output 2>../error &&\n> +\n> +\t\ttest_must_be_empty ../output &&\n> +\t\t# We especially do not want things like\n> +\t\t# \"warning: could not open directory \"\n> +\t\t# appearing in the error output. It is true that directories\n> +\t\t# that are too long cannot be opened, but we should not be\n> +\t\t# recursing into those directories anyway since the very first\n> +\t\t# level is ignored.\n> +\t\ttest_must_be_empty ../error &&\n> +\n> +\t\t# alpine-linux-musl fails to \"rm -rf\" a directory with such\n> +\t\t# a deeply nested hierarchy. Help it out by deleting the\n> +\t\t# leading directories ourselves. Super slow, but, what else\n> +\t\t# can we do? Without this, we will hit a\n> +\t\t# error: Tests passed but test cleanup failed; aborting\n> +\t\t# so do this ugly manual cleanup...\n> +\t\twhile test ! -f directory-random-file.txt; do\n\nDitto.\n\n> +\t\t\tname=$(ls -d directory*) &&\n> +\t\t\tmv $name/* . &&\n> +\t\t\trmdir $name\n> +\t\tdone\n\nHmph, after seeing the discussion thread of v1, I was expecting to\nsee a helper in Perl that cd's down and then comes back up while\nremoving what is in its directory (and I expected something similar\nfor creation side we saw above).\n\n> +\t)\n> +'\n> +\n> test_done\n"},{"id":"423924","messageId":"xmqq8s4pv87t.fsf@gitster.g","threadId":"55639","inReplyTo":"a3bd253fa8e8ae47d19beb35327d8283ffa49289.1620432500.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 1/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-08T10:19:18Z","receivedAt":"2021-05-08T10:19:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> +\t\t# alpine-linux-musl fails to \"rm -rf\" a directory with such\n> +\t\t# a deeply nested hierarchy. Help it out by deleting the\n> +\t\t# leading directories ourselves. Super slow, but, what else\n> +\t\t# can we do? Without this, we will hit a\n> +\t\t# error: Tests passed but test cleanup failed; aborting\n> +\t\t# so do this ugly manual cleanup...\n> +\t\twhile test ! -f directory-random-file.txt; do\n> +\t\t\tname=$(ls -d directory*) &&\n> +\t\t\tmv $name/* . &&\n> +\t\t\trmdir $name\n> +\t\tdone\n\nAnother thing: this not being a test_when_finished handler means it\nwould not help after a test failure.\n\nPerhaps wrap it in a helper\n\n clean_deep_hierarchy () {\n\trm -fr directory* ||\n\twhile test ! -f directory-random-file.txt\n\tdo\n\t\t...\n\tdone\n }\n\nand call it from test_when_finished?\n"},{"id":"423930","messageId":"2b4e818f-9429-83f1-51d6-d8eb0ba618ff@iee.email","threadId":"55639","inReplyTo":"a3bd253fa8e8ae47d19beb35327d8283ffa49289.1620360300.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2021-05-08T11:13:28Z","receivedAt":"2021-05-08T11:13:35Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 07/05/2021 05:04, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n>\n> PNPM \n\nfor me, this was a UNA (un-named abbreviation), can we clarify it, e.g\ns/PNPM/& package manager/\n\n> is apparently creating deeply nested (but ignored) directory\n> structures; traversing them is costly performance-wise, unnecessary, and\n> in some cases is even throwing warnings/errors because the paths are too\n> long to handle on various platforms. Add a testcase that demonstrates\n> this problem.\n>\n> Initial-test-by: Jason Gore \n> Helped-by: brian m. carlson \n> Signed-off-by: Elijah Newren \n> ---\n> t/t7300-clean.sh | 40 ++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 40 insertions(+)\n>\n> diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\n> index a74816ca8b46..5f1dc397c11e 100755\n> --- a/t/t7300-clean.sh\n> +++ b/t/t7300-clean.sh\n> @@ -746,4 +746,44 @@ test_expect_success 'clean untracked paths by pathspec' '\n> \ttest_must_be_empty actual\n> '\n> \n> +test_expect_failure 'avoid traversing into ignored directories' '\n> +\ttest_when_finished rm -f output error &&\n> +\ttest_create_repo avoid-traversing-deep-hierarchy &&\n> +\t(\n> +\t\tcd avoid-traversing-deep-hierarchy &&\n> +\n> +\t\t>directory-random-file.txt &&\n> +\t\t# Put this file under directory400/directory399/.../directory1/\n> +\t\tdepth=400 &&\n> +\t\tfor x in $(test_seq 1 $depth); do\n> +\t\t\tmkdir \"tmpdirectory$x\" &&\n> +\t\t\tmv directory* \"tmpdirectory$x\" &&\n> +\t\t\tmv \"tmpdirectory$x\" \"directory$x\"\n> +\t\tdone &&\n> +\n> +\t\tgit clean -ffdxn -e directory$depth >../output 2>../error &&\n> +\n> +\t\ttest_must_be_empty ../output &&\n> +\t\t# We especially do not want things like\n> +\t\t# \"warning: could not open directory \"\n> +\t\t# appearing in the error output. It is true that directories\n> +\t\t# that are too long cannot be opened, but we should not be\n> +\t\t# recursing into those directories anyway since the very first\n> +\t\t# level is ignored.\n> +\t\ttest_must_be_empty ../error &&\n> +\n> +\t\t# alpine-linux-musl fails to \"rm -rf\" a directory with such\n> +\t\t# a deeply nested hierarchy. Help it out by deleting the\n> +\t\t# leading directories ourselves. Super slow, but, what else\n> +\t\t# can we do? Without this, we will hit a\n> +\t\t# error: Tests passed but test cleanup failed; aborting\n> +\t\t# so do this ugly manual cleanup...\n> +\t\twhile test ! -f directory-random-file.txt; do\n> +\t\t\tname=$(ls -d directory*) &&\n> +\t\t\tmv $name/* . &&\n> +\t\t\trmdir $name\n> +\t\tdone\n> +\t)\n> +'\n> +\n> test_done\n\n"},{"id":"423936","messageId":"CABPp-BE4ccB3cVXMqQ-4ZPn6TrRMrD61zHrXNbtNR9GQ9id98Q@mail.gmail.com","threadId":"55639","inReplyTo":"CAPig+cQw5xBK00rtMVw4b6e_21vgp33yYDfdRape202dpWQjbA@mail.gmail.com","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-08T17:20:01Z","receivedAt":"2021-05-08T17:20:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, May 7, 2021 at 5:11 PM Eric Sunshine wrote:\n>\n> On Fri, May 7, 2021 at 8:04 PM Elijah Newren wrote:\n> > I think I figured it out. I now have the test simplified down to just:\n> >\n> > test_expect_success 'avoid traversing into ignored directories' '\n> > test_when_finished rm -f output error trace.* &&\n> > test_create_repo avoid-traversing-deep-hierarchy &&\n> > (\n> > mkdir -p untracked/subdir/with/a &&\n> > >untracked/subdir/with/a/random-file.txt &&\n> >\n> > GIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n> > git clean -ffdxn -e untracked &&\n> >\n> > grep data.*read_directo.*visited ../trace.output \\\n> > | cut -d \"|\" -f 9 >../trace.relevant &&\n> > cat >../trace.expect <<-EOF &&\n> > directories-visited:1\n> > paths-visited:4\n> > EOF\n> > test_cmp ../trace.expect ../trace.relevant\n> > )\n> > '\n>\n> I believe that you can close the subshell immediately after `git\n> clean`, which would allow you to drop all the \"../\" prefixes on\n> pathnames.\n\nAh, good point. I'll make that fix.\n\n> > This relies on a few extra changes to the code: (1) switching the\n> > existing trace calls in dir.c over to using trace2 variants, and (2)\n> > adding two new counters (visited_directories and visited_paths) that\n> > are output using the trace2 framework. I'm a little unsure if I\n> > should check the paths-visited counter (will some platform have\n> > additional files in every directory besides '.' and '..'? Or not have\n> > one of those?), but it is good to have it check that the code in this\n> > case visits no directories other than the toplevel one (i.e. that\n> > directories-visited is 1).\n>\n> I can't find the reference, but I recall a reply by jrneider (to some\n> proposed patch) that not all platforms are guaranteed to have \".\" and\n> \"..\" entries (but I'm not sure we need to worry about that presently).\n"},{"id":"423937","messageId":"CABPp-BFvTMzsyD1fbt5BxA6Na=MFLqUkcLyCtnxfADcsu9KPjA@mail.gmail.com","threadId":"55639","inReplyTo":"2b4e818f-9429-83f1-51d6-d8eb0ba618ff@iee.email","subject":"Re: [PATCH 1/5] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-08T17:20:40Z","receivedAt":"2021-05-08T17:20:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, May 8, 2021 at 4:13 AM Philip Oakley wrote:\n>\n> On 07/05/2021 05:04, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > PNPM\n>\n> for me, this was a UNA (un-named abbreviation), can we clarify it, e.g\n> s/PNPM/& package manager/\n\nWill do, thanks.\n"},{"id":"423938","messageId":"CABPp-BE5hfMb_Jnwdb5hBjwkZg3EFEWxdqWht9wATRkbz6cjkA@mail.gmail.com","threadId":"55639","inReplyTo":"xmqqczu1v8i0.fsf@gitster.g","subject":"Re: [PATCH v2 1/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-08T17:34:48Z","receivedAt":"2021-05-08T17:35:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, May 8, 2021 at 3:13 AM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > From: Elijah Newren \n> >\n> > PNPM is apparently creating deeply nested (but ignored) directory\n>\n> Sorry, but what's PNPM?\n\na package manager; I'll use Philip Oakley's suggestion to make it more clear.\n\n> > structures; traversing them is costly performance-wise, unnecessary, and\n> > in some cases is even throwing warnings/errors because the paths are too\n> > long to handle on various platforms. Add a testcase that demonstrates\n> > this problem.\n> >\n> > Initial-test-by: Jason Gore \n> > Helped-by: brian m. carlson \n> > Signed-off-by: Elijah Newren \n> > ---\n> > t/t7300-clean.sh | 40 ++++++++++++++++++++++++++++++++++++++++\n> > 1 file changed, 40 insertions(+)\n> >\n> > diff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\n> > index a74816ca8b46..5f1dc397c11e 100755\n> > --- a/t/t7300-clean.sh\n> > +++ b/t/t7300-clean.sh\n> > @@ -746,4 +746,44 @@ test_expect_success 'clean untracked paths by pathspec' '\n> > test_must_be_empty actual\n> > '\n> >\n> > +test_expect_failure 'avoid traversing into ignored directories' '\n> > + test_when_finished rm -f output error &&\n> > + test_create_repo avoid-traversing-deep-hierarchy &&\n> > + (\n> > + cd avoid-traversing-deep-hierarchy &&\n> > +\n> > + >directory-random-file.txt &&\n> > + # Put this file under directory400/directory399/.../directory1/\n> > + depth=400 &&\n> > + for x in $(test_seq 1 $depth); do\n>\n> Style. Lose semicolon, have \"do\" on the next line on its own,\n> aligned with \"for\". Tip: you shouldn't need any semicolon other\n> than the doubled ones in case/esac in your shell script.\n\nThanks.\n\n>\n> > + mkdir \"tmpdirectory$x\" &&\n> > + mv directory* \"tmpdirectory$x\" &&\n> > + mv \"tmpdirectory$x\" \"directory$x\"\n> > + done &&\n> > +\n> > + git clean -ffdxn -e directory$depth >../output 2>../error &&\n> > +\n> > + test_must_be_empty ../output &&\n> > + # We especially do not want things like\n> > + # \"warning: could not open directory \"\n> > + # appearing in the error output. It is true that directories\n> > + # that are too long cannot be opened, but we should not be\n> > + # recursing into those directories anyway since the very first\n> > + # level is ignored.\n> > + test_must_be_empty ../error &&\n> > +\n> > + # alpine-linux-musl fails to \"rm -rf\" a directory with such\n> > + # a deeply nested hierarchy. Help it out by deleting the\n> > + # leading directories ourselves. Super slow, but, what else\n> > + # can we do? Without this, we will hit a\n> > + # error: Tests passed but test cleanup failed; aborting\n> > + # so do this ugly manual cleanup...\n> > + while test ! -f directory-random-file.txt; do\n>\n> Ditto.\n\nYep, sorry.\n\n> > + name=$(ls -d directory*) &&\n> > + mv $name/* . &&\n> > + rmdir $name\n> > + done\n>\n> Hmph, after seeing the discussion thread of v1, I was expecting to\n> see a helper in Perl that cd's down and then comes back up while\n> removing what is in its directory (and I expected something similar\n> for creation side we saw above).\n\nHmm, I was a bit unsure of the alternative route I took in patches 7\nand 8 (switching trace1 to trace2 in dir.c, then using it to get more\nstatistics which would allow a much more shallow directory structure\nfor this test). I wasn't sure if the strategy seemed acceptable, and\nI wanted people to be able to see the two schemes side-by-side, but if\nthat alternative is acceptable, I want to move patch 7 to the front of\nthe series, the code change parts of patch 8 as the second patch, and\nthen squash the rest of patch 8 into this patch vastly simplifying\nthis testcase and obsoleting everyone's comments on it.\n\nMaybe I should have just refactored the series that way anyway. I'll\nsend a reroll that does that, and put all the [RFC] patches first.\n"},{"id":"423939","messageId":"CABPp-BG7Rp7QtSSDCwjmOC=SJZ7Uuw8Uddtu+VHbiFaRQAUJKQ@mail.gmail.com","threadId":"55639","inReplyTo":"xmqq8s4pv87t.fsf@gitster.g","subject":"Re: [PATCH v2 1/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-08T17:41:09Z","receivedAt":"2021-05-08T17:41:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sat, May 8, 2021 at 3:19 AM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > + # alpine-linux-musl fails to \"rm -rf\" a directory with such\n> > + # a deeply nested hierarchy. Help it out by deleting the\n> > + # leading directories ourselves. Super slow, but, what else\n> > + # can we do? Without this, we will hit a\n> > + # error: Tests passed but test cleanup failed; aborting\n> > + # so do this ugly manual cleanup...\n> > + while test ! -f directory-random-file.txt; do\n> > + name=$(ls -d directory*) &&\n> > + mv $name/* . &&\n> > + rmdir $name\n> > + done\n>\n> Another thing: this not being a test_when_finished handler means it\n> would not help after a test failure.\n\ntest failures are irrelevant here; this code is here to help\ntest_done's directory cleanup, which only fires when all tests pass.\n\n\nBut if I restructure the series, this whole section of code\ndisappears. I'll do that...\n"},{"id":"423940","messageId":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v2.git.git.1620432500.gitgitgadget@gmail.com","subject":"[PATCH v3 0/8] Directory traversal fixes","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T19:58:56Z","receivedAt":"2021-05-08T19:59:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patchset fixes a few directory traversal issues, where fill_directory()\nwould traverse into directories that it shouldn't and not traverse into\ndirectories that it should (one of which was originally reported on this\nlist at [1]). And it includes a few cleanups\n\nChanges since v2:\n\n * Move the RFC patches to the front\n * Deletes all the ugly test code that stole reviewer attention away from\n the rest of the series. :-) The RFC patches being first allow the test to\n be dramatically simplified and rewritten.\n * Included cleanups suggested by Phillip Oakley and Eric Sunshine (the\n cleanups suggested by others are obsolete with the test rewrite).\n\nPatches 1-3 are RFC because\n\n * (1) I'm not that familiar with trace1 & trace2; I've only used trace2 for\n region_enter() and region_leave() calls before. And I'm unsure if\n removing trace1 counts as a backward compatibility issue or not, though\n the trace2 documentation claims it's meant to replace trace1.\n * (2) The ls-files -i handling to print an error instead of operating as\n before might be considered a backward incompatible change. I want to hear\n others' opinions on that.\n\nAlso, if anyone has any ideas about a better place to put the \"Some\nsidenotes\" from the sixth commit message rather than keeping them in a\nrandom commit message, that might be helpful too.\n\n[1] See\nhttps://lore.kernel.org/git/DM6PR00MB06829EC5B85E0C5AC595004E894E9@DM6PR00MB0682.namprd00.prod.outlook.com/\nor alternatively https://github.com/git-for-windows/git/issues/2732.\n\nDerrick Stolee (1):\n dir: update stale description of treat_directory()\n\nElijah Newren (7):\n [RFC] dir: convert trace calls to trace2 equivalents\n [RFC] dir: report number of visited directories and paths with trace2\n [RFC] ls-files: error out on -i unless -o or -c are specified\n t7300: add testcase showing unnecessary traversal into ignored\n directory\n t3001, t7300: add testcase showcasing missed directory traversal\n dir: avoid unnecessary traversal into ignored directory\n dir: traverse into untracked directories if they may have ignored\n subfiles\n\n builtin/ls-files.c | 3 +\n dir.c | 103 +++++++++------\n dir.h | 4 +\n t/t1306-xdg-files.sh | 2 +-\n t/t3001-ls-files-others-exclude.sh | 5 +\n t/t3003-ls-files-exclude.sh | 4 +-\n t/t7063-status-untracked-cache.sh | 194 ++++++++++++++++-------------\n t/t7300-clean.sh | 41 ++++++\n t/t7519-status-fsmonitor.sh | 8 +-\n 9 files changed, 238 insertions(+), 126 deletions(-)\n\n\nbase-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1020%2Fnewren%2Fdirectory-traversal-fixes-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1020/newren/directory-traversal-fixes-v3\nPull-Request: https://github.com/git/git/pull/1020\n\nRange-diff vs v2:\n\n 7: 3a2394506a53 = 1: 9f1c0d78d739 [RFC] dir: convert trace calls to trace2 equivalents\n 8: fba4d65b78c7 ! 2: 8b511f228af8 [RFC] dir: reported number of visited directories and paths with trace2\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - [RFC] dir: reported number of visited directories and paths with trace2\n + [RFC] dir: report number of visited directories and paths with trace2\n \n - Previously, tests that wanted to verify that we don't traverse into a\n - deep directory hierarchy that is ignored had no easy way to verify and\n - enforce that behavior. Record information about the number of\n - directories and paths we inspect while traversing the directory\n - hierarchy in read_directory(), and when trace2 is enabled, print these\n - statistics.\n -\n - Make use of these statistics in t7300 to simplify (and vastly improve\n - the performance of) the \"avoid traversing into ignored directories\"\n - test.\n + Provide more statistics in trace2 output that include the number of\n + directories and total paths visited by the directory traversal logic.\n + Subsequent patches will take advantage of this to ensure we do not\n + unnecessarily traverse into ignored directories.\n \n Signed-off-by: Elijah Newren \n \n @@ t/t7063-status-untracked-cache.sh: get_relevant_traces() {\n \t | cut -d \"|\" -f 9 \\\n \t >$OUTPUT_FILE\n }\n -\n - ## t/t7300-clean.sh ##\n -@@ t/t7300-clean.sh: test_expect_success 'clean untracked paths by pathspec' '\n - '\n - \n - test_expect_success 'avoid traversing into ignored directories' '\n --\ttest_when_finished rm -f output error &&\n -+\ttest_when_finished rm -f output error trace.* &&\n - \ttest_create_repo avoid-traversing-deep-hierarchy &&\n - \t(\n - \t\tcd avoid-traversing-deep-hierarchy &&\n - \n --\t\t>directory-random-file.txt &&\n --\t\t# Put this file under directory400/directory399/.../directory1/\n --\t\tdepth=400 &&\n --\t\tfor x in $(test_seq 1 $depth); do\n --\t\t\tmkdir \"tmpdirectory$x\" &&\n --\t\t\tmv directory* \"tmpdirectory$x\" &&\n --\t\t\tmv \"tmpdirectory$x\" \"directory$x\"\n --\t\tdone &&\n --\n --\t\tgit clean -ffdxn -e directory$depth >../output 2>../error &&\n --\n --\t\ttest_must_be_empty ../output &&\n --\t\t# We especially do not want things like\n --\t\t# \"warning: could not open directory \"\n --\t\t# appearing in the error output. It is true that directories\n --\t\t# that are too long cannot be opened, but we should not be\n --\t\t# recursing into those directories anyway since the very first\n --\t\t# level is ignored.\n --\t\ttest_must_be_empty ../error &&\n --\n --\t\t# alpine-linux-musl fails to \"rm -rf\" a directory with such\n --\t\t# a deeply nested hierarchy. Help it out by deleting the\n --\t\t# leading directories ourselves. Super slow, but, what else\n --\t\t# can we do? Without this, we will hit a\n --\t\t# error: Tests passed but test cleanup failed; aborting\n --\t\t# so do this ugly manual cleanup...\n --\t\twhile test ! -f directory-random-file.txt; do\n --\t\t\tname=$(ls -d directory*) &&\n --\t\t\tmv $name/* . &&\n --\t\t\trmdir $name\n --\t\tdone\n -+\t\tmkdir -p untracked/subdir/with/a &&\n -+\t\t>untracked/subdir/with/a/random-file.txt &&\n -+\n -+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n -+\t\tgit clean -ffdxn -e untracked &&\n -+\n -+\t\tgrep data.*read_directo.*visited ../trace.output \\\n -+\t\t\t| cut -d \"|\" -f 9 >../trace.relevant &&\n -+\t\tcat >../trace.expect <<-EOF &&\n -+\t\t directories-visited:1\n -+\t\t paths-visited:4\n -+\t\tEOF\n -+\t\ttest_cmp ../trace.expect ../trace.relevant\n - \t)\n - '\n - \n 5: 3d8dd00ccd10 = 3: 44a1322c4402 [RFC] ls-files: error out on -i unless -o or -c are specified\n 1: a3bd253fa8e8 ! 4: dc3d3f247141 t7300: add testcase showing unnecessary traversal into ignored directory\n @@ Metadata\n ## Commit message ##\n t7300: add testcase showing unnecessary traversal into ignored directory\n \n - PNPM is apparently creating deeply nested (but ignored) directory\n - structures; traversing them is costly performance-wise, unnecessary, and\n - in some cases is even throwing warnings/errors because the paths are too\n - long to handle on various platforms. Add a testcase that demonstrates\n - this problem.\n + The PNPM package manager is apparently creating deeply nested (but\n + ignored) directory structures; traversing them is costly\n + performance-wise, unnecessary, and in some cases is even throwing\n + warnings/errors because the paths are too long to handle on various\n + platforms. Add a testcase that checks for such unnecessary directory\n + traversal.\n \n - Initial-test-by: Jason Gore \n - Helped-by: brian m. carlson \n Signed-off-by: Elijah Newren \n \n ## t/t7300-clean.sh ##\n @@ t/t7300-clean.sh: test_expect_success 'clean untracked paths by pathspec' '\n '\n \n +test_expect_failure 'avoid traversing into ignored directories' '\n -+\ttest_when_finished rm -f output error &&\n ++\ttest_when_finished rm -f output error trace.* &&\n +\ttest_create_repo avoid-traversing-deep-hierarchy &&\n +\t(\n +\t\tcd avoid-traversing-deep-hierarchy &&\n +\n -+\t\t>directory-random-file.txt &&\n -+\t\t# Put this file under directory400/directory399/.../directory1/\n -+\t\tdepth=400 &&\n -+\t\tfor x in $(test_seq 1 $depth); do\n -+\t\t\tmkdir \"tmpdirectory$x\" &&\n -+\t\t\tmv directory* \"tmpdirectory$x\" &&\n -+\t\t\tmv \"tmpdirectory$x\" \"directory$x\"\n -+\t\tdone &&\n ++\t\tmkdir -p untracked/subdir/with/a &&\n ++\t\t>untracked/subdir/with/a/random-file.txt &&\n +\n -+\t\tgit clean -ffdxn -e directory$depth >../output 2>../error &&\n ++\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n ++\t\tgit clean -ffdxn -e untracked\n ++\t) &&\n +\n -+\t\ttest_must_be_empty ../output &&\n -+\t\t# We especially do not want things like\n -+\t\t# \"warning: could not open directory \"\n -+\t\t# appearing in the error output. It is true that directories\n -+\t\t# that are too long cannot be opened, but we should not be\n -+\t\t# recursing into those directories anyway since the very first\n -+\t\t# level is ignored.\n -+\t\ttest_must_be_empty ../error &&\n -+\n -+\t\t# alpine-linux-musl fails to \"rm -rf\" a directory with such\n -+\t\t# a deeply nested hierarchy. Help it out by deleting the\n -+\t\t# leading directories ourselves. Super slow, but, what else\n -+\t\t# can we do? Without this, we will hit a\n -+\t\t# error: Tests passed but test cleanup failed; aborting\n -+\t\t# so do this ugly manual cleanup...\n -+\t\twhile test ! -f directory-random-file.txt; do\n -+\t\t\tname=$(ls -d directory*) &&\n -+\t\t\tmv $name/* . &&\n -+\t\t\trmdir $name\n -+\t\tdone\n -+\t)\n ++\tgrep data.*read_directo.*visited trace.output \\\n ++\t\t| cut -d \"|\" -f 9 >trace.relevant &&\n ++\tcat >trace.expect <<-EOF &&\n ++\t directories-visited:1\n ++\t paths-visited:4\n ++\tEOF\n ++\ttest_cmp trace.expect trace.relevant\n +'\n +\n test_done\n 2: aa3a41e26eca ! 5: 73b03a1e8e05 t3001, t7300: add testcase showcasing missed directory traversal\n @@ t/t3001-ls-files-others-exclude.sh: EOF\n \n ## t/t7300-clean.sh ##\n @@ t/t7300-clean.sh: test_expect_failure 'avoid traversing into ignored directories' '\n - \t)\n + \ttest_cmp trace.expect trace.relevant\n '\n \n +test_expect_failure 'traverse into directories that may have ignored entries' '\n 3: 3c3f6111da13 ! 6: 66ffc7f02d08 dir: avoid unnecessary traversal into ignored directory\n @@ t/t7300-clean.sh: test_expect_success 'clean untracked paths by pathspec' '\n \n -test_expect_failure 'avoid traversing into ignored directories' '\n +test_expect_success 'avoid traversing into ignored directories' '\n - \ttest_when_finished rm -f output error &&\n + \ttest_when_finished rm -f output error trace.* &&\n \ttest_create_repo avoid-traversing-deep-hierarchy &&\n \t(\n 4: fad048339b81 ! 7: acde436b220e dir: traverse into untracked directories if they may have ignored subfiles\n @@ t/t3001-ls-files-others-exclude.sh: EOF\n \n ## t/t7300-clean.sh ##\n @@ t/t7300-clean.sh: test_expect_success 'avoid traversing into ignored directories' '\n - \t)\n + \ttest_cmp trace.expect trace.relevant\n '\n \n -test_expect_failure 'traverse into directories that may have ignored entries' '\n 6: 1d825dfdc70b = 8: 57135c357774 dir: update stale description of treat_directory()\n\n-- \ngitgitgadget\n"},{"id":"423941","messageId":"9f1c0d78d739cc1da8517eea92e81a7bd162578f.1620503945.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","subject":"[PATCH v3 1/8] [RFC] dir: convert trace calls to trace2 equivalents","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T19:58:57Z","receivedAt":"2021-05-08T19:59:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 34 ++++--\n t/t7063-status-untracked-cache.sh | 193 +++++++++++++++++-------------\n t/t7519-status-fsmonitor.sh | 8 +-\n 3 files changed, 135 insertions(+), 100 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3474e67e8f3c..9f7c8debeab3 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2760,12 +2760,29 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \treturn root;\n }\n \n+static void trace2_read_directory_statistics(struct dir_struct *dir,\n+\t\t\t\t\t struct repository *repo)\n+{\n+\tif (!dir->untracked)\n+\t\treturn;\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"node-creation\", dir->untracked->dir_created);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"gitignore-invalidation\",\n+\t\t\t dir->untracked->gitignore_invalidated);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"directory-invalidation\",\n+\t\t\t dir->untracked->dir_invalidated);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"opendir\", dir->untracked->dir_opened);\n+}\n+\n int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t const char *path, int len, const struct pathspec *pathspec)\n {\n \tstruct untracked_cache_dir *untracked;\n \n-\ttrace_performance_enter();\n+\ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n \n \tif (has_symlink_leading_path(path, len)) {\n \t\ttrace_performance_leave(\"read directory %.*s\", len, path);\n@@ -2784,23 +2801,13 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tQSORT(dir->entries, dir->nr, cmp_dir_entry);\n \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n \n-\ttrace_performance_leave(\"read directory %.*s\", len, path);\n+\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n \tif (dir->untracked) {\n \t\tstatic int force_untracked_cache = -1;\n-\t\tstatic struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n \n \t\tif (force_untracked_cache < 0)\n \t\t\tforce_untracked_cache =\n \t\t\t\tgit_env_bool(\"GIT_FORCE_UNTRACKED_CACHE\", 0);\n-\t\ttrace_printf_key(&trace_untracked_stats,\n-\t\t\t\t \"node creation: %u\\n\"\n-\t\t\t\t \"gitignore invalidation: %u\\n\"\n-\t\t\t\t \"directory invalidation: %u\\n\"\n-\t\t\t\t \"opendir: %u\\n\",\n-\t\t\t\t dir->untracked->dir_created,\n-\t\t\t\t dir->untracked->gitignore_invalidated,\n-\t\t\t\t dir->untracked->dir_invalidated,\n-\t\t\t\t dir->untracked->dir_opened);\n \t\tif (force_untracked_cache &&\n \t\t\tdir->untracked == istate->untracked &&\n \t\t (dir->untracked->dir_opened ||\n@@ -2811,6 +2818,9 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t\tFREE_AND_NULL(dir->untracked);\n \t\t}\n \t}\n+\n+\tif (trace2_is_enabled())\n+\t\ttrace2_read_directory_statistics(dir, istate->repo);\n \treturn dir->nr;\n }\n \ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex accefde72fb1..6bce65b439e3 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -57,6 +57,19 @@ iuc () {\n \treturn $ret\n }\n \n+get_relevant_traces() {\n+\t# From the GIT_TRACE2_PERF data of the form\n+\t# $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT\n+\t# extract the $RELEVANT_STAT fields. We don't care about region_enter\n+\t# or region_leave, or stats for things outside read_directory.\n+\tINPUT_FILE=$1\n+\tOUTPUT_FILE=$2\n+\tgrep data.*read_directo $INPUT_FILE \\\n+\t | cut -d \"|\" -f 9 \\\n+\t >$OUTPUT_FILE\n+}\n+\n+\n test_lazy_prereq UNTRACKED_CACHE '\n \t{ git update-index --test-untracked-cache; ret=$?; } &&\n \ttest $ret -ne 1\n@@ -129,19 +142,20 @@ EOF\n \n test_expect_success 'status first time (empty cache)' '\n \tavoid_racy &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <four &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <.gitignore &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <>.git/info/exclude &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n+\t: >../trace.output &&\n \tavoid_racy &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../status.actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <../before\n \t) &&\n@@ -346,12 +346,12 @@ test_expect_success UNTRACKED_CACHE 'ignore .git changes when invalidating UNTR'\n \tEOF\n \t(\n \t\tcd dot-git &&\n-\t\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace-after\" \\\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace-after\" \\\n \t\tgit status &&\n \t\ttest-tool dump-untracked-cache >../after\n \t) &&\n-\tgrep \"directory invalidation\" trace-before >>before &&\n-\tgrep \"directory invalidation\" trace-after >>after &&\n+\tgrep \"directory-invalidation\" trace-before | cut -d\"|\" -f 9 >>before &&\n+\tgrep \"directory-invalidation\" trace-after | cut -d\"|\" -f 9 >>after &&\n \t# UNTR extension unchanged, dir invalidation count unchanged\n \ttest_cmp before after\n '\n-- \ngitgitgadget\n\n"},{"id":"423942","messageId":"8b511f228af8e047d8b38159da658b10174c40e8.1620503945.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","subject":"[PATCH v3 2/8] [RFC] dir: report number of visited directories and paths with trace2","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T19:58:58Z","receivedAt":"2021-05-08T19:59:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nProvide more statistics in trace2 output that include the number of\ndirectories and total paths visited by the directory traversal logic.\nSubsequent patches will take advantage of this to ensure we do not\nunnecessarily traverse into ignored directories.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 8 ++++++++\n dir.h | 4 ++++\n t/t7063-status-untracked-cache.sh | 1 +\n 3 files changed, 13 insertions(+)\n\ndiff --git a/dir.c b/dir.c\nindex 9f7c8debeab3..dfb174227b36 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2440,6 +2440,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \n \tif (open_cached_dir(&cdir, dir, untracked, istate, &path, check_only))\n \t\tgoto out;\n+\tdir->visited_directories++;\n \n \tif (untracked)\n \t\tuntracked->check_only = !!check_only;\n@@ -2448,6 +2449,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t/* check how the file or directory should be treated */\n \t\tstate = treat_path(dir, untracked, &cdir, istate, &path,\n \t\t\t\t baselen, pathspec);\n+\t\tdir->visited_paths++;\n \n \t\tif (state > dir_state)\n \t\t\tdir_state = state;\n@@ -2763,6 +2765,10 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n static void trace2_read_directory_statistics(struct dir_struct *dir,\n \t\t\t\t\t struct repository *repo)\n {\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"directories-visited\", dir->visited_directories);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"paths-visited\", dir->visited_paths);\n \tif (!dir->untracked)\n \t\treturn;\n \ttrace2_data_intmax(\"read_directory\", repo,\n@@ -2783,6 +2789,8 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tstruct untracked_cache_dir *untracked;\n \n \ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n+\tdir->visited_paths = 0;\n+\tdir->visited_directories = 0;\n \n \tif (has_symlink_leading_path(path, len)) {\n \t\ttrace_performance_leave(\"read directory %.*s\", len, path);\ndiff --git a/dir.h b/dir.h\nindex 04d886cfce75..22c67907f689 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -336,6 +336,10 @@ struct dir_struct {\n \tstruct oid_stat ss_info_exclude;\n \tstruct oid_stat ss_excludes_file;\n \tunsigned unmanaged_exclude_files;\n+\n+\t/* Stats about the traversal */\n+\tunsigned visited_paths;\n+\tunsigned visited_directories;\n };\n \n /*Count the number of slashes for string s*/\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 6bce65b439e3..1517c316892f 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -65,6 +65,7 @@ get_relevant_traces() {\n \tINPUT_FILE=$1\n \tOUTPUT_FILE=$2\n \tgrep data.*read_directo $INPUT_FILE \\\n+\t | grep -v visited \\\n \t | cut -d \"|\" -f 9 \\\n \t >$OUTPUT_FILE\n }\n-- \ngitgitgadget\n\n"},{"id":"423943","messageId":"44a1322c44026e675ea254a00f3b50d4955ac56e.1620503945.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","subject":"[PATCH v3 3/8] [RFC] ls-files: error out on -i unless -o or -c are specified","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T19:58:59Z","receivedAt":"2021-05-08T19:59:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nls-files --ignored can be used together with either --others or\n--cached. After being perplexed for a bit and digging in to the code, I\nassumed that ls-files -i was just broken and not printing anything and\nhad a nice patch ready to submit when I finally realized that -i can be\nused with --cached to find tracked ignores.\n\nWhile that was a mistake on my part, and a careful reading of the\ndocumentation could have made this more clear, I suspect this is an\nerror others are likely to make as well. In fact, of two uses in our\ntestsuite, I believe one of the two did make this error. In t1306.13,\nthere are NO tracked files, and all the excludes built up and used in\nthat test and in previous tests thus have to be about untracked files.\nHowever, since they were looking for an empty result, the mistake went\nunnoticed as their erroneous command also just happened to give an empty\nanswer.\n\n-i will most the time be used with -o, which would suggest we could just\nmake -i imply -o in the absence of either a -o or -c, but that would be\na backward incompatible break. Instead, let's just flag -i without\neither a -o or -c as an error, and update the two relevant testcases to\nspecify their intent.\n\nSigned-off-by: Elijah Newren \n---\n builtin/ls-files.c | 3 +++\n t/t1306-xdg-files.sh | 2 +-\n t/t3003-ls-files-exclude.sh | 4 ++--\n 3 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 60a2913a01e9..9f74b1ab2e69 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -748,6 +748,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tif (pathspec.nr && error_unmatch)\n \t\tps_matched = xcalloc(pathspec.nr, 1);\n \n+\tif ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached)\n+\t\tdie(\"ls-files --ignored is usually used with --others, but --cached is the default. Please specify which you want.\");\n+\n \tif ((dir.flags & DIR_SHOW_IGNORED) && !exc_given)\n \t\tdie(\"ls-files --ignored needs some exclude pattern\");\n \ndiff --git a/t/t1306-xdg-files.sh b/t/t1306-xdg-files.sh\nindex dd87b43be1a6..40d3c42618c0 100755\n--- a/t/t1306-xdg-files.sh\n+++ b/t/t1306-xdg-files.sh\n@@ -116,7 +116,7 @@ test_expect_success 'Exclusion in a non-XDG global ignore file' '\n test_expect_success 'Checking XDG ignore file when HOME is unset' '\n \t(sane_unset HOME &&\n \t git config --unset core.excludesfile &&\n-\t git ls-files --exclude-standard --ignored >actual) &&\n+\t git ls-files --exclude-standard --ignored --others >actual) &&\n \ttest_must_be_empty actual\n '\n \ndiff --git a/t/t3003-ls-files-exclude.sh b/t/t3003-ls-files-exclude.sh\nindex d5ec333131f9..c41c4f046abf 100755\n--- a/t/t3003-ls-files-exclude.sh\n+++ b/t/t3003-ls-files-exclude.sh\n@@ -29,11 +29,11 @@ test_expect_success 'add file to gitignore' '\n '\n check_all_output\n \n-test_expect_success 'ls-files -i lists only tracked-but-ignored files' '\n+test_expect_success 'ls-files -i -c lists only tracked-but-ignored files' '\n \techo content >other-file &&\n \tgit add other-file &&\n \techo file >expect &&\n-\tgit ls-files -i --exclude-standard >output &&\n+\tgit ls-files -i -c --exclude-standard >output &&\n \ttest_cmp expect output\n '\n \n-- \ngitgitgadget\n\n"},{"id":"423944","messageId":"dc3d3f2471410aa55da4dbc8e2747192888bce5f.1620503945.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","subject":"[PATCH v3 4/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T19:59:00Z","receivedAt":"2021-05-08T19:59:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe PNPM package manager is apparently creating deeply nested (but\nignored) directory structures; traversing them is costly\nperformance-wise, unnecessary, and in some cases is even throwing\nwarnings/errors because the paths are too long to handle on various\nplatforms. Add a testcase that checks for such unnecessary directory\ntraversal.\n\nSigned-off-by: Elijah Newren \n---\n t/t7300-clean.sh | 22 ++++++++++++++++++++++\n 1 file changed, 22 insertions(+)\n\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex a74816ca8b46..b7c9898fac5b 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,4 +746,26 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_failure 'avoid traversing into ignored directories' '\n+\ttest_when_finished rm -f output error trace.* &&\n+\ttest_create_repo avoid-traversing-deep-hierarchy &&\n+\t(\n+\t\tcd avoid-traversing-deep-hierarchy &&\n+\n+\t\tmkdir -p untracked/subdir/with/a &&\n+\t\t>untracked/subdir/with/a/random-file.txt &&\n+\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n+\t\tgit clean -ffdxn -e untracked\n+\t) &&\n+\n+\tgrep data.*read_directo.*visited trace.output \\\n+\t\t| cut -d \"|\" -f 9 >trace.relevant &&\n+\tcat >trace.expect <<-EOF &&\n+\t directories-visited:1\n+\t paths-visited:4\n+\tEOF\n+\ttest_cmp trace.expect trace.relevant\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"423945","messageId":"73b03a1e8e05fd4e1952c8f4b0bb7b8519a77862.1620503945.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","subject":"[PATCH v3 5/8] t3001, t7300: add testcase showcasing missed directory traversal","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T19:59:01Z","receivedAt":"2021-05-08T19:59:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn the last commit, we added a testcase showing that the directory\ntraversal machinery sometimes traverses into directories unnecessarily.\nHere we show that there are cases where it does the opposite: it does\nnot traverse into directories, despite those directories having\nimportant files that need to be flagged.\n\nAdd a testcase showing that `git ls-files -o -i --directory` can omit\nsome of the files it should be listing, and another showing that `git\nclean -fX` can fail to clean out some of the expected files.\n\nSigned-off-by: Elijah Newren \n---\n t/t3001-ls-files-others-exclude.sh | 5 +++++\n t/t7300-clean.sh | 19 +++++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex 1ec7cb57c7a8..ac05d1a17931 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,6 +292,11 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+\t# Expectation same as previous test\n+\tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n+\ttest_cmp expect actual\n+'\n \n test_expect_success 'ls-files with \"**\" patterns and no slashes' '\n \tgit ls-files -o -i --exclude \"one**a.1\" >actual &&\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex b7c9898fac5b..74d395838708 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -768,4 +768,23 @@ test_expect_failure 'avoid traversing into ignored directories' '\n \ttest_cmp trace.expect trace.relevant\n '\n \n+test_expect_failure 'traverse into directories that may have ignored entries' '\n+\ttest_when_finished rm -f output &&\n+\ttest_create_repo need-to-traverse-into-hierarchy &&\n+\t(\n+\t\tcd need-to-traverse-into-hierarchy &&\n+\t\tmkdir -p modules/foobar/src/generated &&\n+\t\t> modules/foobar/src/generated/code.c &&\n+\t\t> modules/foobar/Makefile &&\n+\t\techo \"/modules/**/src/generated/\" >.gitignore &&\n+\n+\t\tgit clean -fX modules/foobar >../output &&\n+\n+\t\tgrep Removing ../output &&\n+\n+\t\ttest_path_is_missing modules/foobar/src/generated/code.c &&\n+\t\ttest_path_is_file modules/foobar/Makefile\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"423946","messageId":"66ffc7f02d08f3f07cb3cb2605b113a630f1e127.1620503945.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","subject":"[PATCH v3 6/8] dir: avoid unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T19:59:02Z","receivedAt":"2021-05-08T19:59:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe show_other_directories case in treat_directory() tried to handle\nboth excludes and untracked files with the same logic, and mishandled\nboth the excludes and the untracked files in the process, in different\nways. Split that logic apart, and then focus on the logic for the\nexcludes; a subsequent commit will address the logic for untracked\nfiles.\n\nFor show_other_directories, an excluded directory means that\nevery path underneath that directory will also be excluded. Given that\nthe calling code requested to just show directories when everything\nunder a directory had the same state (that's what the\n\"DIR_SHOW_OTHER_DIRECTORIES\" flag means), we generally do not need to\ntraverse into such directories and can just immediately mark them as\nignored (i.e. as path_excluded). The only reason we cannot just\nimmediately return path_excluded is the DIR_HIDE_EMPTY_DIRECTORIES flag\nand the possibility that the ignored directory is an empty directory.\nThe code previously treated DIR_SHOW_IGNORED_TOO in most cases as an\nexception as well, which was wrong. It can sometimes reduce the number\nof cases where we need to recurse (namely if\nDIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set), but should not be able\nto increase the number of cases where we need to recurse. Fix the logic\naccordingly.\n\nSome sidenotes about possible confusion with dir.c:\n\n* \"ignored\" often refers to an untracked ignore\", i.e. a file which is\n not tracked which matches one of the ignore/exclusion rules. But you\n can also have a \"tracked ignore\", a tracked file that happens to match\n one of the ignore/exclusion rules and which dir.c has to worry about\n since \"git ls-files -c -i\" is supposed to list them.\n\n* The dir code often uses \"ignored\" and \"excluded\" interchangeably,\n which you need to keep in mind while reading the code. Sadly, though,\n it can get very confusing since ignore rules can have exclusions, as\n in the last of the following .gitignore rules:\n .gitignore\n *~\n *.log\n !settings.log\n In the last entry above, (pathspec->items[3].magic & PATHSPEC_EXCLUDE)\n will be true due the the '!' negating the rule. Someone might refer\n to this as \"excluded\". That means the file 'settings.log' will not\n match, and thus not be ignored. So we won't return path_excluded for\n it. So it's an exclude rule that prevents the file from being an\n exclude. The non-excluded rules are the ones that result in files\n being excludes. Great fun, eh?\n\nSometimes it feels like dir.c needs its own glossary with its many\ndefinitions, including the multiply-defined terms.\n\nReported-by: Jason Gore \nSigned-off-by: Elijah Newren \n---\n dir.c | 44 +++++++++++++++++++++++++++++---------------\n t/t7300-clean.sh | 2 +-\n 2 files changed, 30 insertions(+), 16 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex dfb174227b36..3f2cfef2c2bb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1844,6 +1844,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t}\n \n \t/* This is the \"show_other_directories\" case */\n+\tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n \t * If we have a pathspec which could match something _below_ this\n@@ -1854,27 +1855,40 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tif (matches_how == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n \t\treturn path_recurse;\n \n+\t/* Special cases for where this directory is excluded/ignored */\n+\tif (excluded) {\n+\t\t/*\n+\t\t * In the show_other_directories case, if we're not\n+\t\t * hiding empty directories, there is no need to\n+\t\t * recurse into an ignored directory.\n+\t\t */\n+\t\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\t\treturn path_excluded;\n+\n+\t\t/*\n+\t\t * Even if we are hiding empty directories, we can still avoid\n+\t\t * recursing into ignored directories for DIR_SHOW_IGNORED_TOO\n+\t\t * if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n+\t\t */\n+\t\tif ((dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n+\t\t\treturn path_excluded;\n+\t}\n+\n \t/*\n-\t * Other than the path_recurse case immediately above, we only need\n-\t * to recurse into untracked/ignored directories if either of the\n-\t * following bits is set:\n+\t * Other than the path_recurse case above, we only need to\n+\t * recurse into untracked directories if either of the following\n+\t * bits is set:\n \t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n \t * there are ignored entries below)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n-\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n-\t\treturn excluded ? path_excluded : path_untracked;\n-\n-\t/*\n-\t * ...and even if DIR_SHOW_IGNORED_TOO is set, we can still avoid\n-\t * recursing into ignored directories if the path is excluded and\n-\t * DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n-\t */\n-\tif (excluded &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n-\t\treturn path_excluded;\n+\tif (!excluded &&\n+\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n+\t\treturn path_untracked;\n+\t}\n \n \t/*\n \t * Even if we don't want to know all the paths under an untracked or\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 74d395838708..a1d695ee9fe9 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,7 +746,7 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_failure 'avoid traversing into ignored directories' '\n+test_expect_success 'avoid traversing into ignored directories' '\n \ttest_when_finished rm -f output error trace.* &&\n \ttest_create_repo avoid-traversing-deep-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"423947","messageId":"acde436b220e9bc9d1e1f3ae499331297ceb721e.1620503946.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","subject":"[PATCH v3 7/8] dir: traverse into untracked directories if they may have ignored subfiles","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T19:59:03Z","receivedAt":"2021-05-08T19:59:22Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA directory that is untracked does not imply that all files under it\nshould be categorized as untracked; in particular, if the caller is\ninterested in ignored files, many files or directories underneath the\nuntracked directory may be ignored. We previously partially handled\nthis right with DIR_SHOW_IGNORED_TOO, but missed DIR_SHOW_IGNORED. It\nwas not obvious, though, because the logic for untracked and excluded\nfiles had been fused together making it harder to reason about. The\nprevious commit split that logic out, making it easier to notice that\nDIR_SHOW_IGNORED was missing. Add it.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 10 ++++++----\n t/t3001-ls-files-others-exclude.sh | 2 +-\n t/t7300-clean.sh | 2 +-\n 3 files changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3f2cfef2c2bb..f5d9732d9e68 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1877,15 +1877,17 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/*\n \t * Other than the path_recurse case above, we only need to\n-\t * recurse into untracked directories if either of the following\n+\t * recurse into untracked directories if any of the following\n \t * bits is set:\n-\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n-\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED (because then we need to determine if\n+\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED_TOO (same as above)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n \tif (!excluded &&\n-\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t !(dir->flags & (DIR_SHOW_IGNORED |\n+\t\t\t DIR_SHOW_IGNORED_TOO |\n \t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n \t\treturn path_untracked;\n \t}\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex ac05d1a17931..516c95ea0e82 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,7 +292,7 @@ EOF\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+test_expect_success 'ls-files with \"**\" patterns and --directory' '\n \t# Expectation same as previous test\n \tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex a1d695ee9fe9..751764c0f1ae 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -768,7 +768,7 @@ test_expect_success 'avoid traversing into ignored directories' '\n \ttest_cmp trace.expect trace.relevant\n '\n \n-test_expect_failure 'traverse into directories that may have ignored entries' '\n+test_expect_success 'traverse into directories that may have ignored entries' '\n \ttest_when_finished rm -f output &&\n \ttest_create_repo need-to-traverse-into-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"423948","messageId":"57135c35777416a91c0ae6bac7f0a57293815ced.1620503946.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","subject":"[PATCH v3 8/8] dir: update stale description of treat_directory()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-08T19:59:04Z","receivedAt":"2021-05-08T19:59:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe documentation comment for treat_directory() was originally written\nin 095952 (Teach directory traversal about subprojects, 2007-04-11)\nwhich was before the 'struct dir_struct' split its bitfield of named\noptions into a 'flags' enum in 7c4c97c0 (Turn the flags in struct\ndir_struct into a single variable, 2009-02-16). When those flags\nchanged, the comment became stale, since members like\n'show_other_directories' transitioned into flags like\nDIR_SHOW_OTHER_DIRECTORIES.\n\nUpdate the comments for treat_directory() to use these flag names rather\nthan the old member names.\n\nSigned-off-by: Derrick Stolee \nReviewed-by: Elijah Newren \n---\n dir.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex f5d9732d9e68..896a9a62b2c7 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1749,13 +1749,13 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n * Case 3: if we didn't have it in the index previously, we\n * have a few sub-cases:\n *\n- * (a) if \"show_other_directories\" is true, we show it as\n- * just a directory, unless \"hide_empty_directories\" is\n+ * (a) if DIR_SHOW_OTHER_DIRECTORIES flag is set, we show it as\n+ * just a directory, unless DIR_HIDE_EMPTY_DIRECTORIES is\n * also true, in which case we need to check if it contains any\n * untracked and / or ignored files.\n- * (b) if it looks like a git directory, and we don't have\n- * 'no_gitlinks' set we treat it as a gitlink, and show it\n- * as a directory.\n+ * (b) if it looks like a git directory and we don't have the\n+ * DIR_NO_GITLINKS flag, then we treat it as a gitlink, and\n+ * show it as a directory.\n * (c) otherwise, we recurse into it.\n */\n static enum path_treatment treat_directory(struct dir_struct *dir,\n@@ -1843,7 +1843,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_recurse;\n \t}\n \n-\t/* This is the \"show_other_directories\" case */\n \tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n@@ -1858,7 +1857,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* Special cases for where this directory is excluded/ignored */\n \tif (excluded) {\n \t\t/*\n-\t\t * In the show_other_directories case, if we're not\n+\t\t * If DIR_SHOW_OTHER_DIRECTORIES is set and we're not\n \t\t * hiding empty directories, there is no need to\n \t\t * recurse into an ignored directory.\n \t\t */\n-- \ngitgitgadget\n"},{"id":"423994","messageId":"xmqqk0o7ry5c.fsf@gitster.g","threadId":"55639","inReplyTo":"9f1c0d78d739cc1da8517eea92e81a7bd162578f.1620503945.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/8] [RFC] dir: convert trace calls to trace2 equivalents","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-10T04:49:35Z","receivedAt":"2021-05-10T04:49:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> +static void trace2_read_directory_statistics(struct dir_struct *dir,\n> +\t\t\t\t\t struct repository *repo)\n> +{\n> +\tif (!dir->untracked)\n> +\t\treturn;\n> +\ttrace2_data_intmax(\"read_directory\", repo,\n> +\t\t\t \"node-creation\", dir->untracked->dir_created);\n> +\ttrace2_data_intmax(\"read_directory\", repo,\n> +\t\t\t \"gitignore-invalidation\",\n> +\t\t\t dir->untracked->gitignore_invalidated);\n> +\ttrace2_data_intmax(\"read_directory\", repo,\n> +\t\t\t \"directory-invalidation\",\n> +\t\t\t dir->untracked->dir_invalidated);\n> +\ttrace2_data_intmax(\"read_directory\", repo,\n> +\t\t\t \"opendir\", dir->untracked->dir_opened);\n> +}\n> +\n\nThis obviously looks like an equivalent to what happens in the\noriginal inside the \"if (dir->untracked)\" block.\n\nAnd we have a performance_{enter,leave} pair replaced with\na region_[enter,leave} pair. \n\n> -\ttrace_performance_enter();\n> +\ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n> ...\n> -\ttrace_performance_leave(\"read directory %.*s\", len, path);\n> +\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n\n> \tif (dir->untracked) {\n> \t\tstatic int force_untracked_cache = -1;\n> -\t\tstatic struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n> \n> \t\tif (force_untracked_cache < 0)\n> \t\t\tforce_untracked_cache =\n> \t\t\t\tgit_env_bool(\"GIT_FORCE_UNTRACKED_CACHE\", 0);\n> -\t\ttrace_printf_key(&trace_untracked_stats,\n> -\t\t\t\t \"node creation: %u\\n\"\n> -\t\t\t\t \"gitignore invalidation: %u\\n\"\n> -\t\t\t\t \"directory invalidation: %u\\n\"\n> -\t\t\t\t \"opendir: %u\\n\",\n> -\t\t\t\t dir->untracked->dir_created,\n> -\t\t\t\t dir->untracked->gitignore_invalidated,\n> -\t\t\t\t dir->untracked->dir_invalidated,\n> -\t\t\t\t dir->untracked->dir_opened);\n> \t\tif (force_untracked_cache &&\n> \t\t\tdir->untracked == istate->untracked &&\n> \t\t (dir->untracked->dir_opened ||\n\nRemoval of the trace_printf() in the middle made the body of this\nif() statement much less distracting, which is good.\n\n> @@ -2811,6 +2818,9 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n> \t\t\tFREE_AND_NULL(dir->untracked);\n> \t\t}\n> \t}\n> +\n> +\tif (trace2_is_enabled())\n> +\t\ttrace2_read_directory_statistics(dir, istate->repo);\n\nThis slightly changes the semantics in that the original did an\nequivalent emitting from inside the \"if (dir->untracked)\" block, but\nthis call is hoisted outside, and the new helper knows how to be\nsilent when untracked thing is not in effect, so the net effect at\nthis step is the same. And if we ever add tracing statics that is\nrelevant when !dir->untracked is true, the new code organization is\neasier to work with.\n\nThe only curious thing is the guard \"if (trace2_is_enabled())\";\ncorrectness-wise, are there bad things going to happen if it is not\nhere, or is this a performance hack, or is it more for its\ndocumentation value (meaning, it would be a bug if we later added\nthings that are irrelevant when trace is not enabled to the helper)?\n\n> @@ -57,6 +57,19 @@ iuc () {\n> \treturn $ret\n> }\n> \n> +get_relevant_traces() {\n\nStyle. SP on both sides of \"()\".\n\n> +\t# From the GIT_TRACE2_PERF data of the form\n> +\t# $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT\n> +\t# extract the $RELEVANT_STAT fields. We don't care about region_enter\n> +\t# or region_leave, or stats for things outside read_directory.\n> +\tINPUT_FILE=$1\n> +\tOUTPUT_FILE=$2\n> +\tgrep data.*read_directo $INPUT_FILE \\\n> +\t | cut -d \"|\" -f 9 \\\n> +\t >$OUTPUT_FILE\n\nStyle. Wrapping the line after pipe '|' will allow you to omit the\nbackslash. Also quote the redirection target, i.e. >\"$OUTPUT_FILE\",\nto help certain vintage of bash.\n\nThose who are more familiar with the trace2 infrastructure may want\nto further comment, but it looked obvious and straightforward to me.\n\nThanks.\n"},{"id":"423995","messageId":"xmqqfsyvrxmh.fsf@gitster.g","threadId":"55639","inReplyTo":"8b511f228af8e047d8b38159da658b10174c40e8.1620503945.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 2/8] [RFC] dir: report number of visited directories and paths with trace2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-10T05:00:54Z","receivedAt":"2021-05-10T05:01:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> From: Elijah Newren \n>\n> Provide more statistics in trace2 output that include the number of\n> directories and total paths visited by the directory traversal logic.\n> Subsequent patches will take advantage of this to ensure we do not\n> unnecessarily traverse into ignored directories.\n\nAnd this change is the reason behind how the call to the trace\nstatistics helper is now outside the \"if (untracked)\" block after\npatch 1/8; makes sense to me.\n\n"},{"id":"423996","messageId":"xmqqbl9jrx8f.fsf@gitster.g","threadId":"55639","inReplyTo":"44a1322c44026e675ea254a00f3b50d4955ac56e.1620503945.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 3/8] [RFC] ls-files: error out on -i unless -o or -c are specified","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-10T05:09:20Z","receivedAt":"2021-05-10T05:09:27Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> @@ -748,6 +748,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n> \tif (pathspec.nr && error_unmatch)\n> \t\tps_matched = xcalloc(pathspec.nr, 1);\n> \n> +\tif ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached)\n> +\t\tdie(\"ls-files --ignored is usually used with --others, but --cached is the default. Please specify which you want.\");\n> +\n\nSo \"git ls-files -i\" would suddenly start erroring out and users are\nto scramble and patch their scripts?\n\nMore importantly, the message does not make much sense. \"I is\nusually used with O\" is very true, but the mention of \"usually\" here\nmeans it is not an error for \"I\" to be used without \"O\". That part\nis very understandable and correct.\n\nBut I do not know what \"but --cached is the default\" part wants to\nsay. If it is the _default_, and (assuming that what I read in the\nproposed log message is correct) the combination of \"-i -c\" is valid,\nthen I would understand the message if the code were more like this:\n\n\tif ((dir.flags & DIR_SHOW_IGNORED) &&\n\t !show_others && !show_cached) {\n\t\tshow_cached = 1; /* default */\n\t\twarning(\"ls-files -i given without -o/-c; defaulting to -i -c\");\n\t}\n\nIf we are not defaulting to cached, then\n\n\tdie(\"ls-files -i must be used with either -o or -c\");\n\nwould also make sense.\n\nThe variant presented in the patch does not make sense to me.\n\n> diff --git a/t/t1306-xdg-files.sh b/t/t1306-xdg-files.sh\n> index dd87b43be1a6..40d3c42618c0 100755\n> --- a/t/t1306-xdg-files.sh\n> +++ b/t/t1306-xdg-files.sh\n> @@ -116,7 +116,7 @@ test_expect_success 'Exclusion in a non-XDG global ignore file' '\n> test_expect_success 'Checking XDG ignore file when HOME is unset' '\n> \t(sane_unset HOME &&\n> \t git config --unset core.excludesfile &&\n> -\t git ls-files --exclude-standard --ignored >actual) &&\n> +\t git ls-files --exclude-standard --ignored --others >actual) &&\n> \ttest_must_be_empty actual\n> '\n> \n> diff --git a/t/t3003-ls-files-exclude.sh b/t/t3003-ls-files-exclude.sh\n> index d5ec333131f9..c41c4f046abf 100755\n> --- a/t/t3003-ls-files-exclude.sh\n> +++ b/t/t3003-ls-files-exclude.sh\n> @@ -29,11 +29,11 @@ test_expect_success 'add file to gitignore' '\n> '\n> check_all_output\n> \n> -test_expect_success 'ls-files -i lists only tracked-but-ignored files' '\n> +test_expect_success 'ls-files -i -c lists only tracked-but-ignored files' '\n> \techo content >other-file &&\n> \tgit add other-file &&\n> \techo file >expect &&\n> -\tgit ls-files -i --exclude-standard >output &&\n> +\tgit ls-files -i -c --exclude-standard >output &&\n> \ttest_cmp expect output\n> '\n"},{"id":"423997","messageId":"xmqq7dk7rwcc.fsf@gitster.g","threadId":"55639","inReplyTo":"dc3d3f2471410aa55da4dbc8e2747192888bce5f.1620503945.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 4/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-10T05:28:35Z","receivedAt":"2021-05-10T05:28:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> +test_expect_failure 'avoid traversing into ignored directories' '\n> +\ttest_when_finished rm -f output error trace.* &&\n> +\ttest_create_repo avoid-traversing-deep-hierarchy &&\n> +\t(\n> +\t\tcd avoid-traversing-deep-hierarchy &&\n> +\n> +\t\tmkdir -p untracked/subdir/with/a &&\n> +\t\t>untracked/subdir/with/a/random-file.txt &&\n> +\n> +\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n> +\t\tgit clean -ffdxn -e untracked\n> +\t) &&\n> +\n> +\tgrep data.*read_directo.*visited trace.output \\\n> +\t\t| cut -d \"|\" -f 9 >trace.relevant &&\n> +\tcat >trace.expect <<-EOF &&\n> +\t directories-visited:1\n> +\t paths-visited:4\n\nAre the origins of '1' and '4' trivially obvious to those who are\nreading the test, or do these deserve comments?\n\nWe create an empty test repository, go there and create a untracked/\nhierarchy with a junk file, and tell \"clean\" that 'untracked' is\n\"also\" in the exclude pattern (but since there is no other exclude\npattern, that is the only one), so everything underneath untracked/\nwe have no reason to inspect.\n\nSo, we do not visit 'untracked' directory. Which ones do we visit?\nIs '1' coming from the top-level of the working tree '.'? What\nabout the number of visited paths '4' (the trace is stored outside\nthis new test repository, so that's not it).\n\nThanks.\n\n> +\tEOF\n> +\ttest_cmp trace.expect trace.relevant\n> +'\n> +\n> test_done\n"},{"id":"423999","messageId":"xmqq35uvrvfv.fsf@gitster.g","threadId":"55639","inReplyTo":"66ffc7f02d08f3f07cb3cb2605b113a630f1e127.1620503945.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 6/8] dir: avoid unnecessary traversal into ignored directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-10T05:48:04Z","receivedAt":"2021-05-10T05:48:08Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> Some sidenotes about possible confusion with dir.c:\n\nThanks for working on untangling this mess ;-)\n\n> * \"ignored\" often refers to an untracked ignore\", i.e. a file which is\n> not tracked which matches one of the ignore/exclusion rules. But you\n> can also have a \"tracked ignore\", a tracked file that happens to match\n> one of the ignore/exclusion rules and which dir.c has to worry about\n> since \"git ls-files -c -i\" is supposed to list them.\n\nOK. This is to find a pattern in .gitignore that is too broad\n(i.e. if the path were to be added as a new thing today, it would\nrequire \"add -f\"), right? The combination of \"-i -c\" does make\nsense for that purpose.\n\n> * The dir code often uses \"ignored\" and \"excluded\" interchangeably,\n> which you need to keep in mind while reading the code. \n\nTrue. In tree .gitignore files are to hold exclude patterns, and\nper repository personal exclude file is called $GIT_DIR/info/exclude\nwhich is confusing.\n\n> Sadly, though,\n> it can get very confusing since ignore rules can have exclusions, as\n> in the last of the following .gitignore rules:\n> .gitignore\n> *~\n> *.log\n> !settings.log\n> In the last entry above, (pathspec->items[3].magic & PATHSPEC_EXCLUDE)\n> will be true due the the '!' negating the rule. Someone might refer\n> to this as \"excluded\".\n\nThat one I've never heard of. As far as I am concerned, that is a\nnegative exclude pattern.\n\nI do wish we started the project with .gitignore files and\n$GIT_DIR/info/ignore both of which holds ignore patterns and\nnegative ignore patterns from day one, but the boat sailed\nlong time ago.\n\n"},{"id":"424140","messageId":"96679919-f6ea-adcc-b8fc-9eaa0134c876@jeffhostetler.com","threadId":"55639","inReplyTo":"9f1c0d78d739cc1da8517eea92e81a7bd162578f.1620503945.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/8] [RFC] dir: convert trace calls to trace2 equivalents","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-05-11T16:17:09Z","receivedAt":"2021-05-11T16:17:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/8/21 3:58 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Signed-off-by: Elijah Newren \n> ---\n> dir.c | 34 ++++--\n> t/t7063-status-untracked-cache.sh | 193 +++++++++++++++++-------------\n> t/t7519-status-fsmonitor.sh | 8 +-\n> 3 files changed, 135 insertions(+), 100 deletions(-)\n> \n> diff --git a/dir.c b/dir.c\n> index 3474e67e8f3c..9f7c8debeab3 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -2760,12 +2760,29 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n> \treturn root;\n> }\n> \n> +static void trace2_read_directory_statistics(struct dir_struct *dir,\n> +\t\t\t\t\t struct repository *repo)\n> +{\n> +\tif (!dir->untracked)\n> +\t\treturn;\n\nIs there value to also printing the path?\nThe existing `trace_performance_leave()` calls were, but\nI'm familiar enough with this code to say if the output\nwasn't always something like \".\".\n\n> +\ttrace2_data_intmax(\"read_directory\", repo,\n> +\t\t\t \"node-creation\", dir->untracked->dir_created);\n> +\ttrace2_data_intmax(\"read_directory\", repo,\n> +\t\t\t \"gitignore-invalidation\",\n> +\t\t\t dir->untracked->gitignore_invalidated);\n> +\ttrace2_data_intmax(\"read_directory\", repo,\n> +\t\t\t \"directory-invalidation\",\n> +\t\t\t dir->untracked->dir_invalidated);\n> +\ttrace2_data_intmax(\"read_directory\", repo,\n> +\t\t\t \"opendir\", dir->untracked->dir_opened);\n> +}\n> +\n\nThe existing code was quite tangled and I think this helps\nmake things more clear.\n\n\n> int read_directory(struct dir_struct *dir, struct index_state *istate,\n> \t\t const char *path, int len, const struct pathspec *pathspec)\n> {\n> \tstruct untracked_cache_dir *untracked;\n> \n> -\ttrace_performance_enter();\n> +\ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n> \n> \tif (has_symlink_leading_path(path, len)) {\n> \t\ttrace_performance_leave(\"read directory %.*s\", len, path);\n\nThis `trace_performance_leave()` inside the `if` needs to be\nconverted too.\n\n\n> @@ -2784,23 +2801,13 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n> \tQSORT(dir->entries, dir->nr, cmp_dir_entry);\n> \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n> \n> -\ttrace_performance_leave(\"read directory %.*s\", len, path);\n> +\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n\nCan we put the call to `trace2_read_directory_statistics()` before\nthe above `trace2_region_leave()` call? Then those stats will\nappear indented between the begin- and end-region events in the output.\n\nThat way, the following `if (dir-untracked) {...}` is only\nconcerned with the untracked cache and/or freeing that data.\n\n> \tif (dir->untracked) {\n> \t\tstatic int force_untracked_cache = -1;\n> -\t\tstatic struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n> \n> \t\tif (force_untracked_cache < 0)\n> \t\t\tforce_untracked_cache =\n> \t\t\t\tgit_env_bool(\"GIT_FORCE_UNTRACKED_CACHE\", 0);\n> -\t\ttrace_printf_key(&trace_untracked_stats,\n> -\t\t\t\t \"node creation: %u\\n\"\n> -\t\t\t\t \"gitignore invalidation: %u\\n\"\n> -\t\t\t\t \"directory invalidation: %u\\n\"\n> -\t\t\t\t \"opendir: %u\\n\",\n> -\t\t\t\t dir->untracked->dir_created,\n> -\t\t\t\t dir->untracked->gitignore_invalidated,\n> -\t\t\t\t dir->untracked->dir_invalidated,\n> -\t\t\t\t dir->untracked->dir_opened);\n> \t\tif (force_untracked_cache &&\n> \t\t\tdir->untracked == istate->untracked &&\n> \t\t (dir->untracked->dir_opened ||\n> @@ -2811,6 +2818,9 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n> \t\t\tFREE_AND_NULL(dir->untracked);\n> \t\t}\n> \t}\n> +\n> +\tif (trace2_is_enabled())\n> +\t\ttrace2_read_directory_statistics(dir, istate->repo);\n\nAlso, I think it'd be ok to move the `trace2_is_enabled()` call\ninside the function. Since we're also testing `!dir->untracked`\ninside the function.\n\nThe more that I look at the before and after versions, the\nmore I think the `trace2_read_directory_statistics()` call\nshould be up before the `trace2_region_leave()`. Here at the\nbottom of the function, we may have already freed `dir->untracked`.\nI'm not familiar enough with this code to know if that is a\ngood or bad thing.\n\n\n> \treturn dir->nr;\n> }\n> \n...\n\n\nThanks,\nJeff\n"},{"id":"424145","messageId":"CABPp-BFk_VNv7jzDP8Ep0Yk2e_nNg7-EAK1rbpS=9y9oNmi0kg@mail.gmail.com","threadId":"55639","inReplyTo":"xmqqk0o7ry5c.fsf@gitster.g","subject":"Re: [PATCH v3 1/8] [RFC] dir: convert trace calls to trace2 equivalents","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-11T17:23:48Z","receivedAt":"2021-05-11T17:24:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, May 9, 2021 at 9:49 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > +static void trace2_read_directory_statistics(struct dir_struct *dir,\n> > + struct repository *repo)\n> > +{\n> > + if (!dir->untracked)\n> > + return;\n> > + trace2_data_intmax(\"read_directory\", repo,\n> > + \"node-creation\", dir->untracked->dir_created);\n> > + trace2_data_intmax(\"read_directory\", repo,\n> > + \"gitignore-invalidation\",\n> > + dir->untracked->gitignore_invalidated);\n> > + trace2_data_intmax(\"read_directory\", repo,\n> > + \"directory-invalidation\",\n> > + dir->untracked->dir_invalidated);\n> > + trace2_data_intmax(\"read_directory\", repo,\n> > + \"opendir\", dir->untracked->dir_opened);\n> > +}\n> > +\n>\n> This obviously looks like an equivalent to what happens in the\n> original inside the \"if (dir->untracked)\" block.\n>\n> And we have a performance_{enter,leave} pair replaced with\n> a region_[enter,leave} pair.\n>\n> > - trace_performance_enter();\n> > + trace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n> > ...\n> > - trace_performance_leave(\"read directory %.*s\", len, path);\n> > + trace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n>\n> > if (dir->untracked) {\n> > static int force_untracked_cache = -1;\n> > - static struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n> >\n> > if (force_untracked_cache < 0)\n> > force_untracked_cache =\n> > git_env_bool(\"GIT_FORCE_UNTRACKED_CACHE\", 0);\n> > - trace_printf_key(&trace_untracked_stats,\n> > - \"node creation: %u\\n\"\n> > - \"gitignore invalidation: %u\\n\"\n> > - \"directory invalidation: %u\\n\"\n> > - \"opendir: %u\\n\",\n> > - dir->untracked->dir_created,\n> > - dir->untracked->gitignore_invalidated,\n> > - dir->untracked->dir_invalidated,\n> > - dir->untracked->dir_opened);\n> > if (force_untracked_cache &&\n> > dir->untracked == istate->untracked &&\n> > (dir->untracked->dir_opened ||\n>\n> Removal of the trace_printf() in the middle made the body of this\n> if() statement much less distracting, which is good.\n>\n> > @@ -2811,6 +2818,9 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n> > FREE_AND_NULL(dir->untracked);\n> > }\n> > }\n> > +\n> > + if (trace2_is_enabled())\n> > + trace2_read_directory_statistics(dir, istate->repo);\n>\n> This slightly changes the semantics in that the original did an\n> equivalent emitting from inside the \"if (dir->untracked)\" block, but\n> this call is hoisted outside, and the new helper knows how to be\n> silent when untracked thing is not in effect, so the net effect at\n> this step is the same. And if we ever add tracing statics that is\n> relevant when !dir->untracked is true, the new code organization is\n> easier to work with.\n>\n> The only curious thing is the guard \"if (trace2_is_enabled())\";\n> correctness-wise, are there bad things going to happen if it is not\n> here, or is this a performance hack, or is it more for its\n> documentation value (meaning, it would be a bug if we later added\n> things that are irrelevant when trace is not enabled to the helper)?\n\nNo, there's nothing bad that would happen here. It was a combination\nof a performance hack and documentation in case\ntrace2_read_directory_statistics() started gaining other code besides\ntrace2_*() calls, but which code was only relevant when trace2 was\nenabled.\n\nTurns out, though, that Jeff's suggestion to also print the path in\nthe statistics is going to require me creating a temporary strbuf so\nthat I can get a NUL-terminated string. We only want to do that when\ntrace2_is_enabled(), so that will make the introduction of that check\na bit more natural.\n\n> > @@ -57,6 +57,19 @@ iuc () {\n> > return $ret\n> > }\n> >\n> > +get_relevant_traces() {\n>\n> Style. SP on both sides of \"()\".\n\nWill fix.\n\n>\n> > + # From the GIT_TRACE2_PERF data of the form\n> > + # $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT\n> > + # extract the $RELEVANT_STAT fields. We don't care about region_enter\n> > + # or region_leave, or stats for things outside read_directory.\n> > + INPUT_FILE=$1\n> > + OUTPUT_FILE=$2\n> > + grep data.*read_directo $INPUT_FILE \\\n> > + | cut -d \"|\" -f 9 \\\n> > + >$OUTPUT_FILE\n>\n> Style. Wrapping the line after pipe '|' will allow you to omit the\n> backslash. Also quote the redirection target, i.e. >\"$OUTPUT_FILE\",\n> to help certain vintage of bash.\n\nWill fix.\n"},{"id":"424147","messageId":"CABPp-BEa0hgrTXWAA0hVAZ3uRhsfLY=jEe17+2X5pQpRKiejMg@mail.gmail.com","threadId":"55639","inReplyTo":"96679919-f6ea-adcc-b8fc-9eaa0134c876@jeffhostetler.com","subject":"Re: [PATCH v3 1/8] [RFC] dir: convert trace calls to trace2 equivalents","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-11T17:29:47Z","receivedAt":"2021-05-11T17:30:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, May 11, 2021 at 9:17 AM Jeff Hostetler wrote:\n>\n> On 5/8/21 3:58 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > dir.c | 34 ++++--\n> > t/t7063-status-untracked-cache.sh | 193 +++++++++++++++++-------------\n> > t/t7519-status-fsmonitor.sh | 8 +-\n> > 3 files changed, 135 insertions(+), 100 deletions(-)\n> >\n> > diff --git a/dir.c b/dir.c\n> > index 3474e67e8f3c..9f7c8debeab3 100644\n> > --- a/dir.c\n> > +++ b/dir.c\n> > @@ -2760,12 +2760,29 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n> > return root;\n> > }\n> >\n> > +static void trace2_read_directory_statistics(struct dir_struct *dir,\n> > + struct repository *repo)\n> > +{\n> > + if (!dir->untracked)\n> > + return;\n>\n> Is there value to also printing the path?\n> The existing `trace_performance_leave()` calls were, but\n> I'm familiar enough with this code to say if the output\n> wasn't always something like \".\".\n\nThe path will most likely just be \"\" (i.e. the empty string) for the\ntoplevel directory, but not always so it may be useful to print it.\nI'll add it.\n\n> > + trace2_data_intmax(\"read_directory\", repo,\n> > + \"node-creation\", dir->untracked->dir_created);\n> > + trace2_data_intmax(\"read_directory\", repo,\n> > + \"gitignore-invalidation\",\n> > + dir->untracked->gitignore_invalidated);\n> > + trace2_data_intmax(\"read_directory\", repo,\n> > + \"directory-invalidation\",\n> > + dir->untracked->dir_invalidated);\n> > + trace2_data_intmax(\"read_directory\", repo,\n> > + \"opendir\", dir->untracked->dir_opened);\n> > +}\n> > +\n>\n> The existing code was quite tangled and I think this helps\n> make things more clear.\n>\n>\n> > int read_directory(struct dir_struct *dir, struct index_state *istate,\n> > const char *path, int len, const struct pathspec *pathspec)\n> > {\n> > struct untracked_cache_dir *untracked;\n> >\n> > - trace_performance_enter();\n> > + trace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n> >\n> > if (has_symlink_leading_path(path, len)) {\n> > trace_performance_leave(\"read directory %.*s\", len, path);\n>\n> This `trace_performance_leave()` inside the `if` needs to be\n> converted too.\n\nOoh, good catch. Will fix.\n\n> > @@ -2784,23 +2801,13 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n> > QSORT(dir->entries, dir->nr, cmp_dir_entry);\n> > QSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n> >\n> > - trace_performance_leave(\"read directory %.*s\", len, path);\n> > + trace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n>\n> Can we put the call to `trace2_read_directory_statistics()` before\n> the above `trace2_region_leave()` call? Then those stats will\n> appear indented between the begin- and end-region events in the output.\n>\n> That way, the following `if (dir-untracked) {...}` is only\n> concerned with the untracked cache and/or freeing that data.\n\nMakes sense, I'll move it.\n\n> > if (dir->untracked) {\n> > static int force_untracked_cache = -1;\n> > - static struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n> >\n> > if (force_untracked_cache < 0)\n> > force_untracked_cache =\n> > git_env_bool(\"GIT_FORCE_UNTRACKED_CACHE\", 0);\n> > - trace_printf_key(&trace_untracked_stats,\n> > - \"node creation: %u\\n\"\n> > - \"gitignore invalidation: %u\\n\"\n> > - \"directory invalidation: %u\\n\"\n> > - \"opendir: %u\\n\",\n> > - dir->untracked->dir_created,\n> > - dir->untracked->gitignore_invalidated,\n> > - dir->untracked->dir_invalidated,\n> > - dir->untracked->dir_opened);\n> > if (force_untracked_cache &&\n> > dir->untracked == istate->untracked &&\n> > (dir->untracked->dir_opened ||\n> > @@ -2811,6 +2818,9 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n> > FREE_AND_NULL(dir->untracked);\n> > }\n> > }\n> > +\n> > + if (trace2_is_enabled())\n> > + trace2_read_directory_statistics(dir, istate->repo);\n>\n> Also, I think it'd be ok to move the `trace2_is_enabled()` call\n> inside the function. Since we're also testing `!dir->untracked`\n> inside the function.\n\nActually, I can't do that. The path passed to this function is not\ngoing to always be (and will often not be) NUL-terminated, but\ntrace2_data_string() expects a NUL-terminated string. So, I'm going\nto make a temporary strbuf and copy the path into it, but of course I\nonly want to spend time doing that if trace2_is_enabled().\n\n> The more that I look at the before and after versions, the\n> more I think the `trace2_read_directory_statistics()` call\n> should be up before the `trace2_region_leave()`. Here at the\n> bottom of the function, we may have already freed `dir->untracked`.\n> I'm not familiar enough with this code to know if that is a\n> good or bad thing.\n\nYeah, the statistics really need to be moved earlier, both for the\nnesting reasons you point out and because otherwise the statistics\nwon't print whenever dir->untracked != istate->untracked. I'll move\nthem.\n"},{"id":"424149","messageId":"CABPp-BE7V233NYJARGXaTUZdt+L=N55B27UrV6pOuKaKOUWRWA@mail.gmail.com","threadId":"55639","inReplyTo":"xmqqbl9jrx8f.fsf@gitster.g","subject":"Re: [PATCH v3 3/8] [RFC] ls-files: error out on -i unless -o or -c are specified","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-11T17:40:45Z","receivedAt":"2021-05-11T17:41:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, May 9, 2021 at 10:09 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > @@ -748,6 +748,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n> > if (pathspec.nr && error_unmatch)\n> > ps_matched = xcalloc(pathspec.nr, 1);\n> >\n> > + if ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached)\n> > + die(\"ls-files --ignored is usually used with --others, but --cached is the default. Please specify which you want.\");\n> > +\n>\n> So \"git ls-files -i\" would suddenly start erroring out and users are\n> to scramble and patch their scripts?\n\nThus the reason I marked this as \"RFC\" and called it out in the cover\nletter for folks to comment on.\n\nI figured that if I was having difficulty using it correctly and even\nour own testsuite showed that 50% of such invocations were wrong\n(despite being reviewed[1]), then it seems likely to me that erroring\nout to inform folks of this problem might be warranted. But, if folks\ndisagree, I can switch it to a warning instead.\n\n[1] https://lore.kernel.org/git/20120724133227.GA14422@sigill.intra.peff.net/#t\n\n> More importantly, the message does not make much sense. \"I is\n> usually used with O\" is very true, but the mention of \"usually\" here\n> means it is not an error for \"I\" to be used without \"O\". That part\n> is very understandable and correct.\n>\n> But I do not know what \"but --cached is the default\" part wants to\n> say. If it is the _default_, and (assuming that what I read in the\n> proposed log message is correct) the combination of \"-i -c\" is valid,\n> then I would understand the message if the code were more like this:\n>\n> if ((dir.flags & DIR_SHOW_IGNORED) &&\n> !show_others && !show_cached) {\n> show_cached = 1; /* default */\n> warning(\"ls-files -i given without -o/-c; defaulting to -i -c\");\n> }\n>\n> If we are not defaulting to cached, then\n>\n> die(\"ls-files -i must be used with either -o or -c\");\n>\n> would also make sense.\n\nOoh, that wording is much nicer. I'll adopt the latter suggestion,\nbut let me know if you'd rather I went the warning route.\n"},{"id":"424150","messageId":"CABPp-BFV6R9GNeZVbiKx_Tazq+6LuBNCzLcsEq63MfG5YD0hFQ@mail.gmail.com","threadId":"55639","inReplyTo":"xmqq7dk7rwcc.fsf@gitster.g","subject":"Re: [PATCH v3 4/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-11T17:45:56Z","receivedAt":"2021-05-11T17:46:09Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, May 9, 2021 at 10:28 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > +test_expect_failure 'avoid traversing into ignored directories' '\n> > + test_when_finished rm -f output error trace.* &&\n> > + test_create_repo avoid-traversing-deep-hierarchy &&\n> > + (\n> > + cd avoid-traversing-deep-hierarchy &&\n> > +\n> > + mkdir -p untracked/subdir/with/a &&\n> > + >untracked/subdir/with/a/random-file.txt &&\n> > +\n> > + GIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n> > + git clean -ffdxn -e untracked\n> > + ) &&\n> > +\n> > + grep data.*read_directo.*visited trace.output \\\n> > + | cut -d \"|\" -f 9 >trace.relevant &&\n> > + cat >trace.expect <<-EOF &&\n> > + directories-visited:1\n> > + paths-visited:4\n>\n> Are the origins of '1' and '4' trivially obvious to those who are\n> reading the test, or do these deserve comments?\n>\n> We create an empty test repository, go there and create a untracked/\n> hierarchy with a junk file, and tell \"clean\" that 'untracked' is\n> \"also\" in the exclude pattern (but since there is no other exclude\n> pattern, that is the only one), so everything underneath untracked/\n> we have no reason to inspect.\n>\n> So, we do not visit 'untracked' directory. Which ones do we visit?\n> Is '1' coming from the top-level of the working tree '.'? What\n> about the number of visited paths '4' (the trace is stored outside\n> this new test repository, so that's not it).\n\nGood points. I'll make a comment that directories-visited:1 is about\nensuring we only went into the toplevel directory, and I'll removed\nthe paths-visited check.\n\nBut to answer your question, the paths we visit are '.', '..', '.git',\nand 'untracked', the first three of which we mark as path_none and\ndon't recurse into because of special rules for those paths, and the\nlast of which we shouldn't recurse into since it is ignored. There\nweren't any non-directory files in the toplevel directory, or those\nwould also be included in the paths-visited count. A later patch in\nthe series will fix the code to not recurse into the 'untracked'\ndirectory, fixing this test.\n"},{"id":"424152","messageId":"CABPp-BE6-qH6NPf4-g5MiWwTsX=58vFGEOY4dTA2sY=zfyaEMg@mail.gmail.com","threadId":"55639","inReplyTo":"xmqq35uvrvfv.fsf@gitster.g","subject":"Re: [PATCH v3 6/8] dir: avoid unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-11T17:57:12Z","receivedAt":"2021-05-11T17:57:26Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, May 9, 2021 at 10:48 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n> > Some sidenotes about possible confusion with dir.c:\n>\n> Thanks for working on untangling this mess ;-)\n>\n> > * \"ignored\" often refers to an untracked ignore\", i.e. a file which is\n> > not tracked which matches one of the ignore/exclusion rules. But you\n> > can also have a \"tracked ignore\", a tracked file that happens to match\n> > one of the ignore/exclusion rules and which dir.c has to worry about\n> > since \"git ls-files -c -i\" is supposed to list them.\n>\n> OK. This is to find a pattern in .gitignore that is too broad\n> (i.e. if the path were to be added as a new thing today, it would\n> require \"add -f\"), right? The combination of \"-i -c\" does make\n> sense for that purpose.\n>\n> > * The dir code often uses \"ignored\" and \"excluded\" interchangeably,\n> > which you need to keep in mind while reading the code.\n>\n> True. In tree .gitignore files are to hold exclude patterns, and\n> per repository personal exclude file is called $GIT_DIR/info/exclude\n> which is confusing.\n>\n> > Sadly, though,\n> > it can get very confusing since ignore rules can have exclusions, as\n> > in the last of the following .gitignore rules:\n> > .gitignore\n> > *~\n> > *.log\n> > !settings.log\n> > In the last entry above, (pathspec->items[3].magic & PATHSPEC_EXCLUDE)\n> > will be true due the the '!' negating the rule. Someone might refer\n> > to this as \"excluded\".\n>\n> That one I've never heard of. As far as I am concerned, that is a\n> negative exclude pattern.\n\nOops, I was mixing up negative exclude patterns and negative (or\nexcluded) pathspecs. So \"exclude\" can refer to \"ignored\" files, or be\nused in \"PATHSPEC_EXCLUDE\" for excluded pathspecs.\n\n...and there's another way it's used. \"exclude\" can also be used to\nrefer to \"exclude\" patterns, meaning the patterns that .gitignore (and\nrelated files) use. However, .git/info/sparse-checkout re-used these\nsame rulesets, but then used them to determine path *inclusion*. At\nmy request, Stolee mostly fixed that up in 65edd96aec (\"treewide:\nrename 'exclude' methods to 'pattern'\", 2019-09-03) but you can still\noccasionally find a code comment referring to an \"exclude\" pattern\nthat might actually be used by the sparse-checkout stuff as an\ninclusion rule.\n\nAnd then we have a myriad of other variables and comments with \"excl\"\nin their name that might be derived from any of the above three...and\nit's sometimes difficult for me to remember which one of the concepts\nsuch a derived variable or comment might be referring to.\n\n*sigh*\n\n> I do wish we started the project with .gitignore files and\n> $GIT_DIR/info/ignore both of which holds ignore patterns and\n> negative ignore patterns from day one, but the boat sailed\n> long time ago.\n"},{"id":"424159","messageId":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v3.git.git.1620503945.gitgitgadget@gmail.com","subject":"[PATCH v4 0/8] Directory traversal fixes","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-11T18:34:00Z","receivedAt":"2021-05-11T18:34:13Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patchset fixes a few directory traversal issues, where fill_directory()\nwould traverse into directories that it shouldn't and not traverse into\ndirectories that it should (one of which was originally reported on this\nlist at [1]). And it includes a few cleanups\n\nChanges since v3, includes numerous cleanups suggested by Junio and Jeff\n(thanks for the reviews!):\n\n * Removed the RFC labels, but if folks want a warning instead of a die on\n ls-files -i (see patch 3), let me know\n * Include the path passed to read_directory() in the printed trace2\n statistics\n * Print trace2 statistics before calling trace2_region_leave()\n * Make sure to convert both trace_performance_leave() calls\n * testcase style fixes\n * left a comment that directories-visited:1 referred to the toplevel\n directory\n * fixed up some commit message comments about \"exclude\" and mentioned yet\n another way it can be confusing.\n\nAs noted in previous versions of this series, if folks would prefer ls-files\n-i to continue running but print a warning rather than making it an error as\nI did in this series, let me know. Also, if anyone has any ideas about a\nbetter place to put the \"Some sidenotes\" from the sixth commit message\nrather than keeping them in a random commit message, that might be helpful\ntoo.\n\n[1] See\nhttps://lore.kernel.org/git/DM6PR00MB06829EC5B85E0C5AC595004E894E9@DM6PR00MB0682.namprd00.prod.outlook.com/\nor alternatively https://github.com/git-for-windows/git/issues/2732.\n\nDerrick Stolee (1):\n dir: update stale description of treat_directory()\n\nElijah Newren (7):\n dir: convert trace calls to trace2 equivalents\n dir: report number of visited directories and paths with trace2\n ls-files: error out on -i unless -o or -c are specified\n t7300: add testcase showing unnecessary traversal into ignored\n directory\n t3001, t7300: add testcase showcasing missed directory traversal\n dir: avoid unnecessary traversal into ignored directory\n dir: traverse into untracked directories if they may have ignored\n subfiles\n\n builtin/ls-files.c | 3 +\n dir.c | 112 +++++++++++-----\n dir.h | 4 +\n t/t1306-xdg-files.sh | 2 +-\n t/t3001-ls-files-others-exclude.sh | 5 +\n t/t3003-ls-files-exclude.sh | 4 +-\n t/t7063-status-untracked-cache.sh | 206 +++++++++++++++++------------\n t/t7300-clean.sh | 42 ++++++\n t/t7519-status-fsmonitor.sh | 8 +-\n 9 files changed, 259 insertions(+), 127 deletions(-)\n\n\nbase-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1020%2Fnewren%2Fdirectory-traversal-fixes-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1020/newren/directory-traversal-fixes-v4\nPull-Request: https://github.com/git/git/pull/1020\n\nRange-diff vs v3:\n\n 1: 9f1c0d78d739 ! 1: 9204e36b7e90 [RFC] dir: convert trace calls to trace2 equivalents\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - [RFC] dir: convert trace calls to trace2 equivalents\n + dir: convert trace calls to trace2 equivalents\n \n Signed-off-by: Elijah Newren \n \n @@ dir.c: static struct untracked_cache_dir *validate_untracked_cache(struct dir_st\n }\n \n +static void trace2_read_directory_statistics(struct dir_struct *dir,\n -+\t\t\t\t\t struct repository *repo)\n ++\t\t\t\t\t struct repository *repo,\n ++\t\t\t\t\t const char *path)\n +{\n +\tif (!dir->untracked)\n +\t\treturn;\n ++\ttrace2_data_string(\"read_directory\", repo, \"path\", path);\n +\ttrace2_data_intmax(\"read_directory\", repo,\n +\t\t\t \"node-creation\", dir->untracked->dir_created);\n +\ttrace2_data_intmax(\"read_directory\", repo,\n @@ dir.c: static struct untracked_cache_dir *validate_untracked_cache(struct dir_st\n +\ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n \n \tif (has_symlink_leading_path(path, len)) {\n - \t\ttrace_performance_leave(\"read directory %.*s\", len, path);\n +-\t\ttrace_performance_leave(\"read directory %.*s\", len, path);\n ++\t\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n + \t\treturn dir->nr;\n + \t}\n + \n @@ dir.c: int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tQSORT(dir->entries, dir->nr, cmp_dir_entry);\n \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n \n -\ttrace_performance_leave(\"read directory %.*s\", len, path);\n ++\tif (trace2_is_enabled()) {\n ++\t\tstruct strbuf tmp = STRBUF_INIT;\n ++\t\tstrbuf_add(&tmp, path, len);\n ++\t\ttrace2_read_directory_statistics(dir, istate->repo, tmp.buf);\n ++\t\tstrbuf_release(&tmp);\n ++\t}\n ++\n +\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n \tif (dir->untracked) {\n \t\tstatic int force_untracked_cache = -1;\n @@ dir.c: int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t}\n \t}\n +\n -+\tif (trace2_is_enabled())\n -+\t\ttrace2_read_directory_statistics(dir, istate->repo);\n \treturn dir->nr;\n }\n \n @@ t/t7063-status-untracked-cache.sh: iuc () {\n \treturn $ret\n }\n \n -+get_relevant_traces() {\n ++get_relevant_traces () {\n +\t# From the GIT_TRACE2_PERF data of the form\n +\t# $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT\n +\t# extract the $RELEVANT_STAT fields. We don't care about region_enter\n +\t# or region_leave, or stats for things outside read_directory.\n +\tINPUT_FILE=$1\n +\tOUTPUT_FILE=$2\n -+\tgrep data.*read_directo $INPUT_FILE \\\n -+\t | cut -d \"|\" -f 9 \\\n -+\t >$OUTPUT_FILE\n ++\tgrep data.*read_directo $INPUT_FILE |\n ++\t cut -d \"|\" -f 9 \\\n ++\t >\"$OUTPUT_FILE\"\n +}\n +\n +\n @@ t/t7063-status-untracked-cache.sh: EOF\n -gitignore invalidation: 1\n -directory invalidation: 0\n -opendir: 4\n -+ ..node-creation:3\n -+ ..gitignore-invalidation:1\n -+ ..directory-invalidation:0\n -+ ..opendir:4\n ++ ....path:\n ++ ....node-creation:3\n ++ ....gitignore-invalidation:1\n ++ ....directory-invalidation:0\n ++ ....opendir:4\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: test_expect_success 'untracked cache after fi\n -gitignore invalidation: 0\n -directory invalidation: 0\n -opendir: 0\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:0\n -+ ..directory-invalidation:0\n -+ ..opendir:0\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:0\n ++ ....directory-invalidation:0\n ++ ....opendir:0\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: A two\n -gitignore invalidation: 0\n -directory invalidation: 1\n -opendir: 1\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:0\n -+ ..directory-invalidation:1\n -+ ..opendir:1\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:0\n ++ ....directory-invalidation:1\n ++ ....opendir:1\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: A two\n -gitignore invalidation: 1\n -directory invalidation: 1\n -opendir: 4\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:1\n -+ ..directory-invalidation:1\n -+ ..opendir:4\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:1\n ++ ....directory-invalidation:1\n ++ ....opendir:4\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: A two\n -gitignore invalidation: 1\n -directory invalidation: 0\n -opendir: 4\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:1\n -+ ..directory-invalidation:0\n -+ ..opendir:4\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:1\n ++ ....directory-invalidation:0\n ++ ....opendir:4\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: A one\n -gitignore invalidation: 0\n -directory invalidation: 0\n -opendir: 1\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:0\n -+ ..directory-invalidation:0\n -+ ..opendir:1\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:0\n ++ ....directory-invalidation:0\n ++ ....opendir:1\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: A two\n -gitignore invalidation: 0\n -directory invalidation: 0\n -opendir: 1\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:0\n -+ ..directory-invalidation:0\n -+ ..opendir:1\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:0\n ++ ....directory-invalidation:0\n ++ ....opendir:1\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: test_expect_success 'status after commit' '\n -gitignore invalidation: 0\n -directory invalidation: 0\n -opendir: 2\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:0\n -+ ..directory-invalidation:0\n -+ ..opendir:2\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:0\n ++ ....directory-invalidation:0\n ++ ....opendir:2\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: test_expect_success 'test sparse status with\n -gitignore invalidation: 1\n -directory invalidation: 2\n -opendir: 2\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:1\n -+ ..directory-invalidation:2\n -+ ..opendir:2\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:1\n ++ ....directory-invalidation:2\n ++ ....opendir:2\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: test_expect_success 'test sparse status again\n -gitignore invalidation: 0\n -directory invalidation: 0\n -opendir: 0\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:0\n -+ ..directory-invalidation:0\n -+ ..opendir:0\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:0\n ++ ....directory-invalidation:0\n ++ ....opendir:0\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: test_expect_success 'test sparse status with\n -gitignore invalidation: 0\n -directory invalidation: 1\n -opendir: 3\n -+ ..node-creation:2\n -+ ..gitignore-invalidation:0\n -+ ..directory-invalidation:1\n -+ ..opendir:3\n ++ ....path:\n ++ ....node-creation:2\n ++ ....gitignore-invalidation:0\n ++ ....directory-invalidation:1\n ++ ....opendir:3\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n @@ t/t7063-status-untracked-cache.sh: EOF\n -gitignore invalidation: 0\n -directory invalidation: 0\n -opendir: 0\n -+ ..node-creation:0\n -+ ..gitignore-invalidation:0\n -+ ..directory-invalidation:0\n -+ ..opendir:0\n ++ ....path:\n ++ ....node-creation:0\n ++ ....gitignore-invalidation:0\n ++ ....directory-invalidation:0\n ++ ....opendir:0\n EOF\n -\ttest_cmp ../trace.expect ../trace\n +\ttest_cmp ../trace.expect ../trace.relevant\n 2: 8b511f228af8 ! 2: 6939253be825 [RFC] dir: report number of visited directories and paths with trace2\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - [RFC] dir: report number of visited directories and paths with trace2\n + dir: report number of visited directories and paths with trace2\n \n Provide more statistics in trace2 output that include the number of\n directories and total paths visited by the directory traversal logic.\n @@ dir.c: static enum path_treatment read_directory_recursive(struct dir_struct *di\n \n \t\tif (state > dir_state)\n \t\t\tdir_state = state;\n -@@ dir.c: static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n - static void trace2_read_directory_statistics(struct dir_struct *dir,\n - \t\t\t\t\t struct repository *repo)\n +@@ dir.c: static void trace2_read_directory_statistics(struct dir_struct *dir,\n + \t\t\t\t\t struct repository *repo,\n + \t\t\t\t\t const char *path)\n {\n +\ttrace2_data_intmax(\"read_directory\", repo,\n +\t\t\t \"directories-visited\", dir->visited_directories);\n @@ dir.c: static struct untracked_cache_dir *validate_untracked_cache(struct dir_st\n +\t\t\t \"paths-visited\", dir->visited_paths);\n \tif (!dir->untracked)\n \t\treturn;\n - \ttrace2_data_intmax(\"read_directory\", repo,\n + \ttrace2_data_string(\"read_directory\", repo, \"path\", path);\n @@ dir.c: int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tstruct untracked_cache_dir *untracked;\n \n @@ dir.c: int read_directory(struct dir_struct *dir, struct index_state *istate,\n +\tdir->visited_directories = 0;\n \n \tif (has_symlink_leading_path(path, len)) {\n - \t\ttrace_performance_leave(\"read directory %.*s\", len, path);\n + \t\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n \n ## dir.h ##\n @@ dir.h: struct dir_struct {\n @@ dir.h: struct dir_struct {\n /*Count the number of slashes for string s*/\n \n ## t/t7063-status-untracked-cache.sh ##\n -@@ t/t7063-status-untracked-cache.sh: get_relevant_traces() {\n +@@ t/t7063-status-untracked-cache.sh: get_relevant_traces () {\n \tINPUT_FILE=$1\n \tOUTPUT_FILE=$2\n - \tgrep data.*read_directo $INPUT_FILE \\\n -+\t | grep -v visited \\\n - \t | cut -d \"|\" -f 9 \\\n - \t >$OUTPUT_FILE\n + \tgrep data.*read_directo $INPUT_FILE |\n +-\t cut -d \"|\" -f 9 \\\n ++\t cut -d \"|\" -f 9 |\n ++\t grep -v visited \\\n + \t >\"$OUTPUT_FILE\"\n }\n + \n 3: 44a1322c4402 ! 3: 8d0ca8104be6 [RFC] ls-files: error out on -i unless -o or -c are specified\n @@ Metadata\n Author: Elijah Newren \n \n ## Commit message ##\n - [RFC] ls-files: error out on -i unless -o or -c are specified\n + ls-files: error out on -i unless -o or -c are specified\n \n ls-files --ignored can be used together with either --others or\n --cached. After being perplexed for a bit and digging in to the code, I\n assumed that ls-files -i was just broken and not printing anything and\n - had a nice patch ready to submit when I finally realized that -i can be\n + I had a nice patch ready to submit when I finally realized that -i can be\n used with --cached to find tracked ignores.\n \n While that was a mistake on my part, and a careful reading of the\n @@ builtin/ls-files.c: int cmd_ls_files(int argc, const char **argv, const char *cm\n \t\tps_matched = xcalloc(pathspec.nr, 1);\n \n +\tif ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached)\n -+\t\tdie(\"ls-files --ignored is usually used with --others, but --cached is the default. Please specify which you want.\");\n ++\t\tdie(\"ls-files -i must be used with either -o or -c\");\n +\n \tif ((dir.flags & DIR_SHOW_IGNORED) && !exc_given)\n \t\tdie(\"ls-files --ignored needs some exclude pattern\");\n 4: dc3d3f247141 ! 4: 317abab3571e t7300: add testcase showing unnecessary traversal into ignored directory\n @@ t/t7300-clean.sh: test_expect_success 'clean untracked paths by pathspec' '\n +\t\tgit clean -ffdxn -e untracked\n +\t) &&\n +\n -+\tgrep data.*read_directo.*visited trace.output \\\n -+\t\t| cut -d \"|\" -f 9 >trace.relevant &&\n ++\t# Make sure we only visited into the top-level directory, and did\n ++\t# not traverse into the \"untracked\" subdirectory since it was excluded\n ++\tgrep data.*read_directo.*directories-visited trace.output |\n ++\t\tcut -d \"|\" -f 9 >trace.relevant &&\n +\tcat >trace.expect <<-EOF &&\n -+\t directories-visited:1\n -+\t paths-visited:4\n ++\t ..directories-visited:1\n +\tEOF\n +\ttest_cmp trace.expect trace.relevant\n +'\n 5: 73b03a1e8e05 = 5: 5eb019327b57 t3001, t7300: add testcase showcasing missed directory traversal\n 6: 66ffc7f02d08 ! 6: 89cc01ef8598 dir: avoid unnecessary traversal into ignored directory\n @@ Commit message\n since \"git ls-files -c -i\" is supposed to list them.\n \n * The dir code often uses \"ignored\" and \"excluded\" interchangeably,\n - which you need to keep in mind while reading the code. Sadly, though,\n - it can get very confusing since ignore rules can have exclusions, as\n - in the last of the following .gitignore rules:\n - .gitignore\n + which you need to keep in mind while reading the code.\n +\n + * \"exclude\" is used multiple ways in the code:\n +\n + * As noted above, \"exclude\" is often a synonym for \"ignored\".\n +\n + * The logic for parsing .gitignore files was re-used in\n + .git/info/sparse-checkout, except there it is used to mark paths that\n + the user wants to *keep*. This was mostly addressed by commit\n + 65edd96aec (\"treewide: rename 'exclude' methods to 'pattern'\",\n + 2019-09-03), but every once in a while you'll find a comment about\n + \"exclude\" referring to these patterns that might in fact be in use\n + by the sparse-checkout machinery for inclusion rules.\n +\n + * The word \"EXCLUDE\" is also used for pathspec negation, as in\n + (pathspec->items[3].magic & PATHSPEC_EXCLUDE)\n + Thus if a user had a .gitignore file containing\n *~\n *.log\n !settings.log\n - In the last entry above, (pathspec->items[3].magic & PATHSPEC_EXCLUDE)\n - will be true due the the '!' negating the rule. Someone might refer\n - to this as \"excluded\". That means the file 'settings.log' will not\n - match, and thus not be ignored. So we won't return path_excluded for\n - it. So it's an exclude rule that prevents the file from being an\n - exclude. The non-excluded rules are the ones that result in files\n - being excludes. Great fun, eh?\n + And then ran\n + git add -- 'settings.*' ':^settings.log'\n + Then :^settings.log is a pathspec negation making settings.log not\n + be requested to be added even though all other settings.* files are\n + being added. Also, !settings.log in the gitignore file is a negative\n + exclude pattern meaning that settings.log is normally a file we\n + want to track even though all other *.log files are ignored.\n \n Sometimes it feels like dir.c needs its own glossary with its many\n definitions, including the multiply-defined terms.\n 7: acde436b220e = 7: 4a561e1229e4 dir: traverse into untracked directories if they may have ignored subfiles\n 8: 57135c357774 = 8: 2945e749f5e3 dir: update stale description of treat_directory()\n\n-- \ngitgitgadget\n"},{"id":"424160","messageId":"9204e36b7e9035c4cdda018d7ced8e8ca7acc8bc.1620758049.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","subject":"[PATCH v4 1/8] dir: convert trace calls to trace2 equivalents","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-11T18:34:01Z","receivedAt":"2021-05-11T18:34:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 43 +++++--\n t/t7063-status-untracked-cache.sh | 205 ++++++++++++++++++------------\n t/t7519-status-fsmonitor.sh | 8 +-\n 3 files changed, 155 insertions(+), 101 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3474e67e8f3c..122fcbffdf89 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2760,15 +2760,34 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \treturn root;\n }\n \n+static void trace2_read_directory_statistics(struct dir_struct *dir,\n+\t\t\t\t\t struct repository *repo,\n+\t\t\t\t\t const char *path)\n+{\n+\tif (!dir->untracked)\n+\t\treturn;\n+\ttrace2_data_string(\"read_directory\", repo, \"path\", path);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"node-creation\", dir->untracked->dir_created);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"gitignore-invalidation\",\n+\t\t\t dir->untracked->gitignore_invalidated);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"directory-invalidation\",\n+\t\t\t dir->untracked->dir_invalidated);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"opendir\", dir->untracked->dir_opened);\n+}\n+\n int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t const char *path, int len, const struct pathspec *pathspec)\n {\n \tstruct untracked_cache_dir *untracked;\n \n-\ttrace_performance_enter();\n+\ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n \n \tif (has_symlink_leading_path(path, len)) {\n-\t\ttrace_performance_leave(\"read directory %.*s\", len, path);\n+\t\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n \t\treturn dir->nr;\n \t}\n \n@@ -2784,23 +2803,20 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tQSORT(dir->entries, dir->nr, cmp_dir_entry);\n \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n \n-\ttrace_performance_leave(\"read directory %.*s\", len, path);\n+\tif (trace2_is_enabled()) {\n+\t\tstruct strbuf tmp = STRBUF_INIT;\n+\t\tstrbuf_add(&tmp, path, len);\n+\t\ttrace2_read_directory_statistics(dir, istate->repo, tmp.buf);\n+\t\tstrbuf_release(&tmp);\n+\t}\n+\n+\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n \tif (dir->untracked) {\n \t\tstatic int force_untracked_cache = -1;\n-\t\tstatic struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n \n \t\tif (force_untracked_cache < 0)\n \t\t\tforce_untracked_cache =\n \t\t\t\tgit_env_bool(\"GIT_FORCE_UNTRACKED_CACHE\", 0);\n-\t\ttrace_printf_key(&trace_untracked_stats,\n-\t\t\t\t \"node creation: %u\\n\"\n-\t\t\t\t \"gitignore invalidation: %u\\n\"\n-\t\t\t\t \"directory invalidation: %u\\n\"\n-\t\t\t\t \"opendir: %u\\n\",\n-\t\t\t\t dir->untracked->dir_created,\n-\t\t\t\t dir->untracked->gitignore_invalidated,\n-\t\t\t\t dir->untracked->dir_invalidated,\n-\t\t\t\t dir->untracked->dir_opened);\n \t\tif (force_untracked_cache &&\n \t\t\tdir->untracked == istate->untracked &&\n \t\t (dir->untracked->dir_opened ||\n@@ -2811,6 +2827,7 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t\tFREE_AND_NULL(dir->untracked);\n \t\t}\n \t}\n+\n \treturn dir->nr;\n }\n \ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex accefde72fb1..9710d33b3cd6 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -57,6 +57,19 @@ iuc () {\n \treturn $ret\n }\n \n+get_relevant_traces () {\n+\t# From the GIT_TRACE2_PERF data of the form\n+\t# $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT\n+\t# extract the $RELEVANT_STAT fields. We don't care about region_enter\n+\t# or region_leave, or stats for things outside read_directory.\n+\tINPUT_FILE=$1\n+\tOUTPUT_FILE=$2\n+\tgrep data.*read_directo $INPUT_FILE |\n+\t cut -d \"|\" -f 9 \\\n+\t >\"$OUTPUT_FILE\"\n+}\n+\n+\n test_lazy_prereq UNTRACKED_CACHE '\n \t{ git update-index --test-untracked-cache; ret=$?; } &&\n \ttest $ret -ne 1\n@@ -129,19 +142,21 @@ EOF\n \n test_expect_success 'status first time (empty cache)' '\n \tavoid_racy &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <four &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <.gitignore &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <>.git/info/exclude &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n+\t: >../trace.output &&\n \tavoid_racy &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../status.actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <../before\n \t) &&\n@@ -346,12 +346,12 @@ test_expect_success UNTRACKED_CACHE 'ignore .git changes when invalidating UNTR'\n \tEOF\n \t(\n \t\tcd dot-git &&\n-\t\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace-after\" \\\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace-after\" \\\n \t\tgit status &&\n \t\ttest-tool dump-untracked-cache >../after\n \t) &&\n-\tgrep \"directory invalidation\" trace-before >>before &&\n-\tgrep \"directory invalidation\" trace-after >>after &&\n+\tgrep \"directory-invalidation\" trace-before | cut -d\"|\" -f 9 >>before &&\n+\tgrep \"directory-invalidation\" trace-after | cut -d\"|\" -f 9 >>after &&\n \t# UNTR extension unchanged, dir invalidation count unchanged\n \ttest_cmp before after\n '\n-- \ngitgitgadget\n\n"},{"id":"424161","messageId":"8d0ca8104be628d02b30065009427062887b5e0a.1620758049.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","subject":"[PATCH v4 3/8] ls-files: error out on -i unless -o or -c are specified","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-11T18:34:03Z","receivedAt":"2021-05-11T18:34:17Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nls-files --ignored can be used together with either --others or\n--cached. After being perplexed for a bit and digging in to the code, I\nassumed that ls-files -i was just broken and not printing anything and\nI had a nice patch ready to submit when I finally realized that -i can be\nused with --cached to find tracked ignores.\n\nWhile that was a mistake on my part, and a careful reading of the\ndocumentation could have made this more clear, I suspect this is an\nerror others are likely to make as well. In fact, of two uses in our\ntestsuite, I believe one of the two did make this error. In t1306.13,\nthere are NO tracked files, and all the excludes built up and used in\nthat test and in previous tests thus have to be about untracked files.\nHowever, since they were looking for an empty result, the mistake went\nunnoticed as their erroneous command also just happened to give an empty\nanswer.\n\n-i will most the time be used with -o, which would suggest we could just\nmake -i imply -o in the absence of either a -o or -c, but that would be\na backward incompatible break. Instead, let's just flag -i without\neither a -o or -c as an error, and update the two relevant testcases to\nspecify their intent.\n\nSigned-off-by: Elijah Newren \n---\n builtin/ls-files.c | 3 +++\n t/t1306-xdg-files.sh | 2 +-\n t/t3003-ls-files-exclude.sh | 4 ++--\n 3 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 60a2913a01e9..e8e25006c647 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -748,6 +748,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tif (pathspec.nr && error_unmatch)\n \t\tps_matched = xcalloc(pathspec.nr, 1);\n \n+\tif ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached)\n+\t\tdie(\"ls-files -i must be used with either -o or -c\");\n+\n \tif ((dir.flags & DIR_SHOW_IGNORED) && !exc_given)\n \t\tdie(\"ls-files --ignored needs some exclude pattern\");\n \ndiff --git a/t/t1306-xdg-files.sh b/t/t1306-xdg-files.sh\nindex dd87b43be1a6..40d3c42618c0 100755\n--- a/t/t1306-xdg-files.sh\n+++ b/t/t1306-xdg-files.sh\n@@ -116,7 +116,7 @@ test_expect_success 'Exclusion in a non-XDG global ignore file' '\n test_expect_success 'Checking XDG ignore file when HOME is unset' '\n \t(sane_unset HOME &&\n \t git config --unset core.excludesfile &&\n-\t git ls-files --exclude-standard --ignored >actual) &&\n+\t git ls-files --exclude-standard --ignored --others >actual) &&\n \ttest_must_be_empty actual\n '\n \ndiff --git a/t/t3003-ls-files-exclude.sh b/t/t3003-ls-files-exclude.sh\nindex d5ec333131f9..c41c4f046abf 100755\n--- a/t/t3003-ls-files-exclude.sh\n+++ b/t/t3003-ls-files-exclude.sh\n@@ -29,11 +29,11 @@ test_expect_success 'add file to gitignore' '\n '\n check_all_output\n \n-test_expect_success 'ls-files -i lists only tracked-but-ignored files' '\n+test_expect_success 'ls-files -i -c lists only tracked-but-ignored files' '\n \techo content >other-file &&\n \tgit add other-file &&\n \techo file >expect &&\n-\tgit ls-files -i --exclude-standard >output &&\n+\tgit ls-files -i -c --exclude-standard >output &&\n \ttest_cmp expect output\n '\n \n-- \ngitgitgadget\n\n"},{"id":"424162","messageId":"6939253be825f7c79dc5fb5fa13e420b876950d7.1620758049.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","subject":"[PATCH v4 2/8] dir: report number of visited directories and paths with trace2","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-11T18:34:02Z","receivedAt":"2021-05-11T18:34:18Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nProvide more statistics in trace2 output that include the number of\ndirectories and total paths visited by the directory traversal logic.\nSubsequent patches will take advantage of this to ensure we do not\nunnecessarily traverse into ignored directories.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 8 ++++++++\n dir.h | 4 ++++\n t/t7063-status-untracked-cache.sh | 3 ++-\n 3 files changed, 14 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 122fcbffdf89..69b8c9d7f9fb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2440,6 +2440,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \n \tif (open_cached_dir(&cdir, dir, untracked, istate, &path, check_only))\n \t\tgoto out;\n+\tdir->visited_directories++;\n \n \tif (untracked)\n \t\tuntracked->check_only = !!check_only;\n@@ -2448,6 +2449,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t/* check how the file or directory should be treated */\n \t\tstate = treat_path(dir, untracked, &cdir, istate, &path,\n \t\t\t\t baselen, pathspec);\n+\t\tdir->visited_paths++;\n \n \t\tif (state > dir_state)\n \t\t\tdir_state = state;\n@@ -2764,6 +2766,10 @@ static void trace2_read_directory_statistics(struct dir_struct *dir,\n \t\t\t\t\t struct repository *repo,\n \t\t\t\t\t const char *path)\n {\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"directories-visited\", dir->visited_directories);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"paths-visited\", dir->visited_paths);\n \tif (!dir->untracked)\n \t\treturn;\n \ttrace2_data_string(\"read_directory\", repo, \"path\", path);\n@@ -2785,6 +2791,8 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tstruct untracked_cache_dir *untracked;\n \n \ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n+\tdir->visited_paths = 0;\n+\tdir->visited_directories = 0;\n \n \tif (has_symlink_leading_path(path, len)) {\n \t\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\ndiff --git a/dir.h b/dir.h\nindex 04d886cfce75..22c67907f689 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -336,6 +336,10 @@ struct dir_struct {\n \tstruct oid_stat ss_info_exclude;\n \tstruct oid_stat ss_excludes_file;\n \tunsigned unmanaged_exclude_files;\n+\n+\t/* Stats about the traversal */\n+\tunsigned visited_paths;\n+\tunsigned visited_directories;\n };\n \n /*Count the number of slashes for string s*/\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 9710d33b3cd6..a0c123b0a77a 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -65,7 +65,8 @@ get_relevant_traces () {\n \tINPUT_FILE=$1\n \tOUTPUT_FILE=$2\n \tgrep data.*read_directo $INPUT_FILE |\n-\t cut -d \"|\" -f 9 \\\n+\t cut -d \"|\" -f 9 |\n+\t grep -v visited \\\n \t >\"$OUTPUT_FILE\"\n }\n \n-- \ngitgitgadget\n\n"},{"id":"424163","messageId":"317abab3571e69394cf0a0dbe971440bca16964b.1620758049.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","subject":"[PATCH v4 4/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-11T18:34:04Z","receivedAt":"2021-05-11T18:34:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe PNPM package manager is apparently creating deeply nested (but\nignored) directory structures; traversing them is costly\nperformance-wise, unnecessary, and in some cases is even throwing\nwarnings/errors because the paths are too long to handle on various\nplatforms. Add a testcase that checks for such unnecessary directory\ntraversal.\n\nSigned-off-by: Elijah Newren \n---\n t/t7300-clean.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex a74816ca8b46..07e8ba2d4b85 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,4 +746,27 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_failure 'avoid traversing into ignored directories' '\n+\ttest_when_finished rm -f output error trace.* &&\n+\ttest_create_repo avoid-traversing-deep-hierarchy &&\n+\t(\n+\t\tcd avoid-traversing-deep-hierarchy &&\n+\n+\t\tmkdir -p untracked/subdir/with/a &&\n+\t\t>untracked/subdir/with/a/random-file.txt &&\n+\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n+\t\tgit clean -ffdxn -e untracked\n+\t) &&\n+\n+\t# Make sure we only visited into the top-level directory, and did\n+\t# not traverse into the \"untracked\" subdirectory since it was excluded\n+\tgrep data.*read_directo.*directories-visited trace.output |\n+\t\tcut -d \"|\" -f 9 >trace.relevant &&\n+\tcat >trace.expect <<-EOF &&\n+\t ..directories-visited:1\n+\tEOF\n+\ttest_cmp trace.expect trace.relevant\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"424164","messageId":"5eb019327b57e710de976301ae3ec7a022645a95.1620758049.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","subject":"[PATCH v4 5/8] t3001, t7300: add testcase showcasing missed directory traversal","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-11T18:34:05Z","receivedAt":"2021-05-11T18:34:19Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn the last commit, we added a testcase showing that the directory\ntraversal machinery sometimes traverses into directories unnecessarily.\nHere we show that there are cases where it does the opposite: it does\nnot traverse into directories, despite those directories having\nimportant files that need to be flagged.\n\nAdd a testcase showing that `git ls-files -o -i --directory` can omit\nsome of the files it should be listing, and another showing that `git\nclean -fX` can fail to clean out some of the expected files.\n\nSigned-off-by: Elijah Newren \n---\n t/t3001-ls-files-others-exclude.sh | 5 +++++\n t/t7300-clean.sh | 19 +++++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex 1ec7cb57c7a8..ac05d1a17931 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,6 +292,11 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+\t# Expectation same as previous test\n+\tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n+\ttest_cmp expect actual\n+'\n \n test_expect_success 'ls-files with \"**\" patterns and no slashes' '\n \tgit ls-files -o -i --exclude \"one**a.1\" >actual &&\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 07e8ba2d4b85..34c08c325407 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -769,4 +769,23 @@ test_expect_failure 'avoid traversing into ignored directories' '\n \ttest_cmp trace.expect trace.relevant\n '\n \n+test_expect_failure 'traverse into directories that may have ignored entries' '\n+\ttest_when_finished rm -f output &&\n+\ttest_create_repo need-to-traverse-into-hierarchy &&\n+\t(\n+\t\tcd need-to-traverse-into-hierarchy &&\n+\t\tmkdir -p modules/foobar/src/generated &&\n+\t\t> modules/foobar/src/generated/code.c &&\n+\t\t> modules/foobar/Makefile &&\n+\t\techo \"/modules/**/src/generated/\" >.gitignore &&\n+\n+\t\tgit clean -fX modules/foobar >../output &&\n+\n+\t\tgrep Removing ../output &&\n+\n+\t\ttest_path_is_missing modules/foobar/src/generated/code.c &&\n+\t\ttest_path_is_file modules/foobar/Makefile\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"424165","messageId":"4a561e1229e46124030856530258d6a89ce5e938.1620758049.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","subject":"[PATCH v4 7/8] dir: traverse into untracked directories if they may have ignored subfiles","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-11T18:34:07Z","receivedAt":"2021-05-11T18:34:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA directory that is untracked does not imply that all files under it\nshould be categorized as untracked; in particular, if the caller is\ninterested in ignored files, many files or directories underneath the\nuntracked directory may be ignored. We previously partially handled\nthis right with DIR_SHOW_IGNORED_TOO, but missed DIR_SHOW_IGNORED. It\nwas not obvious, though, because the logic for untracked and excluded\nfiles had been fused together making it harder to reason about. The\nprevious commit split that logic out, making it easier to notice that\nDIR_SHOW_IGNORED was missing. Add it.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 10 ++++++----\n t/t3001-ls-files-others-exclude.sh | 2 +-\n t/t7300-clean.sh | 2 +-\n 3 files changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 0126e2f08af7..deeff1a58319 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1877,15 +1877,17 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/*\n \t * Other than the path_recurse case above, we only need to\n-\t * recurse into untracked directories if either of the following\n+\t * recurse into untracked directories if any of the following\n \t * bits is set:\n-\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n-\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED (because then we need to determine if\n+\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED_TOO (same as above)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n \tif (!excluded &&\n-\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t !(dir->flags & (DIR_SHOW_IGNORED |\n+\t\t\t DIR_SHOW_IGNORED_TOO |\n \t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n \t\treturn path_untracked;\n \t}\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex ac05d1a17931..516c95ea0e82 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,7 +292,7 @@ EOF\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+test_expect_success 'ls-files with \"**\" patterns and --directory' '\n \t# Expectation same as previous test\n \tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 21e48b3ba591..0399701e6276 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -769,7 +769,7 @@ test_expect_success 'avoid traversing into ignored directories' '\n \ttest_cmp trace.expect trace.relevant\n '\n \n-test_expect_failure 'traverse into directories that may have ignored entries' '\n+test_expect_success 'traverse into directories that may have ignored entries' '\n \ttest_when_finished rm -f output &&\n \ttest_create_repo need-to-traverse-into-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"424166","messageId":"89cc01ef8598b5faa529a4aea7e1e296465ef8e3.1620758049.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","subject":"[PATCH v4 6/8] dir: avoid unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-11T18:34:06Z","receivedAt":"2021-05-11T18:34:21Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe show_other_directories case in treat_directory() tried to handle\nboth excludes and untracked files with the same logic, and mishandled\nboth the excludes and the untracked files in the process, in different\nways. Split that logic apart, and then focus on the logic for the\nexcludes; a subsequent commit will address the logic for untracked\nfiles.\n\nFor show_other_directories, an excluded directory means that\nevery path underneath that directory will also be excluded. Given that\nthe calling code requested to just show directories when everything\nunder a directory had the same state (that's what the\n\"DIR_SHOW_OTHER_DIRECTORIES\" flag means), we generally do not need to\ntraverse into such directories and can just immediately mark them as\nignored (i.e. as path_excluded). The only reason we cannot just\nimmediately return path_excluded is the DIR_HIDE_EMPTY_DIRECTORIES flag\nand the possibility that the ignored directory is an empty directory.\nThe code previously treated DIR_SHOW_IGNORED_TOO in most cases as an\nexception as well, which was wrong. It can sometimes reduce the number\nof cases where we need to recurse (namely if\nDIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set), but should not be able\nto increase the number of cases where we need to recurse. Fix the logic\naccordingly.\n\nSome sidenotes about possible confusion with dir.c:\n\n* \"ignored\" often refers to an untracked ignore\", i.e. a file which is\n not tracked which matches one of the ignore/exclusion rules. But you\n can also have a \"tracked ignore\", a tracked file that happens to match\n one of the ignore/exclusion rules and which dir.c has to worry about\n since \"git ls-files -c -i\" is supposed to list them.\n\n* The dir code often uses \"ignored\" and \"excluded\" interchangeably,\n which you need to keep in mind while reading the code.\n\n* \"exclude\" is used multiple ways in the code:\n\n * As noted above, \"exclude\" is often a synonym for \"ignored\".\n\n * The logic for parsing .gitignore files was re-used in\n .git/info/sparse-checkout, except there it is used to mark paths that\n the user wants to *keep*. This was mostly addressed by commit\n 65edd96aec (\"treewide: rename 'exclude' methods to 'pattern'\",\n 2019-09-03), but every once in a while you'll find a comment about\n \"exclude\" referring to these patterns that might in fact be in use\n by the sparse-checkout machinery for inclusion rules.\n\n * The word \"EXCLUDE\" is also used for pathspec negation, as in\n (pathspec->items[3].magic & PATHSPEC_EXCLUDE)\n Thus if a user had a .gitignore file containing\n *~\n *.log\n !settings.log\n And then ran\n git add -- 'settings.*' ':^settings.log'\n Then :^settings.log is a pathspec negation making settings.log not\n be requested to be added even though all other settings.* files are\n being added. Also, !settings.log in the gitignore file is a negative\n exclude pattern meaning that settings.log is normally a file we\n want to track even though all other *.log files are ignored.\n\nSometimes it feels like dir.c needs its own glossary with its many\ndefinitions, including the multiply-defined terms.\n\nReported-by: Jason Gore \nSigned-off-by: Elijah Newren \n---\n dir.c | 44 +++++++++++++++++++++++++++++---------------\n t/t7300-clean.sh | 2 +-\n 2 files changed, 30 insertions(+), 16 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 69b8c9d7f9fb..0126e2f08af7 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1844,6 +1844,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t}\n \n \t/* This is the \"show_other_directories\" case */\n+\tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n \t * If we have a pathspec which could match something _below_ this\n@@ -1854,27 +1855,40 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tif (matches_how == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n \t\treturn path_recurse;\n \n+\t/* Special cases for where this directory is excluded/ignored */\n+\tif (excluded) {\n+\t\t/*\n+\t\t * In the show_other_directories case, if we're not\n+\t\t * hiding empty directories, there is no need to\n+\t\t * recurse into an ignored directory.\n+\t\t */\n+\t\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\t\treturn path_excluded;\n+\n+\t\t/*\n+\t\t * Even if we are hiding empty directories, we can still avoid\n+\t\t * recursing into ignored directories for DIR_SHOW_IGNORED_TOO\n+\t\t * if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n+\t\t */\n+\t\tif ((dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n+\t\t\treturn path_excluded;\n+\t}\n+\n \t/*\n-\t * Other than the path_recurse case immediately above, we only need\n-\t * to recurse into untracked/ignored directories if either of the\n-\t * following bits is set:\n+\t * Other than the path_recurse case above, we only need to\n+\t * recurse into untracked directories if either of the following\n+\t * bits is set:\n \t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n \t * there are ignored entries below)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n-\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n-\t\treturn excluded ? path_excluded : path_untracked;\n-\n-\t/*\n-\t * ...and even if DIR_SHOW_IGNORED_TOO is set, we can still avoid\n-\t * recursing into ignored directories if the path is excluded and\n-\t * DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n-\t */\n-\tif (excluded &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n-\t\treturn path_excluded;\n+\tif (!excluded &&\n+\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n+\t\treturn path_untracked;\n+\t}\n \n \t/*\n \t * Even if we don't want to know all the paths under an untracked or\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 34c08c325407..21e48b3ba591 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,7 +746,7 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_failure 'avoid traversing into ignored directories' '\n+test_expect_success 'avoid traversing into ignored directories' '\n \ttest_when_finished rm -f output error trace.* &&\n \ttest_create_repo avoid-traversing-deep-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"424167","messageId":"2945e749f5e398ead58a1e97c653145c2cf7cca5.1620758049.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","subject":"[PATCH v4 8/8] dir: update stale description of treat_directory()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-11T18:34:08Z","receivedAt":"2021-05-11T18:34:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe documentation comment for treat_directory() was originally written\nin 095952 (Teach directory traversal about subprojects, 2007-04-11)\nwhich was before the 'struct dir_struct' split its bitfield of named\noptions into a 'flags' enum in 7c4c97c0 (Turn the flags in struct\ndir_struct into a single variable, 2009-02-16). When those flags\nchanged, the comment became stale, since members like\n'show_other_directories' transitioned into flags like\nDIR_SHOW_OTHER_DIRECTORIES.\n\nUpdate the comments for treat_directory() to use these flag names rather\nthan the old member names.\n\nSigned-off-by: Derrick Stolee \nReviewed-by: Elijah Newren \n---\n dir.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex deeff1a58319..993a12145f9d 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1749,13 +1749,13 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n * Case 3: if we didn't have it in the index previously, we\n * have a few sub-cases:\n *\n- * (a) if \"show_other_directories\" is true, we show it as\n- * just a directory, unless \"hide_empty_directories\" is\n+ * (a) if DIR_SHOW_OTHER_DIRECTORIES flag is set, we show it as\n+ * just a directory, unless DIR_HIDE_EMPTY_DIRECTORIES is\n * also true, in which case we need to check if it contains any\n * untracked and / or ignored files.\n- * (b) if it looks like a git directory, and we don't have\n- * 'no_gitlinks' set we treat it as a gitlink, and show it\n- * as a directory.\n+ * (b) if it looks like a git directory and we don't have the\n+ * DIR_NO_GITLINKS flag, then we treat it as a gitlink, and\n+ * show it as a directory.\n * (c) otherwise, we recurse into it.\n */\n static enum path_treatment treat_directory(struct dir_struct *dir,\n@@ -1843,7 +1843,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_recurse;\n \t}\n \n-\t/* This is the \"show_other_directories\" case */\n \tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n@@ -1858,7 +1857,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* Special cases for where this directory is excluded/ignored */\n \tif (excluded) {\n \t\t/*\n-\t\t * In the show_other_directories case, if we're not\n+\t\t * If DIR_SHOW_OTHER_DIRECTORIES is set and we're not\n \t\t * hiding empty directories, there is no need to\n \t\t * recurse into an ignored directory.\n \t\t */\n-- \ngitgitgadget\n"},{"id":"424178","messageId":"df3695ad-5ba7-df22-2a2a-ca799c5d16d8@jeffhostetler.com","threadId":"55639","inReplyTo":"9204e36b7e9035c4cdda018d7ced8e8ca7acc8bc.1620758049.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 1/8] dir: convert trace calls to trace2 equivalents","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-05-11T19:06:03Z","receivedAt":"2021-05-11T19:06:14Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/11/21 2:34 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Signed-off-by: Elijah Newren \n> ---\n> dir.c | 43 +++++--\n> t/t7063-status-untracked-cache.sh | 205 ++++++++++++++++++------------\n> t/t7519-status-fsmonitor.sh | 8 +-\n> 3 files changed, 155 insertions(+), 101 deletions(-)\n> \n> diff --git a/dir.c b/dir.c\n> index 3474e67e8f3c..122fcbffdf89 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -2760,15 +2760,34 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n> \treturn root;\n> }\n> \n> +static void trace2_read_directory_statistics(struct dir_struct *dir,\n> +\t\t\t\t\t struct repository *repo,\n> +\t\t\t\t\t const char *path)\n> +{\n> +\tif (!dir->untracked)\n> +\t\treturn;\n> +\ttrace2_data_string(\"read_directory\", repo, \"path\", path);\n\nI'm probably just nit-picking here, but should this look more like:\n\n\tif (path && *path)\n\t\ttrace2_data_string(...)\n\tif (!dir->untracked)\n\t\treturn;\n\nThen when you add the visitied fields in the next commit,\nyou'll have the path with them (when present).\n\n(and it would let you optionally avoid the tmp strbuf in\nthe caller.)\n\nJeff\n"},{"id":"424192","messageId":"CABPp-BGeOUOvYRD+gX4jVR0kN8O5_icKtT18F9BdeR424DjUmg@mail.gmail.com","threadId":"55639","inReplyTo":"df3695ad-5ba7-df22-2a2a-ca799c5d16d8@jeffhostetler.com","subject":"Re: [PATCH v4 1/8] dir: convert trace calls to trace2 equivalents","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-11T20:12:12Z","receivedAt":"2021-05-11T20:12:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, May 11, 2021 at 12:06 PM Jeff Hostetler wrote:\n>\n> On 5/11/21 2:34 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > dir.c | 43 +++++--\n> > t/t7063-status-untracked-cache.sh | 205 ++++++++++++++++++------------\n> > t/t7519-status-fsmonitor.sh | 8 +-\n> > 3 files changed, 155 insertions(+), 101 deletions(-)\n> >\n> > diff --git a/dir.c b/dir.c\n> > index 3474e67e8f3c..122fcbffdf89 100644\n> > --- a/dir.c\n> > +++ b/dir.c\n> > @@ -2760,15 +2760,34 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n> > return root;\n> > }\n> >\n> > +static void trace2_read_directory_statistics(struct dir_struct *dir,\n> > + struct repository *repo,\n> > + const char *path)\n> > +{\n> > + if (!dir->untracked)\n> > + return;\n> > + trace2_data_string(\"read_directory\", repo, \"path\", path);\n>\n> I'm probably just nit-picking here, but should this look more like:\n\nnit-picking and questions are totally fine. :-) Thanks for reviewing.\n\n>\n> if (path && *path)\n> trace2_data_string(...)\n\npath is always non-NULL (it'd be an error to call read_directory()\nwith a NULL path). So the first part of the check isn't meaningful\nfor this particular code. The second half is interesting. Do we want\nto omit the path when it happens to be the toplevel directory (the\ncase where !*path)? The original trace_performance_leave() calls\ncertainly didn't, and I was just trying to provide the same info they\ndo, as you suggested. I guess people could determine the path by\nknowing that the code doesn't print it when it's empty, but do we want\ntrace2 users to need to read the code to figure out statistics and\ninfo?\n\n> if (!dir->untracked)\n> return;\n>\n> Then when you add the visitied fields in the next commit,\n> you'll have the path with them (when present).\n\nThere is always a path with them, it's just that the empty string\ndenotes the toplevel directory.\n\n> (and it would let you optionally avoid the tmp strbuf in\n> the caller.)\n\nThe path in read_directory() is not necessarily NUL-delimited, so\nattempting to use it as-is, or even with your checks, would cause us\nto possibly print garbage and do out-of-bounds reads. We need the tmp\nstrbuf.\n"},{"id":"424216","messageId":"xmqqy2ckj40i.fsf@gitster.g","threadId":"55639","inReplyTo":"CABPp-BE7V233NYJARGXaTUZdt+L=N55B27UrV6pOuKaKOUWRWA@mail.gmail.com","subject":"Re: [PATCH v3 3/8] [RFC] ls-files: error out on -i unless -o or -c are specified","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-11T22:32:13Z","receivedAt":"2021-05-11T22:32:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n>> If we are not defaulting to cached, then\n>>\n>> die(\"ls-files -i must be used with either -o or -c\");\n>>\n>> would also make sense.\n>\n> Ooh, that wording is much nicer. I'll adopt the latter suggestion,\n> but let me know if you'd rather I went the warning route.\n\nEven though warning would be safer, I have no strong prefeference.\nEither way will resolve my puzzlement.\n\nThanks.\n"},{"id":"424217","messageId":"xmqqsg2sj3id.fsf@gitster.g","threadId":"55639","inReplyTo":"CABPp-BFV6R9GNeZVbiKx_Tazq+6LuBNCzLcsEq63MfG5YD0hFQ@mail.gmail.com","subject":"Re: [PATCH v3 4/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-11T22:43:06Z","receivedAt":"2021-05-11T22:43:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> But to answer your question, the paths we visit are '.', '..', '.git',\n> and 'untracked', the first three of which we mark as path_none and\n> don't recurse into because of special rules for those paths, and the\n> last of which we shouldn't recurse into since it is ignored.\n\nNot a hard requirement, but I wish if we entirely ignored \".\" and\n\"..\" in our code (not just not counting, but making whoever calls\nreaddir() skip and call it again when it gets \".\" or \"..\").\n\n https://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html\n\nseems to imply that readdir() may not give \".\" or \"..\" (if dot or\ndot-dot exists, you are to return them only once, which implies that\nit is perfectly OK for dot or dot-dot to be missing).\n\nSo dropping the test for number of visited paths would be nicer from\nportability's point of view ;-)\n\nThanks.\n"},{"id":"424222","messageId":"a01ffeaf-31de-fdf5-3992-bea40feb1213@jeffhostetler.com","threadId":"55639","inReplyTo":"CABPp-BGeOUOvYRD+gX4jVR0kN8O5_icKtT18F9BdeR424DjUmg@mail.gmail.com","subject":"Re: [PATCH v4 1/8] dir: convert trace calls to trace2 equivalents","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-05-11T23:12:47Z","receivedAt":"2021-05-11T23:12:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/11/21 4:12 PM, Elijah Newren wrote:\n> On Tue, May 11, 2021 at 12:06 PM Jeff Hostetler wrote:\n>>\n>> On 5/11/21 2:34 PM, Elijah Newren via GitGitGadget wrote:\n>>> From: Elijah Newren \n>>>\n>>> Signed-off-by: Elijah Newren \n>>> ---\n>>> dir.c | 43 +++++--\n>>> t/t7063-status-untracked-cache.sh | 205 ++++++++++++++++++------------\n>>> t/t7519-status-fsmonitor.sh | 8 +-\n>>> 3 files changed, 155 insertions(+), 101 deletions(-)\n>>>\n>>> diff --git a/dir.c b/dir.c\n>>> index 3474e67e8f3c..122fcbffdf89 100644\n>>> --- a/dir.c\n>>> +++ b/dir.c\n>>> @@ -2760,15 +2760,34 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n>>> return root;\n>>> }\n>>>\n>>> +static void trace2_read_directory_statistics(struct dir_struct *dir,\n>>> + struct repository *repo,\n>>> + const char *path)\n>>> +{\n>>> + if (!dir->untracked)\n>>> + return;\n>>> + trace2_data_string(\"read_directory\", repo, \"path\", path);\n>>\n>> I'm probably just nit-picking here, but should this look more like:\n> \n> nit-picking and questions are totally fine. :-) Thanks for reviewing.\n> \n>>\n>> if (path && *path)\n>> trace2_data_string(...)\n> \n> path is always non-NULL (it'd be an error to call read_directory()\n> with a NULL path). So the first part of the check isn't meaningful\n> for this particular code. The second half is interesting. Do we want\n> to omit the path when it happens to be the toplevel directory (the\n> case where !*path)? The original trace_performance_leave() calls\n> certainly didn't, and I was just trying to provide the same info they\n> do, as you suggested. I guess people could determine the path by\n> knowing that the code doesn't print it when it's empty, but do we want\n> trace2 users to need to read the code to figure out statistics and\n> info?\n\nthat's fine. it might be easier to just always print it (even if\nblank) so that post-processors know that rather than have to assume\nit.\n\n> \n>> if (!dir->untracked)\n>> return;\n>>\n>> Then when you add the visitied fields in the next commit,\n>> you'll have the path with them (when present).\n> \n> There is always a path with them, it's just that the empty string\n> denotes the toplevel directory.\n> \n>> (and it would let you optionally avoid the tmp strbuf in\n>> the caller.)\n> \n> The path in read_directory() is not necessarily NUL-delimited, so\n> attempting to use it as-is, or even with your checks, would cause us\n> to possibly print garbage and do out-of-bounds reads. We need the tmp\n> strbuf.\n> \n\nI just meant, \"if (!len) pass NULL, else build and pass tmp.buf\".\n\nbut i'm nit-picking again.\n\nJeff\n\n"},{"id":"424237","messageId":"CABPp-BF9ptJio+NSi-i-sQJqjsOiKfFxzVHQ=RTPQkV9hjPrMQ@mail.gmail.com","threadId":"55639","inReplyTo":"a01ffeaf-31de-fdf5-3992-bea40feb1213@jeffhostetler.com","subject":"Re: [PATCH v4 1/8] dir: convert trace calls to trace2 equivalents","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-12T00:44:42Z","receivedAt":"2021-05-12T00:44:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, May 11, 2021 at 4:12 PM Jeff Hostetler wrote:\n>\n> On 5/11/21 4:12 PM, Elijah Newren wrote:\n> > On Tue, May 11, 2021 at 12:06 PM Jeff Hostetler wrote:\n> >>\n> >> On 5/11/21 2:34 PM, Elijah Newren via GitGitGadget wrote:\n> >>> From: Elijah Newren \n> >>>\n> >>> Signed-off-by: Elijah Newren \n> >>> ---\n> >>> dir.c | 43 +++++--\n> >>> t/t7063-status-untracked-cache.sh | 205 ++++++++++++++++++------------\n> >>> t/t7519-status-fsmonitor.sh | 8 +-\n> >>> 3 files changed, 155 insertions(+), 101 deletions(-)\n> >>>\n> >>> diff --git a/dir.c b/dir.c\n> >>> index 3474e67e8f3c..122fcbffdf89 100644\n> >>> --- a/dir.c\n> >>> +++ b/dir.c\n> >>> @@ -2760,15 +2760,34 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n> >>> return root;\n> >>> }\n> >>>\n> >>> +static void trace2_read_directory_statistics(struct dir_struct *dir,\n> >>> + struct repository *repo,\n> >>> + const char *path)\n> >>> +{\n> >>> + if (!dir->untracked)\n> >>> + return;\n> >>> + trace2_data_string(\"read_directory\", repo, \"path\", path);\n> >>\n> >> I'm probably just nit-picking here, but should this look more like:\n> >\n> > nit-picking and questions are totally fine. :-) Thanks for reviewing.\n> >\n> >>\n> >> if (path && *path)\n> >> trace2_data_string(...)\n> >\n> > path is always non-NULL (it'd be an error to call read_directory()\n> > with a NULL path). So the first part of the check isn't meaningful\n> > for this particular code. The second half is interesting. Do we want\n> > to omit the path when it happens to be the toplevel directory (the\n> > case where !*path)? The original trace_performance_leave() calls\n> > certainly didn't, and I was just trying to provide the same info they\n> > do, as you suggested. I guess people could determine the path by\n> > knowing that the code doesn't print it when it's empty, but do we want\n> > trace2 users to need to read the code to figure out statistics and\n> > info?\n>\n> that's fine. it might be easier to just always print it (even if\n> blank) so that post-processors know that rather than have to assume\n> it.\n>\n> >\n> >> if (!dir->untracked)\n> >> return;\n> >>\n> >> Then when you add the visitied fields in the next commit,\n> >> you'll have the path with them (when present).\n> >\n> > There is always a path with them, it's just that the empty string\n> > denotes the toplevel directory.\n> >\n> >> (and it would let you optionally avoid the tmp strbuf in\n> >> the caller.)\n> >\n> > The path in read_directory() is not necessarily NUL-delimited, so\n> > attempting to use it as-is, or even with your checks, would cause us\n> > to possibly print garbage and do out-of-bounds reads. We need the tmp\n> > strbuf.\n> >\n>\n> I just meant, \"if (!len) pass NULL, else build and pass tmp.buf\".\n\nAh, gotcha, that's why you were checking non-NULL.\n\nHowever, what about the other case when len is nonzero. Let's say\nthat len = 8 and path points at\n\"filename*%&#)aWholeBunchOfTotalGarbageAfterTheRealFilenameThatShouldNotBeReadOrIncluded\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\"\n?\n\nHow do you make it print \"filename\" and only \"filename\" without the\nother stuff without using the tmp strbuf?\n"},{"id":"424250","messageId":"CABPp-BEJQ5RQdvm58xrNcWm9VdPrUF8_-bym-M424bs7xR=8Gg@mail.gmail.com","threadId":"55639","inReplyTo":"xmqqsg2sj3id.fsf@gitster.g","subject":"Re: [PATCH v3 4/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-12T02:07:40Z","receivedAt":"2021-05-12T02:07:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Tue, May 11, 2021 at 3:43 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > But to answer your question, the paths we visit are '.', '..', '.git',\n> > and 'untracked', the first three of which we mark as path_none and\n> > don't recurse into because of special rules for those paths, and the\n> > last of which we shouldn't recurse into since it is ignored.\n>\n> Not a hard requirement, but I wish if we entirely ignored \".\" and\n> \"..\" in our code (not just not counting, but making whoever calls\n> readdir() skip and call it again when it gets \".\" or \"..\").\n>\n> https://pubs.opengroup.org/onlinepubs/9699919799/functions/readdir.html\n>\n> seems to imply that readdir() may not give \".\" or \"..\" (if dot or\n> dot-dot exists, you are to return them only once, which implies that\n> it is perfectly OK for dot or dot-dot to be missing).\n\n\nSomething like this?\n\ndiff --git a/dir.c b/dir.c\nindex 993a12145f..7f470bc701 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2341,7 +2341,11 @@ static int read_cached_dir(struct cached_dir *cdir)\n struct dirent *de;\n\n if (cdir->fdir) {\n- de = readdir(cdir->fdir);\n+ while ((de = readdir(cdir->fdir))) {\n+ /* Ignore '.' and '..' by re-looping; handle the rest */\n+ if (!de || !is_dot_or_dotdot(de->d_name))\n+ break;\n+ }\n if (!de) {\n cdir->d_name = NULL;\n cdir->d_type = DT_UNKNOWN;\n\nIt appears that the other two callers of readdir() in dir.c, namely in\nis_empty_dir() and remove_dir_recurse() already have such special\nrepeat-if-is_dot_or_dotdot() logic built into them, so this was\npartially lifted from those.\n\nIf you'd like, I can add another patch in the series with this change\nso that all readdir() calls in dir.c have such ignore '.' and '..'\nlogic. Or, we could perhaps introduce a new readdir() wrapper that\ndoes nothing other than ignore '.' and '..' and have all three of\nthese callsites use that new wrapper.\n\n> So dropping the test for number of visited paths would be nicer from\n> portability's point of view ;-)\n\nYep, makes sense. I already did that in v4, which means it'll\ncontinue to pass with or without the above proposed change to\nread_cached_dir().\n"},{"id":"424265","messageId":"xmqqy2ckhc80.fsf@gitster.g","threadId":"55639","inReplyTo":"CABPp-BEJQ5RQdvm58xrNcWm9VdPrUF8_-bym-M424bs7xR=8Gg@mail.gmail.com","subject":"Re: [PATCH v3 4/8] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-12T03:17:51Z","receivedAt":"2021-05-12T03:17:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> If you'd like, I can add another patch in the series with this change\n> so that all readdir() calls in dir.c have such ignore '.' and '..'\n> logic. Or, we could perhaps introduce a new readdir() wrapper that\n> does nothing other than ignore '.' and '..' and have all three of\n> these callsites use that new wrapper.\n\nYeah, it is good to be consistent (either implementation).\n\n>> So dropping the test for number of visited paths would be nicer from\n>> portability's point of view ;-)\n>\n> Yep, makes sense. I already did that in v4, which means it'll\n> continue to pass with or without the above proposed change to\n> read_cached_dir().\n\nYup.\n"},{"id":"424322","messageId":"d9690dfc-2883-6f0a-e66b-89066b660c8d@jeffhostetler.com","threadId":"55639","inReplyTo":"CABPp-BF9ptJio+NSi-i-sQJqjsOiKfFxzVHQ=RTPQkV9hjPrMQ@mail.gmail.com","subject":"Re: [PATCH v4 1/8] dir: convert trace calls to trace2 equivalents","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2021-05-12T12:26:46Z","receivedAt":"2021-05-12T12:26:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 5/11/21 8:44 PM, Elijah Newren wrote:\n> On Tue, May 11, 2021 at 4:12 PM Jeff Hostetler wrote:\n>>\n>> On 5/11/21 4:12 PM, Elijah Newren wrote:\n>>> On Tue, May 11, 2021 at 12:06 PM Jeff Hostetler wrote:\n>>>>\n>>>> On 5/11/21 2:34 PM, Elijah Newren via GitGitGadget wrote:\n>>>>> From: Elijah Newren \n>>>>>\n>>>>> Signed-off-by: Elijah Newren \n>>>>> ---\n>>>>> dir.c | 43 +++++--\n>>>>> t/t7063-status-untracked-cache.sh | 205 ++++++++++++++++++------------\n>>>>> t/t7519-status-fsmonitor.sh | 8 +-\n>>>>> 3 files changed, 155 insertions(+), 101 deletions(-)\n>>>>>\n>>>>> diff --git a/dir.c b/dir.c\n>>>>> index 3474e67e8f3c..122fcbffdf89 100644\n>>>>> --- a/dir.c\n>>>>> +++ b/dir.c\n>>>>> @@ -2760,15 +2760,34 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n>>>>> return root;\n>>>>> }\n>>>>>\n>>>>> +static void trace2_read_directory_statistics(struct dir_struct *dir,\n>>>>> + struct repository *repo,\n>>>>> + const char *path)\n>>>>> +{\n>>>>> + if (!dir->untracked)\n>>>>> + return;\n>>>>> + trace2_data_string(\"read_directory\", repo, \"path\", path);\n>>>>\n>>>> I'm probably just nit-picking here, but should this look more like:\n>>>\n>>> nit-picking and questions are totally fine. :-) Thanks for reviewing.\n>>>\n>>>>\n>>>> if (path && *path)\n>>>> trace2_data_string(...)\n>>>\n>>> path is always non-NULL (it'd be an error to call read_directory()\n>>> with a NULL path). So the first part of the check isn't meaningful\n>>> for this particular code. The second half is interesting. Do we want\n>>> to omit the path when it happens to be the toplevel directory (the\n>>> case where !*path)? The original trace_performance_leave() calls\n>>> certainly didn't, and I was just trying to provide the same info they\n>>> do, as you suggested. I guess people could determine the path by\n>>> knowing that the code doesn't print it when it's empty, but do we want\n>>> trace2 users to need to read the code to figure out statistics and\n>>> info?\n>>\n>> that's fine. it might be easier to just always print it (even if\n>> blank) so that post-processors know that rather than have to assume\n>> it.\n>>\n>>>\n>>>> if (!dir->untracked)\n>>>> return;\n>>>>\n>>>> Then when you add the visitied fields in the next commit,\n>>>> you'll have the path with them (when present).\n>>>\n>>> There is always a path with them, it's just that the empty string\n>>> denotes the toplevel directory.\n>>>\n>>>> (and it would let you optionally avoid the tmp strbuf in\n>>>> the caller.)\n>>>\n>>> The path in read_directory() is not necessarily NUL-delimited, so\n>>> attempting to use it as-is, or even with your checks, would cause us\n>>> to possibly print garbage and do out-of-bounds reads. We need the tmp\n>>> strbuf.\n>>>\n>>\n>> I just meant, \"if (!len) pass NULL, else build and pass tmp.buf\".\n> \n> Ah, gotcha, that's why you were checking non-NULL.\n> \n> However, what about the other case when len is nonzero. Let's say\n> that len = 8 and path points at\n> \"filename*%&#)aWholeBunchOfTotalGarbageAfterTheRealFilenameThatShouldNotBeReadOrIncluded\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\"\n> ?\n> \n> How do you make it print \"filename\" and only \"filename\" without the\n> other stuff without using the tmp strbuf?\n> \n\nI was still saying to use the \"strbuf tmp\" in the non-zero len case,\nbut just pass NULL (or \"\") for the len==0 case.\n\nAlternatively, since `trace2_read_directory_statistics() a static\nlocal function, we could move all of the path manipulation into it.\n\nstatic void emit_stats(\n\tstruct dir_struct *dir,\n\tstruct repository *repo,\n\tconst char* path_buf,\n\tsize_t path_len)\n{\n\tif (!path_len)\n\t\ttrace2_data_string(\"read_directory\", repo,\n\t\t\t\"path\", \"\");\n\telse {\n\t\tstruct strbuf tmp = STRBUF_INIT;\n\t\tstrbuf_add(&tmp, path_buf, path_len);\n\t\ttrace2_data_string(\"read_directory\", repo,\n\t\t\t\"path\", tmp.buf);\n\t\tstrbuf_release(&tmp);\n\t}\n\t... the rest of intmax stats ...\n}\n\n\nBTW, could we also rename your stats function? I've been trying\nto keep the \"trace2_\" prefix reserved for the Trace2 API.\n\n\nThanks,\nJeff\n"},{"id":"424345","messageId":"CABPp-BHSoEqy2R=X3dRzPykMQH7Pie_bMqo79Y+CR1bUoYXLfA@mail.gmail.com","threadId":"55639","inReplyTo":"d9690dfc-2883-6f0a-e66b-89066b660c8d@jeffhostetler.com","subject":"Re: [PATCH v4 1/8] dir: convert trace calls to trace2 equivalents","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-12T15:24:37Z","receivedAt":"2021-05-12T16:07:28Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, May 12, 2021 at 5:26 AM Jeff Hostetler wrote:\n>\n> On 5/11/21 8:44 PM, Elijah Newren wrote:\n> > On Tue, May 11, 2021 at 4:12 PM Jeff Hostetler wrote:\n> >>\n> >> On 5/11/21 4:12 PM, Elijah Newren wrote:\n> >>> On Tue, May 11, 2021 at 12:06 PM Jeff Hostetler wrote:\n> >>>>\n> >>>> On 5/11/21 2:34 PM, Elijah Newren via GitGitGadget wrote:\n> >>>>> From: Elijah Newren \n> >>>>>\n> >>>>> Signed-off-by: Elijah Newren \n> >>>>> ---\n> >>>>> dir.c | 43 +++++--\n> >>>>> t/t7063-status-untracked-cache.sh | 205 ++++++++++++++++++------------\n> >>>>> t/t7519-status-fsmonitor.sh | 8 +-\n> >>>>> 3 files changed, 155 insertions(+), 101 deletions(-)\n> >>>>>\n> >>>>> diff --git a/dir.c b/dir.c\n> >>>>> index 3474e67e8f3c..122fcbffdf89 100644\n> >>>>> --- a/dir.c\n> >>>>> +++ b/dir.c\n> >>>>> @@ -2760,15 +2760,34 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n> >>>>> return root;\n> >>>>> }\n> >>>>>\n> >>>>> +static void trace2_read_directory_statistics(struct dir_struct *dir,\n> >>>>> + struct repository *repo,\n> >>>>> + const char *path)\n> >>>>> +{\n> >>>>> + if (!dir->untracked)\n> >>>>> + return;\n> >>>>> + trace2_data_string(\"read_directory\", repo, \"path\", path);\n> >>>>\n> >>>> I'm probably just nit-picking here, but should this look more like:\n> >>>\n> >>> nit-picking and questions are totally fine. :-) Thanks for reviewing.\n> >>>\n> >>>>\n> >>>> if (path && *path)\n> >>>> trace2_data_string(...)\n> >>>\n> >>> path is always non-NULL (it'd be an error to call read_directory()\n> >>> with a NULL path). So the first part of the check isn't meaningful\n> >>> for this particular code. The second half is interesting. Do we want\n> >>> to omit the path when it happens to be the toplevel directory (the\n> >>> case where !*path)? The original trace_performance_leave() calls\n> >>> certainly didn't, and I was just trying to provide the same info they\n> >>> do, as you suggested. I guess people could determine the path by\n> >>> knowing that the code doesn't print it when it's empty, but do we want\n> >>> trace2 users to need to read the code to figure out statistics and\n> >>> info?\n> >>\n> >> that's fine. it might be easier to just always print it (even if\n> >> blank) so that post-processors know that rather than have to assume\n> >> it.\n> >>\n> >>>\n> >>>> if (!dir->untracked)\n> >>>> return;\n> >>>>\n> >>>> Then when you add the visitied fields in the next commit,\n> >>>> you'll have the path with them (when present).\n> >>>\n> >>> There is always a path with them, it's just that the empty string\n> >>> denotes the toplevel directory.\n> >>>\n> >>>> (and it would let you optionally avoid the tmp strbuf in\n> >>>> the caller.)\n> >>>\n> >>> The path in read_directory() is not necessarily NUL-delimited, so\n> >>> attempting to use it as-is, or even with your checks, would cause us\n> >>> to possibly print garbage and do out-of-bounds reads. We need the tmp\n> >>> strbuf.\n> >>>\n> >>\n> >> I just meant, \"if (!len) pass NULL, else build and pass tmp.buf\".\n> >\n> > Ah, gotcha, that's why you were checking non-NULL.\n> >\n> > However, what about the other case when len is nonzero. Let's say\n> > that len = 8 and path points at\n> > \"filename*%&#)aWholeBunchOfTotalGarbageAfterTheRealFilenameThatShouldNotBeReadOrIncluded\\0\\0\\0\\0\\0\\0\\0\\0\\0\\0\"\n> > ?\n> >\n> > How do you make it print \"filename\" and only \"filename\" without the\n> > other stuff without using the tmp strbuf?\n> >\n>\n> I was still saying to use the \"strbuf tmp\" in the non-zero len case,\n> but just pass NULL (or \"\") for the len==0 case.\n\nAh, now I see what you were saying. Sorry for not getting it earlier.\n\n> Alternatively, since `trace2_read_directory_statistics() a static\n> local function, we could move all of the path manipulation into it.\n>\n> static void emit_stats(\n> struct dir_struct *dir,\n> struct repository *repo,\n> const char* path_buf,\n> size_t path_len)\n> {\n> if (!path_len)\n> trace2_data_string(\"read_directory\", repo,\n> \"path\", \"\");\n> else {\n> struct strbuf tmp = STRBUF_INIT;\n> strbuf_add(&tmp, path_buf, path_len);\n> trace2_data_string(\"read_directory\", repo,\n> \"path\", tmp.buf);\n> strbuf_release(&tmp);\n> }\n> ... the rest of intmax stats ...\n> }\n\nMakes sense.\n\n> BTW, could we also rename your stats function? I've been trying\n> to keep the \"trace2_\" prefix reserved for the Trace2 API.\n\nSure, will do.\n"},{"id":"424353","messageId":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v4.git.git.1620758049.gitgitgadget@gmail.com","subject":"[PATCH v5 0/9] Directory traversal fixes","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:13Z","receivedAt":"2021-05-12T17:55:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patchset fixes a few directory traversal issues, where fill_directory()\nwould traverse into directories that it shouldn't and not traverse into\ndirectories that it should (one of which was originally reported on this\nlist at [1]). And it includes a few cleanups\n\nChanges since v4:\n\n * Tweak the trace2 statistics emitting a bit, as per suggestions from Jeff.\n * Introduce a new readdir_skip_dot_and_dotdot() helper at the end of the\n series, and use it everywhere we repeat the same code to skip '.' and\n '..' entries from readdir. Also use it in dir.c's read_cached_dir() so we\n can be consistent about skipping it, even for statistics, across\n platforms.\n\nIf anyone has any ideas about a better place to put the \"Some sidenotes\"\nfrom the sixth commit message rather than keeping them in a random commit\nmessage, that might be helpful.\n\n[1] See\nhttps://lore.kernel.org/git/DM6PR00MB06829EC5B85E0C5AC595004E894E9@DM6PR00MB0682.namprd00.prod.outlook.com/\nor alternatively https://github.com/git-for-windows/git/issues/2732.\n\nDerrick Stolee (1):\n dir: update stale description of treat_directory()\n\nElijah Newren (8):\n dir: convert trace calls to trace2 equivalents\n dir: report number of visited directories and paths with trace2\n ls-files: error out on -i unless -o or -c are specified\n t7300: add testcase showing unnecessary traversal into ignored\n directory\n t3001, t7300: add testcase showcasing missed directory traversal\n dir: avoid unnecessary traversal into ignored directory\n dir: traverse into untracked directories if they may have ignored\n subfiles\n dir: introduce readdir_skip_dot_and_dotdot() helper\n\n builtin/clean.c | 4 +-\n builtin/ls-files.c | 3 +\n builtin/worktree.c | 4 +-\n diff-no-index.c | 5 +-\n dir.c | 146 +++++++++++++-------\n dir.h | 6 +\n entry.c | 5 +-\n notes-merge.c | 5 +-\n object-file.c | 4 +-\n packfile.c | 5 +-\n rerere.c | 4 +-\n t/t1306-xdg-files.sh | 2 +-\n t/t3001-ls-files-others-exclude.sh | 5 +\n t/t3003-ls-files-exclude.sh | 4 +-\n t/t7063-status-untracked-cache.sh | 206 +++++++++++++++++------------\n t/t7300-clean.sh | 42 ++++++\n t/t7519-status-fsmonitor.sh | 8 +-\n worktree.c | 12 +-\n 18 files changed, 298 insertions(+), 172 deletions(-)\n\n\nbase-commit: 311531c9de557d25ac087c1637818bd2aad6eb3a\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-1020%2Fnewren%2Fdirectory-traversal-fixes-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-1020/newren/directory-traversal-fixes-v5\nPull-Request: https://github.com/git/git/pull/1020\n\nRange-diff vs v4:\n\n 1: 9204e36b7e90 ! 1: 6b1b4820dd20 dir: convert trace calls to trace2 equivalents\n @@ dir.c: static struct untracked_cache_dir *validate_untracked_cache(struct dir_st\n \treturn root;\n }\n \n -+static void trace2_read_directory_statistics(struct dir_struct *dir,\n -+\t\t\t\t\t struct repository *repo,\n -+\t\t\t\t\t const char *path)\n ++static void emit_traversal_statistics(struct dir_struct *dir,\n ++\t\t\t\t struct repository *repo,\n ++\t\t\t\t const char *path,\n ++\t\t\t\t int path_len)\n +{\n ++\tif (!trace2_is_enabled())\n ++\t\treturn;\n ++\n ++\tif (!path_len) {\n ++\t\ttrace2_data_string(\"read_directory\", repo, \"path\", \"\");\n ++\t} else {\n ++\t\tstruct strbuf tmp = STRBUF_INIT;\n ++\t\tstrbuf_add(&tmp, path, path_len);\n ++\t\ttrace2_data_string(\"read_directory\", repo, \"path\", tmp.buf);\n ++\t\tstrbuf_release(&tmp);\n ++\t}\n ++\n +\tif (!dir->untracked)\n +\t\treturn;\n -+\ttrace2_data_string(\"read_directory\", repo, \"path\", path);\n +\ttrace2_data_intmax(\"read_directory\", repo,\n +\t\t\t \"node-creation\", dir->untracked->dir_created);\n +\ttrace2_data_intmax(\"read_directory\", repo,\n @@ dir.c: int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n \n -\ttrace_performance_leave(\"read directory %.*s\", len, path);\n -+\tif (trace2_is_enabled()) {\n -+\t\tstruct strbuf tmp = STRBUF_INIT;\n -+\t\tstrbuf_add(&tmp, path, len);\n -+\t\ttrace2_read_directory_statistics(dir, istate->repo, tmp.buf);\n -+\t\tstrbuf_release(&tmp);\n -+\t}\n ++\temit_traversal_statistics(dir, istate->repo, path, len);\n +\n +\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n \tif (dir->untracked) {\n 2: 6939253be825 ! 2: cfe2898b7a7e dir: report number of visited directories and paths with trace2\n @@ dir.c: static enum path_treatment read_directory_recursive(struct dir_struct *di\n \n \t\tif (state > dir_state)\n \t\t\tdir_state = state;\n -@@ dir.c: static void trace2_read_directory_statistics(struct dir_struct *dir,\n - \t\t\t\t\t struct repository *repo,\n - \t\t\t\t\t const char *path)\n - {\n +@@ dir.c: static void emit_traversal_statistics(struct dir_struct *dir,\n + \t\tstrbuf_release(&tmp);\n + \t}\n + \n +\ttrace2_data_intmax(\"read_directory\", repo,\n +\t\t\t \"directories-visited\", dir->visited_directories);\n +\ttrace2_data_intmax(\"read_directory\", repo,\n +\t\t\t \"paths-visited\", dir->visited_paths);\n ++\n \tif (!dir->untracked)\n \t\treturn;\n - \ttrace2_data_string(\"read_directory\", repo, \"path\", path);\n + \ttrace2_data_intmax(\"read_directory\", repo,\n @@ dir.c: int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tstruct untracked_cache_dir *untracked;\n \n 3: 8d0ca8104be6 = 3: 279ef30ffbc2 ls-files: error out on -i unless -o or -c are specified\n 4: 317abab3571e = 4: 5a8807a1992c t7300: add testcase showing unnecessary traversal into ignored directory\n 5: 5eb019327b57 = 5: b014ccbbaf3e t3001, t7300: add testcase showcasing missed directory traversal\n 6: 89cc01ef8598 = 6: ae1c9e37b21b dir: avoid unnecessary traversal into ignored directory\n 7: 4a561e1229e4 = 7: 6fa1e85edf2f dir: traverse into untracked directories if they may have ignored subfiles\n 8: 2945e749f5e3 = 8: 179f992edc92 dir: update stale description of treat_directory()\n -: ------------ > 9: b7c6176560bd dir: introduce readdir_skip_dot_and_dotdot() helper\n\n-- \ngitgitgadget\n"},{"id":"424355","messageId":"6b1b4820dd20ed0911e049f0edadc2adcc391e9d.1620840502.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"[PATCH v5 1/9] dir: convert trace calls to trace2 equivalents","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:14Z","receivedAt":"2021-05-12T17:55:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 50 ++++++--\n t/t7063-status-untracked-cache.sh | 205 ++++++++++++++++++------------\n t/t7519-status-fsmonitor.sh | 8 +-\n 3 files changed, 162 insertions(+), 101 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3474e67e8f3c..cf19a83d3e2c 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2760,15 +2760,46 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \treturn root;\n }\n \n+static void emit_traversal_statistics(struct dir_struct *dir,\n+\t\t\t\t struct repository *repo,\n+\t\t\t\t const char *path,\n+\t\t\t\t int path_len)\n+{\n+\tif (!trace2_is_enabled())\n+\t\treturn;\n+\n+\tif (!path_len) {\n+\t\ttrace2_data_string(\"read_directory\", repo, \"path\", \"\");\n+\t} else {\n+\t\tstruct strbuf tmp = STRBUF_INIT;\n+\t\tstrbuf_add(&tmp, path, path_len);\n+\t\ttrace2_data_string(\"read_directory\", repo, \"path\", tmp.buf);\n+\t\tstrbuf_release(&tmp);\n+\t}\n+\n+\tif (!dir->untracked)\n+\t\treturn;\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"node-creation\", dir->untracked->dir_created);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"gitignore-invalidation\",\n+\t\t\t dir->untracked->gitignore_invalidated);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"directory-invalidation\",\n+\t\t\t dir->untracked->dir_invalidated);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"opendir\", dir->untracked->dir_opened);\n+}\n+\n int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t const char *path, int len, const struct pathspec *pathspec)\n {\n \tstruct untracked_cache_dir *untracked;\n \n-\ttrace_performance_enter();\n+\ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n \n \tif (has_symlink_leading_path(path, len)) {\n-\t\ttrace_performance_leave(\"read directory %.*s\", len, path);\n+\t\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n \t\treturn dir->nr;\n \t}\n \n@@ -2784,23 +2815,15 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tQSORT(dir->entries, dir->nr, cmp_dir_entry);\n \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n \n-\ttrace_performance_leave(\"read directory %.*s\", len, path);\n+\temit_traversal_statistics(dir, istate->repo, path, len);\n+\n+\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\n \tif (dir->untracked) {\n \t\tstatic int force_untracked_cache = -1;\n-\t\tstatic struct trace_key trace_untracked_stats = TRACE_KEY_INIT(UNTRACKED_STATS);\n \n \t\tif (force_untracked_cache < 0)\n \t\t\tforce_untracked_cache =\n \t\t\t\tgit_env_bool(\"GIT_FORCE_UNTRACKED_CACHE\", 0);\n-\t\ttrace_printf_key(&trace_untracked_stats,\n-\t\t\t\t \"node creation: %u\\n\"\n-\t\t\t\t \"gitignore invalidation: %u\\n\"\n-\t\t\t\t \"directory invalidation: %u\\n\"\n-\t\t\t\t \"opendir: %u\\n\",\n-\t\t\t\t dir->untracked->dir_created,\n-\t\t\t\t dir->untracked->gitignore_invalidated,\n-\t\t\t\t dir->untracked->dir_invalidated,\n-\t\t\t\t dir->untracked->dir_opened);\n \t\tif (force_untracked_cache &&\n \t\t\tdir->untracked == istate->untracked &&\n \t\t (dir->untracked->dir_opened ||\n@@ -2811,6 +2834,7 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t\tFREE_AND_NULL(dir->untracked);\n \t\t}\n \t}\n+\n \treturn dir->nr;\n }\n \ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex accefde72fb1..9710d33b3cd6 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -57,6 +57,19 @@ iuc () {\n \treturn $ret\n }\n \n+get_relevant_traces () {\n+\t# From the GIT_TRACE2_PERF data of the form\n+\t# $TIME $FILE:$LINE | d0 | main | data | r1 | ? | ? | read_directo | $RELEVANT_STAT\n+\t# extract the $RELEVANT_STAT fields. We don't care about region_enter\n+\t# or region_leave, or stats for things outside read_directory.\n+\tINPUT_FILE=$1\n+\tOUTPUT_FILE=$2\n+\tgrep data.*read_directo $INPUT_FILE |\n+\t cut -d \"|\" -f 9 \\\n+\t >\"$OUTPUT_FILE\"\n+}\n+\n+\n test_lazy_prereq UNTRACKED_CACHE '\n \t{ git update-index --test-untracked-cache; ret=$?; } &&\n \ttest $ret -ne 1\n@@ -129,19 +142,21 @@ EOF\n \n test_expect_success 'status first time (empty cache)' '\n \tavoid_racy &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <four &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <.gitignore &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <>.git/info/exclude &&\n-\t: >../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n+\t: >../trace.output &&\n \tavoid_racy &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n-\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n+\t: >../trace.output &&\n+\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status --porcelain >../status.actual &&\n \tiuc status --porcelain >../status.iuc &&\n \ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../status.actual &&\n+\tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <../before\n \t) &&\n@@ -346,12 +346,12 @@ test_expect_success UNTRACKED_CACHE 'ignore .git changes when invalidating UNTR'\n \tEOF\n \t(\n \t\tcd dot-git &&\n-\t\tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace-after\" \\\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace-after\" \\\n \t\tgit status &&\n \t\ttest-tool dump-untracked-cache >../after\n \t) &&\n-\tgrep \"directory invalidation\" trace-before >>before &&\n-\tgrep \"directory invalidation\" trace-after >>after &&\n+\tgrep \"directory-invalidation\" trace-before | cut -d\"|\" -f 9 >>before &&\n+\tgrep \"directory-invalidation\" trace-after | cut -d\"|\" -f 9 >>after &&\n \t# UNTR extension unchanged, dir invalidation count unchanged\n \ttest_cmp before after\n '\n-- \ngitgitgadget\n\n"},{"id":"424354","messageId":"cfe2898b7a7e08e01ef5b71f913f6f34bebf38f0.1620840502.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"[PATCH v5 2/9] dir: report number of visited directories and paths with trace2","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:15Z","receivedAt":"2021-05-12T17:55:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nProvide more statistics in trace2 output that include the number of\ndirectories and total paths visited by the directory traversal logic.\nSubsequent patches will take advantage of this to ensure we do not\nunnecessarily traverse into ignored directories.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 9 +++++++++\n dir.h | 4 ++++\n t/t7063-status-untracked-cache.sh | 3 ++-\n 3 files changed, 15 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex cf19a83d3e2c..f6dec5fd4a78 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2440,6 +2440,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \n \tif (open_cached_dir(&cdir, dir, untracked, istate, &path, check_only))\n \t\tgoto out;\n+\tdir->visited_directories++;\n \n \tif (untracked)\n \t\tuntracked->check_only = !!check_only;\n@@ -2448,6 +2449,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t/* check how the file or directory should be treated */\n \t\tstate = treat_path(dir, untracked, &cdir, istate, &path,\n \t\t\t\t baselen, pathspec);\n+\t\tdir->visited_paths++;\n \n \t\tif (state > dir_state)\n \t\t\tdir_state = state;\n@@ -2777,6 +2779,11 @@ static void emit_traversal_statistics(struct dir_struct *dir,\n \t\tstrbuf_release(&tmp);\n \t}\n \n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"directories-visited\", dir->visited_directories);\n+\ttrace2_data_intmax(\"read_directory\", repo,\n+\t\t\t \"paths-visited\", dir->visited_paths);\n+\n \tif (!dir->untracked)\n \t\treturn;\n \ttrace2_data_intmax(\"read_directory\", repo,\n@@ -2797,6 +2804,8 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tstruct untracked_cache_dir *untracked;\n \n \ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n+\tdir->visited_paths = 0;\n+\tdir->visited_directories = 0;\n \n \tif (has_symlink_leading_path(path, len)) {\n \t\ttrace2_region_leave(\"dir\", \"read_directory\", istate->repo);\ndiff --git a/dir.h b/dir.h\nindex 04d886cfce75..22c67907f689 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -336,6 +336,10 @@ struct dir_struct {\n \tstruct oid_stat ss_info_exclude;\n \tstruct oid_stat ss_excludes_file;\n \tunsigned unmanaged_exclude_files;\n+\n+\t/* Stats about the traversal */\n+\tunsigned visited_paths;\n+\tunsigned visited_directories;\n };\n \n /*Count the number of slashes for string s*/\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 9710d33b3cd6..a0c123b0a77a 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -65,7 +65,8 @@ get_relevant_traces () {\n \tINPUT_FILE=$1\n \tOUTPUT_FILE=$2\n \tgrep data.*read_directo $INPUT_FILE |\n-\t cut -d \"|\" -f 9 \\\n+\t cut -d \"|\" -f 9 |\n+\t grep -v visited \\\n \t >\"$OUTPUT_FILE\"\n }\n \n-- \ngitgitgadget\n\n"},{"id":"424357","messageId":"5a8807a1992cd012a8271fd157a757c07254b27a.1620840502.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"[PATCH v5 4/9] t7300: add testcase showing unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:17Z","receivedAt":"2021-05-12T17:55:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe PNPM package manager is apparently creating deeply nested (but\nignored) directory structures; traversing them is costly\nperformance-wise, unnecessary, and in some cases is even throwing\nwarnings/errors because the paths are too long to handle on various\nplatforms. Add a testcase that checks for such unnecessary directory\ntraversal.\n\nSigned-off-by: Elijah Newren \n---\n t/t7300-clean.sh | 23 +++++++++++++++++++++++\n 1 file changed, 23 insertions(+)\n\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex a74816ca8b46..07e8ba2d4b85 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,4 +746,27 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_failure 'avoid traversing into ignored directories' '\n+\ttest_when_finished rm -f output error trace.* &&\n+\ttest_create_repo avoid-traversing-deep-hierarchy &&\n+\t(\n+\t\tcd avoid-traversing-deep-hierarchy &&\n+\n+\t\tmkdir -p untracked/subdir/with/a &&\n+\t\t>untracked/subdir/with/a/random-file.txt &&\n+\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n+\t\tgit clean -ffdxn -e untracked\n+\t) &&\n+\n+\t# Make sure we only visited into the top-level directory, and did\n+\t# not traverse into the \"untracked\" subdirectory since it was excluded\n+\tgrep data.*read_directo.*directories-visited trace.output |\n+\t\tcut -d \"|\" -f 9 >trace.relevant &&\n+\tcat >trace.expect <<-EOF &&\n+\t ..directories-visited:1\n+\tEOF\n+\ttest_cmp trace.expect trace.relevant\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"424358","messageId":"ae1c9e37b21bc2f1421747dd02a311487f483f0d.1620840502.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"[PATCH v5 6/9] dir: avoid unnecessary traversal into ignored directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:19Z","receivedAt":"2021-05-12T17:55:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe show_other_directories case in treat_directory() tried to handle\nboth excludes and untracked files with the same logic, and mishandled\nboth the excludes and the untracked files in the process, in different\nways. Split that logic apart, and then focus on the logic for the\nexcludes; a subsequent commit will address the logic for untracked\nfiles.\n\nFor show_other_directories, an excluded directory means that\nevery path underneath that directory will also be excluded. Given that\nthe calling code requested to just show directories when everything\nunder a directory had the same state (that's what the\n\"DIR_SHOW_OTHER_DIRECTORIES\" flag means), we generally do not need to\ntraverse into such directories and can just immediately mark them as\nignored (i.e. as path_excluded). The only reason we cannot just\nimmediately return path_excluded is the DIR_HIDE_EMPTY_DIRECTORIES flag\nand the possibility that the ignored directory is an empty directory.\nThe code previously treated DIR_SHOW_IGNORED_TOO in most cases as an\nexception as well, which was wrong. It can sometimes reduce the number\nof cases where we need to recurse (namely if\nDIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set), but should not be able\nto increase the number of cases where we need to recurse. Fix the logic\naccordingly.\n\nSome sidenotes about possible confusion with dir.c:\n\n* \"ignored\" often refers to an untracked ignore\", i.e. a file which is\n not tracked which matches one of the ignore/exclusion rules. But you\n can also have a \"tracked ignore\", a tracked file that happens to match\n one of the ignore/exclusion rules and which dir.c has to worry about\n since \"git ls-files -c -i\" is supposed to list them.\n\n* The dir code often uses \"ignored\" and \"excluded\" interchangeably,\n which you need to keep in mind while reading the code.\n\n* \"exclude\" is used multiple ways in the code:\n\n * As noted above, \"exclude\" is often a synonym for \"ignored\".\n\n * The logic for parsing .gitignore files was re-used in\n .git/info/sparse-checkout, except there it is used to mark paths that\n the user wants to *keep*. This was mostly addressed by commit\n 65edd96aec (\"treewide: rename 'exclude' methods to 'pattern'\",\n 2019-09-03), but every once in a while you'll find a comment about\n \"exclude\" referring to these patterns that might in fact be in use\n by the sparse-checkout machinery for inclusion rules.\n\n * The word \"EXCLUDE\" is also used for pathspec negation, as in\n (pathspec->items[3].magic & PATHSPEC_EXCLUDE)\n Thus if a user had a .gitignore file containing\n *~\n *.log\n !settings.log\n And then ran\n git add -- 'settings.*' ':^settings.log'\n Then :^settings.log is a pathspec negation making settings.log not\n be requested to be added even though all other settings.* files are\n being added. Also, !settings.log in the gitignore file is a negative\n exclude pattern meaning that settings.log is normally a file we\n want to track even though all other *.log files are ignored.\n\nSometimes it feels like dir.c needs its own glossary with its many\ndefinitions, including the multiply-defined terms.\n\nReported-by: Jason Gore \nSigned-off-by: Elijah Newren \n---\n dir.c | 44 +++++++++++++++++++++++++++++---------------\n t/t7300-clean.sh | 2 +-\n 2 files changed, 30 insertions(+), 16 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex f6dec5fd4a78..db2ae516a3aa 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1844,6 +1844,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t}\n \n \t/* This is the \"show_other_directories\" case */\n+\tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n \t * If we have a pathspec which could match something _below_ this\n@@ -1854,27 +1855,40 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tif (matches_how == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n \t\treturn path_recurse;\n \n+\t/* Special cases for where this directory is excluded/ignored */\n+\tif (excluded) {\n+\t\t/*\n+\t\t * In the show_other_directories case, if we're not\n+\t\t * hiding empty directories, there is no need to\n+\t\t * recurse into an ignored directory.\n+\t\t */\n+\t\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\t\treturn path_excluded;\n+\n+\t\t/*\n+\t\t * Even if we are hiding empty directories, we can still avoid\n+\t\t * recursing into ignored directories for DIR_SHOW_IGNORED_TOO\n+\t\t * if DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n+\t\t */\n+\t\tif ((dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n+\t\t\treturn path_excluded;\n+\t}\n+\n \t/*\n-\t * Other than the path_recurse case immediately above, we only need\n-\t * to recurse into untracked/ignored directories if either of the\n-\t * following bits is set:\n+\t * Other than the path_recurse case above, we only need to\n+\t * recurse into untracked directories if either of the following\n+\t * bits is set:\n \t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n \t * there are ignored entries below)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n-\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n-\t\treturn excluded ? path_excluded : path_untracked;\n-\n-\t/*\n-\t * ...and even if DIR_SHOW_IGNORED_TOO is set, we can still avoid\n-\t * recursing into ignored directories if the path is excluded and\n-\t * DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n-\t */\n-\tif (excluded &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n-\t\treturn path_excluded;\n+\tif (!excluded &&\n+\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n+\t\treturn path_untracked;\n+\t}\n \n \t/*\n \t * Even if we don't want to know all the paths under an untracked or\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 34c08c325407..21e48b3ba591 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -746,7 +746,7 @@ test_expect_success 'clean untracked paths by pathspec' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_failure 'avoid traversing into ignored directories' '\n+test_expect_success 'avoid traversing into ignored directories' '\n \ttest_when_finished rm -f output error trace.* &&\n \ttest_create_repo avoid-traversing-deep-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"424359","messageId":"179f992edc9254803252ae10e5d692f3755a40f3.1620840502.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"[PATCH v5 8/9] dir: update stale description of treat_directory()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:21Z","receivedAt":"2021-05-12T17:55:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe documentation comment for treat_directory() was originally written\nin 095952 (Teach directory traversal about subprojects, 2007-04-11)\nwhich was before the 'struct dir_struct' split its bitfield of named\noptions into a 'flags' enum in 7c4c97c0 (Turn the flags in struct\ndir_struct into a single variable, 2009-02-16). When those flags\nchanged, the comment became stale, since members like\n'show_other_directories' transitioned into flags like\nDIR_SHOW_OTHER_DIRECTORIES.\n\nUpdate the comments for treat_directory() to use these flag names rather\nthan the old member names.\n\nSigned-off-by: Derrick Stolee \nReviewed-by: Elijah Newren \n---\n dir.c | 13 ++++++-------\n 1 file changed, 6 insertions(+), 7 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex c0233bbba36c..4794c822b47f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1749,13 +1749,13 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n * Case 3: if we didn't have it in the index previously, we\n * have a few sub-cases:\n *\n- * (a) if \"show_other_directories\" is true, we show it as\n- * just a directory, unless \"hide_empty_directories\" is\n+ * (a) if DIR_SHOW_OTHER_DIRECTORIES flag is set, we show it as\n+ * just a directory, unless DIR_HIDE_EMPTY_DIRECTORIES is\n * also true, in which case we need to check if it contains any\n * untracked and / or ignored files.\n- * (b) if it looks like a git directory, and we don't have\n- * 'no_gitlinks' set we treat it as a gitlink, and show it\n- * as a directory.\n+ * (b) if it looks like a git directory and we don't have the\n+ * DIR_NO_GITLINKS flag, then we treat it as a gitlink, and\n+ * show it as a directory.\n * (c) otherwise, we recurse into it.\n */\n static enum path_treatment treat_directory(struct dir_struct *dir,\n@@ -1843,7 +1843,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_recurse;\n \t}\n \n-\t/* This is the \"show_other_directories\" case */\n \tassert(dir->flags & DIR_SHOW_OTHER_DIRECTORIES);\n \n \t/*\n@@ -1858,7 +1857,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* Special cases for where this directory is excluded/ignored */\n \tif (excluded) {\n \t\t/*\n-\t\t * In the show_other_directories case, if we're not\n+\t\t * If DIR_SHOW_OTHER_DIRECTORIES is set and we're not\n \t\t * hiding empty directories, there is no need to\n \t\t * recurse into an ignored directory.\n \t\t */\n-- \ngitgitgadget\n\n"},{"id":"424360","messageId":"6fa1e85edf2fbfbf902d4f726c9aaee6aa9df308.1620840502.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"[PATCH v5 7/9] dir: traverse into untracked directories if they may have ignored subfiles","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:20Z","receivedAt":"2021-05-12T17:55:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nA directory that is untracked does not imply that all files under it\nshould be categorized as untracked; in particular, if the caller is\ninterested in ignored files, many files or directories underneath the\nuntracked directory may be ignored. We previously partially handled\nthis right with DIR_SHOW_IGNORED_TOO, but missed DIR_SHOW_IGNORED. It\nwas not obvious, though, because the logic for untracked and excluded\nfiles had been fused together making it harder to reason about. The\nprevious commit split that logic out, making it easier to notice that\nDIR_SHOW_IGNORED was missing. Add it.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 10 ++++++----\n t/t3001-ls-files-others-exclude.sh | 2 +-\n t/t7300-clean.sh | 2 +-\n 3 files changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex db2ae516a3aa..c0233bbba36c 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1877,15 +1877,17 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/*\n \t * Other than the path_recurse case above, we only need to\n-\t * recurse into untracked directories if either of the following\n+\t * recurse into untracked directories if any of the following\n \t * bits is set:\n-\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n-\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED (because then we need to determine if\n+\t * there are ignored entries below)\n+\t * - DIR_SHOW_IGNORED_TOO (same as above)\n \t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n \t * the directory is empty)\n \t */\n \tif (!excluded &&\n-\t !(dir->flags & (DIR_SHOW_IGNORED_TOO |\n+\t !(dir->flags & (DIR_SHOW_IGNORED |\n+\t\t\t DIR_SHOW_IGNORED_TOO |\n \t\t\t DIR_HIDE_EMPTY_DIRECTORIES))) {\n \t\treturn path_untracked;\n \t}\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex ac05d1a17931..516c95ea0e82 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,7 +292,7 @@ EOF\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+test_expect_success 'ls-files with \"**\" patterns and --directory' '\n \t# Expectation same as previous test\n \tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 21e48b3ba591..0399701e6276 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -769,7 +769,7 @@ test_expect_success 'avoid traversing into ignored directories' '\n \ttest_cmp trace.expect trace.relevant\n '\n \n-test_expect_failure 'traverse into directories that may have ignored entries' '\n+test_expect_success 'traverse into directories that may have ignored entries' '\n \ttest_when_finished rm -f output &&\n \ttest_create_repo need-to-traverse-into-hierarchy &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"424361","messageId":"279ef30ffbc2145b0bd8edc377366808bd96ff50.1620840502.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"[PATCH v5 3/9] ls-files: error out on -i unless -o or -c are specified","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:16Z","receivedAt":"2021-05-12T17:55:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nls-files --ignored can be used together with either --others or\n--cached. After being perplexed for a bit and digging in to the code, I\nassumed that ls-files -i was just broken and not printing anything and\nI had a nice patch ready to submit when I finally realized that -i can be\nused with --cached to find tracked ignores.\n\nWhile that was a mistake on my part, and a careful reading of the\ndocumentation could have made this more clear, I suspect this is an\nerror others are likely to make as well. In fact, of two uses in our\ntestsuite, I believe one of the two did make this error. In t1306.13,\nthere are NO tracked files, and all the excludes built up and used in\nthat test and in previous tests thus have to be about untracked files.\nHowever, since they were looking for an empty result, the mistake went\nunnoticed as their erroneous command also just happened to give an empty\nanswer.\n\n-i will most the time be used with -o, which would suggest we could just\nmake -i imply -o in the absence of either a -o or -c, but that would be\na backward incompatible break. Instead, let's just flag -i without\neither a -o or -c as an error, and update the two relevant testcases to\nspecify their intent.\n\nSigned-off-by: Elijah Newren \n---\n builtin/ls-files.c | 3 +++\n t/t1306-xdg-files.sh | 2 +-\n t/t3003-ls-files-exclude.sh | 4 ++--\n 3 files changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex 60a2913a01e9..e8e25006c647 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -748,6 +748,9 @@ int cmd_ls_files(int argc, const char **argv, const char *cmd_prefix)\n \tif (pathspec.nr && error_unmatch)\n \t\tps_matched = xcalloc(pathspec.nr, 1);\n \n+\tif ((dir.flags & DIR_SHOW_IGNORED) && !show_others && !show_cached)\n+\t\tdie(\"ls-files -i must be used with either -o or -c\");\n+\n \tif ((dir.flags & DIR_SHOW_IGNORED) && !exc_given)\n \t\tdie(\"ls-files --ignored needs some exclude pattern\");\n \ndiff --git a/t/t1306-xdg-files.sh b/t/t1306-xdg-files.sh\nindex dd87b43be1a6..40d3c42618c0 100755\n--- a/t/t1306-xdg-files.sh\n+++ b/t/t1306-xdg-files.sh\n@@ -116,7 +116,7 @@ test_expect_success 'Exclusion in a non-XDG global ignore file' '\n test_expect_success 'Checking XDG ignore file when HOME is unset' '\n \t(sane_unset HOME &&\n \t git config --unset core.excludesfile &&\n-\t git ls-files --exclude-standard --ignored >actual) &&\n+\t git ls-files --exclude-standard --ignored --others >actual) &&\n \ttest_must_be_empty actual\n '\n \ndiff --git a/t/t3003-ls-files-exclude.sh b/t/t3003-ls-files-exclude.sh\nindex d5ec333131f9..c41c4f046abf 100755\n--- a/t/t3003-ls-files-exclude.sh\n+++ b/t/t3003-ls-files-exclude.sh\n@@ -29,11 +29,11 @@ test_expect_success 'add file to gitignore' '\n '\n check_all_output\n \n-test_expect_success 'ls-files -i lists only tracked-but-ignored files' '\n+test_expect_success 'ls-files -i -c lists only tracked-but-ignored files' '\n \techo content >other-file &&\n \tgit add other-file &&\n \techo file >expect &&\n-\tgit ls-files -i --exclude-standard >output &&\n+\tgit ls-files -i -c --exclude-standard >output &&\n \ttest_cmp expect output\n '\n \n-- \ngitgitgadget\n\n"},{"id":"424363","messageId":"b014ccbbaf3e4bbee310ac19c09e467d6cea2cef.1620840502.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"[PATCH v5 5/9] t3001, t7300: add testcase showcasing missed directory traversal","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:18Z","receivedAt":"2021-05-12T17:55:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIn the last commit, we added a testcase showing that the directory\ntraversal machinery sometimes traverses into directories unnecessarily.\nHere we show that there are cases where it does the opposite: it does\nnot traverse into directories, despite those directories having\nimportant files that need to be flagged.\n\nAdd a testcase showing that `git ls-files -o -i --directory` can omit\nsome of the files it should be listing, and another showing that `git\nclean -fX` can fail to clean out some of the expected files.\n\nSigned-off-by: Elijah Newren \n---\n t/t3001-ls-files-others-exclude.sh | 5 +++++\n t/t7300-clean.sh | 19 +++++++++++++++++++\n 2 files changed, 24 insertions(+)\n\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex 1ec7cb57c7a8..ac05d1a17931 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -292,6 +292,11 @@ EOF\n \ttest_cmp expect actual\n '\n \n+test_expect_failure 'ls-files with \"**\" patterns and --directory' '\n+\t# Expectation same as previous test\n+\tgit ls-files --directory -o -i --exclude \"**/a.1\" >actual &&\n+\ttest_cmp expect actual\n+'\n \n test_expect_success 'ls-files with \"**\" patterns and no slashes' '\n \tgit ls-files -o -i --exclude \"one**a.1\" >actual &&\ndiff --git a/t/t7300-clean.sh b/t/t7300-clean.sh\nindex 07e8ba2d4b85..34c08c325407 100755\n--- a/t/t7300-clean.sh\n+++ b/t/t7300-clean.sh\n@@ -769,4 +769,23 @@ test_expect_failure 'avoid traversing into ignored directories' '\n \ttest_cmp trace.expect trace.relevant\n '\n \n+test_expect_failure 'traverse into directories that may have ignored entries' '\n+\ttest_when_finished rm -f output &&\n+\ttest_create_repo need-to-traverse-into-hierarchy &&\n+\t(\n+\t\tcd need-to-traverse-into-hierarchy &&\n+\t\tmkdir -p modules/foobar/src/generated &&\n+\t\t> modules/foobar/src/generated/code.c &&\n+\t\t> modules/foobar/Makefile &&\n+\t\techo \"/modules/**/src/generated/\" >.gitignore &&\n+\n+\t\tgit clean -fX modules/foobar >../output &&\n+\n+\t\tgrep Removing ../output &&\n+\n+\t\ttest_path_is_missing modules/foobar/src/generated/code.c &&\n+\t\ttest_path_is_file modules/foobar/Makefile\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"424356","messageId":"b7c6176560bda67e146d0402f927180dace534a2.1620840502.git.gitgitgadget@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"[PATCH v5 9/9] dir: introduce readdir_skip_dot_and_dotdot() helper","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2021-05-12T17:28:22Z","receivedAt":"2021-05-12T17:55:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany places in the code were doing\n while ((d = readdir(dir)) != NULL) {\n if (is_dot_or_dotdot(d->d_name))\n continue;\n ...process d...\n }\nIntroduce a readdir_skip_dot_and_dotdot() helper to make that a one-liner:\n while ((d = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n ...process d...\n }\n\nThis helper particularly simplifies checks for empty directories.\n\nAlso use this helper in read_cached_dir() so that our statistics are\nconsistent across platforms. (In other words, read_cached_dir() should\nhave been using is_dot_or_dotdot() and skipping such entries, but did\nnot and left it to treat_path() to detect and mark such entries as\npath_none.)\n\nSigned-off-by: Elijah Newren \n---\n builtin/clean.c | 4 +---\n builtin/worktree.c | 4 +---\n diff-no-index.c | 5 ++---\n dir.c | 26 +++++++++++++++++---------\n dir.h | 2 ++\n entry.c | 5 +----\n notes-merge.c | 5 +----\n object-file.c | 4 +---\n packfile.c | 5 +----\n rerere.c | 4 +---\n worktree.c | 12 +++---------\n 11 files changed, 31 insertions(+), 45 deletions(-)\n\ndiff --git a/builtin/clean.c b/builtin/clean.c\nindex 995053b79173..a1a57476153b 100644\n--- a/builtin/clean.c\n+++ b/builtin/clean.c\n@@ -189,10 +189,8 @@ static int remove_dirs(struct strbuf *path, const char *prefix, int force_flag,\n \tstrbuf_complete(path, '/');\n \n \tlen = path->len;\n-\twhile ((e = readdir(dir)) != NULL) {\n+\twhile ((e = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n \t\tstruct stat st;\n-\t\tif (is_dot_or_dotdot(e->d_name))\n-\t\t\tcontinue;\n \n \t\tstrbuf_setlen(path, len);\n \t\tstrbuf_addstr(path, e->d_name);\ndiff --git a/builtin/worktree.c b/builtin/worktree.c\nindex 877145349381..ae28249e0f0b 100644\n--- a/builtin/worktree.c\n+++ b/builtin/worktree.c\n@@ -118,10 +118,8 @@ static void prune_worktrees(void)\n \tstruct dirent *d;\n \tif (!dir)\n \t\treturn;\n-\twhile ((d = readdir(dir)) != NULL) {\n+\twhile ((d = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n \t\tchar *path;\n-\t\tif (is_dot_or_dotdot(d->d_name))\n-\t\t\tcontinue;\n \t\tstrbuf_reset(&reason);\n \t\tif (should_prune_worktree(d->d_name, &reason, &path, expire))\n \t\t\tprune_worktree(d->d_name, reason.buf);\ndiff --git a/diff-no-index.c b/diff-no-index.c\nindex 7814eabfe028..e5cc87837143 100644\n--- a/diff-no-index.c\n+++ b/diff-no-index.c\n@@ -26,9 +26,8 @@ static int read_directory_contents(const char *path, struct string_list *list)\n \tif (!(dir = opendir(path)))\n \t\treturn error(\"Could not open directory %s\", path);\n \n-\twhile ((e = readdir(dir)))\n-\t\tif (!is_dot_or_dotdot(e->d_name))\n-\t\t\tstring_list_insert(list, e->d_name);\n+\twhile ((e = readdir_skip_dot_and_dotdot(dir)))\n+\t\tstring_list_insert(list, e->d_name);\n \n \tclosedir(dir);\n \treturn 0;\ndiff --git a/dir.c b/dir.c\nindex 4794c822b47f..66c8518947dd 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -59,6 +59,18 @@ void dir_init(struct dir_struct *dir)\n \tmemset(dir, 0, sizeof(*dir));\n }\n \n+struct dirent *\n+readdir_skip_dot_and_dotdot(DIR *dirp)\n+{\n+\tstruct dirent *e;\n+\n+\twhile ((e = readdir(dirp)) != NULL) {\n+\t\tif (!is_dot_or_dotdot(e->d_name))\n+\t\t\tbreak;\n+\t}\n+\treturn e;\n+}\n+\n int count_slashes(const char *s)\n {\n \tint cnt = 0;\n@@ -2341,7 +2353,7 @@ static int read_cached_dir(struct cached_dir *cdir)\n \tstruct dirent *de;\n \n \tif (cdir->fdir) {\n-\t\tde = readdir(cdir->fdir);\n+\t\tde = readdir_skip_dot_and_dotdot(cdir->fdir);\n \t\tif (!de) {\n \t\t\tcdir->d_name = NULL;\n \t\t\tcdir->d_type = DT_UNKNOWN;\n@@ -2940,11 +2952,9 @@ int is_empty_dir(const char *path)\n \tif (!dir)\n \t\treturn 0;\n \n-\twhile ((e = readdir(dir)) != NULL)\n-\t\tif (!is_dot_or_dotdot(e->d_name)) {\n-\t\t\tret = 0;\n-\t\t\tbreak;\n-\t\t}\n+\te = readdir_skip_dot_and_dotdot(dir);\n+\tif (e)\n+\t\tret = 0;\n \n \tclosedir(dir);\n \treturn ret;\n@@ -2984,10 +2994,8 @@ static int remove_dir_recurse(struct strbuf *path, int flag, int *kept_up)\n \tstrbuf_complete(path, '/');\n \n \tlen = path->len;\n-\twhile ((e = readdir(dir)) != NULL) {\n+\twhile ((e = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n \t\tstruct stat st;\n-\t\tif (is_dot_or_dotdot(e->d_name))\n-\t\t\tcontinue;\n \n \t\tstrbuf_setlen(path, len);\n \t\tstrbuf_addstr(path, e->d_name);\ndiff --git a/dir.h b/dir.h\nindex 22c67907f689..a704e466afd5 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -342,6 +342,8 @@ struct dir_struct {\n \tunsigned visited_directories;\n };\n \n+struct dirent *readdir_skip_dot_and_dotdot(DIR *dirp);\n+\n /*Count the number of slashes for string s*/\n int count_slashes(const char *s);\n \ndiff --git a/entry.c b/entry.c\nindex 2dc94ba5cc2a..6da589696770 100644\n--- a/entry.c\n+++ b/entry.c\n@@ -57,12 +57,9 @@ static void remove_subtree(struct strbuf *path)\n \n \tif (!dir)\n \t\tdie_errno(\"cannot opendir '%s'\", path->buf);\n-\twhile ((de = readdir(dir)) != NULL) {\n+\twhile ((de = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n \t\tstruct stat st;\n \n-\t\tif (is_dot_or_dotdot(de->d_name))\n-\t\t\tcontinue;\n-\n \t\tstrbuf_addch(path, '/');\n \t\tstrbuf_addstr(path, de->d_name);\n \t\tif (lstat(path->buf, &st))\ndiff --git a/notes-merge.c b/notes-merge.c\nindex d2771fa3d43c..e9d6f86d3428 100644\n--- a/notes-merge.c\n+++ b/notes-merge.c\n@@ -695,13 +695,10 @@ int notes_merge_commit(struct notes_merge_options *o,\n \n \tstrbuf_addch(&path, '/');\n \tbaselen = path.len;\n-\twhile ((e = readdir(dir)) != NULL) {\n+\twhile ((e = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n \t\tstruct stat st;\n \t\tstruct object_id obj_oid, blob_oid;\n \n-\t\tif (is_dot_or_dotdot(e->d_name))\n-\t\t\tcontinue;\n-\n \t\tif (get_oid_hex(e->d_name, &obj_oid)) {\n \t\t\tif (o->verbosity >= 3)\n \t\t\t\tprintf(\"Skipping non-SHA1 entry '%s%s'\\n\",\ndiff --git a/object-file.c b/object-file.c\nindex 624af408cdcd..77bdcfd21bc8 100644\n--- a/object-file.c\n+++ b/object-file.c\n@@ -2304,10 +2304,8 @@ int for_each_file_in_obj_subdir(unsigned int subdir_nr,\n \tstrbuf_addch(path, '/');\n \tbaselen = path->len;\n \n-\twhile ((de = readdir(dir))) {\n+\twhile ((de = readdir_skip_dot_and_dotdot(dir))) {\n \t\tsize_t namelen;\n-\t\tif (is_dot_or_dotdot(de->d_name))\n-\t\t\tcontinue;\n \n \t\tnamelen = strlen(de->d_name);\n \t\tstrbuf_setlen(path, baselen);\ndiff --git a/packfile.c b/packfile.c\nindex 8668345d9309..7c8f1b7202ca 100644\n--- a/packfile.c\n+++ b/packfile.c\n@@ -813,10 +813,7 @@ void for_each_file_in_pack_dir(const char *objdir,\n \t}\n \tstrbuf_addch(&path, '/');\n \tdirnamelen = path.len;\n-\twhile ((de = readdir(dir)) != NULL) {\n-\t\tif (is_dot_or_dotdot(de->d_name))\n-\t\t\tcontinue;\n-\n+\twhile ((de = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n \t\tstrbuf_setlen(&path, dirnamelen);\n \t\tstrbuf_addstr(&path, de->d_name);\n \ndiff --git a/rerere.c b/rerere.c\nindex dee60dc6df63..d83d58df4fbc 100644\n--- a/rerere.c\n+++ b/rerere.c\n@@ -1190,13 +1190,11 @@ void rerere_gc(struct repository *r, struct string_list *rr)\n \tif (!dir)\n \t\tdie_errno(_(\"unable to open rr-cache directory\"));\n \t/* Collect stale conflict IDs ... */\n-\twhile ((e = readdir(dir))) {\n+\twhile ((e = readdir_skip_dot_and_dotdot(dir))) {\n \t\tstruct rerere_dir *rr_dir;\n \t\tstruct rerere_id id;\n \t\tint now_empty;\n \n-\t\tif (is_dot_or_dotdot(e->d_name))\n-\t\t\tcontinue;\n \t\tif (!is_rr_cache_dirname(e->d_name))\n \t\t\tcontinue; /* or should we remove e->d_name? */\n \ndiff --git a/worktree.c b/worktree.c\nindex f35ac40a84a5..237517baee67 100644\n--- a/worktree.c\n+++ b/worktree.c\n@@ -128,10 +128,8 @@ struct worktree **get_worktrees(void)\n \tdir = opendir(path.buf);\n \tstrbuf_release(&path);\n \tif (dir) {\n-\t\twhile ((d = readdir(dir)) != NULL) {\n+\t\twhile ((d = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n \t\t\tstruct worktree *linked = NULL;\n-\t\t\tif (is_dot_or_dotdot(d->d_name))\n-\t\t\t\tcontinue;\n \n \t\t\tif ((linked = get_linked_worktree(d->d_name))) {\n \t\t\t\tALLOC_GROW(list, counter + 1, alloc);\n@@ -486,13 +484,9 @@ int submodule_uses_worktrees(const char *path)\n \tif (!dir)\n \t\treturn 0;\n \n-\twhile ((d = readdir(dir)) != NULL) {\n-\t\tif (is_dot_or_dotdot(d->d_name))\n-\t\t\tcontinue;\n-\n+\td = readdir_skip_dot_and_dotdot(dir);\n+\tif (d != NULL)\n \t\tret = 1;\n-\t\tbreak;\n-\t}\n \tclosedir(dir);\n \treturn ret;\n }\n-- \ngitgitgadget\n"},{"id":"424767","messageId":"bac325b9-8370-601b-a348-a7270df8b824@gmail.com","threadId":"55639","inReplyTo":"179f992edc9254803252ae10e5d692f3755a40f3.1620840502.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 8/9] dir: update stale description of treat_directory()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-17T17:20:56Z","receivedAt":"2021-05-17T17:21:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/12/2021 1:28 PM, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The documentation comment for treat_directory() was originally written\n> in 095952 (Teach directory traversal about subprojects, 2007-04-11)\n> which was before the 'struct dir_struct' split its bitfield of named\n> options into a 'flags' enum in 7c4c97c0 (Turn the flags in struct\n> dir_struct into a single variable, 2009-02-16). When those flags\n> changed, the comment became stale, since members like\n> 'show_other_directories' transitioned into flags like\n> DIR_SHOW_OTHER_DIRECTORIES.\n> \n> Update the comments for treat_directory() to use these flag names rather\n> than the old member names.\n> \n> Signed-off-by: Derrick Stolee \n> Reviewed-by: Elijah Newren \n\nI think you want the \"Reviewed-by\" before the \"Signed-off-by\",\nfollowed by your own sign-off.\n\nThanks,\n-Stolee\n"},{"id":"424768","messageId":"2b7ba27c-5ffa-0ef0-6c7a-4da45f979b9c@gmail.com","threadId":"55639","inReplyTo":"b7c6176560bda67e146d0402f927180dace534a2.1620840502.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 9/9] dir: introduce readdir_skip_dot_and_dotdot() helper","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-17T17:22:48Z","receivedAt":"2021-05-17T17:22:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/12/2021 1:28 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Many places in the code were doing\n> while ((d = readdir(dir)) != NULL) {\n> if (is_dot_or_dotdot(d->d_name))\n> continue;\n> ...process d...\n> }\n> Introduce a readdir_skip_dot_and_dotdot() helper to make that a one-liner:\n> while ((d = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n> ...process d...\n> }\n> \n> This helper particularly simplifies checks for empty directories.\n> \n> Also use this helper in read_cached_dir() so that our statistics are\n> consistent across platforms. (In other words, read_cached_dir() should\n> have been using is_dot_or_dotdot() and skipping such entries, but did\n> not and left it to treat_path() to detect and mark such entries as\n> path_none.)\n\nI like the idea of this helper!\n \n> +struct dirent *\n> +readdir_skip_dot_and_dotdot(DIR *dirp)\n\nnit: This seems like an accidental newline between the\nreturn type and the method name.\n\nOtherwise, patch LGTM.\n\n-Stolee\n"},{"id":"424770","messageId":"e03ea953-5d43-fe33-3219-364ef69691b1@gmail.com","threadId":"55639","inReplyTo":"pull.1020.v5.git.git.1620840502.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 0/9] Directory traversal fixes","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2021-05-17T17:23:41Z","receivedAt":"2021-05-17T17:23:45Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/12/2021 1:28 PM, Elijah Newren via GitGitGadget wrote:\n> This patchset fixes a few directory traversal issues, where fill_directory()\n> would traverse into directories that it shouldn't and not traverse into\n> directories that it should (one of which was originally reported on this\n> list at [1]). And it includes a few cleanups\n\nSorry that I've been sleeping on this series since v1. I re-read this\nversion from scratch and only found a couple nitpicks.\n\n> If anyone has any ideas about a better place to put the \"Some sidenotes\"\n> from the sixth commit message rather than keeping them in a random commit\n> message, that might be helpful.\n\nI don't have any better ideas, sorry.\n\nThanks,\n-Stolee\n"},{"id":"424777","messageId":"xmqqa6ot6t7m.fsf@gitster.g","threadId":"55639","inReplyTo":"bac325b9-8370-601b-a348-a7270df8b824@gmail.com","subject":"Re: [PATCH v5 8/9] dir: update stale description of treat_directory()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-17T19:44:29Z","receivedAt":"2021-05-17T19:44:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 5/12/2021 1:28 PM, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>> \n>> The documentation comment for treat_directory() was originally written\n>> in 095952 (Teach directory traversal about subprojects, 2007-04-11)\n>> which was before the 'struct dir_struct' split its bitfield of named\n>> options into a 'flags' enum in 7c4c97c0 (Turn the flags in struct\n>> dir_struct into a single variable, 2009-02-16). When those flags\n>> changed, the comment became stale, since members like\n>> 'show_other_directories' transitioned into flags like\n>> DIR_SHOW_OTHER_DIRECTORIES.\n>> \n>> Update the comments for treat_directory() to use these flag names rather\n>> than the old member names.\n>> \n>> Signed-off-by: Derrick Stolee \n>> Reviewed-by: Elijah Newren \n>\n> I think you want the \"Reviewed-by\" before the \"Signed-off-by\",\n> followed by your own sign-off.\n\nGrabbing somebody else's signed-off patch, and forwarding it (with\nor without tweaks and enhancements) with your own sign-off would be\na sufficient sign that you've inspected the patch deeply enough to\nbe confident that it is worth forwarding. So I think you can even\nlose the reviewed-by.\n\nBut as long as you are relaying somebody else's patch, DCO asks you\nto sign it off yourself.\n\nThanks.\n"},{"id":"424814","messageId":"CABPp-BH1MzWb-Xchm=DiUj+M73W9zzR6RUG91A7WO908MCt=6A@mail.gmail.com","threadId":"55639","inReplyTo":"xmqqa6ot6t7m.fsf@gitster.g","subject":"Re: [PATCH v5 8/9] dir: update stale description of treat_directory()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-18T03:32:09Z","receivedAt":"2021-05-18T03:32:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, May 17, 2021 at 12:44 PM Junio C Hamano wrote:\n>\n> Derrick Stolee writes:\n>\n> > On 5/12/2021 1:28 PM, Derrick Stolee via GitGitGadget wrote:\n> >> From: Derrick Stolee \n> >>\n> >> The documentation comment for treat_directory() was originally written\n> >> in 095952 (Teach directory traversal about subprojects, 2007-04-11)\n> >> which was before the 'struct dir_struct' split its bitfield of named\n> >> options into a 'flags' enum in 7c4c97c0 (Turn the flags in struct\n> >> dir_struct into a single variable, 2009-02-16). When those flags\n> >> changed, the comment became stale, since members like\n> >> 'show_other_directories' transitioned into flags like\n> >> DIR_SHOW_OTHER_DIRECTORIES.\n> >>\n> >> Update the comments for treat_directory() to use these flag names rather\n> >> than the old member names.\n> >>\n> >> Signed-off-by: Derrick Stolee \n> >> Reviewed-by: Elijah Newren \n> >\n> > I think you want the \"Reviewed-by\" before the \"Signed-off-by\",\n> > followed by your own sign-off.\n>\n> Grabbing somebody else's signed-off patch, and forwarding it (with\n> or without tweaks and enhancements) with your own sign-off would be\n> a sufficient sign that you've inspected the patch deeply enough to\n> be confident that it is worth forwarding. So I think you can even\n> lose the reviewed-by.\n>\n> But as long as you are relaying somebody else's patch, DCO asks you\n> to sign it off yourself.\n>\n> Thanks.\n\nI was going to go fix this up, but it looks like en/dir-traversal has\nalready merged down to next.\n\nWe could revert the last two patches of the series out of next\n(allowing the first seven with the important fixes to merge down) and\nthen I could resubmit just the last two patches. Or we could just let\nthem all merge down as-is. Preferences?\n"},{"id":"424815","messageId":"CABPp-BF7PrSK=yFvAVQe+vpNTv_j1ZTgh_HQeJVeGZeGk=499Q@mail.gmail.com","threadId":"55639","inReplyTo":"2b7ba27c-5ffa-0ef0-6c7a-4da45f979b9c@gmail.com","subject":"Re: [PATCH v5 9/9] dir: introduce readdir_skip_dot_and_dotdot() helper","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2021-05-18T03:34:39Z","receivedAt":"2021-05-18T03:35:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, May 17, 2021 at 10:22 AM Derrick Stolee wrote:\n>\n> On 5/12/2021 1:28 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Many places in the code were doing\n> > while ((d = readdir(dir)) != NULL) {\n> > if (is_dot_or_dotdot(d->d_name))\n> > continue;\n> > ...process d...\n> > }\n> > Introduce a readdir_skip_dot_and_dotdot() helper to make that a one-liner:\n> > while ((d = readdir_skip_dot_and_dotdot(dir)) != NULL) {\n> > ...process d...\n> > }\n> >\n> > This helper particularly simplifies checks for empty directories.\n> >\n> > Also use this helper in read_cached_dir() so that our statistics are\n> > consistent across platforms. (In other words, read_cached_dir() should\n> > have been using is_dot_or_dotdot() and skipping such entries, but did\n> > not and left it to treat_path() to detect and mark such entries as\n> > path_none.)\n>\n> I like the idea of this helper!\n>\n> > +struct dirent *\n> > +readdir_skip_dot_and_dotdot(DIR *dirp)\n>\n> nit: This seems like an accidental newline between the\n> return type and the method name.\n\nI would fix this, but the patch is already in next. If Junio decides\nto revert the last two patches out of next because of the\nSigned-off-by issue on the previous patch, then I'll resubmit this\npatch as well with this issue fixed.\n"},{"id":"424913","messageId":"xmqqtumzzedd.fsf@gitster.g","threadId":"55639","inReplyTo":"CABPp-BH1MzWb-Xchm=DiUj+M73W9zzR6RUG91A7WO908MCt=6A@mail.gmail.com","subject":"Re: [PATCH v5 8/9] dir: update stale description of treat_directory()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2021-05-19T01:44:30Z","receivedAt":"2021-05-19T01:44:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> We could revert the last two patches of the series out of next\n> (allowing the first seven with the important fixes to merge down) and\n> then I could resubmit just the last two patches. Or we could just let\n> them all merge down as-is. Preferences?\n\nThe former, thanks.\n"}]}