{"thread":{"id":"52713","subject":"[PATCH 0/6] Avoid multiple recursive calls for same path in read_directory_recursive()","startedAt":"2020-01-29T22:03:49Z","lastAt":"2020-07-20T20:25:57Z","messageCount":76,"participants":["Elijah Newren via GitGitGadget","Derrick Stolee","Elijah Newren","SZEDER Gábor","Derrick Stolee via GitGitGadget","Junio C Hamano","Andreas Schwab","Martin Ågren"],"isPatch":true,"patchVersion":1,"patchTotal":6},"messages":[{"id":"390750","messageId":"pull.700.git.git.1580335424.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":null,"subject":"[PATCH 0/6] Avoid multiple recursive calls for same path in read_directory_recursive()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-29T22:03:37Z","receivedAt":"2020-01-29T22:03:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series builds on en/fill-directory-fixes-more. This series should\nbe considered an RFC because of the untracked-cache changes (see the last\ntwo commits), for which I'm hoping to get an untracked-cache expert to\ncomment. This series does provide some modest speedups (see second to last\ncommit message), and should allow 'git status --ignored' to complete in a\nmore reasonable timeframe for Martin Melka (see \nhttps://lore.kernel.org/git/CANt4O2L_DZnMqVxZzTBMvr=BTWqB6L0uyORkoN_yMHLmUX7yHw@mail.gmail.com/\n)\n\nElijah Newren (6):\n dir: consolidate treat_path() and treat_one_path()\n dir: fix broken comment\n dir: fix confusion based on variable tense\n dir: move setting of nested_repo next to its actual usage\n dir: replace exponential algorithm with a linear one\n t7063: blindly accept diffs\n\n dir.c | 295 +++++++++++++++++-------------\n t/t7063-status-untracked-cache.sh | 50 ++---\n 2 files changed, 191 insertions(+), 154 deletions(-)\n\n\nbase-commit: 0cbb60574e741e8255ba457606c4c90898cfc755\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-700%2Fnewren%2Ffill-directory-exponential-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-700/newren/fill-directory-exponential-v1\nPull-Request: https://github.com/git/git/pull/700\n-- \ngitgitgadget\n"},{"id":"390751","messageId":"2ceb64ae61eccd662922f6156e00d4044bef515c.1580335424.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.git.git.1580335424.gitgitgadget@gmail.com","subject":"[PATCH 2/6] dir: fix broken comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-29T22:03:39Z","receivedAt":"2020-01-29T22:03:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 68c56aeddb..c358158f55 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2259,7 +2259,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\t/* skip the dir_add_* part */\n+\t\t\t/* skip the add_path_to_appropriate_result_list() */\n \t\t\tcontinue;\n \t\t}\n \n-- \ngitgitgadget\n\n"},{"id":"390752","messageId":"27bc1357964662faeeb1f4cadd31772ed6caf109.1580335424.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.git.git.1580335424.gitgitgadget@gmail.com","subject":"[PATCH 1/6] dir: consolidate treat_path() and treat_one_path()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-29T22:03:38Z","receivedAt":"2020-01-29T22:03:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCommit 16e2cfa90993 (\"read_directory(): further split treat_path()\",\n2010-01-08) split treat_one_path() out of treat_path(), because\ntreat_leading_path() would not have access to a dirent but wanted to\nre-use as much of treat_path() as possible. Not re-using all of\ntreat_path() caused other bugs, as noted in commit b9670c1f5e6b (\"dir:\nfix checks on common prefix directory\", 2019-12-19). Finally, in commit\nad6f2157f951 (\"dir: restructure in a way to avoid passing around a\nstruct dirent\", 2020-01-16), dirents were removed from treat_path() and\nother functions entirely.\n\nSince the only reason for splitting these functions was the lack of a\ndirent -- which no longer applies to either function -- and since the\nsplit caused problems in the past resulting in us not using\ntreat_one_path() separately anymore, just undo the split.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 121 ++++++++++++++++++++++++++--------------------------------\n 1 file changed, 55 insertions(+), 66 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex b460211e61..68c56aeddb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1863,21 +1863,65 @@ static int resolve_dtype(int dtype, struct index_state *istate,\n \treturn dtype;\n }\n \n-static enum path_treatment treat_one_path(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec,\n-\t\t\t\t\t int dtype)\n-{\n-\tint exclude;\n-\tint has_path_in_index = !!index_file_exists(istate, path->buf, path->len, ignore_case);\n+static enum path_treatment treat_path_fast(struct dir_struct *dir,\n+\t\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t\t struct index_state *istate,\n+\t\t\t\t\t struct strbuf *path,\n+\t\t\t\t\t int baselen,\n+\t\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tstrbuf_setlen(path, baselen);\n+\tif (!cdir->ucd) {\n+\t\tstrbuf_addstr(path, cdir->file);\n+\t\treturn path_untracked;\n+\t}\n+\tstrbuf_addstr(path, cdir->ucd->name);\n+\t/* treat_one_path() does this before it calls treat_directory() */\n+\tstrbuf_complete(path, '/');\n+\tif (cdir->ucd->check_only)\n+\t\t/*\n+\t\t * check_only is set as a result of treat_directory() getting\n+\t\t * to its bottom. Verify again the same set of directories\n+\t\t * with check_only set.\n+\t\t */\n+\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n+\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n+\t/*\n+\t * We get path_recurse in the first run when\n+\t * directory_exists_in_index() returns index_nonexistent. We\n+\t * are sure that new changes in the index does not impact the\n+\t * outcome. Return now.\n+\t */\n+\treturn path_recurse;\n+}\n+\n+static enum path_treatment treat_path(struct dir_struct *dir,\n+\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t struct index_state *istate,\n+\t\t\t\t struct strbuf *path,\n+\t\t\t\t int baselen,\n+\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tint has_path_in_index, dtype, exclude;\n \tenum path_treatment path_treatment;\n \n-\tdtype = resolve_dtype(dtype, istate, path->buf, path->len);\n+\tif (!cdir->d_name)\n+\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n+\t\t\t\t baselen, pathspec);\n+\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n+\t\treturn path_none;\n+\tstrbuf_setlen(path, baselen);\n+\tstrbuf_addstr(path, cdir->d_name);\n+\tif (simplify_away(path->buf, path->len, pathspec))\n+\t\treturn path_none;\n+\n+\tdtype = resolve_dtype(cdir->d_type, istate, path->buf, path->len);\n \n \t/* Always exclude indexed files */\n+\thas_path_in_index = !!index_file_exists(istate, path->buf, path->len,\n+\t\t\t\t\t\tignore_case);\n \tif (dtype != DT_DIR && has_path_in_index)\n \t\treturn path_none;\n \n@@ -1942,61 +1986,6 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t}\n }\n \n-static enum path_treatment treat_path_fast(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tstrbuf_setlen(path, baselen);\n-\tif (!cdir->ucd) {\n-\t\tstrbuf_addstr(path, cdir->file);\n-\t\treturn path_untracked;\n-\t}\n-\tstrbuf_addstr(path, cdir->ucd->name);\n-\t/* treat_one_path() does this before it calls treat_directory() */\n-\tstrbuf_complete(path, '/');\n-\tif (cdir->ucd->check_only)\n-\t\t/*\n-\t\t * check_only is set as a result of treat_directory() getting\n-\t\t * to its bottom. Verify again the same set of directories\n-\t\t * with check_only set.\n-\t\t */\n-\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n-\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n-\t/*\n-\t * We get path_recurse in the first run when\n-\t * directory_exists_in_index() returns index_nonexistent. We\n-\t * are sure that new changes in the index does not impact the\n-\t * outcome. Return now.\n-\t */\n-\treturn path_recurse;\n-}\n-\n-static enum path_treatment treat_path(struct dir_struct *dir,\n-\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t struct index_state *istate,\n-\t\t\t\t struct strbuf *path,\n-\t\t\t\t int baselen,\n-\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tif (!cdir->d_name)\n-\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n-\t\t\t\t baselen, pathspec);\n-\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n-\t\treturn path_none;\n-\tstrbuf_setlen(path, baselen);\n-\tstrbuf_addstr(path, cdir->d_name);\n-\tif (simplify_away(path->buf, path->len, pathspec))\n-\t\treturn path_none;\n-\n-\treturn treat_one_path(dir, untracked, istate, path, baselen, pathspec,\n-\t\t\t cdir->d_type);\n-}\n-\n static void add_untracked(struct untracked_cache_dir *dir, const char *name)\n {\n \tif (!dir)\n-- \ngitgitgadget\n\n"},{"id":"390753","messageId":"3b2ec5eaf65c9fe44c4337a4cc2fc3dae6203d54.1580335424.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.git.git.1580335424.gitgitgadget@gmail.com","subject":"[PATCH 4/6] dir: move setting of nested_repo next to its actual usage","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-29T22:03:41Z","receivedAt":"2020-01-29T22:03:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 225f0bc082..ef3307718a 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1659,7 +1659,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n-\tint nested_repo = 0;\n+\tint nested_repo;\n \n \t/* The \"len-1\" is to strip the final '/' */\n \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n@@ -1670,6 +1670,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_none;\n \n \tcase index_nonexistent:\n+\t\tnested_repo = 0;\n \t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n \t\t !(dir->flags & DIR_NO_GITLINKS)) {\n \t\t\tstruct strbuf sb = STRBUF_INIT;\n-- \ngitgitgadget\n\n"},{"id":"390754","messageId":"e6d21228d126d62fafdde185c180f9f5ba64c458.1580335424.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.git.git.1580335424.gitgitgadget@gmail.com","subject":"[PATCH 3/6] dir: fix confusion based on variable tense","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-29T22:03:40Z","receivedAt":"2020-01-29T22:03:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nDespite having contributed several fixes in this area, I have for months\n(years?) assumed that the \"exclude\" variable was a directive; this\ncaused me to think of it as a different mode we operate in and left me\nconfused as I tried to build up a mental model around why we'd need such\na directive. I mostly tried to ignore it while focusing on the pieces I\nwas trying to understand.\n\nThen I finally traced this variable all back to a call to is_excluded(),\nmeaning it was actually functioning as an adjective. In particular, it\nwas a checked property (\"Does this path match a rule in .gitignore?\"),\nrather than a mode passed in from the caller. Change the variable name\nto match the part of speech used by the function called to define it,\nwhich will hopefully make these bits of code slightly clearer to the\nnext reader.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 26 +++++++++++++-------------\n 1 file changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex c358158f55..225f0bc082 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1656,7 +1656,7 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n static enum path_treatment treat_directory(struct dir_struct *dir,\n \tstruct index_state *istate,\n \tstruct untracked_cache_dir *untracked,\n-\tconst char *dirname, int len, int baselen, int exclude,\n+\tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n@@ -1679,13 +1679,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\t}\n \t\tif (nested_repo)\n \t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(exclude ? path_excluded : path_untracked));\n+\t\t\t\t(excluded ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n \t\t\tbreak;\n-\t\tif (exclude &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n+\t\tif (excluded &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n \n \t\t\t/*\n \t\t\t * This is an excluded directory and we are\n@@ -1713,7 +1713,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* This is the \"show_other_directories\" case */\n \n \tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n@@ -1723,7 +1723,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t * the directory contains any files.\n \t */\n \treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, exclude, pathspec);\n+\t\t\t\t\tuntracked, 1, excluded, pathspec);\n }\n \n /*\n@@ -1904,7 +1904,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t int baselen,\n \t\t\t\t const struct pathspec *pathspec)\n {\n-\tint has_path_in_index, dtype, exclude;\n+\tint has_path_in_index, dtype, excluded;\n \tenum path_treatment path_treatment;\n \n \tif (!cdir->d_name)\n@@ -1949,13 +1949,13 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t (directory_exists_in_index(istate, path->buf, path->len) == index_nonexistent))\n \t\treturn path_none;\n \n-\texclude = is_excluded(dir, istate, path->buf, &dtype);\n+\texcluded = is_excluded(dir, istate, path->buf, &dtype);\n \n \t/*\n \t * Excluded? If we don't explicitly want to show\n \t * ignored files, ignore it\n \t */\n-\tif (exclude && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n+\tif (excluded && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n \t\treturn path_excluded;\n \n \tswitch (dtype) {\n@@ -1965,7 +1965,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\tstrbuf_addch(path, '/');\n \t\tpath_treatment = treat_directory(dir, istate, untracked,\n \t\t\t\t\t\t path->buf, path->len,\n-\t\t\t\t\t\t baselen, exclude, pathspec);\n+\t\t\t\t\t\t baselen, excluded, pathspec);\n \t\t/*\n \t\t * If 1) we only want to return directories that\n \t\t * match an exclude pattern and 2) this directory does\n@@ -1974,7 +1974,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t * recurse into this directory (instead of marking the\n \t\t * directory itself as an ignored path).\n \t\t */\n-\t\tif (!exclude &&\n+\t\tif (!excluded &&\n \t\t path_treatment == path_excluded &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n@@ -1982,7 +1982,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\treturn path_treatment;\n \tcase DT_REG:\n \tcase DT_LNK:\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"390755","messageId":"7fb8063541248b7b91d9559dbca1445c634a87f1.1580335424.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.git.git.1580335424.gitgitgadget@gmail.com","subject":"[PATCH 6/6] t7063: blindly accept diffs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-29T22:03:43Z","receivedAt":"2020-01-29T22:03:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAssuming that the changes I made in the last commit to drastically\nmodify how and when and especially how frequently untracked paths are\nvisited should result in changes to the untracked-cache, this commit\nsimply updates the t7063 testcases to match what the code now reports.\n\nIf this is correct, this commit should be squashed into the previous\none.\n\nIt'd be nice if I could get an untracked-cache expert to comment on\nthis...\n\nSigned-off-by: Elijah Newren \n---\n t/t7063-status-untracked-cache.sh | 50 ++++++++++++-------------------\n 1 file changed, 19 insertions(+), 31 deletions(-)\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 190ae149cf..c1b0fd0540 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -85,9 +85,7 @@ dtwo/\n three\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n-three\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \n test_expect_success 'status first time (empty cache)' '\n@@ -140,8 +138,6 @@ test_expect_success 'modify in root directory, one dir invalidation' '\n A done/one\n A one\n A two\n-?? dthree/\n-?? dtwo/\n ?? four\n ?? three\n EOF\n@@ -164,15 +160,11 @@ core.excludesfile 0000000000000000000000000000000000000000\n exclude_per_dir .gitignore\n flags 00000006\n / 0000000000000000000000000000000000000000 recurse valid\n-dthree/\n-dtwo/\n four\n three\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n-three\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -217,9 +209,7 @@ dtwo/\n three\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n-three\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -235,6 +225,7 @@ A done/one\n A one\n A two\n ?? .gitignore\n+?? dthree/\n ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../actual &&\n@@ -256,11 +247,11 @@ exclude_per_dir .gitignore\n flags 00000006\n / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid\n .gitignore\n+dthree/\n dtwo/\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -277,7 +268,6 @@ flags 00000006\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -290,7 +280,6 @@ test_expect_success 'status after the move' '\n A done/one\n A one\n ?? .gitignore\n-?? dtwo/\n ?? two\n EOF\n \ttest_cmp ../status.expect ../actual &&\n@@ -312,12 +301,10 @@ exclude_per_dir .gitignore\n flags 00000006\n / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid\n .gitignore\n-dtwo/\n two\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -334,7 +321,6 @@ flags 00000006\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -348,7 +334,6 @@ A done/one\n A one\n A two\n ?? .gitignore\n-?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../actual &&\n \tcat >../status.expect <../trace.expect <../trace.expect <../trace.expect <../trace.expect <../actual &&\n+\tcat >../expect-from-test-dump <\n\ndir's read_directory_recursive() naturally operates recursively in order\nto walk the directory tree. Treating of directories is sometimes weird\nbecause there are so many different permutations about how to handle\ndirectories. Some examples:\n\n * 'git ls-files -o --directory' only needs to know that a directory\n itself is untracked; it doesn't need to recurse into it to see what\n is underneath.\n\n * 'git status' needs to recurse into an untracked directory, but only\n to determine whether or not it is empty. If there are no files\n underneath, the directory itself will be omitted from the output.\n If it is not empty, only the directory will be listed.\n\n * 'git status --ignored' needs to recurse into untracked directories\n and report all the ignored entries and then report the directory as\n untracked -- UNLESS all the entries under the directory are\n ignored, in which case we don't print any of the entries under the\n directory and just report the directory itself as ignored.\n\n * For 'git clean', we may need to recurse into a directory that\n doesn't match any specified pathspecs, if it's possible that there\n is an entry underneath the directory that can match one of the\n pathspecs. In such a case, we need to be careful to omit the\n directory itself from the list of paths (see e.g. commit\n 404ebceda01c (\"dir: also check directories for matching pathspecs\",\n 2019-09-17))\n\nPart of the tension noted above is that the treatment of a directory can\nchanged based on the files within it, and based on the various settings\nin dir->flags. Trying to keep this in mind while reading over the code,\nit is easy to (accidentally?) think in terms of \"treat_directory() tells\nus what to do with a directory, and read_directory_recursive() is the\nthing that recurses\". Since we need to look into a directory to know\nhow to treat it, though, it was quite easy to decide to recurse into the\ndirectory from treat_directory() by adding a read_directory_recursive()\ncall. Adding such a call is actually fine, IF we didn't also cause\nread_directory_recursive() to recurse into the same directory again.\n\nUnfortunately, commit df5bcdf83aeb (\"dir: recurse into untracked dirs\nfor ignored files\", 2017-05-18), added exactly such a case to the code,\nmeaning we'd have two calls to read_directory_recursive() for an\nuntracked directory. So, if we had a file named\n one/two/three/four/five/somefile.txt\nand nothing in one/ was tracked, then 'git status --ignored' would\ncall read_directory_recursive() twice on the directory 'one/', and\neach of those would call read_directory_recursive() twice on the\ndirectory 'one/two/', and so on until read_directory_recursive() was\ncalled 2^5 times for 'one/two/three/four/five/'.\n\nAvoid calling read_directory_recursive() twice per level by moving a\nlot of the special logic into treat_directory().\n\nSince dir.c is somewhat complex, extra cruft built up around this over\ntime. While trying to unravel it, I noticed several instances where the\nfirst call to read_directory_recursive() would return e.g.\npath_untracked for a some directory and a later one would return e.g.\npath_none, and the code relied on the side-effect of the first adding\nuntracked entries to dir->entries in order to get the correct output\ndespite the supposed override in return value by the later call.\n\nI am somewhat concerned that there are still bugs and maybe even\ntestcases with the wrong expectation. I have tried to carefully\ndocument treat_directory() since it becomes more complex after this\nchange (though much of this complexity came from elsewhere that probably\ndeserved better comments to begin with). However, much of my work felt\nmore like a game of whackamole while attempting to make the code match\nthe existing regression tests than an attempt to create an\nimplementation that matched some clear design. That seems wrong to me,\nbut the rules of existing behavior had so many special cases that I had\na hard time coming up with some overarching rules about what correct\nbehavior is for all cases, forcing me to hope that the regression tests\nare correct and sufficient. (I'll note that this turmoil makes working\nwith dir.c extremely unpleasant for me; I keep hoping it'll get better,\nbut it never seems to.)\n\nHowever, on the positive side, it does make the code much faster. For\nthe following simple shell loop in an empty repository:\n\n for depth in $(seq 10 25)\n do\n dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n rm -rf dir\n mkdir -p $dirs\n >$dirs/untracked-file\n /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n done\n\nI saw the following timings, in seconds (note that the numbers are a\nlittle noisy from run-to-run, but the trend is very clear with every\nrun):\n\n 10: 0.03\n 11: 0.05\n 12: 0.08\n 13: 0.19\n 14: 0.29\n 15: 0.50\n 16: 1.05\n 17: 2.11\n 18: 4.11\n 19: 8.60\n 20: 17.55\n 21: 33.87\n 22: 68.71\n 23: 140.05\n 24: 274.45\n 25: 551.15\n\nAfter this fix, those drop to:\n\n 10: 0.00\n 11: 0.00\n 12: 0.00\n 13: 0.00\n 14: 0.00\n 15: 0.00\n 16: 0.00\n 17: 0.00\n 18: 0.00\n 19: 0.00\n 20: 0.00\n 21: 0.00\n 22: 0.00\n 23: 0.00\n 24: 0.00\n 25: 0.00\n\nIn fact, it isn't until a depth of 190 nested directories that it\nsometimes starts reporting a time of 0.01 seconds and doesn't\nconsistently report 0.01 seconds until there are 240 nested directories.\nThe previous code would have taken\n 17.55 * 2^220 / (60*60*24*365) = 9.4 * 10^59 YEARS\nto have completed the 240 nested directories case. It's not often\nthat you get to speed something up by a factor of 3*10^69.\n\nWARNING: This change breaks t7063. I don't know whether that is to be expected\n(I now intentionally visit untracked directories differently so naturally the\nuntracked cache should change), or if I've broken something. I'm hoping to get\nan untracked cache expert to chime in...\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 151 ++++++++++++++++++++++++++++++++++++++++------------------\n 1 file changed, 105 insertions(+), 46 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex ef3307718a..aaf038a9c4 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1659,7 +1659,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n-\tint nested_repo;\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n+\tenum path_treatment state;\n+\tint nested_repo, old_ignored_nr, stop_early;\n \n \t/* The \"len-1\" is to strip the final '/' */\n \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n@@ -1713,18 +1719,101 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/* This is the \"show_other_directories\" case */\n \n-\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t/*\n+\t * We only need to recurse into untracked/ignored directories if\n+\t * either of the following bits is set:\n+\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n+\t * there are ignored directories below)\n+\t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n+\t * the directory is empty)\n+\t */\n+\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n \t\treturn excluded ? path_excluded : path_untracked;\n \n+\t/*\n+\t * If we only want to determine if dirname is empty, then we can\n+\t * stop at the first file we find underneath that directory rather\n+\t * than continuing to recurse beyond it. If DIR_SHOW_IGNORED_TOO\n+\t * is set, then we want MORE than just determining if dirname is\n+\t * empty.\n+\t */\n+\tstop_early = ((dir->flags & DIR_HIDE_EMPTY_DIRECTORIES) &&\n+\t\t !(dir->flags & DIR_SHOW_IGNORED_TOO));\n+\n+\t/*\n+\t * If /every/ file within an untracked directory is ignored, then\n+\t * we want to treat the directory as ignored (for e.g. status\n+\t * --porcelain), without listing the individual ignored files\n+\t * underneath. To do so, we'll save the current ignored_nr, and\n+\t * pop all the ones added after it if it turns out the entire\n+\t * directory is ignored.\n+\t */\n+\told_ignored_nr = dir->ignored_nr;\n+\n+\t/* Actually recurse into dirname now, we'll fixup the state later. */\n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n+\tstate = read_directory_recursive(dir, istate, dirname, len, untracked,\n+\t\t\t\t\t stop_early, stop_early, pathspec);\n+\n+\t/* There are a variety of reasons we may need to fixup the state... */\n+\tif (state == path_excluded) {\n+\t\tint i;\n+\n+\t\t/*\n+\t\t * When stop_early is set, read_directory_recursive() will\n+\t\t * never return path_untracked regardless of whether\n+\t\t * underlying paths were untracked or ignored (because\n+\t\t * returning early means it excluded some paths, or\n+\t\t * something like that -- see commit 5aaa7fd39aaf (\"Improve\n+\t\t * performance of git status --ignored\", 2017-09-18)).\n+\t\t * However, we're not really concerned with the status of\n+\t\t * files under the directory, we just wanted to know\n+\t\t * whether the directory was empty (state == path_none) or\n+\t\t * not (state == path_excluded), and if not, we'd return\n+\t\t * our original status based on whether the untracked\n+\t\t * directory matched an exclusion pattern.\n+\t\t */\n+\t\tif (stop_early)\n+\t\t\tstate = excluded ? path_excluded : path_untracked;\n+\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * When\n+\t\t\t * !stop_early && state == path_excluded\n+\t\t\t * then all paths under dirname were ignored. For\n+\t\t\t * this case, git status --porcelain wants to just\n+\t\t\t * list the directory itself as ignored and not\n+\t\t\t * list the individual paths underneath. Remove\n+\t\t\t * the individual paths underneath.\n+\t\t\t */\n+\t\t\tfor (i = old_ignored_nr + 1; iignored_nr; ++i)\n+\t\t\t\tfree(dir->ignored[i]);\n+\t\t\tdir->ignored_nr = old_ignored_nr;\n+\t\t}\n+\t}\n \n \t/*\n-\t * If this is an excluded directory, then we only need to check if\n-\t * the directory contains any files.\n+\t * If there is nothing under the current directory and we are not\n+\t * hiding empty directories, then we need to report on the\n+\t * untracked or ignored status of the directory itself.\n \t */\n-\treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, excluded, pathspec);\n+\tif (state == path_none && !(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\tstate = excluded ? path_excluded : path_untracked;\n+\n+\t/*\n+\t * We can recurse into untracked directories that don't match any\n+\t * of the given pathspecs when some file underneath the directory\n+\t * might match one of the pathspecs. If so, we should make sure\n+\t * to note that the directory itself did not match.\n+\t */\n+\tif (pathspec &&\n+\t !match_pathspec(istate, pathspec, dirname, len,\n+\t\t\t 0 /* prefix */, NULL,\n+\t\t\t 0 /* do NOT special case dirs */))\n+\t\tstate = path_none;\n+\n+\treturn state;\n }\n \n /*\n@@ -1872,6 +1961,11 @@ static enum path_treatment treat_path_fast(struct dir_struct *dir,\n \t\t\t\t\t int baselen,\n \t\t\t\t\t const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n \tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n \t\tstrbuf_addstr(path, cdir->file);\n@@ -2177,14 +2271,10 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n \t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in treat_leading_path(). See the commit message for the\n-\t * commit adding this warning as well as the commit preceding it\n-\t * for details.\n+\t * WARNING: Do NOT call recurse unless path_recurse is returned\n+\t * from treat_path(). Recursing on any other return value\n+\t * results in exponential slowdown.\n \t */\n-\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -2206,13 +2296,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tdir_state = state;\n \n \t\t/* recurse into subdir if instructed by treat_path */\n-\t\tif ((state == path_recurse) ||\n-\t\t\t((state == path_untracked) &&\n-\t\t\t (resolve_dtype(cdir.d_type, istate, path.buf, path.len) == DT_DIR) &&\n-\t\t\t ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t (pathspec &&\n-\t\t\t do_match_pathspec(istate, pathspec, path.buf, path.len,\n-\t\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {\n+\t\tif (state == path_recurse) {\n \t\t\tstruct untracked_cache_dir *ud;\n \t\t\tud = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t\t path.buf + baselen,\n@@ -2296,15 +2380,6 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n {\n-\t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in read_directory_recursive(). See 777b420347 (dir:\n-\t * synchronize treat_leading_path() and read_directory_recursive(),\n-\t * 2019-12-19) and its parent commit for details.\n-\t */\n-\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct strbuf subdir = STRBUF_INIT;\n \tint prevlen, baselen;\n@@ -2355,23 +2430,7 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tstrbuf_reset(&subdir);\n \t\tstrbuf_add(&subdir, path+prevlen, baselen-prevlen);\n \t\tcdir.d_name = subdir.buf;\n-\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n-\t\t\t\t pathspec);\n-\t\tif (state == path_untracked &&\n-\t\t resolve_dtype(cdir.d_type, istate, sb.buf, sb.len) == DT_DIR &&\n-\t\t (dir->flags & DIR_SHOW_IGNORED_TOO ||\n-\t\t do_match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)) {\n-\t\t\tif (!match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t\t 0 /* prefix */, NULL,\n-\t\t\t\t\t 0 /* do NOT special case dirs */))\n-\t\t\t\tstate = path_none;\n-\t\t\tadd_path_to_appropriate_result_list(dir, NULL, &cdir,\n-\t\t\t\t\t\t\t istate,\n-\t\t\t\t\t\t\t &sb, baselen,\n-\t\t\t\t\t\t\t pathspec, state);\n-\t\t\tstate = path_recurse;\n-\t\t}\n+\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen, pathspec);\n \n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n-- \ngitgitgadget\n\n"},{"id":"390811","messageId":"183dfd55-7693-3a96-dbe3-418580a706ea@gmail.com","threadId":"52713","inReplyTo":"e6d21228d126d62fafdde185c180f9f5ba64c458.1580335424.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/6] dir: fix confusion based on variable tense","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-30T15:20:02Z","receivedAt":"2020-01-30T15:20:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/29/2020 5:03 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Despite having contributed several fixes in this area, I have for months\n> (years?) assumed that the \"exclude\" variable was a directive; this\n> caused me to think of it as a different mode we operate in and left me\n> confused as I tried to build up a mental model around why we'd need such\n> a directive. I mostly tried to ignore it while focusing on the pieces I\n> was trying to understand.\n> \n> Then I finally traced this variable all back to a call to is_excluded(),\n> meaning it was actually functioning as an adjective. In particular, it\n> was a checked property (\"Does this path match a rule in .gitignore?\"),\n> rather than a mode passed in from the caller. Change the variable name\n> to match the part of speech used by the function called to define it,\n> which will hopefully make these bits of code slightly clearer to the\n> next reader.\n\nI agree that some of the terminology in the .gitignore is confusing,\nespecially when the terminology was used in the opposite sense for\nthe sparse-checkout feature. I think this rename is worth the noise.\n\nFor reference, here are some commits from ds/include-exclude that\nperformed similar refactors:\n\n468ce99b77 unpack-trees: rename 'is_excluded_from_list()'\n65edd96aec treewide: rename 'exclude' methods to 'pattern'\n4ff89ee52c treewide: rename 'EXCL_FLAG_' to 'PATTERN_FLAG_'\ncaa3d55444 treewide: rename 'struct exclude_list' to 'struct pattern_list'\nab8db61390 treewide: rename 'struct exclude' to 'struct path_pattern'\n\nThanks,\n-Stolee\n"},{"id":"390814","messageId":"8d9a6047-26be-ff78-cf1a-9f2b35f927e0@gmail.com","threadId":"52713","inReplyTo":"3b2ec5eaf65c9fe44c4337a4cc2fc3dae6203d54.1580335424.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 4/6] dir: move setting of nested_repo next to its actual usage","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-30T15:33:30Z","receivedAt":"2020-01-30T15:33:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/29/2020 5:03 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Signed-off-by: Elijah Newren \n> ---\n> dir.c | 3 ++-\n> 1 file changed, 2 insertions(+), 1 deletion(-)\n> \n> diff --git a/dir.c b/dir.c\n> index 225f0bc082..ef3307718a 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1659,7 +1659,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> \tconst char *dirname, int len, int baselen, int excluded,\n> \tconst struct pathspec *pathspec)\n> {\n> -\tint nested_repo = 0;\n> +\tint nested_repo;\n> \n> \t/* The \"len-1\" is to strip the final '/' */\n> \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n> @@ -1670,6 +1670,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> \t\treturn path_none;\n> \n> \tcase index_nonexistent:\n> +\t\tnested_repo = 0;\n\nI had to look at this code in-full from en/fill-directory-fixes-more to\nbe sure that this case was the only use of nested_repo. However, I found\nthat this switch statement is unnecessarily complicated. By converting\nthe switch to multiple \"if\" statements, I noticed that the third case\nactually has a \"break\" statement that can lead to the final \"fourth case\"\noutside the switch statement.\n\nHopefully the patch below is a worthy replacement for this one:\n\n-->8--\n\nFrom b5c04e6e028cb6c7f9e78fbdd2182383d928fe6d Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Thu, 30 Jan 2020 15:28:39 +0000\nSubject: [PATCH] dir: refactor treat_directory to clarify variable scope\n\nThe nested_repo variable in treat_directory() is created and\ninitialized before a multi-case switch statement, but is only\nused by one case. In fact, this switch is very asymmetrical,\nas the first two cases are simple but the third is more\ncomplicated than the rest of the method.\n\nExtract the switch statement into a series of \"if\" statements.\nThis simplifies the trivial cases, while highlighting the fact\nthat a \"break\" statement in a condition of the third case\nactually leads to jumping to the fourth case (after the switch).\nThis assists a reader who provides an initial scan to notice\nthere is a second way to approach the \"show_other_directories\"\ncase than simply the response from directory_exists_in_index().\n\nSigned-off-by: Derrick Stolee \n---\n dir.c | 17 ++++++++---------\n 1 file changed, 8 insertions(+), 9 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex b460211e61..e48812efe6 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1659,17 +1659,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int exclude,\n \tconst struct pathspec *pathspec)\n {\n-\tint nested_repo = 0;\n-\n \t/* The \"len-1\" is to strip the final '/' */\n-\tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n-\tcase index_directory:\n-\t\treturn path_recurse;\n+\tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n-\tcase index_gitdir:\n+\tif (status == index_directory)\n+\t\treturn path_recurse;\n+\tif (status == index_gitdir)\n \t\treturn path_none;\n \n-\tcase index_nonexistent:\n+\tif (status == index_nonexistent) {\n+\t\tint nested_repo = 0;\n \t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n \t\t !(dir->flags & DIR_NO_GITLINKS)) {\n \t\t\tstruct strbuf sb = STRBUF_INIT;\n@@ -1682,7 +1681,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\t\t\t(exclude ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n-\t\t\tbreak;\n+\t\t\tgoto show_other_directories;\n \t\tif (exclude &&\n \t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n@@ -1711,7 +1710,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t}\n \n \t/* This is the \"show_other_directories\" case */\n-\n+show_other_directories:\n \tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n \t\treturn exclude ? path_excluded : path_untracked;\n \n-- \n2.25.0.vfs.1.1\n\n\n"},{"id":"390815","messageId":"CABPp-BFbXJRW38CeGy78b22MfZ8cNizexCM-+n-ODqy+fOo2uw@mail.gmail.com","threadId":"52713","inReplyTo":"8d9a6047-26be-ff78-cf1a-9f2b35f927e0@gmail.com","subject":"Re: [PATCH 4/6] dir: move setting of nested_repo next to its actual usage","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-01-30T15:45:35Z","receivedAt":"2020-01-30T15:45:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jan 30, 2020 at 7:33 AM Derrick Stolee wrote:\n>\n> On 1/29/2020 5:03 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Signed-off-by: Elijah Newren \n> > ---\n> > dir.c | 3 ++-\n> > 1 file changed, 2 insertions(+), 1 deletion(-)\n> >\n> > diff --git a/dir.c b/dir.c\n> > index 225f0bc082..ef3307718a 100644\n> > --- a/dir.c\n> > +++ b/dir.c\n> > @@ -1659,7 +1659,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> > const char *dirname, int len, int baselen, int excluded,\n> > const struct pathspec *pathspec)\n> > {\n> > - int nested_repo = 0;\n> > + int nested_repo;\n> >\n> > /* The \"len-1\" is to strip the final '/' */\n> > switch (directory_exists_in_index(istate, dirname, len-1)) {\n> > @@ -1670,6 +1670,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> > return path_none;\n> >\n> > case index_nonexistent:\n> > + nested_repo = 0;\n>\n> I had to look at this code in-full from en/fill-directory-fixes-more to\n> be sure that this case was the only use of nested_repo. However, I found\n> that this switch statement is unnecessarily complicated. By converting\n> the switch to multiple \"if\" statements, I noticed that the third case\n> actually has a \"break\" statement that can lead to the final \"fourth case\"\n> outside the switch statement.\n>\n> Hopefully the patch below is a worthy replacement for this one:\n>\n> -->8--\n>\n> From b5c04e6e028cb6c7f9e78fbdd2182383d928fe6d Mon Sep 17 00:00:00 2001\n> From: Derrick Stolee \n> Date: Thu, 30 Jan 2020 15:28:39 +0000\n> Subject: [PATCH] dir: refactor treat_directory to clarify variable scope\n>\n> The nested_repo variable in treat_directory() is created and\n> initialized before a multi-case switch statement, but is only\n> used by one case. In fact, this switch is very asymmetrical,\n> as the first two cases are simple but the third is more\n> complicated than the rest of the method.\n>\n> Extract the switch statement into a series of \"if\" statements.\n> This simplifies the trivial cases, while highlighting the fact\n> that a \"break\" statement in a condition of the third case\n> actually leads to jumping to the fourth case (after the switch).\n> This assists a reader who provides an initial scan to notice\n> there is a second way to approach the \"show_other_directories\"\n> case than simply the response from directory_exists_in_index().\n\nWait, I'm lost. Wasn't that break statement the only way to get to\nthe \"show_other_directories\" block of code after the switch statement?\n I can't see where the second way is; am I missing something?\n\nThat is, unless directory_exists_in_index() suddenly starts returning\nsome value other than the three current possibilities. Perhaps we\nshould throw a BUG() if we get anything other than index_directory,\nindex_gitdir, or index_nonexistent.\n\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> dir.c | 17 ++++++++---------\n> 1 file changed, 8 insertions(+), 9 deletions(-)\n>\n> diff --git a/dir.c b/dir.c\n> index b460211e61..e48812efe6 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1659,17 +1659,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> const char *dirname, int len, int baselen, int exclude,\n> const struct pathspec *pathspec)\n> {\n> - int nested_repo = 0;\n> -\n> /* The \"len-1\" is to strip the final '/' */\n> - switch (directory_exists_in_index(istate, dirname, len-1)) {\n> - case index_directory:\n> - return path_recurse;\n> + enum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n>\n> - case index_gitdir:\n> + if (status == index_directory)\n> + return path_recurse;\n> + if (status == index_gitdir)\n> return path_none;\n>\n> - case index_nonexistent:\n> + if (status == index_nonexistent) {\n> + int nested_repo = 0;\n> if ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n> !(dir->flags & DIR_NO_GITLINKS)) {\n> struct strbuf sb = STRBUF_INIT;\n> @@ -1682,7 +1681,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> (exclude ? path_excluded : path_untracked));\n>\n> if (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n> - break;\n> + goto show_other_directories;\n> if (exclude &&\n> (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n> (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n> @@ -1711,7 +1710,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> }\n\nI'd say we'd want to add a BUG(\"Unhandled value for\ndirectory_exists_in_index: %d\\n\", status); right here.\n\n>\n> /* This is the \"show_other_directories\" case */\n> -\n> +show_other_directories:\n> if (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n> return exclude ? path_excluded : path_untracked;\n>\n> --\n> 2.25.0.vfs.1.1\n\nOtherwise, the patch looks good to me and I'll be happy to replace my\npatch with this one.\n"},{"id":"390816","messageId":"59b5b766-29e2-a709-b407-56bf6ea60b47@gmail.com","threadId":"52713","inReplyTo":"40b378e7adbbff5ecfd95fd888465fd0f99791c8.1580335424.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/6] dir: replace exponential algorithm with a linear one","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-30T15:55:50Z","receivedAt":"2020-01-30T15:55:55Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"I am very enticed by the subject!\n\nOn 1/29/2020 5:03 PM, Elijah Newren via GitGitGadget wrote:\n> Unfortunately, commit df5bcdf83aeb (\"dir: recurse into untracked dirs\n> for ignored files\", 2017-05-18), added exactly such a case to the code,\n\nI was disappointed that the commit you mention did not add a test for\nthe new behavior, but then found a test change in the following commit\nfb89888849 (dir: hide untracked contents of untracked dirs, 2017-05-18).\nThis makes me feel better that your changes are less likely to un-do\nthe intention of df5bcdf83aeb.\n\n> meaning we'd have two calls to read_directory_recursive() for an\n> untracked directory. So, if we had a file named\n> one/two/three/four/five/somefile.txt\n> and nothing in one/ was tracked, then 'git status --ignored' would\n> call read_directory_recursive() twice on the directory 'one/', and\n> each of those would call read_directory_recursive() twice on the\n> directory 'one/two/', and so on until read_directory_recursive() was\n> called 2^5 times for 'one/two/three/four/five/'.\n\nWow! Good find. \"Accidentally exponential\" is a lot worse than\n\"accidentally quadratic\". At least the N here _usually_ does not\ngrow too quickly, but the constant here (lstat-ing directories and\nfiles) is significant enough that 2^3 or 2^4 is enough to notice\nthe difference.\n\n> Avoid calling read_directory_recursive() twice per level by moving a\n> lot of the special logic into treat_directory().\n> \n> Since dir.c is somewhat complex, extra cruft built up around this over\n> time. While trying to unravel it, I noticed several instances where the\n> first call to read_directory_recursive() would return e.g.\n> path_untracked for a some directory and a later one would return e.g.\n> path_none, and the code relied on the side-effect of the first adding\n> untracked entries to dir->entries in order to get the correct output\n> despite the supposed override in return value by the later call.\n>\n> I am somewhat concerned that there are still bugs and maybe even\n> testcases with the wrong expectation. I have tried to carefully\n> document treat_directory() since it becomes more complex after this\n> change (though much of this complexity came from elsewhere that probably\n> deserved better comments to begin with). However, much of my work felt\n> more like a game of whackamole while attempting to make the code match\n> the existing regression tests than an attempt to create an\n> implementation that matched some clear design. That seems wrong to me,\n> but the rules of existing behavior had so many special cases that I had\n> a hard time coming up with some overarching rules about what correct\n> behavior is for all cases, forcing me to hope that the regression tests\n> are correct and sufficient. (I'll note that this turmoil makes working\n> with dir.c extremely unpleasant for me; I keep hoping it'll get better,\n> but it never seems to.)\n\nKeep fighting the good fight! It appears that some of our most-important\ncode has these complicated cases and side-effects because it has grown\nso organically over time. It's unlikely that someone _could_ rewrite it\nto avoid that pain, as dir.c contains a lot of accumulated knowledge from\nthe many special-cases Git handles. I suppose the only thing we can do\nis try to write as many detailed tests as possible.\n\n> However, on the positive side, it does make the code much faster. For\n> the following simple shell loop in an empty repository:\n> \n> for depth in $(seq 10 25)\n> do\n> dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n> rm -rf dir\n> mkdir -p $dirs\n> >$dirs/untracked-file\n> /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n> done\n> \n> I saw the following timings, in seconds (note that the numbers are a\n> little noisy from run-to-run, but the trend is very clear with every\n> run):\n> \n> 10: 0.03\n> 11: 0.05\n> 12: 0.08\n> 13: 0.19\n> 14: 0.29\n> 15: 0.50\n> 16: 1.05\n> 17: 2.11\n> 18: 4.11\n> 19: 8.60\n> 20: 17.55\n> 21: 33.87\n> 22: 68.71\n> 23: 140.05\n> 24: 274.45\n> 25: 551.15\n\nAre these timings on Linux? I imagine that the timings will increase\nmuch more quickly on Windows.\n\n> After this fix, those drop to:\n> \n> 10: 0.00\n...\n> 25: 0.00\n\nNice. I wonder if presenting these 0.00 values as a table is worth\nthe space? At least the effect is dramatic.\n\n> In fact, it isn't until a depth of 190 nested directories that it\n> sometimes starts reporting a time of 0.01 seconds and doesn't\n> consistently report 0.01 seconds until there are 240 nested directories.\n> The previous code would have taken\n> 17.55 * 2^220 / (60*60*24*365) = 9.4 * 10^59 YEARS\n> to have completed the 240 nested directories case. It's not often\n> that you get to speed something up by a factor of 3*10^69.\n\nAwesome.\n\n> WARNING: This change breaks t7063. I don't know whether that is to be expected\n> (I now intentionally visit untracked directories differently so naturally the\n> untracked cache should change), or if I've broken something. I'm hoping to get\n> an untracked cache expert to chime in...\n\nI suppose that when the untracked cache is enabled, your expectation that we\ndo not need to recurse into an untracked directory is incorrect: we actually\nwant to explore that directory. Is there a mode we can check to see if we\nare REALLY REALLY collecting _all_ untracked paths? Perhaps we need to create\none?\n\nI'm CC'ing Kevin Willford because he is more familiar with the Git index\nthan me, and perhaps the untracked cache in particular.\n\n> Signed-off-by: Elijah Newren \n> ---\n> dir.c | 151 ++++++++++++++++++++++++++++++++++++++++------------------\n> 1 file changed, 105 insertions(+), 46 deletions(-)\n> \n> diff --git a/dir.c b/dir.c\n> index ef3307718a..aaf038a9c4 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1659,7 +1659,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> \tconst char *dirname, int len, int baselen, int excluded,\n> \tconst struct pathspec *pathspec)\n> {\n> -\tint nested_repo;\n> +\t/*\n> +\t * WARNING: From this function, you can return path_recurse or you\n> +\t * can call read_directory_recursive() (or neither), but\n> +\t * you CAN'T DO BOTH.\n> +\t */\n> +\tenum path_treatment state;\n> +\tint nested_repo, old_ignored_nr, stop_early;\n> \n> \t/* The \"len-1\" is to strip the final '/' */\n> \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n> @@ -1713,18 +1719,101 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> \n> \t/* This is the \"show_other_directories\" case */\n> \n> -\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n> +\t/*\n> +\t * We only need to recurse into untracked/ignored directories if\n> +\t * either of the following bits is set:\n> +\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n> +\t * there are ignored directories below)\n> +\t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n> +\t * the directory is empty)\n\nPerhaps here is where you could also have a DIR_LIST_ALL_UNTRACKED\nflag to ensure the untracked cache loads all untracked paths?\n\n> +\t */\n> +\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n> \t\treturn excluded ? path_excluded : path_untracked;\n> \n> +\t/*\n> +\t * If we only want to determine if dirname is empty, then we can\n> +\t * stop at the first file we find underneath that directory rather\n> +\t * than continuing to recurse beyond it. If DIR_SHOW_IGNORED_TOO\n> +\t * is set, then we want MORE than just determining if dirname is\n> +\t * empty.\n> +\t */\n> +\tstop_early = ((dir->flags & DIR_HIDE_EMPTY_DIRECTORIES) &&\n> +\t\t !(dir->flags & DIR_SHOW_IGNORED_TOO));\n> +\n> +\t/*\n> +\t * If /every/ file within an untracked directory is ignored, then\n> +\t * we want to treat the directory as ignored (for e.g. status\n> +\t * --porcelain), without listing the individual ignored files\n> +\t * underneath. To do so, we'll save the current ignored_nr, and\n> +\t * pop all the ones added after it if it turns out the entire\n> +\t * directory is ignored.\n\nHere is a question for an untracked cache expert: Do we store ignored\npaths in the untracked cache?\n\n> +\t */\n> +\told_ignored_nr = dir->ignored_nr;\n> +\n> +\t/* Actually recurse into dirname now, we'll fixup the state later. */\n> \tuntracked = lookup_untracked(dir->untracked, untracked,\n> \t\t\t\t dirname + baselen, len - baselen);\n> +\tstate = read_directory_recursive(dir, istate, dirname, len, untracked,\n> +\t\t\t\t\t stop_early, stop_early, pathspec);\n> +\n> +\t/* There are a variety of reasons we may need to fixup the state... */\n> +\tif (state == path_excluded) {\n> +\t\tint i;\n> +\n> +\t\t/*\n> +\t\t * When stop_early is set, read_directory_recursive() will\n> +\t\t * never return path_untracked regardless of whether\n> +\t\t * underlying paths were untracked or ignored (because\n> +\t\t * returning early means it excluded some paths, or\n> +\t\t * something like that -- see commit 5aaa7fd39aaf (\"Improve\n> +\t\t * performance of git status --ignored\", 2017-09-18)).\n> +\t\t * However, we're not really concerned with the status of\n> +\t\t * files under the directory, we just wanted to know\n> +\t\t * whether the directory was empty (state == path_none) or\n> +\t\t * not (state == path_excluded), and if not, we'd return\n> +\t\t * our original status based on whether the untracked\n> +\t\t * directory matched an exclusion pattern.\n> +\t\t */\n> +\t\tif (stop_early)\n> +\t\t\tstate = excluded ? path_excluded : path_untracked;\n> +\n> +\t\telse {\n> +\t\t\t/*\n> +\t\t\t * When\n> +\t\t\t * !stop_early && state == path_excluded\n> +\t\t\t * then all paths under dirname were ignored. For\n> +\t\t\t * this case, git status --porcelain wants to just\n> +\t\t\t * list the directory itself as ignored and not\n> +\t\t\t * list the individual paths underneath. Remove\n> +\t\t\t * the individual paths underneath.\n> +\t\t\t */\n> +\t\t\tfor (i = old_ignored_nr + 1; iignored_nr; ++i)\n> +\t\t\t\tfree(dir->ignored[i]);\n> +\t\t\tdir->ignored_nr = old_ignored_nr;\n> +\t\t}\n> +\t}\n> \n> \t/*\n> -\t * If this is an excluded directory, then we only need to check if\n> -\t * the directory contains any files.\n> +\t * If there is nothing under the current directory and we are not\n> +\t * hiding empty directories, then we need to report on the\n> +\t * untracked or ignored status of the directory itself.\n> \t */\n> -\treturn read_directory_recursive(dir, istate, dirname, len,\n> -\t\t\t\t\tuntracked, 1, excluded, pathspec);\n> +\tif (state == path_none && !(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n> +\t\tstate = excluded ? path_excluded : path_untracked;\n> +\n> +\t/*\n> +\t * We can recurse into untracked directories that don't match any\n> +\t * of the given pathspecs when some file underneath the directory\n> +\t * might match one of the pathspecs. If so, we should make sure\n> +\t * to note that the directory itself did not match.\n> +\t */\n> +\tif (pathspec &&\n> +\t !match_pathspec(istate, pathspec, dirname, len,\n> +\t\t\t 0 /* prefix */, NULL,\n> +\t\t\t 0 /* do NOT special case dirs */))\n> +\t\tstate = path_none;\n> +\n> +\treturn state;\n> }\n\nThis is certainly a substantial change, and I'm not able to read it\ncarefully right now. I hope to return to it soon, but hopefully I've\npointed out some places that may lead you to resolve your untracked\ncache issues.\n\nThanks,\n-Stolee\n\n"},{"id":"390817","messageId":"1bc41d94-5d4d-4157-fc00-08b97fb20738@gmail.com","threadId":"52713","inReplyTo":"CABPp-BFbXJRW38CeGy78b22MfZ8cNizexCM-+n-ODqy+fOo2uw@mail.gmail.com","subject":"Re: [PATCH 4/6] dir: move setting of nested_repo next to its actual usage","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-30T16:00:48Z","receivedAt":"2020-01-30T16:00:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2020 10:45 AM, Elijah Newren wrote:\n> On Thu, Jan 30, 2020 at 7:33 AM Derrick Stolee wrote:\n>>\n>> On 1/29/2020 5:03 PM, Elijah Newren via GitGitGadget wrote:\n>>> From: Elijah Newren \n>>>\n>>> Signed-off-by: Elijah Newren \n>>> ---\n>>> dir.c | 3 ++-\n>>> 1 file changed, 2 insertions(+), 1 deletion(-)\n>>>\n>>> diff --git a/dir.c b/dir.c\n>>> index 225f0bc082..ef3307718a 100644\n>>> --- a/dir.c\n>>> +++ b/dir.c\n>>> @@ -1659,7 +1659,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n>>> const char *dirname, int len, int baselen, int excluded,\n>>> const struct pathspec *pathspec)\n>>> {\n>>> - int nested_repo = 0;\n>>> + int nested_repo;\n>>>\n>>> /* The \"len-1\" is to strip the final '/' */\n>>> switch (directory_exists_in_index(istate, dirname, len-1)) {\n>>> @@ -1670,6 +1670,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n>>> return path_none;\n>>>\n>>> case index_nonexistent:\n>>> + nested_repo = 0;\n>>\n>> I had to look at this code in-full from en/fill-directory-fixes-more to\n>> be sure that this case was the only use of nested_repo. However, I found\n>> that this switch statement is unnecessarily complicated. By converting\n>> the switch to multiple \"if\" statements, I noticed that the third case\n>> actually has a \"break\" statement that can lead to the final \"fourth case\"\n>> outside the switch statement.\n>>\n>> Hopefully the patch below is a worthy replacement for this one:\n>>\n>> -->8--\n>>\n>> From b5c04e6e028cb6c7f9e78fbdd2182383d928fe6d Mon Sep 17 00:00:00 2001\n>> From: Derrick Stolee \n>> Date: Thu, 30 Jan 2020 15:28:39 +0000\n>> Subject: [PATCH] dir: refactor treat_directory to clarify variable scope\n>>\n>> The nested_repo variable in treat_directory() is created and\n>> initialized before a multi-case switch statement, but is only\n>> used by one case. In fact, this switch is very asymmetrical,\n>> as the first two cases are simple but the third is more\n>> complicated than the rest of the method.\n>>\n>> Extract the switch statement into a series of \"if\" statements.\n>> This simplifies the trivial cases, while highlighting the fact\n>> that a \"break\" statement in a condition of the third case\n>> actually leads to jumping to the fourth case (after the switch).\n>> This assists a reader who provides an initial scan to notice\n>> there is a second way to approach the \"show_other_directories\"\n>> case than simply the response from directory_exists_in_index().\n> \n> Wait, I'm lost. Wasn't that break statement the only way to get to\n> the \"show_other_directories\" block of code after the switch statement?\n> I can't see where the second way is; am I missing something?\n\nAh, I guess I didn't realize that exist_status didn't have a fourth\nmode. I was assuming that normally the switch would not hit any of\nthe case statements was the way you would _assume_ to hit the block\nafter the switch.\n\nSo yes, my statement is incorrect, but the intention is correct:\nthe flow of this method is very confusing.\n\n> That is, unless directory_exists_in_index() suddenly starts returning\n> some value other than the three current possibilities. Perhaps we\n> should throw a BUG() if we get anything other than index_directory,\n> index_gitdir, or index_nonexistent.\n> \n>>\n>> Signed-off-by: Derrick Stolee \n>> ---\n>> dir.c | 17 ++++++++---------\n>> 1 file changed, 8 insertions(+), 9 deletions(-)\n>>\n>> diff --git a/dir.c b/dir.c\n>> index b460211e61..e48812efe6 100644\n>> --- a/dir.c\n>> +++ b/dir.c\n>> @@ -1659,17 +1659,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n>> const char *dirname, int len, int baselen, int exclude,\n>> const struct pathspec *pathspec)\n>> {\n>> - int nested_repo = 0;\n>> -\n>> /* The \"len-1\" is to strip the final '/' */\n>> - switch (directory_exists_in_index(istate, dirname, len-1)) {\n>> - case index_directory:\n>> - return path_recurse;\n>> + enum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n>>\n>> - case index_gitdir:\n>> + if (status == index_directory)\n>> + return path_recurse;\n>> + if (status == index_gitdir)\n>> return path_none;\n>>\n>> - case index_nonexistent:\n>> + if (status == index_nonexistent) {\n\nSince exist_status only has three options, this \"if\" is redundant.\n\n>> + int nested_repo = 0;\n>> if ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n>> !(dir->flags & DIR_NO_GITLINKS)) {\n>> struct strbuf sb = STRBUF_INIT;\n>> @@ -1682,7 +1681,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n>> (exclude ? path_excluded : path_untracked));\n>>\n>> if (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n>> - break;\n>> + goto show_other_directories;\n\nIt would be better to nest the rest of this block in an \nif (!(dir->flags & DIR_SHOW_OTHER_DIRECTORIES))\n\n>> if (exclude &&\n>> (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n>> (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n>> @@ -1711,7 +1710,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n>> }\n> \n> I'd say we'd want to add a BUG(\"Unhandled value for\n> directory_exists_in_index: %d\\n\", status); right here.\n> \n>>\n>> /* This is the \"show_other_directories\" case */\n>> -\n>> +show_other_directories:\n\n...allowing us to drop this.\n\n>> if (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n>> return exclude ? path_excluded : path_untracked;\n>>\n>> --\n>> 2.25.0.vfs.1.1\n> \n> Otherwise, the patch looks good to me and I'll be happy to replace my\n> patch with this one.\n \nLet me send a v2 of this patch now that you've pointed out my error. It\nis worth making this method clearer before you expand substantially on\nthis final case.\n\nThanks,\n-Stolee\n"},{"id":"390818","messageId":"d67eb395-3b6d-9558-0288-cd3878290c9e@gmail.com","threadId":"52713","inReplyTo":"1bc41d94-5d4d-4157-fc00-08b97fb20738@gmail.com","subject":"Re: [PATCH 4/6] dir: move setting of nested_repo next to its actual usage","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-30T16:10:28Z","receivedAt":"2020-01-30T16:10:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2020 11:00 AM, Derrick Stolee wrote:\n> \n> Let me send a v2 of this patch now that you've pointed out my error. It\n> is worth making this method clearer before you expand substantially on\n> this final case.\n\nHere we are:\n\n-->8--\n\nFrom 3fb4fdda25affe9fe6b3e91050e8ad105bcb6fe0 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Thu, 30 Jan 2020 15:28:39 +0000\nSubject: [PATCH v2] dir: refactor treat_directory to clarify control flow\n\nThe logic in treat_directory() is handled by a multi-case\nswitch statement, but this switch is very asymmetrical, as\nthe first two cases are simple but the third is more\ncomplicated than the rest of the method. In fact, the third\ncase includes a \"break\" statement that leads to the block\nof code outside the switch statement. That is the only way\nto reach that block, as the switch handles all possible\nvalues from directory_exists_in_index();\n\nExtract the switch statement into a series of \"if\" statements.\nThis simplifies the trivial cases, while clarifying how to\nreach the \"show_other_directories\" case. This is particularly\nimportant as the \"show_other_directories\" case will expand\nin a later change.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \n---\n dir.c | 33 +++++++++++++++------------------\n 1 file changed, 15 insertions(+), 18 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex b460211e61..0989558ae6 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1660,29 +1660,26 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n-\n \t/* The \"len-1\" is to strip the final '/' */\n-\tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n-\tcase index_directory:\n-\t\treturn path_recurse;\n+\tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n-\tcase index_gitdir:\n+\tif (status == index_directory)\n+\t\treturn path_recurse;\n+\tif (status == index_gitdir)\n \t\treturn path_none;\n \n-\tcase index_nonexistent:\n-\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n-\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n-\t\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\t\tstrbuf_addstr(&sb, dirname);\n-\t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n-\t\t\tstrbuf_release(&sb);\n-\t\t}\n-\t\tif (nested_repo)\n-\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(exclude ? path_excluded : path_untracked));\n+\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n+\t\t!(dir->flags & DIR_NO_GITLINKS)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tstrbuf_addstr(&sb, dirname);\n+\t\tnested_repo = is_nonbare_repository_dir(&sb);\n+\t\tstrbuf_release(&sb);\n+\t}\n+\tif (nested_repo)\n+\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n+\t\t\t(exclude ? path_excluded : path_untracked));\n \n-\t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n-\t\t\tbreak;\n+\tif (!(dir->flags & DIR_SHOW_OTHER_DIRECTORIES)) {\n \t\tif (exclude &&\n \t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n-- \n2.25.0.vfs.1.1\n\n\n\n"},{"id":"390819","messageId":"CABPp-BGe7ae+pf+HUdLE-xDoXEPDJ=5vWH4cnU9N6pZM2o1Q-g@mail.gmail.com","threadId":"52713","inReplyTo":"d67eb395-3b6d-9558-0288-cd3878290c9e@gmail.com","subject":"Re: [PATCH 4/6] dir: move setting of nested_repo next to its actual usage","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-01-30T16:20:33Z","receivedAt":"2020-01-30T16:20:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jan 30, 2020 at 8:10 AM Derrick Stolee wrote:\n>\n> On 1/30/2020 11:00 AM, Derrick Stolee wrote:\n> >\n> > Let me send a v2 of this patch now that you've pointed out my error. It\n> > is worth making this method clearer before you expand substantially on\n> > this final case.\n>\n> Here we are:\n>\n> -->8--\n>\n> From 3fb4fdda25affe9fe6b3e91050e8ad105bcb6fe0 Mon Sep 17 00:00:00 2001\n> From: Derrick Stolee \n> Date: Thu, 30 Jan 2020 15:28:39 +0000\n> Subject: [PATCH v2] dir: refactor treat_directory to clarify control flow\n>\n> The logic in treat_directory() is handled by a multi-case\n> switch statement, but this switch is very asymmetrical, as\n> the first two cases are simple but the third is more\n> complicated than the rest of the method. In fact, the third\n> case includes a \"break\" statement that leads to the block\n> of code outside the switch statement. That is the only way\n> to reach that block, as the switch handles all possible\n> values from directory_exists_in_index();\n>\n> Extract the switch statement into a series of \"if\" statements.\n> This simplifies the trivial cases, while clarifying how to\n> reach the \"show_other_directories\" case. This is particularly\n> important as the \"show_other_directories\" case will expand\n> in a later change.\n>\n> Helped-by: Elijah Newren \n> Signed-off-by: Derrick Stolee \n> ---\n> dir.c | 33 +++++++++++++++------------------\n> 1 file changed, 15 insertions(+), 18 deletions(-)\n>\n> diff --git a/dir.c b/dir.c\n> index b460211e61..0989558ae6 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -1660,29 +1660,26 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> const struct pathspec *pathspec)\n> {\n> int nested_repo = 0;\n> -\n> /* The \"len-1\" is to strip the final '/' */\n> - switch (directory_exists_in_index(istate, dirname, len-1)) {\n> - case index_directory:\n> - return path_recurse;\n> + enum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n>\n> - case index_gitdir:\n> + if (status == index_directory)\n> + return path_recurse;\n> + if (status == index_gitdir)\n> return path_none;\n\nI think right here we should add:\n\n if (status != index_nonexistent):\n BUG(\"Unhandled value for directory_exists_in_index:\n%d\\n\", status);\n\nfor future-proofing, since both you and I had to look up what\npossibilities existed as a return status from\ndirectory_exists_in_index(), and I'd rather a large warning was thrown\nif someone ever adds a fourth option to that function rather than\nassume treat_directory() is fine and only needs to special case two\nchoices.\n\nOr we could add an assert or a code comment, just so long as we\ndocument to future readers that the remainder of the code is assuming\nstatus==index_nonexistent.\n\n> - case index_nonexistent:\n> - if ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n> - !(dir->flags & DIR_NO_GITLINKS)) {\n> - struct strbuf sb = STRBUF_INIT;\n> - strbuf_addstr(&sb, dirname);\n> - nested_repo = is_nonbare_repository_dir(&sb);\n> - strbuf_release(&sb);\n> - }\n> - if (nested_repo)\n> - return ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n> - (exclude ? path_excluded : path_untracked));\n> + if ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n> + !(dir->flags & DIR_NO_GITLINKS)) {\n> + struct strbuf sb = STRBUF_INIT;\n> + strbuf_addstr(&sb, dirname);\n> + nested_repo = is_nonbare_repository_dir(&sb);\n> + strbuf_release(&sb);\n> + }\n> + if (nested_repo)\n> + return ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n> + (exclude ? path_excluded : path_untracked));\n>\n> - if (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n> - break;\n> + if (!(dir->flags & DIR_SHOW_OTHER_DIRECTORIES)) {\n> if (exclude &&\n> (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n> (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n> --\n> 2.25.0.vfs.1.1\n\nOtherwise, I'm quite happy with these changes.\n"},{"id":"390820","messageId":"CABPp-BEQ5s=+6Rnb-A+pdEaoPXxfo-hMSegSe1eai=RE74A3Og@mail.gmail.com","threadId":"52713","inReplyTo":"59b5b766-29e2-a709-b407-56bf6ea60b47@gmail.com","subject":"Re: [PATCH 5/6] dir: replace exponential algorithm with a linear one","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-01-30T17:13:21Z","receivedAt":"2020-01-30T17:13:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jan 30, 2020 at 7:55 AM Derrick Stolee wrote:\n> > However, on the positive side, it does make the code much faster. For\n> > the following simple shell loop in an empty repository:\n> >\n> > for depth in $(seq 10 25)\n> > do\n> > dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n> > rm -rf dir\n> > mkdir -p $dirs\n> > >$dirs/untracked-file\n> > /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n> > done\n> >\n> > I saw the following timings, in seconds (note that the numbers are a\n> > little noisy from run-to-run, but the trend is very clear with every\n> > run):\n> >\n> > 10: 0.03\n> > 11: 0.05\n> > 12: 0.08\n> > 13: 0.19\n> > 14: 0.29\n> > 15: 0.50\n> > 16: 1.05\n> > 17: 2.11\n> > 18: 4.11\n> > 19: 8.60\n> > 20: 17.55\n> > 21: 33.87\n> > 22: 68.71\n> > 23: 140.05\n> > 24: 274.45\n> > 25: 551.15\n>\n> Are these timings on Linux? I imagine that the timings will increase\n> much more quickly on Windows.\n\nYes, on Linux, with an SSD for the hard drive in this case (though I\nsuspect OS caching of the directories would probably eliminate any\ndifferences between an SSD and a spinny disk since the same\ndirectories are visited so many times).\n\n> > After this fix, those drop to:\n> >\n> > 10: 0.00\n> ...\n> > 25: 0.00\n>\n> Nice. I wonder if presenting these 0.00 values as a table is worth\n> the space? At least the effect is dramatic.\n\nI first considered a table, but then noted it didn't match the code\nsnippet I provided and was worried I'd have to spend more time\nexplaining how I post-processed the output from two runs than we'd\ngain from compressing the number of lines of the commit message.\nAssuming reader time was more valuable, I opted to just keep the two\nsnippets of output.\n\n> > WARNING: This change breaks t7063. I don't know whether that is to be expected\n> > (I now intentionally visit untracked directories differently so naturally the\n> > untracked cache should change), or if I've broken something. I'm hoping to get\n> > an untracked cache expert to chime in...\n>\n> I suppose that when the untracked cache is enabled, your expectation that we\n> do not need to recurse into an untracked directory is incorrect: we actually\n> want to explore that directory. Is there a mode we can check to see if we\n> are REALLY REALLY collecting _all_ untracked paths? Perhaps we need to create\n> one?\n\nI don't think I made any significant changes about using the untracked\ncache versus traversing; the primary differences should be that I\ntraverse each directory once instead of 2^N times. However, the\nprevious code would traverse with both check_only=0 and check_only=1,\nand to avoid the whole 2^N thing I only traverse once. That\nfundamentally means I only won't traverse with both settings of that\nflag.\n\nThe output in t7063 seems to suggest to me that the check_only flag\nmatters to what the untracked-cache stores (\"check_only\" literally\nappears as part of the expected output), and the output also suggests\nthat the untracked-cache is recording when entries are visited\nmultiple times somehow. Or maybe I'm just totally misunderstanding\nthe expected output in t7063. I really have no clue about that stuff.\n\n> I'm CC'ing Kevin Willford because he is more familiar with the Git index\n> than me, and perhaps the untracked cache in particular.\n\nGetting another set of eyes, even if they only know enough to provide\nhunches or guesses, would be very welcome.\n\n> > Signed-off-by: Elijah Newren \n> > ---\n> > dir.c | 151 ++++++++++++++++++++++++++++++++++++++++------------------\n> > 1 file changed, 105 insertions(+), 46 deletions(-)\n> >\n> > diff --git a/dir.c b/dir.c\n> > index ef3307718a..aaf038a9c4 100644\n> > --- a/dir.c\n> > +++ b/dir.c\n> > @@ -1659,7 +1659,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> > const char *dirname, int len, int baselen, int excluded,\n> > const struct pathspec *pathspec)\n> > {\n> > - int nested_repo;\n> > + /*\n> > + * WARNING: From this function, you can return path_recurse or you\n> > + * can call read_directory_recursive() (or neither), but\n> > + * you CAN'T DO BOTH.\n> > + */\n> > + enum path_treatment state;\n> > + int nested_repo, old_ignored_nr, stop_early;\n> >\n> > /* The \"len-1\" is to strip the final '/' */\n> > switch (directory_exists_in_index(istate, dirname, len-1)) {\n> > @@ -1713,18 +1719,101 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> >\n> > /* This is the \"show_other_directories\" case */\n> >\n> > - if (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n> > + /*\n> > + * We only need to recurse into untracked/ignored directories if\n> > + * either of the following bits is set:\n> > + * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n> > + * there are ignored directories below)\n> > + * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n> > + * the directory is empty)\n>\n> Perhaps here is where you could also have a DIR_LIST_ALL_UNTRACKED\n> flag to ensure the untracked cache loads all untracked paths?\n\nDo you mean DIR_KEEP_UNTRACKED_CONTENTS (which is documented in dir.h\nas only having meaning when DIR_SHOW_IGNORED_TOO is also set, and thus\ncaused me to not list it separately)?\n\nSpeaking of DIR_KEEP_UNTRACKED_CONTENTS, though, its handling as a\npost-processing step in read_directory() is now inconsistent with how\nwe handle squashing a directory full of ignores into just marking the\ncontaining directory as ignored. I think I should move the\nread_directory() logic for DIR_KEEP_UNTRACKED_CONTENTS to\ntreat_directory() and use another counter similar to old_ignored_nr.\nIt should be more efficient that way, too.\n\n>\n> > + */\n> > + if (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n> > return excluded ? path_excluded : path_untracked;\n> >\n> > + /*\n> > + * If we only want to determine if dirname is empty, then we can\n> > + * stop at the first file we find underneath that directory rather\n> > + * than continuing to recurse beyond it. If DIR_SHOW_IGNORED_TOO\n> > + * is set, then we want MORE than just determining if dirname is\n> > + * empty.\n> > + */\n> > + stop_early = ((dir->flags & DIR_HIDE_EMPTY_DIRECTORIES) &&\n> > + !(dir->flags & DIR_SHOW_IGNORED_TOO));\n> > +\n> > + /*\n> > + * If /every/ file within an untracked directory is ignored, then\n> > + * we want to treat the directory as ignored (for e.g. status\n> > + * --porcelain), without listing the individual ignored files\n> > + * underneath. To do so, we'll save the current ignored_nr, and\n> > + * pop all the ones added after it if it turns out the entire\n> > + * directory is ignored.\n>\n> Here is a question for an untracked cache expert: Do we store ignored\n> paths in the untracked cache?\n\nAccording to 0dcb8d7fe0ec (\"untracked cache: record .gitignore\ninformation and dir hierarchy\", 2015-03-08), no:\n\n This cached output is about untracked files only, not ignored files\n because the number of tracked files is usually small, so small cache\n overhead, while the number of ignored files could go really high\n (e.g. *.o files mixing with source code).\n\n...unless, of course, someone came along later and changed the design goals.\n\n[...]\n\n> This is certainly a substantial change, and I'm not able to read it\n> carefully right now. I hope to return to it soon, but hopefully I've\n> pointed out some places that may lead you to resolve your untracked\n> cache issues.\n\nYeah, it's pretty hard to reason about; personally I needed lots of\ndumps of state during traversals just to partially make sense of it.\n\nI had dumps of output from both before and after my changes printing\nout return values of treat_directory() and paths and a bunch of other\nstuff and was doing lots of comparisons (and repeatedly did this for\nmany, many different testcases with different toplevel git commands).\nIt was particularly annoying that the old stuff would traverse\neverything 2^N times, half the time with check_only on and half the\ntime with it off. It would return different state values for the same\npath from different calls, often depending on the side effects of\ndir.entries having had more entries added by the first recursion to\nget the right output, despite the fact that the \"wrong\" state was\nreturned by treat_directory() for later visits to the same path (e.g.\npath_untracked returned for the first time it was visited, then\npath_none later, and it was a case where path_untracked was correct in\nmy view).\n\nDespite those difficulties, having an extra set of eyes try to reason\nabout it and pointing out anything that looks amiss or even that just\nlooks hard to understand would be very welcome.\n"},{"id":"390821","messageId":"CABPp-BHR7mFP1rKJ6UBETHbv+R6GbDKO8wA31RJ7f4VqpJ77DQ@mail.gmail.com","threadId":"52713","inReplyTo":"CABPp-BEQ5s=+6Rnb-A+pdEaoPXxfo-hMSegSe1eai=RE74A3Og@mail.gmail.com","subject":"Re: [PATCH 5/6] dir: replace exponential algorithm with a linear one","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-01-30T17:45:35Z","receivedAt":"2020-01-30T17:45:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Jan 30, 2020 at 9:13 AM Elijah Newren wrote:\n>\n> On Thu, Jan 30, 2020 at 7:55 AM Derrick Stolee wrote:\n[...]\n> > > @@ -1713,18 +1719,101 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> > >\n> > > /* This is the \"show_other_directories\" case */\n> > >\n> > > - if (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n> > > + /*\n> > > + * We only need to recurse into untracked/ignored directories if\n> > > + * either of the following bits is set:\n> > > + * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n> > > + * there are ignored directories below)\n> > > + * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n> > > + * the directory is empty)\n> >\n> > Perhaps here is where you could also have a DIR_LIST_ALL_UNTRACKED\n> > flag to ensure the untracked cache loads all untracked paths?\n>\n> Do you mean DIR_KEEP_UNTRACKED_CONTENTS (which is documented in dir.h\n> as only having meaning when DIR_SHOW_IGNORED_TOO is also set, and thus\n> caused me to not list it separately)?\n>\n> Speaking of DIR_KEEP_UNTRACKED_CONTENTS, though, its handling as a\n> post-processing step in read_directory() is now inconsistent with how\n> we handle squashing a directory full of ignores into just marking the\n> containing directory as ignored. I think I should move the\n> read_directory() logic for DIR_KEEP_UNTRACKED_CONTENTS to\n> treat_directory() and use another counter similar to old_ignored_nr.\n> It should be more efficient that way, too.\n\nOh, actually, I think I understand what you're getting at so let me\nclear it up. With DIR_SHOW_IGNORED_TOO, we always recurse to the\nbottom, because it's needed to find any files that might be ignored.\n(Maybe we could do something clever with checking .gitignore entries\nand seeing if it's impossible for them to match anything below the\ncurrent directory, but the code doesn't do anything that clever.) As\na side effect, we'll get all untracked files whenever that flag is\nset. As such, the only question is whether we want to keep all those\nextra untracked files that we found or not, which is the purpose of\nDIR_KEEP_UNTRACKED_CONTENTS. Without DIR_SHOW_IGNORED_TOO, there's no\nneed or want to visit all untracked files without also learning of all\nignored files (and, in fact, git-clean is currently the only one that\nwants to know about all untracked files).\n\nAs far as a simple test goes, in a simple repository with a file named\n one/two/three/four/five/untracked-file\nand with nothing else under one/:\n\nBefore my changes:\n $ strace -e trace=file git status --ignored 2>&1 | grep\n'open(\"one/' | grep -v gitignore.*ENOENT | wc -l\n 62\nNote that 62 == 2^5 + 2^4 + 2^3 + 2^2 + 2^1, showing how many\ndirectories we open and read.\n\nAfter my changes:\n $ strace -e trace=file git status --ignored 2>&1 | grep\n'open(\"one/' | grep -v gitignore.*ENOENT | wc -l\n 5\nshowing that it does open and read each directory, but does so only once.\n"},{"id":"390823","messageId":"21d3d07a-d83d-a46c-ba5f-56c90326cb48@gmail.com","threadId":"52713","inReplyTo":"CABPp-BGe7ae+pf+HUdLE-xDoXEPDJ=5vWH4cnU9N6pZM2o1Q-g@mail.gmail.com","subject":"Re: [PATCH 4/6] dir: move setting of nested_repo next to its actual usage","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-01-30T18:17:25Z","receivedAt":"2020-01-30T18:17:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 1/30/2020 11:20 AM, Elijah Newren wrote:\n> On Thu, Jan 30, 2020 at 8:10 AM Derrick Stolee wrote:\n>> diff --git a/dir.c b/dir.c\n>> index b460211e61..0989558ae6 100644\n>> --- a/dir.c\n>> +++ b/dir.c\n>> @@ -1660,29 +1660,26 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n>> const struct pathspec *pathspec)\n>> {\n>> int nested_repo = 0;\n>> -\n>> /* The \"len-1\" is to strip the final '/' */\n>> - switch (directory_exists_in_index(istate, dirname, len-1)) {\n>> - case index_directory:\n>> - return path_recurse;\n>> + enum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n>>\n>> - case index_gitdir:\n>> + if (status == index_directory)\n>> + return path_recurse;\n>> + if (status == index_gitdir)\n>> return path_none;\n> \n> I think right here we should add:\n> \n> if (status != index_nonexistent):\n> BUG(\"Unhandled value for directory_exists_in_index:\n> %d\\n\", status);\n> \n> for future-proofing, since both you and I had to look up what\n> possibilities existed as a return status from\n> directory_exists_in_index(), and I'd rather a large warning was thrown\n> if someone ever adds a fourth option to that function rather than\n> assume treat_directory() is fine and only needs to special case two\n> choices.\n> \n> Or we could add an assert or a code comment, just so long as we\n> document to future readers that the remainder of the code is assuming\n> status==index_nonexistent.\n\nI'm happy if you squash this into the commit. Thanks!\n\n"},{"id":"390916","messageId":"20200131171341.GH10482@szeder.dev","threadId":"52713","inReplyTo":"40b378e7adbbff5ecfd95fd888465fd0f99791c8.1580335424.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/6] dir: replace exponential algorithm with a linear one","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-01-31T17:13:41Z","receivedAt":"2020-01-31T17:13:49Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 29, 2020 at 10:03:42PM +0000, Elijah Newren via GitGitGadget wrote:\n> Part of the tension noted above is that the treatment of a directory can\n> changed based on the files within it, and based on the various settings\n\ns/changed/change/, or perhaps s/changed/be changed/ ?\n\n> Since dir.c is somewhat complex, extra cruft built up around this over\n> time. While trying to unravel it, I noticed several instances where the\n> first call to read_directory_recursive() would return e.g.\n> path_untracked for a some directory and a later one would return e.g.\n\ns/for a some/for some/\n\n> However, on the positive side, it does make the code much faster. For\n> the following simple shell loop in an empty repository:\n> \n> for depth in $(seq 10 25)\n> do\n> dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n> rm -rf dir\n> mkdir -p $dirs\n> >$dirs/untracked-file\n> /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n> done\n> \n> I saw the following timings, in seconds (note that the numbers are a\n> little noisy from run-to-run, but the trend is very clear with every\n> run):\n> \n> 10: 0.03\n> 11: 0.05\n> 12: 0.08\n> 13: 0.19\n> 14: 0.29\n> 15: 0.50\n> 16: 1.05\n> 17: 2.11\n> 18: 4.11\n> 19: 8.60\n> 20: 17.55\n> 21: 33.87\n> 22: 68.71\n> 23: 140.05\n> 24: 274.45\n> 25: 551.15\n> \n> After this fix, those drop to:\n> \n> 10: 0.00\n> 11: 0.00\n> 12: 0.00\n> 13: 0.00\n> 14: 0.00\n> 15: 0.00\n> 16: 0.00\n> 17: 0.00\n> 18: 0.00\n> 19: 0.00\n> 20: 0.00\n> 21: 0.00\n> 22: 0.00\n> 23: 0.00\n> 24: 0.00\n> 25: 0.00\n\nI agree with Derrick here: if you just said that all these report\n0.00, I would have taken your word for it.\n\nHaving said that... I don't know how to get more decimal places out\nof /use/bin/time, but our trace performance facility uses nanosecond\nresolution timestamps. So using this command in the loop above:\n\n GIT_TRACE_PERFORMANCE=2 git status --ignored 2>&1 >/dev/null |\n sed -n -e \"s/.* performance: \\(.*\\): git command.*/$depth: \\1/p\"\n\ngave me this:\n\n 1: 0.000574302 s\n 2: 0.000584995 s\n 3: 0.000608684 s\n 4: 0.000951336 s\n 5: 0.000762019 s\n 6: 0.000816685 s\n 7: 0.000672516 s\n 8: 0.000912628 s\n 9: 0.000661538 s\n 10: 0.000687465 s\n 11: 0.000708880 s\n 12: 0.000693754 s\n 13: 0.000726120 s\n 14: 0.000737334 s\n 15: 0.000787362 s\n 16: 0.000856687 s\n 17: 0.000780892 s\n 18: 0.000790798 s\n 19: 0.000834411 s\n 20: 0.000859094 s\n 21: 0.001230912 s\n 22: 0.001048852 s\n 23: 0.000891057 s\n 24: 0.000934097 s\n 25: 0.001051704 s\n\nNot sure it's worth including, though.\n\n"},{"id":"390917","messageId":"CABPp-BG8wAmOL03mU2Tf8woVKdWw7o6muXenMH4G-Z6_kBEajQ@mail.gmail.com","threadId":"52713","inReplyTo":"20200131171341.GH10482@szeder.dev","subject":"Re: [PATCH 5/6] dir: replace exponential algorithm with a linear one","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-01-31T17:47:45Z","receivedAt":"2020-01-31T17:48:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Jan 31, 2020 at 9:13 AM SZEDER Gábor wrote:\n>\n> On Wed, Jan 29, 2020 at 10:03:42PM +0000, Elijah Newren via GitGitGadget wrote:\n> > Part of the tension noted above is that the treatment of a directory can\n> > changed based on the files within it, and based on the various settings\n>\n> s/changed/change/, or perhaps s/changed/be changed/ ?\n>\n> > Since dir.c is somewhat complex, extra cruft built up around this over\n> > time. While trying to unravel it, I noticed several instances where the\n> > first call to read_directory_recursive() would return e.g.\n> > path_untracked for a some directory and a later one would return e.g.\n>\n> s/for a some/for some/\n>\n> > However, on the positive side, it does make the code much faster. For\n> > the following simple shell loop in an empty repository:\n> >\n> > for depth in $(seq 10 25)\n> > do\n> > dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n> > rm -rf dir\n> > mkdir -p $dirs\n> > >$dirs/untracked-file\n> > /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n> > done\n> >\n> > I saw the following timings, in seconds (note that the numbers are a\n> > little noisy from run-to-run, but the trend is very clear with every\n> > run):\n> >\n> > 10: 0.03\n> > 11: 0.05\n> > 12: 0.08\n> > 13: 0.19\n> > 14: 0.29\n> > 15: 0.50\n> > 16: 1.05\n> > 17: 2.11\n> > 18: 4.11\n> > 19: 8.60\n> > 20: 17.55\n> > 21: 33.87\n> > 22: 68.71\n> > 23: 140.05\n> > 24: 274.45\n> > 25: 551.15\n> >\n> > After this fix, those drop to:\n> >\n> > 10: 0.00\n> > 11: 0.00\n> > 12: 0.00\n> > 13: 0.00\n> > 14: 0.00\n> > 15: 0.00\n> > 16: 0.00\n> > 17: 0.00\n> > 18: 0.00\n> > 19: 0.00\n> > 20: 0.00\n> > 21: 0.00\n> > 22: 0.00\n> > 23: 0.00\n> > 24: 0.00\n> > 25: 0.00\n>\n> I agree with Derrick here: if you just said that all these report\n> 0.00, I would have taken your word for it.\n\nThanks, I'll include all these fixes. Good timing too, as I was about\nto send a re-roll.\n\n> Having said that... I don't know how to get more decimal places out\n> of /use/bin/time, but our trace performance facility uses nanosecond\n> resolution timestamps. So using this command in the loop above:\n>\n> GIT_TRACE_PERFORMANCE=2 git status --ignored 2>&1 >/dev/null |\n> sed -n -e \"s/.* performance: \\(.*\\): git command.*/$depth: \\1/p\"\n>\n> gave me this:\n>\n> 1: 0.000574302 s\n> 2: 0.000584995 s\n> 3: 0.000608684 s\n> 4: 0.000951336 s\n> 5: 0.000762019 s\n> 6: 0.000816685 s\n> 7: 0.000672516 s\n> 8: 0.000912628 s\n> 9: 0.000661538 s\n> 10: 0.000687465 s\n> 11: 0.000708880 s\n> 12: 0.000693754 s\n> 13: 0.000726120 s\n> 14: 0.000737334 s\n> 15: 0.000787362 s\n> 16: 0.000856687 s\n> 17: 0.000780892 s\n> 18: 0.000790798 s\n> 19: 0.000834411 s\n> 20: 0.000859094 s\n> 21: 0.001230912 s\n> 22: 0.001048852 s\n> 23: 0.000891057 s\n> 24: 0.000934097 s\n> 25: 0.001051704 s\n>\n> Not sure it's worth including, though.\n\nYeah, I'm afraid people will spend time trying to analyze it and the\nnumbers are extremely noisy. I instead included some words about\ncounting the number of untracked files opened according to strace,\nwhich shows before we had 2^(1+$depth)-2 untracked directories get\nopened and after we had exactly $depth get opened.\n"},{"id":"390918","messageId":"20200131180409.GI10482@szeder.dev","threadId":"52713","inReplyTo":"e6d21228d126d62fafdde185c180f9f5ba64c458.1580335424.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/6] dir: fix confusion based on variable tense","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-01-31T18:04:09Z","receivedAt":"2020-01-31T18:04:19Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jan 29, 2020 at 10:03:40PM +0000, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Despite having contributed several fixes in this area, I have for months\n> (years?) assumed that the \"exclude\" variable was a directive; this\n> caused me to think of it as a different mode we operate in and left me\n> confused as I tried to build up a mental model around why we'd need such\n> a directive. I mostly tried to ignore it while focusing on the pieces I\n> was trying to understand.\n> \n> Then I finally traced this variable all back to a call to is_excluded(),\n> meaning it was actually functioning as an adjective. In particular, it\n> was a checked property (\"Does this path match a rule in .gitignore?\"),\n> rather than a mode passed in from the caller. Change the variable name\n> to match the part of speech used by the function called to define it,\n> which will hopefully make these bits of code slightly clearer to the\n> next reader.\n\nSlightly related questions: Does 'excluded' always mean ignored? Or\nis it possible for a file to be excluded but for some other reason\nthan being ignored?\n\nI'm never really sure, and of course it doesn't help that we have both\n'.gitignore' and '.git/info/exclude' files and conditions like:\n\n> +\t\tif (excluded &&\n> +\t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n> +\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n\n"},{"id":"390921","messageId":"CABPp-BHy1Yi7RzRcRGCCFfzOnDbgBeHEQz5wjypM=V1hbGjw6g@mail.gmail.com","threadId":"52713","inReplyTo":"20200131180409.GI10482@szeder.dev","subject":"Re: [PATCH 3/6] dir: fix confusion based on variable tense","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-01-31T18:17:54Z","receivedAt":"2020-01-31T18:18:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Jan 31, 2020 at 10:04 AM SZEDER Gábor wrote:\n>\n> On Wed, Jan 29, 2020 at 10:03:40PM +0000, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > Despite having contributed several fixes in this area, I have for months\n> > (years?) assumed that the \"exclude\" variable was a directive; this\n> > caused me to think of it as a different mode we operate in and left me\n> > confused as I tried to build up a mental model around why we'd need such\n> > a directive. I mostly tried to ignore it while focusing on the pieces I\n> > was trying to understand.\n> >\n> > Then I finally traced this variable all back to a call to is_excluded(),\n> > meaning it was actually functioning as an adjective. In particular, it\n> > was a checked property (\"Does this path match a rule in .gitignore?\"),\n> > rather than a mode passed in from the caller. Change the variable name\n> > to match the part of speech used by the function called to define it,\n> > which will hopefully make these bits of code slightly clearer to the\n> > next reader.\n>\n> Slightly related questions: Does 'excluded' always mean ignored? Or\n> is it possible for a file to be excluded but for some other reason\n> than being ignored?\n>\n> I'm never really sure, and of course it doesn't help that we have both\n> '.gitignore' and '.git/info/exclude' files and conditions like:\n>\n> > + if (excluded &&\n> > + (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n> > + (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n>\n\nGood question; no idea. You can start digging into is_excluded() and\nthe pattern list stored in the dir struct and try to trace it back to\nsee if it's just the combination of ignore rules in .gitignore and\n.git/info/exclude and core.excludesFile, or if there is something else\nmeant here.\n"},{"id":"390923","messageId":"27bc1357964662faeeb1f4cadd31772ed6caf109.1580495486.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v2.git.git.1580495486.gitgitgadget@gmail.com","subject":"[PATCH v2 1/6] dir: consolidate treat_path() and treat_one_path()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T18:31:21Z","receivedAt":"2020-01-31T18:31:32Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCommit 16e2cfa90993 (\"read_directory(): further split treat_path()\",\n2010-01-08) split treat_one_path() out of treat_path(), because\ntreat_leading_path() would not have access to a dirent but wanted to\nre-use as much of treat_path() as possible. Not re-using all of\ntreat_path() caused other bugs, as noted in commit b9670c1f5e6b (\"dir:\nfix checks on common prefix directory\", 2019-12-19). Finally, in commit\nad6f2157f951 (\"dir: restructure in a way to avoid passing around a\nstruct dirent\", 2020-01-16), dirents were removed from treat_path() and\nother functions entirely.\n\nSince the only reason for splitting these functions was the lack of a\ndirent -- which no longer applies to either function -- and since the\nsplit caused problems in the past resulting in us not using\ntreat_one_path() separately anymore, just undo the split.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 121 ++++++++++++++++++++++++++--------------------------------\n 1 file changed, 55 insertions(+), 66 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex b460211e61..68c56aeddb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1863,21 +1863,65 @@ static int resolve_dtype(int dtype, struct index_state *istate,\n \treturn dtype;\n }\n \n-static enum path_treatment treat_one_path(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec,\n-\t\t\t\t\t int dtype)\n-{\n-\tint exclude;\n-\tint has_path_in_index = !!index_file_exists(istate, path->buf, path->len, ignore_case);\n+static enum path_treatment treat_path_fast(struct dir_struct *dir,\n+\t\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t\t struct index_state *istate,\n+\t\t\t\t\t struct strbuf *path,\n+\t\t\t\t\t int baselen,\n+\t\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tstrbuf_setlen(path, baselen);\n+\tif (!cdir->ucd) {\n+\t\tstrbuf_addstr(path, cdir->file);\n+\t\treturn path_untracked;\n+\t}\n+\tstrbuf_addstr(path, cdir->ucd->name);\n+\t/* treat_one_path() does this before it calls treat_directory() */\n+\tstrbuf_complete(path, '/');\n+\tif (cdir->ucd->check_only)\n+\t\t/*\n+\t\t * check_only is set as a result of treat_directory() getting\n+\t\t * to its bottom. Verify again the same set of directories\n+\t\t * with check_only set.\n+\t\t */\n+\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n+\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n+\t/*\n+\t * We get path_recurse in the first run when\n+\t * directory_exists_in_index() returns index_nonexistent. We\n+\t * are sure that new changes in the index does not impact the\n+\t * outcome. Return now.\n+\t */\n+\treturn path_recurse;\n+}\n+\n+static enum path_treatment treat_path(struct dir_struct *dir,\n+\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t struct index_state *istate,\n+\t\t\t\t struct strbuf *path,\n+\t\t\t\t int baselen,\n+\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tint has_path_in_index, dtype, exclude;\n \tenum path_treatment path_treatment;\n \n-\tdtype = resolve_dtype(dtype, istate, path->buf, path->len);\n+\tif (!cdir->d_name)\n+\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n+\t\t\t\t baselen, pathspec);\n+\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n+\t\treturn path_none;\n+\tstrbuf_setlen(path, baselen);\n+\tstrbuf_addstr(path, cdir->d_name);\n+\tif (simplify_away(path->buf, path->len, pathspec))\n+\t\treturn path_none;\n+\n+\tdtype = resolve_dtype(cdir->d_type, istate, path->buf, path->len);\n \n \t/* Always exclude indexed files */\n+\thas_path_in_index = !!index_file_exists(istate, path->buf, path->len,\n+\t\t\t\t\t\tignore_case);\n \tif (dtype != DT_DIR && has_path_in_index)\n \t\treturn path_none;\n \n@@ -1942,61 +1986,6 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t}\n }\n \n-static enum path_treatment treat_path_fast(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tstrbuf_setlen(path, baselen);\n-\tif (!cdir->ucd) {\n-\t\tstrbuf_addstr(path, cdir->file);\n-\t\treturn path_untracked;\n-\t}\n-\tstrbuf_addstr(path, cdir->ucd->name);\n-\t/* treat_one_path() does this before it calls treat_directory() */\n-\tstrbuf_complete(path, '/');\n-\tif (cdir->ucd->check_only)\n-\t\t/*\n-\t\t * check_only is set as a result of treat_directory() getting\n-\t\t * to its bottom. Verify again the same set of directories\n-\t\t * with check_only set.\n-\t\t */\n-\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n-\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n-\t/*\n-\t * We get path_recurse in the first run when\n-\t * directory_exists_in_index() returns index_nonexistent. We\n-\t * are sure that new changes in the index does not impact the\n-\t * outcome. Return now.\n-\t */\n-\treturn path_recurse;\n-}\n-\n-static enum path_treatment treat_path(struct dir_struct *dir,\n-\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t struct index_state *istate,\n-\t\t\t\t struct strbuf *path,\n-\t\t\t\t int baselen,\n-\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tif (!cdir->d_name)\n-\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n-\t\t\t\t baselen, pathspec);\n-\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n-\t\treturn path_none;\n-\tstrbuf_setlen(path, baselen);\n-\tstrbuf_addstr(path, cdir->d_name);\n-\tif (simplify_away(path->buf, path->len, pathspec))\n-\t\treturn path_none;\n-\n-\treturn treat_one_path(dir, untracked, istate, path, baselen, pathspec,\n-\t\t\t cdir->d_type);\n-}\n-\n static void add_untracked(struct untracked_cache_dir *dir, const char *name)\n {\n \tif (!dir)\n-- \ngitgitgadget\n\n"},{"id":"390924","messageId":"2ceb64ae61eccd662922f6156e00d4044bef515c.1580495486.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v2.git.git.1580495486.gitgitgadget@gmail.com","subject":"[PATCH v2 2/6] dir: fix broken comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T18:31:22Z","receivedAt":"2020-01-31T18:31:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 68c56aeddb..c358158f55 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2259,7 +2259,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\t/* skip the dir_add_* part */\n+\t\t\t/* skip the add_path_to_appropriate_result_list() */\n \t\t\tcontinue;\n \t\t}\n \n-- \ngitgitgadget\n\n"},{"id":"390925","messageId":"e6d21228d126d62fafdde185c180f9f5ba64c458.1580495486.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v2.git.git.1580495486.gitgitgadget@gmail.com","subject":"[PATCH v2 3/6] dir: fix confusion based on variable tense","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T18:31:23Z","receivedAt":"2020-01-31T18:31:35Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nDespite having contributed several fixes in this area, I have for months\n(years?) assumed that the \"exclude\" variable was a directive; this\ncaused me to think of it as a different mode we operate in and left me\nconfused as I tried to build up a mental model around why we'd need such\na directive. I mostly tried to ignore it while focusing on the pieces I\nwas trying to understand.\n\nThen I finally traced this variable all back to a call to is_excluded(),\nmeaning it was actually functioning as an adjective. In particular, it\nwas a checked property (\"Does this path match a rule in .gitignore?\"),\nrather than a mode passed in from the caller. Change the variable name\nto match the part of speech used by the function called to define it,\nwhich will hopefully make these bits of code slightly clearer to the\nnext reader.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 26 +++++++++++++-------------\n 1 file changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex c358158f55..225f0bc082 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1656,7 +1656,7 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n static enum path_treatment treat_directory(struct dir_struct *dir,\n \tstruct index_state *istate,\n \tstruct untracked_cache_dir *untracked,\n-\tconst char *dirname, int len, int baselen, int exclude,\n+\tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n@@ -1679,13 +1679,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\t}\n \t\tif (nested_repo)\n \t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(exclude ? path_excluded : path_untracked));\n+\t\t\t\t(excluded ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n \t\t\tbreak;\n-\t\tif (exclude &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n+\t\tif (excluded &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n \n \t\t\t/*\n \t\t\t * This is an excluded directory and we are\n@@ -1713,7 +1713,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* This is the \"show_other_directories\" case */\n \n \tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n@@ -1723,7 +1723,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t * the directory contains any files.\n \t */\n \treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, exclude, pathspec);\n+\t\t\t\t\tuntracked, 1, excluded, pathspec);\n }\n \n /*\n@@ -1904,7 +1904,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t int baselen,\n \t\t\t\t const struct pathspec *pathspec)\n {\n-\tint has_path_in_index, dtype, exclude;\n+\tint has_path_in_index, dtype, excluded;\n \tenum path_treatment path_treatment;\n \n \tif (!cdir->d_name)\n@@ -1949,13 +1949,13 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t (directory_exists_in_index(istate, path->buf, path->len) == index_nonexistent))\n \t\treturn path_none;\n \n-\texclude = is_excluded(dir, istate, path->buf, &dtype);\n+\texcluded = is_excluded(dir, istate, path->buf, &dtype);\n \n \t/*\n \t * Excluded? If we don't explicitly want to show\n \t * ignored files, ignore it\n \t */\n-\tif (exclude && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n+\tif (excluded && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n \t\treturn path_excluded;\n \n \tswitch (dtype) {\n@@ -1965,7 +1965,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\tstrbuf_addch(path, '/');\n \t\tpath_treatment = treat_directory(dir, istate, untracked,\n \t\t\t\t\t\t path->buf, path->len,\n-\t\t\t\t\t\t baselen, exclude, pathspec);\n+\t\t\t\t\t\t baselen, excluded, pathspec);\n \t\t/*\n \t\t * If 1) we only want to return directories that\n \t\t * match an exclude pattern and 2) this directory does\n@@ -1974,7 +1974,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t * recurse into this directory (instead of marking the\n \t\t * directory itself as an ignored path).\n \t\t */\n-\t\tif (!exclude &&\n+\t\tif (!excluded &&\n \t\t path_treatment == path_excluded &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n@@ -1982,7 +1982,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\treturn path_treatment;\n \tcase DT_REG:\n \tcase DT_LNK:\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"390926","messageId":"pull.700.v2.git.git.1580495486.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.git.git.1580335424.gitgitgadget@gmail.com","subject":"[PATCH v2 0/6] Avoid multiple recursive calls for same path in read_directory_recursive()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T18:31:20Z","receivedAt":"2020-01-31T18:31:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This patch series builds on en/fill-directory-fixes-more. This series should\nbe considered an RFC because of the untracked-cache changes (see the last\ntwo commits), for which I'm hoping to get an untracked-cache expert to\ncomment. This series does provide some modest speedups (see second to last\ncommit message), and should allow 'git status --ignored' to complete in a\nmore reasonable timeframe for Martin Melka (see \nhttps://lore.kernel.org/git/CANt4O2L_DZnMqVxZzTBMvr=BTWqB6L0uyORkoN_yMHLmUX7yHw@mail.gmail.com/\n)\n\nChanges since v1:\n\n * Replaced patch 4 with improved version from Stolee (with additional\n improvement of my own)\n * Clarifications, wording fixes, and more about linear perf in commit\n message to patch 5\n * More detail in patch 5 about why \"whackamole\" particularly makes me\n uneasy for dir.c\n\nStuff clearly still missing from v2:\n\n * I didn't make the DIR_KEEP_UNTRACKED_CONTENTS changes I mentioned in \n https://lore.kernel.org/git/CABPp-BEQ5s=+6Rnb-A+pdEaoPXxfo-hMSegSe1eai=RE74A3Og@mail.gmail.com/ \n which I think would make the code cleaner & clearer.\n * I still have not addressed the untracked-cache issue mentioned in the\n last two commits. I looked at it very, very briefly, but I was really\n close to doing something similar to [1] and just dropping my patches in\n this series before even submitting them on Wednesday[2] (dir.c is a\n really unpleasant to work in). Other than wording fixes, I just need a\n week or two off from this area before I dig further, unless someone else\n wants to dive in and needs me to provide pointers on what I've done so\n far.\n\n[1] \nhttps://lore.kernel.org/git/pull.676.v3.git.git.1576571586.gitgitgadget@gmail.com/\n[2] I was inches from doing that Wednesday morning. I had done several\nrounds of \"Okay, I fixed all the tests that broke with my changes last time,\nlet's re-run the testsuite -- wow, four totally different tests from\ntestfiles I hadn't looked at before now break\", and decided that I would\nonly do one more before dropping it an maybe coming back in a month or two.\nThat time happened to work, minus the untracked-cache, so I decided to put\nit in front of other eyeballs.\n\nDerrick Stolee (1):\n dir: refactor treat_directory to clarify control flow\n\nElijah Newren (5):\n dir: consolidate treat_path() and treat_one_path()\n dir: fix broken comment\n dir: fix confusion based on variable tense\n dir: replace exponential algorithm with a linear one\n t7063: blindly accept diffs\n\n dir.c | 331 +++++++++++++++++-------------\n t/t7063-status-untracked-cache.sh | 50 ++---\n 2 files changed, 208 insertions(+), 173 deletions(-)\n\n\nbase-commit: 0cbb60574e741e8255ba457606c4c90898cfc755\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-700%2Fnewren%2Ffill-directory-exponential-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-700/newren/fill-directory-exponential-v2\nPull-Request: https://github.com/git/git/pull/700\n\nRange-diff vs v1:\n\n 1: 27bc135796 = 1: 27bc135796 dir: consolidate treat_path() and treat_one_path()\n 2: 2ceb64ae61 = 2: 2ceb64ae61 dir: fix broken comment\n 3: e6d21228d1 = 3: e6d21228d1 dir: fix confusion based on variable tense\n 4: 3b2ec5eaf6 ! 4: f73f0d66d1 dir: move setting of nested_repo next to its actual usage\n @@ -1,26 +1,73 @@\n -Author: Elijah Newren \n +Author: Derrick Stolee \n \n - dir: move setting of nested_repo next to its actual usage\n + dir: refactor treat_directory to clarify control flow\n \n + The logic in treat_directory() is handled by a multi-case\n + switch statement, but this switch is very asymmetrical, as\n + the first two cases are simple but the third is more\n + complicated than the rest of the method. In fact, the third\n + case includes a \"break\" statement that leads to the block\n + of code outside the switch statement. That is the only way\n + to reach that block, as the switch handles all possible\n + values from directory_exists_in_index();\n +\n + Extract the switch statement into a series of \"if\" statements.\n + This simplifies the trivial cases, while clarifying how to\n + reach the \"show_other_directories\" case. This is particularly\n + important as the \"show_other_directories\" case will expand\n + in a later change.\n +\n + Helped-by: Elijah Newren \n + Signed-off-by: Derrick Stolee \n Signed-off-by: Elijah Newren \n \n diff --git a/dir.c b/dir.c\n --- a/dir.c\n +++ b/dir.c\n @@\n - \tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n --\tint nested_repo = 0;\n -+\tint nested_repo;\n - \n + \tint nested_repo = 0;\n +-\n \t/* The \"len-1\" is to strip the final '/' */\n - \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n -@@\n +-\tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n +-\tcase index_directory:\n +-\t\treturn path_recurse;\n ++\tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n + \n +-\tcase index_gitdir:\n ++\tif (status == index_directory)\n ++\t\treturn path_recurse;\n ++\tif (status == index_gitdir)\n \t\treturn path_none;\n ++\tif (status != index_nonexistent)\n ++\t\tBUG(\"Unhandled value for directory_exists_in_index: %d\\n\", status);\n + \n +-\tcase index_nonexistent:\n +-\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n +-\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n +-\t\t\tstruct strbuf sb = STRBUF_INIT;\n +-\t\t\tstrbuf_addstr(&sb, dirname);\n +-\t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n +-\t\t\tstrbuf_release(&sb);\n +-\t\t}\n +-\t\tif (nested_repo)\n +-\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n +-\t\t\t\t(excluded ? path_excluded : path_untracked));\n ++\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n ++\t\t!(dir->flags & DIR_NO_GITLINKS)) {\n ++\t\tstruct strbuf sb = STRBUF_INIT;\n ++\t\tstrbuf_addstr(&sb, dirname);\n ++\t\tnested_repo = is_nonbare_repository_dir(&sb);\n ++\t\tstrbuf_release(&sb);\n ++\t}\n ++\tif (nested_repo)\n ++\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n ++\t\t\t(excluded ? path_excluded : path_untracked));\n \n - \tcase index_nonexistent:\n -+\t\tnested_repo = 0;\n - \t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n - \t\t !(dir->flags & DIR_NO_GITLINKS)) {\n - \t\t\tstruct strbuf sb = STRBUF_INIT;\n +-\t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n +-\t\t\tbreak;\n ++\tif (!(dir->flags & DIR_SHOW_OTHER_DIRECTORIES)) {\n + \t\tif (excluded &&\n + \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n + \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n 5: 40b378e7ad ! 5: d3136ef52f dir: replace exponential algorithm with a linear one\n @@ -20,26 +20,29 @@\n and report all the ignored entries and then report the directory as\n untracked -- UNLESS all the entries under the directory are\n ignored, in which case we don't print any of the entries under the\n - directory and just report the directory itself as ignored.\n + directory and just report the directory itself as ignored. (Note\n + that although this forces us to walk all untracked files underneath\n + the directory as well, we strip them from the output, except for\n + users like 'git clean' who also set DIR_KEEP_TRACKED_CONTENTS.)\n \n * For 'git clean', we may need to recurse into a directory that\n doesn't match any specified pathspecs, if it's possible that there\n is an entry underneath the directory that can match one of the\n pathspecs. In such a case, we need to be careful to omit the\n - directory itself from the list of paths (see e.g. commit\n - 404ebceda01c (\"dir: also check directories for matching pathspecs\",\n - 2019-09-17))\n + directory itself from the list of paths (see commit 404ebceda01c\n + (\"dir: also check directories for matching pathspecs\", 2019-09-17))\n \n Part of the tension noted above is that the treatment of a directory can\n - changed based on the files within it, and based on the various settings\n + change based on the files within it, and based on the various settings\n in dir->flags. Trying to keep this in mind while reading over the code,\n - it is easy to (accidentally?) think in terms of \"treat_directory() tells\n - us what to do with a directory, and read_directory_recursive() is the\n - thing that recurses\". Since we need to look into a directory to know\n - how to treat it, though, it was quite easy to decide to recurse into the\n + it is easy to think in terms of \"treat_directory() tells us what to do\n + with a directory, and read_directory_recursive() is the thing that\n + recurses\". Since we need to look into a directory to know how to treat\n + it, though, it is quite easy to decide to (also) recurse into the\n directory from treat_directory() by adding a read_directory_recursive()\n - call. Adding such a call is actually fine, IF we didn't also cause\n - read_directory_recursive() to recurse into the same directory again.\n + call. Adding such a call is actually fine, IF we make sure that\n + read_directory_recursive() does not also recurse into that same\n + directory.\n \n Unfortunately, commit df5bcdf83aeb (\"dir: recurse into untracked dirs\n for ignored files\", 2017-05-18), added exactly such a case to the code,\n @@ -58,10 +61,12 @@\n Since dir.c is somewhat complex, extra cruft built up around this over\n time. While trying to unravel it, I noticed several instances where the\n first call to read_directory_recursive() would return e.g.\n - path_untracked for a some directory and a later one would return e.g.\n - path_none, and the code relied on the side-effect of the first adding\n - untracked entries to dir->entries in order to get the correct output\n - despite the supposed override in return value by the later call.\n + path_untracked for some directory and a later one would return e.g.\n + path_none, despite the fact that the directory clearly should have been\n + considered untracked. The code happened to work due to the side-effect\n + from the first invocation of adding untracked entries to dir->entries;\n + this allowed it to get the correct output despite the supposed override\n + in return value by the later call.\n \n I am somewhat concerned that there are still bugs and maybe even\n testcases with the wrong expectation. I have tried to carefully\n @@ -74,9 +79,40 @@\n but the rules of existing behavior had so many special cases that I had\n a hard time coming up with some overarching rules about what correct\n behavior is for all cases, forcing me to hope that the regression tests\n - are correct and sufficient. (I'll note that this turmoil makes working\n - with dir.c extremely unpleasant for me; I keep hoping it'll get better,\n - but it never seems to.)\n + are correct and sufficient. Such a hope seems likely to be ill-founded,\n + given my experience with dir.c-related testcases in the last few months:\n +\n + Examples where the documentation was hard to parse or even just wrong:\n + * 3aca58045f4f (git-clean.txt: do not claim we will delete files with\n + -n/--dry-run, 2019-09-17)\n + * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n + repository, 2019-09-17)\n + * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n + Examples where testcases were declared wrong and changed:\n + * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n + repository, 2019-09-17)\n + * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n + * a2b13367fe55 (Revert \"dir.c: make 'git-status --ignored' work within\n + leading directories\", 2019-12-10)\n + Examples where testcases were clearly inadequate:\n + * 502c386ff944 (t7300-clean: demonstrate deleting nested repo with an\n + ignored file breakage, 2019-08-25)\n + * 7541cc530239 (t7300: add testcases showing failure to clean specified\n + pathspecs, 2019-09-17)\n + * a5e916c7453b (dir: fix off-by-one error in match_pathspec_item,\n + 2019-09-17)\n + * 404ebceda01c (dir: also check directories for matching pathspecs,\n + 2019-09-17)\n + * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n + repository, 2019-09-17)\n + * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n + * 452efd11fbf6 (t3011: demonstrate directory traversal failures,\n + 2019-12-10)\n + * b9670c1f5e6b (dir: fix checks on common prefix directory, 2019-12-19)\n + Examples where \"correct behavior\" was unclear to everyone:\n + https://lore.kernel.org/git/20190905154735.29784-1-newren@gmail.com/\n + Other commits of note:\n + * 902b90cf42bc (clean: fix theoretical path corruption, 2019-09-17)\n \n However, on the positive side, it does make the code much faster. For\n the following simple shell loop in an empty repository:\n @@ -111,27 +147,14 @@\n 24: 274.45\n 25: 551.15\n \n - After this fix, those drop to:\n -\n - 10: 0.00\n - 11: 0.00\n - 12: 0.00\n - 13: 0.00\n - 14: 0.00\n - 15: 0.00\n - 16: 0.00\n - 17: 0.00\n - 18: 0.00\n - 19: 0.00\n - 20: 0.00\n - 21: 0.00\n - 22: 0.00\n - 23: 0.00\n - 24: 0.00\n - 25: 0.00\n + For the above run, using strace I can look for the number of untracked\n + directories opened and can verify that it matches the expected\n + 2^($depth+1)-2 (the sum of 2^1 + 2^2 + 2^3 + ... + 2^$depth).\n \n - In fact, it isn't until a depth of 190 nested directories that it\n - sometimes starts reporting a time of 0.01 seconds and doesn't\n + After this fix, with strace I can verify that the number of untracked\n + directories that are opened drops to just $depth, and the timings all\n + drop to 0.00. In fact, it isn't until a depth of 190 nested directories\n + that it sometimes starts reporting a time of 0.01 seconds and doesn't\n consistently report 0.01 seconds until there are 240 nested directories.\n The previous code would have taken\n 17.55 * 2^220 / (60*60*24*365) = 9.4 * 10^59 YEARS\n @@ -152,17 +175,17 @@\n \tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n --\tint nested_repo;\n +-\tint nested_repo = 0;\n +\t/*\n +\t * WARNING: From this function, you can return path_recurse or you\n +\t * can call read_directory_recursive() (or neither), but\n +\t * you CAN'T DO BOTH.\n +\t */\n +\tenum path_treatment state;\n -+\tint nested_repo, old_ignored_nr, stop_early;\n - \n ++\tint nested_repo = 0, old_ignored_nr, stop_early;\n \t/* The \"len-1\" is to strip the final '/' */\n - \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n + \tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n + \n @@\n \n \t/* This is the \"show_other_directories\" case */\n @@ -292,9 +315,9 @@\n -\t * updates in treat_leading_path(). See the commit message for the\n -\t * commit adding this warning as well as the commit preceding it\n -\t * for details.\n -+\t * WARNING: Do NOT call recurse unless path_recurse is returned\n -+\t * from treat_path(). Recursing on any other return value\n -+\t * results in exponential slowdown.\n ++\t * WARNING: Do NOT recurse unless path_recurse is returned from\n ++\t * treat_path(). Recursing on any other return value\n ++\t * can result in exponential slowdown.\n \t */\n -\n \tstruct cached_dir cdir;\n 6: 7fb8063541 = 6: 9a3f20656e t7063: blindly accept diffs\n\n-- \ngitgitgadget\n"},{"id":"390927","messageId":"9a3f20656e20c48096cba32884462920c1eb75ef.1580495486.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v2.git.git.1580495486.gitgitgadget@gmail.com","subject":"[PATCH v2 6/6] t7063: blindly accept diffs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T18:31:26Z","receivedAt":"2020-01-31T18:31:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAssuming that the changes I made in the last commit to drastically\nmodify how and when and especially how frequently untracked paths are\nvisited should result in changes to the untracked-cache, this commit\nsimply updates the t7063 testcases to match what the code now reports.\n\nIf this is correct, this commit should be squashed into the previous\none.\n\nIt'd be nice if I could get an untracked-cache expert to comment on\nthis...\n\nSigned-off-by: Elijah Newren \n---\n t/t7063-status-untracked-cache.sh | 50 ++++++++++++-------------------\n 1 file changed, 19 insertions(+), 31 deletions(-)\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 190ae149cf..c1b0fd0540 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -85,9 +85,7 @@ dtwo/\n three\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n-three\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \n test_expect_success 'status first time (empty cache)' '\n@@ -140,8 +138,6 @@ test_expect_success 'modify in root directory, one dir invalidation' '\n A done/one\n A one\n A two\n-?? dthree/\n-?? dtwo/\n ?? four\n ?? three\n EOF\n@@ -164,15 +160,11 @@ core.excludesfile 0000000000000000000000000000000000000000\n exclude_per_dir .gitignore\n flags 00000006\n / 0000000000000000000000000000000000000000 recurse valid\n-dthree/\n-dtwo/\n four\n three\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n-three\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -217,9 +209,7 @@ dtwo/\n three\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n-three\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -235,6 +225,7 @@ A done/one\n A one\n A two\n ?? .gitignore\n+?? dthree/\n ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../actual &&\n@@ -256,11 +247,11 @@ exclude_per_dir .gitignore\n flags 00000006\n / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid\n .gitignore\n+dthree/\n dtwo/\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -277,7 +268,6 @@ flags 00000006\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -290,7 +280,6 @@ test_expect_success 'status after the move' '\n A done/one\n A one\n ?? .gitignore\n-?? dtwo/\n ?? two\n EOF\n \ttest_cmp ../status.expect ../actual &&\n@@ -312,12 +301,10 @@ exclude_per_dir .gitignore\n flags 00000006\n / e6fcc8f2ee31bae321d66afd183fcb7237afae6e recurse valid\n .gitignore\n-dtwo/\n two\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -334,7 +321,6 @@ flags 00000006\n /done/ 0000000000000000000000000000000000000000 recurse valid\n /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \ttest_cmp ../expect ../actual\n '\n@@ -348,7 +334,6 @@ A done/one\n A one\n A two\n ?? .gitignore\n-?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../actual &&\n \tcat >../status.expect <../trace.expect <../trace.expect <../trace.expect <../trace.expect <../actual &&\n+\tcat >../expect-from-test-dump <\n\ndir's read_directory_recursive() naturally operates recursively in order\nto walk the directory tree. Treating of directories is sometimes weird\nbecause there are so many different permutations about how to handle\ndirectories. Some examples:\n\n * 'git ls-files -o --directory' only needs to know that a directory\n itself is untracked; it doesn't need to recurse into it to see what\n is underneath.\n\n * 'git status' needs to recurse into an untracked directory, but only\n to determine whether or not it is empty. If there are no files\n underneath, the directory itself will be omitted from the output.\n If it is not empty, only the directory will be listed.\n\n * 'git status --ignored' needs to recurse into untracked directories\n and report all the ignored entries and then report the directory as\n untracked -- UNLESS all the entries under the directory are\n ignored, in which case we don't print any of the entries under the\n directory and just report the directory itself as ignored. (Note\n that although this forces us to walk all untracked files underneath\n the directory as well, we strip them from the output, except for\n users like 'git clean' who also set DIR_KEEP_TRACKED_CONTENTS.)\n\n * For 'git clean', we may need to recurse into a directory that\n doesn't match any specified pathspecs, if it's possible that there\n is an entry underneath the directory that can match one of the\n pathspecs. In such a case, we need to be careful to omit the\n directory itself from the list of paths (see commit 404ebceda01c\n (\"dir: also check directories for matching pathspecs\", 2019-09-17))\n\nPart of the tension noted above is that the treatment of a directory can\nchange based on the files within it, and based on the various settings\nin dir->flags. Trying to keep this in mind while reading over the code,\nit is easy to think in terms of \"treat_directory() tells us what to do\nwith a directory, and read_directory_recursive() is the thing that\nrecurses\". Since we need to look into a directory to know how to treat\nit, though, it is quite easy to decide to (also) recurse into the\ndirectory from treat_directory() by adding a read_directory_recursive()\ncall. Adding such a call is actually fine, IF we make sure that\nread_directory_recursive() does not also recurse into that same\ndirectory.\n\nUnfortunately, commit df5bcdf83aeb (\"dir: recurse into untracked dirs\nfor ignored files\", 2017-05-18), added exactly such a case to the code,\nmeaning we'd have two calls to read_directory_recursive() for an\nuntracked directory. So, if we had a file named\n one/two/three/four/five/somefile.txt\nand nothing in one/ was tracked, then 'git status --ignored' would\ncall read_directory_recursive() twice on the directory 'one/', and\neach of those would call read_directory_recursive() twice on the\ndirectory 'one/two/', and so on until read_directory_recursive() was\ncalled 2^5 times for 'one/two/three/four/five/'.\n\nAvoid calling read_directory_recursive() twice per level by moving a\nlot of the special logic into treat_directory().\n\nSince dir.c is somewhat complex, extra cruft built up around this over\ntime. While trying to unravel it, I noticed several instances where the\nfirst call to read_directory_recursive() would return e.g.\npath_untracked for some directory and a later one would return e.g.\npath_none, despite the fact that the directory clearly should have been\nconsidered untracked. The code happened to work due to the side-effect\nfrom the first invocation of adding untracked entries to dir->entries;\nthis allowed it to get the correct output despite the supposed override\nin return value by the later call.\n\nI am somewhat concerned that there are still bugs and maybe even\ntestcases with the wrong expectation. I have tried to carefully\ndocument treat_directory() since it becomes more complex after this\nchange (though much of this complexity came from elsewhere that probably\ndeserved better comments to begin with). However, much of my work felt\nmore like a game of whackamole while attempting to make the code match\nthe existing regression tests than an attempt to create an\nimplementation that matched some clear design. That seems wrong to me,\nbut the rules of existing behavior had so many special cases that I had\na hard time coming up with some overarching rules about what correct\nbehavior is for all cases, forcing me to hope that the regression tests\nare correct and sufficient. Such a hope seems likely to be ill-founded,\ngiven my experience with dir.c-related testcases in the last few months:\n\n Examples where the documentation was hard to parse or even just wrong:\n * 3aca58045f4f (git-clean.txt: do not claim we will delete files with\n -n/--dry-run, 2019-09-17)\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n Examples where testcases were declared wrong and changed:\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n * a2b13367fe55 (Revert \"dir.c: make 'git-status --ignored' work within\n leading directories\", 2019-12-10)\n Examples where testcases were clearly inadequate:\n * 502c386ff944 (t7300-clean: demonstrate deleting nested repo with an\n ignored file breakage, 2019-08-25)\n * 7541cc530239 (t7300: add testcases showing failure to clean specified\n pathspecs, 2019-09-17)\n * a5e916c7453b (dir: fix off-by-one error in match_pathspec_item,\n 2019-09-17)\n * 404ebceda01c (dir: also check directories for matching pathspecs,\n 2019-09-17)\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n * 452efd11fbf6 (t3011: demonstrate directory traversal failures,\n 2019-12-10)\n * b9670c1f5e6b (dir: fix checks on common prefix directory, 2019-12-19)\n Examples where \"correct behavior\" was unclear to everyone:\n https://lore.kernel.org/git/20190905154735.29784-1-newren@gmail.com/\n Other commits of note:\n * 902b90cf42bc (clean: fix theoretical path corruption, 2019-09-17)\n\nHowever, on the positive side, it does make the code much faster. For\nthe following simple shell loop in an empty repository:\n\n for depth in $(seq 10 25)\n do\n dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n rm -rf dir\n mkdir -p $dirs\n >$dirs/untracked-file\n /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n done\n\nI saw the following timings, in seconds (note that the numbers are a\nlittle noisy from run-to-run, but the trend is very clear with every\nrun):\n\n 10: 0.03\n 11: 0.05\n 12: 0.08\n 13: 0.19\n 14: 0.29\n 15: 0.50\n 16: 1.05\n 17: 2.11\n 18: 4.11\n 19: 8.60\n 20: 17.55\n 21: 33.87\n 22: 68.71\n 23: 140.05\n 24: 274.45\n 25: 551.15\n\nFor the above run, using strace I can look for the number of untracked\ndirectories opened and can verify that it matches the expected\n2^($depth+1)-2 (the sum of 2^1 + 2^2 + 2^3 + ... + 2^$depth).\n\nAfter this fix, with strace I can verify that the number of untracked\ndirectories that are opened drops to just $depth, and the timings all\ndrop to 0.00. In fact, it isn't until a depth of 190 nested directories\nthat it sometimes starts reporting a time of 0.01 seconds and doesn't\nconsistently report 0.01 seconds until there are 240 nested directories.\nThe previous code would have taken\n 17.55 * 2^220 / (60*60*24*365) = 9.4 * 10^59 YEARS\nto have completed the 240 nested directories case. It's not often\nthat you get to speed something up by a factor of 3*10^69.\n\nWARNING: This change breaks t7063. I don't know whether that is to be expected\n(I now intentionally visit untracked directories differently so naturally the\nuntracked cache should change), or if I've broken something. I'm hoping to get\nan untracked cache expert to chime in...\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 151 ++++++++++++++++++++++++++++++++++++++++------------------\n 1 file changed, 105 insertions(+), 46 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 6867356a31..9816fa31d9 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1659,7 +1659,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n-\tint nested_repo = 0;\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n+\tenum path_treatment state;\n+\tint nested_repo = 0, old_ignored_nr, stop_early;\n \t/* The \"len-1\" is to strip the final '/' */\n \tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n@@ -1711,18 +1717,101 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/* This is the \"show_other_directories\" case */\n \n-\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t/*\n+\t * We only need to recurse into untracked/ignored directories if\n+\t * either of the following bits is set:\n+\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n+\t * there are ignored directories below)\n+\t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n+\t * the directory is empty)\n+\t */\n+\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n \t\treturn excluded ? path_excluded : path_untracked;\n \n+\t/*\n+\t * If we only want to determine if dirname is empty, then we can\n+\t * stop at the first file we find underneath that directory rather\n+\t * than continuing to recurse beyond it. If DIR_SHOW_IGNORED_TOO\n+\t * is set, then we want MORE than just determining if dirname is\n+\t * empty.\n+\t */\n+\tstop_early = ((dir->flags & DIR_HIDE_EMPTY_DIRECTORIES) &&\n+\t\t !(dir->flags & DIR_SHOW_IGNORED_TOO));\n+\n+\t/*\n+\t * If /every/ file within an untracked directory is ignored, then\n+\t * we want to treat the directory as ignored (for e.g. status\n+\t * --porcelain), without listing the individual ignored files\n+\t * underneath. To do so, we'll save the current ignored_nr, and\n+\t * pop all the ones added after it if it turns out the entire\n+\t * directory is ignored.\n+\t */\n+\told_ignored_nr = dir->ignored_nr;\n+\n+\t/* Actually recurse into dirname now, we'll fixup the state later. */\n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n+\tstate = read_directory_recursive(dir, istate, dirname, len, untracked,\n+\t\t\t\t\t stop_early, stop_early, pathspec);\n+\n+\t/* There are a variety of reasons we may need to fixup the state... */\n+\tif (state == path_excluded) {\n+\t\tint i;\n+\n+\t\t/*\n+\t\t * When stop_early is set, read_directory_recursive() will\n+\t\t * never return path_untracked regardless of whether\n+\t\t * underlying paths were untracked or ignored (because\n+\t\t * returning early means it excluded some paths, or\n+\t\t * something like that -- see commit 5aaa7fd39aaf (\"Improve\n+\t\t * performance of git status --ignored\", 2017-09-18)).\n+\t\t * However, we're not really concerned with the status of\n+\t\t * files under the directory, we just wanted to know\n+\t\t * whether the directory was empty (state == path_none) or\n+\t\t * not (state == path_excluded), and if not, we'd return\n+\t\t * our original status based on whether the untracked\n+\t\t * directory matched an exclusion pattern.\n+\t\t */\n+\t\tif (stop_early)\n+\t\t\tstate = excluded ? path_excluded : path_untracked;\n+\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * When\n+\t\t\t * !stop_early && state == path_excluded\n+\t\t\t * then all paths under dirname were ignored. For\n+\t\t\t * this case, git status --porcelain wants to just\n+\t\t\t * list the directory itself as ignored and not\n+\t\t\t * list the individual paths underneath. Remove\n+\t\t\t * the individual paths underneath.\n+\t\t\t */\n+\t\t\tfor (i = old_ignored_nr + 1; iignored_nr; ++i)\n+\t\t\t\tfree(dir->ignored[i]);\n+\t\t\tdir->ignored_nr = old_ignored_nr;\n+\t\t}\n+\t}\n \n \t/*\n-\t * If this is an excluded directory, then we only need to check if\n-\t * the directory contains any files.\n+\t * If there is nothing under the current directory and we are not\n+\t * hiding empty directories, then we need to report on the\n+\t * untracked or ignored status of the directory itself.\n \t */\n-\treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, excluded, pathspec);\n+\tif (state == path_none && !(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\tstate = excluded ? path_excluded : path_untracked;\n+\n+\t/*\n+\t * We can recurse into untracked directories that don't match any\n+\t * of the given pathspecs when some file underneath the directory\n+\t * might match one of the pathspecs. If so, we should make sure\n+\t * to note that the directory itself did not match.\n+\t */\n+\tif (pathspec &&\n+\t !match_pathspec(istate, pathspec, dirname, len,\n+\t\t\t 0 /* prefix */, NULL,\n+\t\t\t 0 /* do NOT special case dirs */))\n+\t\tstate = path_none;\n+\n+\treturn state;\n }\n \n /*\n@@ -1870,6 +1959,11 @@ static enum path_treatment treat_path_fast(struct dir_struct *dir,\n \t\t\t\t\t int baselen,\n \t\t\t\t\t const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n \tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n \t\tstrbuf_addstr(path, cdir->file);\n@@ -2175,14 +2269,10 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n \t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in treat_leading_path(). See the commit message for the\n-\t * commit adding this warning as well as the commit preceding it\n-\t * for details.\n+\t * WARNING: Do NOT recurse unless path_recurse is returned from\n+\t * treat_path(). Recursing on any other return value\n+\t * can result in exponential slowdown.\n \t */\n-\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -2204,13 +2294,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tdir_state = state;\n \n \t\t/* recurse into subdir if instructed by treat_path */\n-\t\tif ((state == path_recurse) ||\n-\t\t\t((state == path_untracked) &&\n-\t\t\t (resolve_dtype(cdir.d_type, istate, path.buf, path.len) == DT_DIR) &&\n-\t\t\t ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t (pathspec &&\n-\t\t\t do_match_pathspec(istate, pathspec, path.buf, path.len,\n-\t\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {\n+\t\tif (state == path_recurse) {\n \t\t\tstruct untracked_cache_dir *ud;\n \t\t\tud = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t\t path.buf + baselen,\n@@ -2294,15 +2378,6 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n {\n-\t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in read_directory_recursive(). See 777b420347 (dir:\n-\t * synchronize treat_leading_path() and read_directory_recursive(),\n-\t * 2019-12-19) and its parent commit for details.\n-\t */\n-\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct strbuf subdir = STRBUF_INIT;\n \tint prevlen, baselen;\n@@ -2353,23 +2428,7 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tstrbuf_reset(&subdir);\n \t\tstrbuf_add(&subdir, path+prevlen, baselen-prevlen);\n \t\tcdir.d_name = subdir.buf;\n-\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n-\t\t\t\t pathspec);\n-\t\tif (state == path_untracked &&\n-\t\t resolve_dtype(cdir.d_type, istate, sb.buf, sb.len) == DT_DIR &&\n-\t\t (dir->flags & DIR_SHOW_IGNORED_TOO ||\n-\t\t do_match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)) {\n-\t\t\tif (!match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t\t 0 /* prefix */, NULL,\n-\t\t\t\t\t 0 /* do NOT special case dirs */))\n-\t\t\t\tstate = path_none;\n-\t\t\tadd_path_to_appropriate_result_list(dir, NULL, &cdir,\n-\t\t\t\t\t\t\t istate,\n-\t\t\t\t\t\t\t &sb, baselen,\n-\t\t\t\t\t\t\t pathspec, state);\n-\t\t\tstate = path_recurse;\n-\t\t}\n+\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen, pathspec);\n \n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n-- \ngitgitgadget\n\n"},{"id":"390929","messageId":"f73f0d66d14f50bbe7a9b14f0932fa4045644e2e.1580495486.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v2.git.git.1580495486.gitgitgadget@gmail.com","subject":"[PATCH v2 4/6] dir: refactor treat_directory to clarify control flow","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-01-31T18:31:24Z","receivedAt":"2020-01-31T18:31:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe logic in treat_directory() is handled by a multi-case\nswitch statement, but this switch is very asymmetrical, as\nthe first two cases are simple but the third is more\ncomplicated than the rest of the method. In fact, the third\ncase includes a \"break\" statement that leads to the block\nof code outside the switch statement. That is the only way\nto reach that block, as the switch handles all possible\nvalues from directory_exists_in_index();\n\nExtract the switch statement into a series of \"if\" statements.\nThis simplifies the trivial cases, while clarifying how to\nreach the \"show_other_directories\" case. This is particularly\nimportant as the \"show_other_directories\" case will expand\nin a later change.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n dir.c | 35 +++++++++++++++++------------------\n 1 file changed, 17 insertions(+), 18 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 225f0bc082..6867356a31 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1660,29 +1660,28 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n-\n \t/* The \"len-1\" is to strip the final '/' */\n-\tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n-\tcase index_directory:\n-\t\treturn path_recurse;\n+\tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n-\tcase index_gitdir:\n+\tif (status == index_directory)\n+\t\treturn path_recurse;\n+\tif (status == index_gitdir)\n \t\treturn path_none;\n+\tif (status != index_nonexistent)\n+\t\tBUG(\"Unhandled value for directory_exists_in_index: %d\\n\", status);\n \n-\tcase index_nonexistent:\n-\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n-\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n-\t\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\t\tstrbuf_addstr(&sb, dirname);\n-\t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n-\t\t\tstrbuf_release(&sb);\n-\t\t}\n-\t\tif (nested_repo)\n-\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(excluded ? path_excluded : path_untracked));\n+\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n+\t\t!(dir->flags & DIR_NO_GITLINKS)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tstrbuf_addstr(&sb, dirname);\n+\t\tnested_repo = is_nonbare_repository_dir(&sb);\n+\t\tstrbuf_release(&sb);\n+\t}\n+\tif (nested_repo)\n+\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n+\t\t\t(excluded ? path_excluded : path_untracked));\n \n-\t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n-\t\t\tbreak;\n+\tif (!(dir->flags & DIR_SHOW_OTHER_DIRECTORIES)) {\n \t\tif (excluded &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n-- \ngitgitgadget\n\n"},{"id":"393998","messageId":"b20bc3b9afdc23ad758801f8cfa1e1d997334b60.1585164718.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v3.git.git.1585164718.gitgitgadget@gmail.com","subject":"[PATCH v3 2/7] dir: fix simple typo in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-25T19:31:52Z","receivedAt":"2020-03-25T19:32:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex b460211e614..b505ba747bb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2174,7 +2174,7 @@ static void add_path_to_appropriate_result_list(struct dir_struct *dir,\n * If 'stop_at_first_file' is specified, 'path_excluded' is returned\n * to signal that a file was found. This is the least significant value that\n * indicates that a file was encountered that does not depend on the order of\n- * whether an untracked or exluded path was encountered first.\n+ * whether an untracked or excluded path was encountered first.\n *\n * Returns the most significant path_treatment value encountered in the scan.\n * If 'stop_at_first_file' is specified, `path_excluded` is the most\n-- \ngitgitgadget\n\n"},{"id":"393999","messageId":"d4fe5d335771e89dad40f717bf4623854d1efa9e.1585164718.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v3.git.git.1585164718.gitgitgadget@gmail.com","subject":"[PATCH v3 1/7] t7063: correct broken test expectation","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-25T19:31:51Z","receivedAt":"2020-03-25T19:32:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe untracked cache is caching wrong information, resulting in commands\nlike `git status --porcelain` producing erroneous answers. The tests in\nt7063 actually have a wide enough test to catch a relevant case, in\nparticular surrounding the directory 'dthree/', but it appears the\nanswers were not checked quite closely enough and the tests were coded\nwith the wrong expectation. Once the wrong info got into the cache in\nan early test, since later tests built on it, many others have a wrong\nexpectation as well. This affects just over a third of the tests in\nt7063.\n\nThe error can be seen starting at t7063.12 (the first one switched from\nexpect_success to expect_failure in this patch). That test runs in a\ndirectory with the following files present:\n done/one\n dthree/three\n dtwo/two\n four\n .gitignore\n one\n three\n two\n\nOf those files, the following files are tracked:\n done/one\n one\n two\n\nand the contents of .gitignore are:\n four\n\nand the contents of .git/info/exclude are:\n three\n\nAnd there is no core.excludesfile. Therefore, the following should be\nuntracked:\n .gitignore\n dthree/\n dtwo/\nIndeed, these three paths are reported if you run\n git ls-files -o --directory --exclude-standard\nwithin this directory. However, 'git status --porcelain' was reporting\nfor this test:\n A done/one\n A one\n A two\n ?? .gitignore\n ?? dtwo/\nwhich was clearly wrong -- dthree/ should also be listed as untracked.\nThis appears to have been broken since the test was introduced with\ncommit a3ddcefd97 (\"t7063: tests for untracked cache\", 2015-03-08).\nCorrect the test to expect the right output, marking the test as failed\nfor now. Make the same change throughout the remainder of the testsuite\nto reflect that dthree/ remains an untracked directory throughout and\nshould be recognized as such.\n\nSigned-off-by: Elijah Newren \n---\n t/t7063-status-untracked-cache.sh | 51 ++++++++++++++++++++-----------\n 1 file changed, 33 insertions(+), 18 deletions(-)\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 190ae149cf3..41705ec1526 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -224,7 +224,7 @@ EOF\n \ttest_cmp ../expect ../actual\n '\n \n-test_expect_success 'new info/exclude invalidates everything' '\n+test_expect_failure 'new info/exclude invalidates everything' '\n \tavoid_racy &&\n \techo three >>.git/info/exclude &&\n \t: >../trace &&\n@@ -235,6 +235,7 @@ A done/one\n A one\n A two\n ?? .gitignore\n+?? dthree/\n ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../actual &&\n@@ -247,7 +248,7 @@ EOF\n \ttest_cmp ../trace.expect ../trace\n '\n \n-test_expect_success 'verify untracked cache dump' '\n+test_expect_failure 'verify untracked cache dump' '\n \ttest-tool dump-untracked-cache >../actual &&\n \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n@@ -290,6 +292,7 @@ test_expect_success 'status after the move' '\n A done/one\n A one\n ?? .gitignore\n+?? dthree/\n ?? dtwo/\n ?? two\n EOF\n@@ -303,7 +306,7 @@ EOF\n \ttest_cmp ../trace.expect ../trace\n '\n \n-test_expect_success 'verify untracked cache dump' '\n+test_expect_failure 'verify untracked cache dump' '\n \ttest-tool dump-untracked-cache >../actual &&\n \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n@@ -348,6 +352,7 @@ A done/one\n A one\n A two\n ?? .gitignore\n+?? dthree/\n ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../actual &&\n@@ -360,7 +365,7 @@ EOF\n \ttest_cmp ../trace.expect ../trace\n '\n \n-test_expect_success 'verify untracked cache dump' '\n+test_expect_failure 'verify untracked cache dump' '\n \ttest-tool dump-untracked-cache >../actual &&\n \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n \tcat >../status.expect <../actual &&\n \tcat >../expect <../trace &&\n \tavoid_racy &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n@@ -451,6 +459,7 @@ test_expect_success 'test sparse status with untracked cache' '\n M done/two\n ?? .gitignore\n ?? done/five\n+?? dthree/\n ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../status.actual &&\n@@ -463,7 +472,7 @@ EOF\n \ttest_cmp ../trace.expect ../trace\n '\n \n-test_expect_success 'untracked cache correct after status' '\n+test_expect_failure 'untracked cache correct after status' '\n \ttest-tool dump-untracked-cache >../actual &&\n \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n@@ -491,6 +501,7 @@ test_expect_success 'test sparse status again with untracked cache' '\n M done/two\n ?? .gitignore\n ?? done/five\n+?? dthree/\n ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../status.actual &&\n@@ -509,7 +520,7 @@ test_expect_success 'set up for test of subdir and sparse checkouts' '\n \techo \"sub\" > done/sub/sub/file\n '\n \n-test_expect_success 'test sparse status with untracked cache and subdir' '\n+test_expect_failure 'test sparse status with untracked cache and subdir' '\n \tavoid_racy &&\n \t: >../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n@@ -519,6 +530,7 @@ test_expect_success 'test sparse status with untracked cache and subdir' '\n ?? .gitignore\n ?? done/five\n ?? done/sub/\n+?? dthree/\n ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../status.actual &&\n@@ -531,7 +543,7 @@ EOF\n \ttest_cmp ../trace.expect ../trace\n '\n \n-test_expect_success 'verify untracked cache dump (sparse/subdirs)' '\n+test_expect_failure 'verify untracked cache dump (sparse/subdirs)' '\n \ttest-tool dump-untracked-cache >../actual &&\n \tcat >../expect-from-test-dump <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n@@ -570,7 +583,7 @@ EOF\n \ttest_cmp ../trace.expect ../trace\n '\n \n-test_expect_success 'move entry in subdir from untracked to cached' '\n+test_expect_failure 'move entry in subdir from untracked to cached' '\n \tgit add dtwo/two &&\n \tgit status --porcelain >../status.actual &&\n \tcat >../status.expect <../status.actual &&\n \tcat >../status.expect <../actual &&\n \ttest_cmp ../expect-no-uc ../actual &&\n@@ -642,7 +657,7 @@ test_expect_success 'using --untracked-cache does not fail when core.untrackedCa\n \ttest_cmp ../expect-empty ../actual\n '\n \n-test_expect_success 'setting core.untrackedCache to keep' '\n+test_expect_failure 'setting core.untrackedCache to keep' '\n \tgit config core.untrackedCache keep &&\n \tgit update-index --untracked-cache &&\n \ttest-tool dump-untracked-cache >../actual &&\n-- \ngitgitgadget\n\n"},{"id":"394000","messageId":"02e652d1869b6ed48ab363316d0f676474d57c4a.1585164718.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v3.git.git.1585164718.gitgitgadget@gmail.com","subject":"[PATCH v3 4/7] dir: fix broken comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-25T19:31:54Z","receivedAt":"2020-03-25T19:32:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex d0f3d660850..3a367683661 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2259,7 +2259,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\t/* skip the dir_add_* part */\n+\t\t\t/* skip the add_path_to_appropriate_result_list() */\n \t\t\tcontinue;\n \t\t}\n \n-- \ngitgitgadget\n\n"},{"id":"394002","messageId":"pull.700.v3.git.git.1585164718.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v2.git.git.1580495486.gitgitgadget@gmail.com","subject":"[PATCH v3 0/7] Avoid multiple recursive calls for same path in read_directory_recursive()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-25T19:31:50Z","receivedAt":"2020-03-25T19:32:07Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Sorry for the really long delay on this series; I think I finally figured\nout the untracked cache stuff, though it comes with bad news: existing users\nof untracked cache are dealing with a buggy implementation that may be\ncausing git commands that list untracked files to omit expected paths from\nthe output. This series fixes it, though with a big hammer (partial\ndisabling of the cache; see the final commit in the series).\n\nAlso, as before, this series provides some \"modest\" speedups (see last\ncommit message), and should allow 'git status --ignored' to complete in a\nmore reasonable timeframe for Martin Melka (see \nhttps://lore.kernel.org/git/CANt4O2L_DZnMqVxZzTBMvr=BTWqB6L0uyORkoN_yMHLmUX7yHw@mail.gmail.com/\n).\n\nChanges since v2:\n\n * Added a patch at the beginning which highlights how the untracked cache\n has been broken from the beginning. Using it will result in other\n commands giving erroneous output. At least, before this series it did.\n * Added another patch at the beginning of the series to fix a simple\n comment typo.\n * Dropped the final patch of the previous series, and instead squashed in a\n fix for the untracked cache problems to what is now the final patch of\n the series. I would have liked to have made that a separate commit\n earlier in the series, but the fix depended on both disabling the\n check_only portion of the cache and the avoid-exponential-visiting. If I\n move the partial disabling earlier, nothing is fixed and stuff is still\n visited. If I move the partial disabling later, then I have to\n temporarily mark lots of extra tests with test_expect_failure. It's only\n three extra one-line changes to dir.c, which you can probably spot in the\n range-diff.\n\nStuff still missing from v3:\n\n * I didn't make the DIR_KEEP_UNTRACKED_CONTENTS changes I mentioned in \n https://lore.kernel.org/git/CABPp-BEQ5s=+6Rnb-A+pdEaoPXxfo-hMSegSe1eai=RE74A3Og@mail.gmail.com/ \n which I think would make the code cleaner & clearer. I guess I'm leaving\n that for future work.\n\nAs per the commit message of the final patch, this series has some risk.\nExtra eyes would be greatly appreciated. Also, we should probably merge it\nearly in some cycle, either this one or a later one.\n\nDerrick Stolee (1):\n dir: refactor treat_directory to clarify control flow\n\nElijah Newren (6):\n t7063: correct broken test expectation\n dir: fix simple typo in comment\n dir: consolidate treat_path() and treat_one_path()\n dir: fix broken comment\n dir: fix confusion based on variable tense\n dir: replace exponential algorithm with a linear one, fix untracked\n cache\n\n dir.c | 339 +++++++++++++++++-------------\n t/t7063-status-untracked-cache.sh | 79 +++----\n 2 files changed, 220 insertions(+), 198 deletions(-)\n\n\nbase-commit: 0cbb60574e741e8255ba457606c4c90898cfc755\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-700%2Fnewren%2Ffill-directory-exponential-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-700/newren/fill-directory-exponential-v3\nPull-Request: https://github.com/git/git/pull/700\n\nRange-diff vs v2:\n\n 6: 9a3f20656e2 ! 1: d4fe5d33577 t7063: blindly accept diffs\n @@ -1,17 +1,61 @@\n Author: Elijah Newren \n \n - t7063: blindly accept diffs\n + t7063: correct broken test expectation\n \n - Assuming that the changes I made in the last commit to drastically\n - modify how and when and especially how frequently untracked paths are\n - visited should result in changes to the untracked-cache, this commit\n - simply updates the t7063 testcases to match what the code now reports.\n + The untracked cache is caching wrong information, resulting in commands\n + like `git status --porcelain` producing erroneous answers. The tests in\n + t7063 actually have a wide enough test to catch a relevant case, in\n + particular surrounding the directory 'dthree/', but it appears the\n + answers were not checked quite closely enough and the tests were coded\n + with the wrong expectation. Once the wrong info got into the cache in\n + an early test, since later tests built on it, many others have a wrong\n + expectation as well. This affects just over a third of the tests in\n + t7063.\n \n - If this is correct, this commit should be squashed into the previous\n - one.\n + The error can be seen starting at t7063.12 (the first one switched from\n + expect_success to expect_failure in this patch). That test runs in a\n + directory with the following files present:\n + done/one\n + dthree/three\n + dtwo/two\n + four\n + .gitignore\n + one\n + three\n + two\n \n - It'd be nice if I could get an untracked-cache expert to comment on\n - this...\n + Of those files, the following files are tracked:\n + done/one\n + one\n + two\n +\n + and the contents of .gitignore are:\n + four\n +\n + and the contents of .git/info/exclude are:\n + three\n +\n + And there is no core.excludesfile. Therefore, the following should be\n + untracked:\n + .gitignore\n + dthree/\n + dtwo/\n + Indeed, these three paths are reported if you run\n + git ls-files -o --directory --exclude-standard\n + within this directory. However, 'git status --porcelain' was reporting\n + for this test:\n + A done/one\n + A one\n + A two\n + ?? .gitignore\n + ?? dtwo/\n + which was clearly wrong -- dthree/ should also be listed as untracked.\n + This appears to have been broken since the test was introduced with\n + commit a3ddcefd97 (\"t7063: tests for untracked cache\", 2015-03-08).\n + Correct the test to expect the right output, marking the test as failed\n + for now. Make the same change throughout the remainder of the testsuite\n + to reflect that dthree/ remains an untracked directory throughout and\n + should be recognized as such.\n \n Signed-off-by: Elijah Newren \n \n @@ -19,50 +63,14 @@\n --- a/t/t7063-status-untracked-cache.sh\n +++ b/t/t7063-status-untracked-cache.sh\n @@\n - three\n - /done/ 0000000000000000000000000000000000000000 recurse valid\n - /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n --three\n - /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n --two\n - EOF\n - \n - test_expect_success 'status first time (empty cache)' '\n -@@\n - A done/one\n - A one\n - A two\n --?? dthree/\n --?? dtwo/\n - ?? four\n - ?? three\n - EOF\n -@@\n - exclude_per_dir .gitignore\n - flags 00000006\n - / 0000000000000000000000000000000000000000 recurse valid\n --dthree/\n --dtwo/\n - four\n - three\n - /done/ 0000000000000000000000000000000000000000 recurse valid\n - /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n --three\n - /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n --two\n - EOF\n - \ttest_cmp ../expect ../actual\n - '\n -@@\n - three\n - /done/ 0000000000000000000000000000000000000000 recurse valid\n - /dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n --three\n - /dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n --two\n - EOF\n \ttest_cmp ../expect ../actual\n '\n + \n +-test_expect_success 'new info/exclude invalidates everything' '\n ++test_expect_failure 'new info/exclude invalidates everything' '\n + \tavoid_racy &&\n + \techo three >>.git/info/exclude &&\n + \t: >../trace &&\n @@\n A one\n A two\n @@ -71,6 +79,15 @@\n ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../actual &&\n +@@\n + \ttest_cmp ../trace.expect ../trace\n + '\n + \n +-test_expect_success 'verify untracked cache dump' '\n ++test_expect_failure 'verify untracked cache dump' '\n + \ttest-tool dump-untracked-cache >../actual &&\n + \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n @@\n A done/one\n A one\n ?? .gitignore\n --?? dtwo/\n ++?? dthree/\n + ?? dtwo/\n ?? two\n EOF\n - \ttest_cmp ../status.expect ../actual &&\n +@@\n + \ttest_cmp ../trace.expect ../trace\n + '\n + \n +-test_expect_success 'verify untracked cache dump' '\n ++test_expect_failure 'verify untracked cache dump' '\n + \ttest-tool dump-untracked-cache >../actual &&\n + \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n @@\n A one\n A two\n ?? .gitignore\n --?? dtwo/\n ++?? dthree/\n + ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../actual &&\n - \tcat >../trace.expect <../actual &&\n + \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n \tcat >../status.expect <../trace.expect <../actual &&\n + \tcat >../expect <../trace &&\n + \tavoid_racy &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n @@\n M done/two\n ?? .gitignore\n ?? done/five\n --?? dtwo/\n ++?? dthree/\n + ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../status.actual &&\n - \tcat >../trace.expect <../actual &&\n + \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n @@\n M done/two\n ?? .gitignore\n ?? done/five\n --?? dtwo/\n ++?? dthree/\n + ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../status.actual &&\n - \tcat >../trace.expect < done/sub/sub/file\n + '\n + \n +-test_expect_success 'test sparse status with untracked cache and subdir' '\n ++test_expect_failure 'test sparse status with untracked cache and subdir' '\n + \tavoid_racy &&\n + \t: >../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n @@\n ?? .gitignore\n ?? done/five\n ?? done/sub/\n --?? dtwo/\n ++?? dthree/\n + ?? dtwo/\n EOF\n \ttest_cmp ../status.expect ../status.actual &&\n - \tcat >../trace.expect <../actual &&\n + \tcat >../expect-from-test-dump <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n @@\n - \ttest_cmp ../expect-no-uc ../actual &&\n - \tgit status &&\n + \ttest_cmp ../trace.expect ../trace\n + '\n + \n +-test_expect_success 'move entry in subdir from untracked to cached' '\n ++test_expect_failure 'move entry in subdir from untracked to cached' '\n + \tgit add dtwo/two &&\n + \tgit status --porcelain >../status.actual &&\n + \tcat >../status.expect <../status.actual &&\n + \tcat >../status.expect <../actual &&\n -+\tcat >../expect-from-test-dump <../actual &&\n -: ----------- > 2: b20bc3b9afd dir: fix simple typo in comment\n 1: 27bc1357964 = 3: fa9035949e0 dir: consolidate treat_path() and treat_one_path()\n 2: 2ceb64ae61e = 4: 02e652d1869 dir: fix broken comment\n 3: e6d21228d12 = 5: 705c008d993 dir: fix confusion based on variable tense\n 4: f73f0d66d14 = 6: f5d69102946 dir: refactor treat_directory to clarify control flow\n 5: d3136ef52f3 ! 7: 6cfca619e2c dir: replace exponential algorithm with a linear one\n @@ -1,6 +1,6 @@\n Author: Elijah Newren \n \n - dir: replace exponential algorithm with a linear one\n + dir: replace exponential algorithm with a linear one, fix untracked cache\n \n dir's read_directory_recursive() naturally operates recursively in order\n to walk the directory tree. Treating of directories is sometimes weird\n @@ -161,10 +161,27 @@\n to have completed the 240 nested directories case. It's not often\n that you get to speed something up by a factor of 3*10^69.\n \n - WARNING: This change breaks t7063. I don't know whether that is to be expected\n - (I now intentionally visit untracked directories differently so naturally the\n - untracked cache should change), or if I've broken something. I'm hoping to get\n - an untracked cache expert to chime in...\n + Finally, this also fixes the untracked cache, as noted by the test fixes\n + in t7063. Unfortunately, it does so by passing stop_at_first_file to\n + close_cached_dir() in order to disable the caching of whether\n + directories were empty (this caching was only relevant for directories\n + that we knew we didn't need to walk all the entries under but just\n + needed to know whether the directory had any entries within it in order\n + to know if the directory itself should be marked as path_none or\n + path_untracked). I'm not convinced that disabling the is-the-dir-empty\n + check is necessary; there is probably some way to still cache that and\n + not get erroneous results. However, I have not figured out how to do\n + so. If I revert the change to close_cached_dir() in this patch (thus\n + continuing to cache cases where stop_at_first_file is true meaning we\n + continue to cache whether directories are empty), then the untracked\n + cache breakage in t7063 becomes more prevalant. With my change to\n + close_cached_dir() and the other changes to avoid traversing directories\n + 2^n times in this patch, I not only avoid making the untracked_cache\n + breakage in t7063 worse but actually fix the existing breakage. Update\n + the test results in t7063 to no longer expect check_only cache entries,\n + to reflect that we have to do a bit more work in terms of how many\n + directories we have to open, and to reflect that we fixed the 1/3 of\n + tests that were broken in that testsuite.\n \n Signed-off-by: Elijah Newren \n \n @@ -305,6 +322,24 @@\n \tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n \t\tstrbuf_addstr(path, cdir->file);\n +@@\n + \treturn -1;\n + }\n + \n +-static void close_cached_dir(struct cached_dir *cdir)\n ++static void close_cached_dir(struct cached_dir *cdir, int stop_at_first_file)\n + {\n + \tif (cdir->fdir)\n + \t\tclosedir(cdir->fdir);\n +@@\n + \t * We have gone through this directory and found no untracked\n + \t * entries. Mark it valid.\n + \t */\n +-\tif (cdir->untracked) {\n ++\tif (!stop_at_first_file && cdir->untracked) {\n + \t\tcdir->untracked->valid = 1;\n + \t\tcdir->untracked->recurse = 1;\n + \t}\n @@\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n @@ -338,6 +373,15 @@\n \t\t\tstruct untracked_cache_dir *ud;\n \t\t\tud = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t\t path.buf + baselen,\n +@@\n + \t\t\t\t\t\t istate, &path, baselen,\n + \t\t\t\t\t\t pathspec, state);\n + \t}\n +-\tclose_cached_dir(&cdir);\n ++\tclose_cached_dir(&cdir, stop_at_first_file);\n + out:\n + \tstrbuf_release(&path);\n + \n @@\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n @@ -379,3 +423,342 @@\n \n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n +\n + diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\n + --- a/t/t7063-status-untracked-cache.sh\n + +++ b/t/t7063-status-untracked-cache.sh\n +@@\n + dtwo/\n + three\n + /done/ 0000000000000000000000000000000000000000 recurse valid\n +-/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n +-three\n +-/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n +-two\n + EOF\n + \n + test_expect_success 'status first time (empty cache)' '\n +@@\n + EOF\n + \ttest_cmp ../status.expect ../actual &&\n + \tcat >../trace.expect <../trace.expect <>.git/info/exclude &&\n + \t: >../trace &&\n +@@\n + EOF\n + \ttest_cmp ../status.expect ../actual &&\n + \tcat >../trace.expect <../actual &&\n + \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n +@@\n + EOF\n + \ttest_cmp ../status.expect ../actual &&\n + \tcat >../trace.expect <../actual &&\n + \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n +@@\n + EOF\n + \ttest_cmp ../status.expect ../actual &&\n + \tcat >../trace.expect <../actual &&\n + \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n +@@\n + EOF\n + \ttest_cmp ../status.expect ../actual &&\n + \tcat >../trace.expect <../actual &&\n + \tcat >../expect <../trace &&\n + \tavoid_racy &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n +@@\n + EOF\n + \ttest_cmp ../status.expect ../status.actual &&\n + \tcat >../trace.expect <../actual &&\n + \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n +@@\n + \techo \"sub\" > done/sub/sub/file\n + '\n + \n +-test_expect_failure 'test sparse status with untracked cache and subdir' '\n ++test_expect_success 'test sparse status with untracked cache and subdir' '\n + \tavoid_racy &&\n + \t: >../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n +@@\n + \ttest_cmp ../trace.expect ../trace\n + '\n + \n +-test_expect_failure 'verify untracked cache dump (sparse/subdirs)' '\n ++test_expect_success 'verify untracked cache dump (sparse/subdirs)' '\n + \ttest-tool dump-untracked-cache >../actual &&\n + \tcat >../expect-from-test-dump <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n +@@\n + \ttest_cmp ../trace.expect ../trace\n + '\n + \n +-test_expect_failure 'move entry in subdir from untracked to cached' '\n ++test_expect_success 'move entry in subdir from untracked to cached' '\n + \tgit add dtwo/two &&\n + \tgit status --porcelain >../status.actual &&\n + \tcat >../status.expect <../status.actual &&\n + \tcat >../status.expect <../actual &&\n + \ttest_cmp ../expect-no-uc ../actual &&\n +@@\n + \ttest_cmp ../expect-empty ../actual\n + '\n + \n +-test_expect_failure 'setting core.untrackedCache to keep' '\n ++test_expect_success 'setting core.untrackedCache to keep' '\n + \tgit config core.untrackedCache keep &&\n + \tgit update-index --untracked-cache &&\n + \ttest-tool dump-untracked-cache >../actual &&\n\n-- \ngitgitgadget\n"},{"id":"394001","messageId":"705c008d9934cb7f158ba6fe78eb0e773dcad537.1585164718.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v3.git.git.1585164718.gitgitgadget@gmail.com","subject":"[PATCH v3 5/7] dir: fix confusion based on variable tense","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-25T19:31:55Z","receivedAt":"2020-03-25T19:32:08Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nDespite having contributed several fixes in this area, I have for months\n(years?) assumed that the \"exclude\" variable was a directive; this\ncaused me to think of it as a different mode we operate in and left me\nconfused as I tried to build up a mental model around why we'd need such\na directive. I mostly tried to ignore it while focusing on the pieces I\nwas trying to understand.\n\nThen I finally traced this variable all back to a call to is_excluded(),\nmeaning it was actually functioning as an adjective. In particular, it\nwas a checked property (\"Does this path match a rule in .gitignore?\"),\nrather than a mode passed in from the caller. Change the variable name\nto match the part of speech used by the function called to define it,\nwhich will hopefully make these bits of code slightly clearer to the\nnext reader.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 26 +++++++++++++-------------\n 1 file changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3a367683661..8074e651e6f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1656,7 +1656,7 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n static enum path_treatment treat_directory(struct dir_struct *dir,\n \tstruct index_state *istate,\n \tstruct untracked_cache_dir *untracked,\n-\tconst char *dirname, int len, int baselen, int exclude,\n+\tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n@@ -1679,13 +1679,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\t}\n \t\tif (nested_repo)\n \t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(exclude ? path_excluded : path_untracked));\n+\t\t\t\t(excluded ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n \t\t\tbreak;\n-\t\tif (exclude &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n+\t\tif (excluded &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n \n \t\t\t/*\n \t\t\t * This is an excluded directory and we are\n@@ -1713,7 +1713,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* This is the \"show_other_directories\" case */\n \n \tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n@@ -1723,7 +1723,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t * the directory contains any files.\n \t */\n \treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, exclude, pathspec);\n+\t\t\t\t\tuntracked, 1, excluded, pathspec);\n }\n \n /*\n@@ -1904,7 +1904,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t int baselen,\n \t\t\t\t const struct pathspec *pathspec)\n {\n-\tint has_path_in_index, dtype, exclude;\n+\tint has_path_in_index, dtype, excluded;\n \tenum path_treatment path_treatment;\n \n \tif (!cdir->d_name)\n@@ -1949,13 +1949,13 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t (directory_exists_in_index(istate, path->buf, path->len) == index_nonexistent))\n \t\treturn path_none;\n \n-\texclude = is_excluded(dir, istate, path->buf, &dtype);\n+\texcluded = is_excluded(dir, istate, path->buf, &dtype);\n \n \t/*\n \t * Excluded? If we don't explicitly want to show\n \t * ignored files, ignore it\n \t */\n-\tif (exclude && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n+\tif (excluded && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n \t\treturn path_excluded;\n \n \tswitch (dtype) {\n@@ -1965,7 +1965,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\tstrbuf_addch(path, '/');\n \t\tpath_treatment = treat_directory(dir, istate, untracked,\n \t\t\t\t\t\t path->buf, path->len,\n-\t\t\t\t\t\t baselen, exclude, pathspec);\n+\t\t\t\t\t\t baselen, excluded, pathspec);\n \t\t/*\n \t\t * If 1) we only want to return directories that\n \t\t * match an exclude pattern and 2) this directory does\n@@ -1974,7 +1974,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t * recurse into this directory (instead of marking the\n \t\t * directory itself as an ignored path).\n \t\t */\n-\t\tif (!exclude &&\n+\t\tif (!excluded &&\n \t\t path_treatment == path_excluded &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n@@ -1982,7 +1982,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\treturn path_treatment;\n \tcase DT_REG:\n \tcase DT_LNK:\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"394003","messageId":"f5d69102946e65506406a06a2206aefe05007275.1585164718.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v3.git.git.1585164718.gitgitgadget@gmail.com","subject":"[PATCH v3 6/7] dir: refactor treat_directory to clarify control flow","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-25T19:31:56Z","receivedAt":"2020-03-25T19:32:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe logic in treat_directory() is handled by a multi-case\nswitch statement, but this switch is very asymmetrical, as\nthe first two cases are simple but the third is more\ncomplicated than the rest of the method. In fact, the third\ncase includes a \"break\" statement that leads to the block\nof code outside the switch statement. That is the only way\nto reach that block, as the switch handles all possible\nvalues from directory_exists_in_index();\n\nExtract the switch statement into a series of \"if\" statements.\nThis simplifies the trivial cases, while clarifying how to\nreach the \"show_other_directories\" case. This is particularly\nimportant as the \"show_other_directories\" case will expand\nin a later change.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n dir.c | 35 +++++++++++++++++------------------\n 1 file changed, 17 insertions(+), 18 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 8074e651e6f..d9bcb7e19b6 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1660,29 +1660,28 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n-\n \t/* The \"len-1\" is to strip the final '/' */\n-\tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n-\tcase index_directory:\n-\t\treturn path_recurse;\n+\tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n-\tcase index_gitdir:\n+\tif (status == index_directory)\n+\t\treturn path_recurse;\n+\tif (status == index_gitdir)\n \t\treturn path_none;\n+\tif (status != index_nonexistent)\n+\t\tBUG(\"Unhandled value for directory_exists_in_index: %d\\n\", status);\n \n-\tcase index_nonexistent:\n-\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n-\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n-\t\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\t\tstrbuf_addstr(&sb, dirname);\n-\t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n-\t\t\tstrbuf_release(&sb);\n-\t\t}\n-\t\tif (nested_repo)\n-\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(excluded ? path_excluded : path_untracked));\n+\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n+\t\t!(dir->flags & DIR_NO_GITLINKS)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tstrbuf_addstr(&sb, dirname);\n+\t\tnested_repo = is_nonbare_repository_dir(&sb);\n+\t\tstrbuf_release(&sb);\n+\t}\n+\tif (nested_repo)\n+\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n+\t\t\t(excluded ? path_excluded : path_untracked));\n \n-\t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n-\t\t\tbreak;\n+\tif (!(dir->flags & DIR_SHOW_OTHER_DIRECTORIES)) {\n \t\tif (excluded &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n-- \ngitgitgadget\n\n"},{"id":"394004","messageId":"fa9035949e002d4ce0a489a0c46a4eb7ba81fe16.1585164718.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v3.git.git.1585164718.gitgitgadget@gmail.com","subject":"[PATCH v3 3/7] dir: consolidate treat_path() and treat_one_path()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-25T19:31:53Z","receivedAt":"2020-03-25T19:32:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCommit 16e2cfa90993 (\"read_directory(): further split treat_path()\",\n2010-01-08) split treat_one_path() out of treat_path(), because\ntreat_leading_path() would not have access to a dirent but wanted to\nre-use as much of treat_path() as possible. Not re-using all of\ntreat_path() caused other bugs, as noted in commit b9670c1f5e6b (\"dir:\nfix checks on common prefix directory\", 2019-12-19). Finally, in commit\nad6f2157f951 (\"dir: restructure in a way to avoid passing around a\nstruct dirent\", 2020-01-16), dirents were removed from treat_path() and\nother functions entirely.\n\nSince the only reason for splitting these functions was the lack of a\ndirent -- which no longer applies to either function -- and since the\nsplit caused problems in the past resulting in us not using\ntreat_one_path() separately anymore, just undo the split.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 121 ++++++++++++++++++++++++++--------------------------------\n 1 file changed, 55 insertions(+), 66 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex b505ba747bb..d0f3d660850 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1863,21 +1863,65 @@ static int resolve_dtype(int dtype, struct index_state *istate,\n \treturn dtype;\n }\n \n-static enum path_treatment treat_one_path(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec,\n-\t\t\t\t\t int dtype)\n-{\n-\tint exclude;\n-\tint has_path_in_index = !!index_file_exists(istate, path->buf, path->len, ignore_case);\n+static enum path_treatment treat_path_fast(struct dir_struct *dir,\n+\t\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t\t struct index_state *istate,\n+\t\t\t\t\t struct strbuf *path,\n+\t\t\t\t\t int baselen,\n+\t\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tstrbuf_setlen(path, baselen);\n+\tif (!cdir->ucd) {\n+\t\tstrbuf_addstr(path, cdir->file);\n+\t\treturn path_untracked;\n+\t}\n+\tstrbuf_addstr(path, cdir->ucd->name);\n+\t/* treat_one_path() does this before it calls treat_directory() */\n+\tstrbuf_complete(path, '/');\n+\tif (cdir->ucd->check_only)\n+\t\t/*\n+\t\t * check_only is set as a result of treat_directory() getting\n+\t\t * to its bottom. Verify again the same set of directories\n+\t\t * with check_only set.\n+\t\t */\n+\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n+\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n+\t/*\n+\t * We get path_recurse in the first run when\n+\t * directory_exists_in_index() returns index_nonexistent. We\n+\t * are sure that new changes in the index does not impact the\n+\t * outcome. Return now.\n+\t */\n+\treturn path_recurse;\n+}\n+\n+static enum path_treatment treat_path(struct dir_struct *dir,\n+\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t struct index_state *istate,\n+\t\t\t\t struct strbuf *path,\n+\t\t\t\t int baselen,\n+\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tint has_path_in_index, dtype, exclude;\n \tenum path_treatment path_treatment;\n \n-\tdtype = resolve_dtype(dtype, istate, path->buf, path->len);\n+\tif (!cdir->d_name)\n+\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n+\t\t\t\t baselen, pathspec);\n+\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n+\t\treturn path_none;\n+\tstrbuf_setlen(path, baselen);\n+\tstrbuf_addstr(path, cdir->d_name);\n+\tif (simplify_away(path->buf, path->len, pathspec))\n+\t\treturn path_none;\n+\n+\tdtype = resolve_dtype(cdir->d_type, istate, path->buf, path->len);\n \n \t/* Always exclude indexed files */\n+\thas_path_in_index = !!index_file_exists(istate, path->buf, path->len,\n+\t\t\t\t\t\tignore_case);\n \tif (dtype != DT_DIR && has_path_in_index)\n \t\treturn path_none;\n \n@@ -1942,61 +1986,6 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t}\n }\n \n-static enum path_treatment treat_path_fast(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tstrbuf_setlen(path, baselen);\n-\tif (!cdir->ucd) {\n-\t\tstrbuf_addstr(path, cdir->file);\n-\t\treturn path_untracked;\n-\t}\n-\tstrbuf_addstr(path, cdir->ucd->name);\n-\t/* treat_one_path() does this before it calls treat_directory() */\n-\tstrbuf_complete(path, '/');\n-\tif (cdir->ucd->check_only)\n-\t\t/*\n-\t\t * check_only is set as a result of treat_directory() getting\n-\t\t * to its bottom. Verify again the same set of directories\n-\t\t * with check_only set.\n-\t\t */\n-\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n-\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n-\t/*\n-\t * We get path_recurse in the first run when\n-\t * directory_exists_in_index() returns index_nonexistent. We\n-\t * are sure that new changes in the index does not impact the\n-\t * outcome. Return now.\n-\t */\n-\treturn path_recurse;\n-}\n-\n-static enum path_treatment treat_path(struct dir_struct *dir,\n-\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t struct index_state *istate,\n-\t\t\t\t struct strbuf *path,\n-\t\t\t\t int baselen,\n-\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tif (!cdir->d_name)\n-\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n-\t\t\t\t baselen, pathspec);\n-\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n-\t\treturn path_none;\n-\tstrbuf_setlen(path, baselen);\n-\tstrbuf_addstr(path, cdir->d_name);\n-\tif (simplify_away(path->buf, path->len, pathspec))\n-\t\treturn path_none;\n-\n-\treturn treat_one_path(dir, untracked, istate, path, baselen, pathspec,\n-\t\t\t cdir->d_type);\n-}\n-\n static void add_untracked(struct untracked_cache_dir *dir, const char *name)\n {\n \tif (!dir)\n-- \ngitgitgadget\n\n"},{"id":"394005","messageId":"6cfca619e2c5bc37d1847d49496d1be42f4061ce.1585164718.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v3.git.git.1585164718.gitgitgadget@gmail.com","subject":"[PATCH v3 7/7] dir: replace exponential algorithm with a linear one, fix untracked cache","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-25T19:31:57Z","receivedAt":"2020-03-25T19:32:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ndir's read_directory_recursive() naturally operates recursively in order\nto walk the directory tree. Treating of directories is sometimes weird\nbecause there are so many different permutations about how to handle\ndirectories. Some examples:\n\n * 'git ls-files -o --directory' only needs to know that a directory\n itself is untracked; it doesn't need to recurse into it to see what\n is underneath.\n\n * 'git status' needs to recurse into an untracked directory, but only\n to determine whether or not it is empty. If there are no files\n underneath, the directory itself will be omitted from the output.\n If it is not empty, only the directory will be listed.\n\n * 'git status --ignored' needs to recurse into untracked directories\n and report all the ignored entries and then report the directory as\n untracked -- UNLESS all the entries under the directory are\n ignored, in which case we don't print any of the entries under the\n directory and just report the directory itself as ignored. (Note\n that although this forces us to walk all untracked files underneath\n the directory as well, we strip them from the output, except for\n users like 'git clean' who also set DIR_KEEP_TRACKED_CONTENTS.)\n\n * For 'git clean', we may need to recurse into a directory that\n doesn't match any specified pathspecs, if it's possible that there\n is an entry underneath the directory that can match one of the\n pathspecs. In such a case, we need to be careful to omit the\n directory itself from the list of paths (see commit 404ebceda01c\n (\"dir: also check directories for matching pathspecs\", 2019-09-17))\n\nPart of the tension noted above is that the treatment of a directory can\nchange based on the files within it, and based on the various settings\nin dir->flags. Trying to keep this in mind while reading over the code,\nit is easy to think in terms of \"treat_directory() tells us what to do\nwith a directory, and read_directory_recursive() is the thing that\nrecurses\". Since we need to look into a directory to know how to treat\nit, though, it is quite easy to decide to (also) recurse into the\ndirectory from treat_directory() by adding a read_directory_recursive()\ncall. Adding such a call is actually fine, IF we make sure that\nread_directory_recursive() does not also recurse into that same\ndirectory.\n\nUnfortunately, commit df5bcdf83aeb (\"dir: recurse into untracked dirs\nfor ignored files\", 2017-05-18), added exactly such a case to the code,\nmeaning we'd have two calls to read_directory_recursive() for an\nuntracked directory. So, if we had a file named\n one/two/three/four/five/somefile.txt\nand nothing in one/ was tracked, then 'git status --ignored' would\ncall read_directory_recursive() twice on the directory 'one/', and\neach of those would call read_directory_recursive() twice on the\ndirectory 'one/two/', and so on until read_directory_recursive() was\ncalled 2^5 times for 'one/two/three/four/five/'.\n\nAvoid calling read_directory_recursive() twice per level by moving a\nlot of the special logic into treat_directory().\n\nSince dir.c is somewhat complex, extra cruft built up around this over\ntime. While trying to unravel it, I noticed several instances where the\nfirst call to read_directory_recursive() would return e.g.\npath_untracked for some directory and a later one would return e.g.\npath_none, despite the fact that the directory clearly should have been\nconsidered untracked. The code happened to work due to the side-effect\nfrom the first invocation of adding untracked entries to dir->entries;\nthis allowed it to get the correct output despite the supposed override\nin return value by the later call.\n\nI am somewhat concerned that there are still bugs and maybe even\ntestcases with the wrong expectation. I have tried to carefully\ndocument treat_directory() since it becomes more complex after this\nchange (though much of this complexity came from elsewhere that probably\ndeserved better comments to begin with). However, much of my work felt\nmore like a game of whackamole while attempting to make the code match\nthe existing regression tests than an attempt to create an\nimplementation that matched some clear design. That seems wrong to me,\nbut the rules of existing behavior had so many special cases that I had\na hard time coming up with some overarching rules about what correct\nbehavior is for all cases, forcing me to hope that the regression tests\nare correct and sufficient. Such a hope seems likely to be ill-founded,\ngiven my experience with dir.c-related testcases in the last few months:\n\n Examples where the documentation was hard to parse or even just wrong:\n * 3aca58045f4f (git-clean.txt: do not claim we will delete files with\n -n/--dry-run, 2019-09-17)\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n Examples where testcases were declared wrong and changed:\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n * a2b13367fe55 (Revert \"dir.c: make 'git-status --ignored' work within\n leading directories\", 2019-12-10)\n Examples where testcases were clearly inadequate:\n * 502c386ff944 (t7300-clean: demonstrate deleting nested repo with an\n ignored file breakage, 2019-08-25)\n * 7541cc530239 (t7300: add testcases showing failure to clean specified\n pathspecs, 2019-09-17)\n * a5e916c7453b (dir: fix off-by-one error in match_pathspec_item,\n 2019-09-17)\n * 404ebceda01c (dir: also check directories for matching pathspecs,\n 2019-09-17)\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n * 452efd11fbf6 (t3011: demonstrate directory traversal failures,\n 2019-12-10)\n * b9670c1f5e6b (dir: fix checks on common prefix directory, 2019-12-19)\n Examples where \"correct behavior\" was unclear to everyone:\n https://lore.kernel.org/git/20190905154735.29784-1-newren@gmail.com/\n Other commits of note:\n * 902b90cf42bc (clean: fix theoretical path corruption, 2019-09-17)\n\nHowever, on the positive side, it does make the code much faster. For\nthe following simple shell loop in an empty repository:\n\n for depth in $(seq 10 25)\n do\n dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n rm -rf dir\n mkdir -p $dirs\n >$dirs/untracked-file\n /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n done\n\nI saw the following timings, in seconds (note that the numbers are a\nlittle noisy from run-to-run, but the trend is very clear with every\nrun):\n\n 10: 0.03\n 11: 0.05\n 12: 0.08\n 13: 0.19\n 14: 0.29\n 15: 0.50\n 16: 1.05\n 17: 2.11\n 18: 4.11\n 19: 8.60\n 20: 17.55\n 21: 33.87\n 22: 68.71\n 23: 140.05\n 24: 274.45\n 25: 551.15\n\nFor the above run, using strace I can look for the number of untracked\ndirectories opened and can verify that it matches the expected\n2^($depth+1)-2 (the sum of 2^1 + 2^2 + 2^3 + ... + 2^$depth).\n\nAfter this fix, with strace I can verify that the number of untracked\ndirectories that are opened drops to just $depth, and the timings all\ndrop to 0.00. In fact, it isn't until a depth of 190 nested directories\nthat it sometimes starts reporting a time of 0.01 seconds and doesn't\nconsistently report 0.01 seconds until there are 240 nested directories.\nThe previous code would have taken\n 17.55 * 2^220 / (60*60*24*365) = 9.4 * 10^59 YEARS\nto have completed the 240 nested directories case. It's not often\nthat you get to speed something up by a factor of 3*10^69.\n\nFinally, this also fixes the untracked cache, as noted by the test fixes\nin t7063. Unfortunately, it does so by passing stop_at_first_file to\nclose_cached_dir() in order to disable the caching of whether\ndirectories were empty (this caching was only relevant for directories\nthat we knew we didn't need to walk all the entries under but just\nneeded to know whether the directory had any entries within it in order\nto know if the directory itself should be marked as path_none or\npath_untracked). I'm not convinced that disabling the is-the-dir-empty\ncheck is necessary; there is probably some way to still cache that and\nnot get erroneous results. However, I have not figured out how to do\nso. If I revert the change to close_cached_dir() in this patch (thus\ncontinuing to cache cases where stop_at_first_file is true meaning we\ncontinue to cache whether directories are empty), then the untracked\ncache breakage in t7063 becomes more prevalant. With my change to\nclose_cached_dir() and the other changes to avoid traversing directories\n2^n times in this patch, I not only avoid making the untracked_cache\nbreakage in t7063 worse but actually fix the existing breakage. Update\nthe test results in t7063 to no longer expect check_only cache entries,\nto reflect that we have to do a bit more work in terms of how many\ndirectories we have to open, and to reflect that we fixed the 1/3 of\ntests that were broken in that testsuite.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 157 ++++++++++++++++++++----------\n t/t7063-status-untracked-cache.sh | 100 ++++++-------------\n 2 files changed, 138 insertions(+), 119 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex d9bcb7e19b6..803e2851964 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1659,7 +1659,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n-\tint nested_repo = 0;\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n+\tenum path_treatment state;\n+\tint nested_repo = 0, old_ignored_nr, stop_early;\n \t/* The \"len-1\" is to strip the final '/' */\n \tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n@@ -1711,18 +1717,101 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/* This is the \"show_other_directories\" case */\n \n-\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t/*\n+\t * We only need to recurse into untracked/ignored directories if\n+\t * either of the following bits is set:\n+\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n+\t * there are ignored directories below)\n+\t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n+\t * the directory is empty)\n+\t */\n+\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n \t\treturn excluded ? path_excluded : path_untracked;\n \n+\t/*\n+\t * If we only want to determine if dirname is empty, then we can\n+\t * stop at the first file we find underneath that directory rather\n+\t * than continuing to recurse beyond it. If DIR_SHOW_IGNORED_TOO\n+\t * is set, then we want MORE than just determining if dirname is\n+\t * empty.\n+\t */\n+\tstop_early = ((dir->flags & DIR_HIDE_EMPTY_DIRECTORIES) &&\n+\t\t !(dir->flags & DIR_SHOW_IGNORED_TOO));\n+\n+\t/*\n+\t * If /every/ file within an untracked directory is ignored, then\n+\t * we want to treat the directory as ignored (for e.g. status\n+\t * --porcelain), without listing the individual ignored files\n+\t * underneath. To do so, we'll save the current ignored_nr, and\n+\t * pop all the ones added after it if it turns out the entire\n+\t * directory is ignored.\n+\t */\n+\told_ignored_nr = dir->ignored_nr;\n+\n+\t/* Actually recurse into dirname now, we'll fixup the state later. */\n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n+\tstate = read_directory_recursive(dir, istate, dirname, len, untracked,\n+\t\t\t\t\t stop_early, stop_early, pathspec);\n+\n+\t/* There are a variety of reasons we may need to fixup the state... */\n+\tif (state == path_excluded) {\n+\t\tint i;\n+\n+\t\t/*\n+\t\t * When stop_early is set, read_directory_recursive() will\n+\t\t * never return path_untracked regardless of whether\n+\t\t * underlying paths were untracked or ignored (because\n+\t\t * returning early means it excluded some paths, or\n+\t\t * something like that -- see commit 5aaa7fd39aaf (\"Improve\n+\t\t * performance of git status --ignored\", 2017-09-18)).\n+\t\t * However, we're not really concerned with the status of\n+\t\t * files under the directory, we just wanted to know\n+\t\t * whether the directory was empty (state == path_none) or\n+\t\t * not (state == path_excluded), and if not, we'd return\n+\t\t * our original status based on whether the untracked\n+\t\t * directory matched an exclusion pattern.\n+\t\t */\n+\t\tif (stop_early)\n+\t\t\tstate = excluded ? path_excluded : path_untracked;\n+\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * When\n+\t\t\t * !stop_early && state == path_excluded\n+\t\t\t * then all paths under dirname were ignored. For\n+\t\t\t * this case, git status --porcelain wants to just\n+\t\t\t * list the directory itself as ignored and not\n+\t\t\t * list the individual paths underneath. Remove\n+\t\t\t * the individual paths underneath.\n+\t\t\t */\n+\t\t\tfor (i = old_ignored_nr + 1; iignored_nr; ++i)\n+\t\t\t\tfree(dir->ignored[i]);\n+\t\t\tdir->ignored_nr = old_ignored_nr;\n+\t\t}\n+\t}\n \n \t/*\n-\t * If this is an excluded directory, then we only need to check if\n-\t * the directory contains any files.\n+\t * If there is nothing under the current directory and we are not\n+\t * hiding empty directories, then we need to report on the\n+\t * untracked or ignored status of the directory itself.\n \t */\n-\treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, excluded, pathspec);\n+\tif (state == path_none && !(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\tstate = excluded ? path_excluded : path_untracked;\n+\n+\t/*\n+\t * We can recurse into untracked directories that don't match any\n+\t * of the given pathspecs when some file underneath the directory\n+\t * might match one of the pathspecs. If so, we should make sure\n+\t * to note that the directory itself did not match.\n+\t */\n+\tif (pathspec &&\n+\t !match_pathspec(istate, pathspec, dirname, len,\n+\t\t\t 0 /* prefix */, NULL,\n+\t\t\t 0 /* do NOT special case dirs */))\n+\t\tstate = path_none;\n+\n+\treturn state;\n }\n \n /*\n@@ -1870,6 +1959,11 @@ static enum path_treatment treat_path_fast(struct dir_struct *dir,\n \t\t\t\t\t int baselen,\n \t\t\t\t\t const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n \tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n \t\tstrbuf_addstr(path, cdir->file);\n@@ -2102,7 +2196,7 @@ static int read_cached_dir(struct cached_dir *cdir)\n \treturn -1;\n }\n \n-static void close_cached_dir(struct cached_dir *cdir)\n+static void close_cached_dir(struct cached_dir *cdir, int stop_at_first_file)\n {\n \tif (cdir->fdir)\n \t\tclosedir(cdir->fdir);\n@@ -2110,7 +2204,7 @@ static void close_cached_dir(struct cached_dir *cdir)\n \t * We have gone through this directory and found no untracked\n \t * entries. Mark it valid.\n \t */\n-\tif (cdir->untracked) {\n+\tif (!stop_at_first_file && cdir->untracked) {\n \t\tcdir->untracked->valid = 1;\n \t\tcdir->untracked->recurse = 1;\n \t}\n@@ -2175,14 +2269,10 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n \t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in treat_leading_path(). See the commit message for the\n-\t * commit adding this warning as well as the commit preceding it\n-\t * for details.\n+\t * WARNING: Do NOT recurse unless path_recurse is returned from\n+\t * treat_path(). Recursing on any other return value\n+\t * can result in exponential slowdown.\n \t */\n-\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -2204,13 +2294,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tdir_state = state;\n \n \t\t/* recurse into subdir if instructed by treat_path */\n-\t\tif ((state == path_recurse) ||\n-\t\t\t((state == path_untracked) &&\n-\t\t\t (resolve_dtype(cdir.d_type, istate, path.buf, path.len) == DT_DIR) &&\n-\t\t\t ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t (pathspec &&\n-\t\t\t do_match_pathspec(istate, pathspec, path.buf, path.len,\n-\t\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {\n+\t\tif (state == path_recurse) {\n \t\t\tstruct untracked_cache_dir *ud;\n \t\t\tud = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t\t path.buf + baselen,\n@@ -2266,7 +2350,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t\t\t istate, &path, baselen,\n \t\t\t\t\t\t pathspec, state);\n \t}\n-\tclose_cached_dir(&cdir);\n+\tclose_cached_dir(&cdir, stop_at_first_file);\n out:\n \tstrbuf_release(&path);\n \n@@ -2294,15 +2378,6 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n {\n-\t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in read_directory_recursive(). See 777b420347 (dir:\n-\t * synchronize treat_leading_path() and read_directory_recursive(),\n-\t * 2019-12-19) and its parent commit for details.\n-\t */\n-\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct strbuf subdir = STRBUF_INIT;\n \tint prevlen, baselen;\n@@ -2353,23 +2428,7 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tstrbuf_reset(&subdir);\n \t\tstrbuf_add(&subdir, path+prevlen, baselen-prevlen);\n \t\tcdir.d_name = subdir.buf;\n-\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n-\t\t\t\t pathspec);\n-\t\tif (state == path_untracked &&\n-\t\t resolve_dtype(cdir.d_type, istate, sb.buf, sb.len) == DT_DIR &&\n-\t\t (dir->flags & DIR_SHOW_IGNORED_TOO ||\n-\t\t do_match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)) {\n-\t\t\tif (!match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t\t 0 /* prefix */, NULL,\n-\t\t\t\t\t 0 /* do NOT special case dirs */))\n-\t\t\t\tstate = path_none;\n-\t\t\tadd_path_to_appropriate_result_list(dir, NULL, &cdir,\n-\t\t\t\t\t\t\t istate,\n-\t\t\t\t\t\t\t &sb, baselen,\n-\t\t\t\t\t\t\t pathspec, state);\n-\t\t\tstate = path_recurse;\n-\t\t}\n+\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen, pathspec);\n \n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 41705ec1526..72b6877837b 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -84,10 +84,6 @@ dthree/\n dtwo/\n three\n /done/ 0000000000000000000000000000000000000000 recurse valid\n-/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n-three\n-/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n-two\n EOF\n \n test_expect_success 'status first time (empty cache)' '\n@@ -147,10 +143,10 @@ A two\n EOF\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../trace.expect <>.git/info/exclude &&\n \t: >../trace &&\n@@ -240,7 +228,7 @@ A two\n EOF\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../actual &&\n \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n@@ -298,15 +280,15 @@ A one\n EOF\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../actual &&\n \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n@@ -357,15 +333,15 @@ A two\n EOF\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../actual &&\n \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n@@ -403,15 +376,15 @@ test_expect_failure 'status after commit' '\n EOF\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../actual &&\n \tcat >../expect <../trace &&\n \tavoid_racy &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n@@ -464,15 +434,15 @@ test_expect_failure 'test sparse status with untracked cache' '\n EOF\n \ttest_cmp ../status.expect ../status.actual &&\n \tcat >../trace.expect <../actual &&\n \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n@@ -520,7 +487,7 @@ test_expect_success 'set up for test of subdir and sparse checkouts' '\n \techo \"sub\" > done/sub/sub/file\n '\n \n-test_expect_failure 'test sparse status with untracked cache and subdir' '\n+test_expect_success 'test sparse status with untracked cache and subdir' '\n \tavoid_racy &&\n \t: >../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n@@ -543,7 +510,7 @@ EOF\n \ttest_cmp ../trace.expect ../trace\n '\n \n-test_expect_failure 'verify untracked cache dump (sparse/subdirs)' '\n+test_expect_success 'verify untracked cache dump (sparse/subdirs)' '\n \ttest-tool dump-untracked-cache >../actual &&\n \tcat >../expect-from-test-dump <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n@@ -583,7 +543,7 @@ EOF\n \ttest_cmp ../trace.expect ../trace\n '\n \n-test_expect_failure 'move entry in subdir from untracked to cached' '\n+test_expect_success 'move entry in subdir from untracked to cached' '\n \tgit add dtwo/two &&\n \tgit status --porcelain >../status.actual &&\n \tcat >../status.expect <../status.actual &&\n \tcat >../status.expect <../actual &&\n \ttest_cmp ../expect-no-uc ../actual &&\n@@ -657,7 +617,7 @@ test_expect_success 'using --untracked-cache does not fail when core.untrackedCa\n \ttest_cmp ../expect-empty ../actual\n '\n \n-test_expect_failure 'setting core.untrackedCache to keep' '\n+test_expect_success 'setting core.untrackedCache to keep' '\n \tgit config core.untrackedCache keep &&\n \tgit update-index --untracked-cache &&\n \ttest-tool dump-untracked-cache >../actual &&\n-- \ngitgitgadget\n"},{"id":"394087","messageId":"7ba7311c-08b8-8383-0281-79e58798e0be@gmail.com","threadId":"52713","inReplyTo":"d4fe5d335771e89dad40f717bf4623854d1efa9e.1585164718.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 1/7] t7063: correct broken test expectation","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-26T13:02:22Z","receivedAt":"2020-03-26T13:02:30Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/25/2020 3:31 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> The untracked cache is caching wrong information, resulting in commands\n> like `git status --porcelain` producing erroneous answers. The tests in\n> t7063 actually have a wide enough test to catch a relevant case, in\n> particular surrounding the directory 'dthree/', but it appears the\n> answers were not checked quite closely enough and the tests were coded\n> with the wrong expectation. Once the wrong info got into the cache in\n> an early test, since later tests built on it, many others have a wrong\n> expectation as well. This affects just over a third of the tests in\n> t7063.\n\nWow. Good find.\n\n> The error can be seen starting at t7063.12 (the first one switched from\n> expect_success to expect_failure in this patch). That test runs in a\n> directory with the following files present:\n> done/one\n> dthree/three\n> dtwo/two\n> four\n> .gitignore\n> one\n> three\n> two\n> \n> Of those files, the following files are tracked:\n> done/one\n> one\n> two\n> \n> and the contents of .gitignore are:\n> four\n> \n> and the contents of .git/info/exclude are:\n> three\n> \n> And there is no core.excludesfile. Therefore, the following should be\n> untracked:\n> .gitignore\n> dthree/\n> dtwo/\n> Indeed, these three paths are reported if you run\n> git ls-files -o --directory --exclude-standard\n> within this directory. However, 'git status --porcelain' was reporting\n> for this test:\n> A done/one\n> A one\n> A two\n> ?? .gitignore\n> ?? dtwo/\n> which was clearly wrong -- dthree/ should also be listed as untracked.\n> This appears to have been broken since the test was introduced with\n> commit a3ddcefd97 (\"t7063: tests for untracked cache\", 2015-03-08).\n> Correct the test to expect the right output, marking the test as failed\n> for now. Make the same change throughout the remainder of the testsuite\n> to reflect that dthree/ remains an untracked directory throughout and\n> should be recognized as such.\n\nI wonder if we could simultaneously verify these \"expected\" results match\nusing another command without the untracked cache? It's good that we have\nthe expected outputs explicitly, but perhaps double-checking the command\nwith `-c core.untrackedCache=false` would help us know these are the correct\nexpected outputs?\n\nThanks,\n-Stolee\n"},{"id":"394088","messageId":"a81cb70f-1a3a-6f0f-1500-66e63138a848@gmail.com","threadId":"52713","inReplyTo":"6cfca619e2c5bc37d1847d49496d1be42f4061ce.1585164718.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 7/7] dir: replace exponential algorithm with a linear one, fix untracked cache","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-26T13:13:51Z","receivedAt":"2020-03-26T13:13:58Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/25/2020 3:31 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> dir's read_directory_recursive() naturally operates recursively in order\n> to walk the directory tree. Treating of directories is sometimes weird\n> because there are so many different permutations about how to handle\n> directories. Some examples:\n> \n> * 'git ls-files -o --directory' only needs to know that a directory\n> itself is untracked; it doesn't need to recurse into it to see what\n> is underneath.\n> \n> * 'git status' needs to recurse into an untracked directory, but only\n> to determine whether or not it is empty. If there are no files\n> underneath, the directory itself will be omitted from the output.\n> If it is not empty, only the directory will be listed.\n> \n> * 'git status --ignored' needs to recurse into untracked directories\n> and report all the ignored entries and then report the directory as\n> untracked -- UNLESS all the entries under the directory are\n> ignored, in which case we don't print any of the entries under the\n> directory and just report the directory itself as ignored. (Note\n> that although this forces us to walk all untracked files underneath\n> the directory as well, we strip them from the output, except for\n> users like 'git clean' who also set DIR_KEEP_TRACKED_CONTENTS.)\n> \n> * For 'git clean', we may need to recurse into a directory that\n> doesn't match any specified pathspecs, if it's possible that there\n> is an entry underneath the directory that can match one of the\n> pathspecs. In such a case, we need to be careful to omit the\n> directory itself from the list of paths (see commit 404ebceda01c\n> (\"dir: also check directories for matching pathspecs\", 2019-09-17))\n> \n> Part of the tension noted above is that the treatment of a directory can\n> change based on the files within it, and based on the various settings\n> in dir->flags. Trying to keep this in mind while reading over the code,\n> it is easy to think in terms of \"treat_directory() tells us what to do\n> with a directory, and read_directory_recursive() is the thing that\n> recurses\". Since we need to look into a directory to know how to treat\n> it, though, it is quite easy to decide to (also) recurse into the\n> directory from treat_directory() by adding a read_directory_recursive()\n> call. Adding such a call is actually fine, IF we make sure that\n> read_directory_recursive() does not also recurse into that same\n> directory.\n> \n> Unfortunately, commit df5bcdf83aeb (\"dir: recurse into untracked dirs\n> for ignored files\", 2017-05-18), added exactly such a case to the code,\n> meaning we'd have two calls to read_directory_recursive() for an\n> untracked directory. So, if we had a file named\n> one/two/three/four/five/somefile.txt\n> and nothing in one/ was tracked, then 'git status --ignored' would\n> call read_directory_recursive() twice on the directory 'one/', and\n> each of those would call read_directory_recursive() twice on the\n> directory 'one/two/', and so on until read_directory_recursive() was\n> called 2^5 times for 'one/two/three/four/five/'.\n> \n> Avoid calling read_directory_recursive() twice per level by moving a\n> lot of the special logic into treat_directory().\n> \n> Since dir.c is somewhat complex, extra cruft built up around this over\n> time. While trying to unravel it, I noticed several instances where the\n> first call to read_directory_recursive() would return e.g.\n> path_untracked for some directory and a later one would return e.g.\n> path_none, despite the fact that the directory clearly should have been\n> considered untracked. The code happened to work due to the side-effect\n> from the first invocation of adding untracked entries to dir->entries;\n> this allowed it to get the correct output despite the supposed override\n> in return value by the later call.\n> \n> I am somewhat concerned that there are still bugs and maybe even\n> testcases with the wrong expectation.\n\nFor my part, I recently set up draft PRs to test the 'next' branch in\nScalar [1] and VFS for Git [2]. I'll create a Git installer using these\npatches as well so I can run our functional test suite for a little extra\ncheck of the behavior here.\n\n[1] https://github.com/microsoft/scalar/pull/354/files\n[2] https://github.com/microsoft/VFSForGit/pull/1645\n\n> I have tried to carefully\n> document treat_directory() since it becomes more complex after this\n> change (though much of this complexity came from elsewhere that probably\n> deserved better comments to begin with).\n\nI do enjoy your warning comments.\n\n> However, much of my work felt\n> more like a game of whackamole while attempting to make the code match\n> the existing regression tests than an attempt to create an\n> implementation that matched some clear design. That seems wrong to me,\n> but the rules of existing behavior had so many special cases that I had\n> a hard time coming up with some overarching rules about what correct\n> behavior is for all cases, forcing me to hope that the regression tests\n> are correct and sufficient. Such a hope seems likely to be ill-founded,\n> given my experience with dir.c-related testcases in the last few months:\n> \n> Examples where the documentation was hard to parse or even just wrong:\n> * 3aca58045f4f (git-clean.txt: do not claim we will delete files with\n> -n/--dry-run, 2019-09-17)\n> * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n> repository, 2019-09-17)\n> * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n> Examples where testcases were declared wrong and changed:\n> * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n> repository, 2019-09-17)\n> * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n> * a2b13367fe55 (Revert \"dir.c: make 'git-status --ignored' work within\n> leading directories\", 2019-12-10)\n> Examples where testcases were clearly inadequate:\n> * 502c386ff944 (t7300-clean: demonstrate deleting nested repo with an\n> ignored file breakage, 2019-08-25)\n> * 7541cc530239 (t7300: add testcases showing failure to clean specified\n> pathspecs, 2019-09-17)\n> * a5e916c7453b (dir: fix off-by-one error in match_pathspec_item,\n> 2019-09-17)\n> * 404ebceda01c (dir: also check directories for matching pathspecs,\n> 2019-09-17)\n> * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n> repository, 2019-09-17)\n> * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n> * 452efd11fbf6 (t3011: demonstrate directory traversal failures,\n> 2019-12-10)\n> * b9670c1f5e6b (dir: fix checks on common prefix directory, 2019-12-19)\n> Examples where \"correct behavior\" was unclear to everyone:\n> https://lore.kernel.org/git/20190905154735.29784-1-newren@gmail.com/\n> Other commits of note:\n> * 902b90cf42bc (clean: fix theoretical path corruption, 2019-09-17)\n\nThanks for all of these pointers. These will be helpful if we ever do find\na regression that bisects to this patch.\n\n> However, on the positive side, it does make the code much faster. For\n> the following simple shell loop in an empty repository:\n> \n> for depth in $(seq 10 25)\n> do\n> dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n> rm -rf dir\n> mkdir -p $dirs\n> >$dirs/untracked-file\n> /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n> done\n> \n> I saw the following timings, in seconds (note that the numbers are a\n> little noisy from run-to-run, but the trend is very clear with every\n> run):\n> \n> 10: 0.03\n> 11: 0.05\n> 12: 0.08\n> 13: 0.19\n> 14: 0.29\n> 15: 0.50\n> 16: 1.05\n> 17: 2.11\n> 18: 4.11\n> 19: 8.60\n> 20: 17.55\n> 21: 33.87\n> 22: 68.71\n> 23: 140.05\n> 24: 274.45\n> 25: 551.15\n\nThese are still impressive.\n\n> For the above run, using strace I can look for the number of untracked\n> directories opened and can verify that it matches the expected\n> 2^($depth+1)-2 (the sum of 2^1 + 2^2 + 2^3 + ... + 2^$depth).\n> \n> After this fix, with strace I can verify that the number of untracked\n> directories that are opened drops to just $depth, and the timings all\n> drop to 0.00. In fact, it isn't until a depth of 190 nested directories\n> that it sometimes starts reporting a time of 0.01 seconds and doesn't\n> consistently report 0.01 seconds until there are 240 nested directories.\n> The previous code would have taken\n> 17.55 * 2^220 / (60*60*24*365) = 9.4 * 10^59 YEARS\n> to have completed the 240 nested directories case. It's not often\n> that you get to speed something up by a factor of 3*10^69.\n> \n> Finally, this also fixes the untracked cache, as noted by the test fixes\n> in t7063. Unfortunately, it does so by passing stop_at_first_file to\n> close_cached_dir() in order to disable the caching of whether\n> directories were empty (this caching was only relevant for directories\n> that we knew we didn't need to walk all the entries under but just\n> needed to know whether the directory had any entries within it in order\n> to know if the directory itself should be marked as path_none or\n> path_untracked). I'm not convinced that disabling the is-the-dir-empty\n> check is necessary; there is probably some way to still cache that and\n> not get erroneous results. However, I have not figured out how to do\n> so. If I revert the change to close_cached_dir() in this patch (thus\n> continuing to cache cases where stop_at_first_file is true meaning we\n> continue to cache whether directories are empty), then the untracked\n> cache breakage in t7063 becomes more prevalant. With my change to\n> close_cached_dir() and the other changes to avoid traversing directories\n> 2^n times in this patch, I not only avoid making the untracked_cache\n> breakage in t7063 worse but actually fix the existing breakage. Update\n> the test results in t7063 to no longer expect check_only cache entries,\n> to reflect that we have to do a bit more work in terms of how many\n> directories we have to open, and to reflect that we fixed the 1/3 of\n> tests that were broken in that testsuite.\n\nWe use the untracked cache in Scalar, so we should get some coverage\nof that, too.\n\nI'll let you know when the tests are done, and then do a review.\n\n-Stolee\n"},{"id":"394114","messageId":"CABPp-BEWhavmtDi-ahT+QMWtD6Fe-Ey7cn_82nbetWEQJL=hRA@mail.gmail.com","threadId":"52713","inReplyTo":"7ba7311c-08b8-8383-0281-79e58798e0be@gmail.com","subject":"Re: [PATCH v3 1/7] t7063: correct broken test expectation","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-26T21:18:39Z","receivedAt":"2020-03-26T21:18:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Thu, Mar 26, 2020 at 6:02 AM Derrick Stolee wrote:\n>\n> On 3/25/2020 3:31 PM, Elijah Newren via GitGitGadget wrote:\n> > From: Elijah Newren \n> >\n> > The untracked cache is caching wrong information, resulting in commands\n> > like `git status --porcelain` producing erroneous answers. The tests in\n> > t7063 actually have a wide enough test to catch a relevant case, in\n> > particular surrounding the directory 'dthree/', but it appears the\n> > answers were not checked quite closely enough and the tests were coded\n> > with the wrong expectation. Once the wrong info got into the cache in\n> > an early test, since later tests built on it, many others have a wrong\n> > expectation as well. This affects just over a third of the tests in\n> > t7063.\n>\n> Wow. Good find.\n\nor maybe not...\n\n> > The error can be seen starting at t7063.12 (the first one switched from\n> > expect_success to expect_failure in this patch). That test runs in a\n> > directory with the following files present:\n> > done/one\n> > dthree/three\n> > dtwo/two\n> > four\n> > .gitignore\n> > one\n> > three\n> > two\n> >\n> > Of those files, the following files are tracked:\n> > done/one\n> > one\n> > two\n> >\n> > and the contents of .gitignore are:\n> > four\n> >\n> > and the contents of .git/info/exclude are:\n> > three\n> >\n> > And there is no core.excludesfile. Therefore, the following should be\n> > untracked:\n> > .gitignore\n> > dthree/\n> > dtwo/\n> > Indeed, these three paths are reported if you run\n> > git ls-files -o --directory --exclude-standard\n> > within this directory. However, 'git status --porcelain' was reporting\n> > for this test:\n> > A done/one\n> > A one\n> > A two\n> > ?? .gitignore\n> > ?? dtwo/\n> > which was clearly wrong -- dthree/ should also be listed as untracked.\n> > This appears to have been broken since the test was introduced with\n> > commit a3ddcefd97 (\"t7063: tests for untracked cache\", 2015-03-08).\n> > Correct the test to expect the right output, marking the test as failed\n> > for now. Make the same change throughout the remainder of the testsuite\n> > to reflect that dthree/ remains an untracked directory throughout and\n> > should be recognized as such.\n>\n> I wonder if we could simultaneously verify these \"expected\" results match\n> using another command without the untracked cache? It's good that we have\n> the expected outputs explicitly, but perhaps double-checking the command\n> with `-c core.untrackedCache=false` would help us know these are the correct\n> expected outputs?\n\nThis was an *awesome* idea, even if the implementation doesn't quite\nwork. It turns out that -c core.untrackedCache=false does not\ninstruct status to ignore the untracked cache, it instructs status to\ndelete it. Since we had subsequent tests that depended on the\nuntrackedCache created in previous tests, this would break a number of\ntests. But I can introduce a helper to workaround that:\n\n# Ignore_Untracked_Cache, abbreviated to 3 letters because then people can\n# compare commands side-by-side, e.g.\n# iuc status --porcelain >expect &&\n# git status --porcelain >actual &&\n# test_cmp expect actual\niuc() {\n git ls-files -s >../current-index-entries\n git ls-files -t | grep ^S | sed -e s/^S.// >../current-sparse-entries\n\n GIT_INDEX_FILE=.git/tmp_index\n export GIT_INDEX_FILE\n git update-index --index-info <../current-index-entries\n git update-index --skip-worktree $(cat ../current-sparse-entries)\n\n git -c core.untrackedCache=false \"$@\"\n ret=$?\n\n rm ../current-index-entries\n rm $GIT_INDEX_FILE\n unset GIT_INDEX_FILE\n\n return $ret\n}\n\n\nDoing that helped me discover that the test didn't have a wrong\nexpectation; I did. When a directory that is not tracked is filled\nentirely with files that are ignored, then status --porcelain treats\nthe directory itself as ignored...and thus doesn't display it. (`git\nstatus --porcelain --ignored` will show it). I had seen that\nsomewhere, but hadn't fully understood the check_only and\nstop_at_first_file pieces related to it. Anyway, with this helpful\nhint:\n\n * I can say that there was not a bug in the untracked cache (at\nleast not any that I'm aware of)\n * I can update my first patch to do more thorough checking instead\nof changing the expectation\n * I found the bug in my final patch that had been evading me\n * I added a huge comment explaining check_only and\nstop_at_first_file, how they're used, and what they mean for the\nfuture reader\n * I also no longer need to partially disable the untracked cache in\nmy changes.\n\nNew patches incoming...\n"},{"id":"394115","messageId":"a4287d690be27eb80313337b02437441ad046a33.1585258061.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v4.git.git.1585258061.gitgitgadget@gmail.com","subject":"[PATCH v4 2/7] dir: fix simple typo in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-26T21:27:35Z","receivedAt":"2020-03-26T21:27:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex b460211e614..b505ba747bb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2174,7 +2174,7 @@ static void add_path_to_appropriate_result_list(struct dir_struct *dir,\n * If 'stop_at_first_file' is specified, 'path_excluded' is returned\n * to signal that a file was found. This is the least significant value that\n * indicates that a file was encountered that does not depend on the order of\n- * whether an untracked or exluded path was encountered first.\n+ * whether an untracked or excluded path was encountered first.\n *\n * Returns the most significant path_treatment value encountered in the scan.\n * If 'stop_at_first_file' is specified, `path_excluded` is the most\n-- \ngitgitgadget\n\n"},{"id":"394116","messageId":"752403e339bae098104f41c541c2b0e684586c1f.1585258061.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v4.git.git.1585258061.gitgitgadget@gmail.com","subject":"[PATCH v4 1/7] t7063: more thorough status checking","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-26T21:27:34Z","receivedAt":"2020-03-26T21:27:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIt turns out the t7063 has some testcases that even without using the\nuntracked cache cover situations that nothing else in the testsuite\nhandles. Checking the results of\n git status --porcelain\nboth with and without the untracked cache, and comparing both against\nour expected results helped uncover a critical bug in some dir.c\nrestructuring.\n\nUnfortunately, it's not easy to run status and tell it to ignore the\nuntracked cache; the only knob we have it to instruct it to *delete*\n(and ignore) the untracked cache.\n\nCreate a simple helper that will create a clone of the index that is\nmissing the untracked cache bits, and use it to compare that the results\nwith the untracked cache match the results we get without the untracked\ncache.\n\nSigned-off-by: Elijah Newren \n---\n t/t7063-status-untracked-cache.sh | 52 +++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 190ae149cf3..156d06c34e8 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -30,6 +30,30 @@ status_is_clean() {\n \ttest_must_be_empty ../status.actual\n }\n \n+# Ignore_Untracked_Cache, abbreviated to 3 letters because then people can\n+# compare commands side-by-side, e.g.\n+# iuc status --porcelain >expect &&\n+# git status --porcelain >actual &&\n+# test_cmp expect actual\n+iuc() {\n+\tgit ls-files -s >../current-index-entries\n+\tgit ls-files -t | grep ^S | sed -e s/^S.// >../current-sparse-entries\n+\n+\tGIT_INDEX_FILE=.git/tmp_index\n+\texport GIT_INDEX_FILE\n+\tgit update-index --index-info <../current-index-entries\n+\tgit update-index --skip-worktree $(cat ../current-sparse-entries)\n+\n+\tgit -c core.untrackedCache=false \"$@\"\n+\tret=$?\n+\n+\trm ../current-index-entries\n+\trm $GIT_INDEX_FILE\n+\tunset GIT_INDEX_FILE\n+\n+\treturn $ret\n+}\n+\n test_lazy_prereq UNTRACKED_CACHE '\n \t{ git update-index --test-untracked-cache; ret=$?; } &&\n \ttest $ret -ne 1\n@@ -95,6 +119,8 @@ test_expect_success 'status first time (empty cache)' '\n \t: >../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n+\ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n+\ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n+\ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../status.actual &&\n \tcat >../trace.expect <../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex d0f3d660850..3a367683661 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2259,7 +2259,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\t/* skip the dir_add_* part */\n+\t\t\t/* skip the add_path_to_appropriate_result_list() */\n \t\t\tcontinue;\n \t\t}\n \n-- \ngitgitgadget\n\n"},{"id":"394118","messageId":"48f37e5b11476d2b59dcd7496b44c6a862311f57.1585258061.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v4.git.git.1585258061.gitgitgadget@gmail.com","subject":"[PATCH v4 3/7] dir: consolidate treat_path() and treat_one_path()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-26T21:27:36Z","receivedAt":"2020-03-26T21:27:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCommit 16e2cfa90993 (\"read_directory(): further split treat_path()\",\n2010-01-08) split treat_one_path() out of treat_path(), because\ntreat_leading_path() would not have access to a dirent but wanted to\nre-use as much of treat_path() as possible. Not re-using all of\ntreat_path() caused other bugs, as noted in commit b9670c1f5e6b (\"dir:\nfix checks on common prefix directory\", 2019-12-19). Finally, in commit\nad6f2157f951 (\"dir: restructure in a way to avoid passing around a\nstruct dirent\", 2020-01-16), dirents were removed from treat_path() and\nother functions entirely.\n\nSince the only reason for splitting these functions was the lack of a\ndirent -- which no longer applies to either function -- and since the\nsplit caused problems in the past resulting in us not using\ntreat_one_path() separately anymore, just undo the split.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 121 ++++++++++++++++++++++++++--------------------------------\n 1 file changed, 55 insertions(+), 66 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex b505ba747bb..d0f3d660850 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1863,21 +1863,65 @@ static int resolve_dtype(int dtype, struct index_state *istate,\n \treturn dtype;\n }\n \n-static enum path_treatment treat_one_path(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec,\n-\t\t\t\t\t int dtype)\n-{\n-\tint exclude;\n-\tint has_path_in_index = !!index_file_exists(istate, path->buf, path->len, ignore_case);\n+static enum path_treatment treat_path_fast(struct dir_struct *dir,\n+\t\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t\t struct index_state *istate,\n+\t\t\t\t\t struct strbuf *path,\n+\t\t\t\t\t int baselen,\n+\t\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tstrbuf_setlen(path, baselen);\n+\tif (!cdir->ucd) {\n+\t\tstrbuf_addstr(path, cdir->file);\n+\t\treturn path_untracked;\n+\t}\n+\tstrbuf_addstr(path, cdir->ucd->name);\n+\t/* treat_one_path() does this before it calls treat_directory() */\n+\tstrbuf_complete(path, '/');\n+\tif (cdir->ucd->check_only)\n+\t\t/*\n+\t\t * check_only is set as a result of treat_directory() getting\n+\t\t * to its bottom. Verify again the same set of directories\n+\t\t * with check_only set.\n+\t\t */\n+\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n+\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n+\t/*\n+\t * We get path_recurse in the first run when\n+\t * directory_exists_in_index() returns index_nonexistent. We\n+\t * are sure that new changes in the index does not impact the\n+\t * outcome. Return now.\n+\t */\n+\treturn path_recurse;\n+}\n+\n+static enum path_treatment treat_path(struct dir_struct *dir,\n+\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t struct index_state *istate,\n+\t\t\t\t struct strbuf *path,\n+\t\t\t\t int baselen,\n+\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tint has_path_in_index, dtype, exclude;\n \tenum path_treatment path_treatment;\n \n-\tdtype = resolve_dtype(dtype, istate, path->buf, path->len);\n+\tif (!cdir->d_name)\n+\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n+\t\t\t\t baselen, pathspec);\n+\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n+\t\treturn path_none;\n+\tstrbuf_setlen(path, baselen);\n+\tstrbuf_addstr(path, cdir->d_name);\n+\tif (simplify_away(path->buf, path->len, pathspec))\n+\t\treturn path_none;\n+\n+\tdtype = resolve_dtype(cdir->d_type, istate, path->buf, path->len);\n \n \t/* Always exclude indexed files */\n+\thas_path_in_index = !!index_file_exists(istate, path->buf, path->len,\n+\t\t\t\t\t\tignore_case);\n \tif (dtype != DT_DIR && has_path_in_index)\n \t\treturn path_none;\n \n@@ -1942,61 +1986,6 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t}\n }\n \n-static enum path_treatment treat_path_fast(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tstrbuf_setlen(path, baselen);\n-\tif (!cdir->ucd) {\n-\t\tstrbuf_addstr(path, cdir->file);\n-\t\treturn path_untracked;\n-\t}\n-\tstrbuf_addstr(path, cdir->ucd->name);\n-\t/* treat_one_path() does this before it calls treat_directory() */\n-\tstrbuf_complete(path, '/');\n-\tif (cdir->ucd->check_only)\n-\t\t/*\n-\t\t * check_only is set as a result of treat_directory() getting\n-\t\t * to its bottom. Verify again the same set of directories\n-\t\t * with check_only set.\n-\t\t */\n-\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n-\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n-\t/*\n-\t * We get path_recurse in the first run when\n-\t * directory_exists_in_index() returns index_nonexistent. We\n-\t * are sure that new changes in the index does not impact the\n-\t * outcome. Return now.\n-\t */\n-\treturn path_recurse;\n-}\n-\n-static enum path_treatment treat_path(struct dir_struct *dir,\n-\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t struct index_state *istate,\n-\t\t\t\t struct strbuf *path,\n-\t\t\t\t int baselen,\n-\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tif (!cdir->d_name)\n-\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n-\t\t\t\t baselen, pathspec);\n-\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n-\t\treturn path_none;\n-\tstrbuf_setlen(path, baselen);\n-\tstrbuf_addstr(path, cdir->d_name);\n-\tif (simplify_away(path->buf, path->len, pathspec))\n-\t\treturn path_none;\n-\n-\treturn treat_one_path(dir, untracked, istate, path, baselen, pathspec,\n-\t\t\t cdir->d_type);\n-}\n-\n static void add_untracked(struct untracked_cache_dir *dir, const char *name)\n {\n \tif (!dir)\n-- \ngitgitgadget\n\n"},{"id":"394120","messageId":"pull.700.v4.git.git.1585258061.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v3.git.git.1585164718.gitgitgadget@gmail.com","subject":"[PATCH v4 0/7] Avoid multiple recursive calls for same path in read_directory_recursive()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-26T21:27:33Z","receivedAt":"2020-03-26T21:27:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series provides some \"modest\" speedups (see last commit message), and\nshould allow 'git status --ignored' to complete in a more reasonable\ntimeframe for Martin Melka (see \nhttps://lore.kernel.org/git/CANt4O2L_DZnMqVxZzTBMvr=BTWqB6L0uyORkoN_yMHLmUX7yHw@mail.gmail.com/\n).\n\nChanges since v3:\n\n * Turns out I was wrong about the untracked cache stuff and had some bugs\n around untracked directories with nothing bug ignored sub-entries.\n * First patch now is no longer a change of expectation of the untracked\n cache, but some more thorough testing/verification in that test that\n helped explain my misunderstanding and uncover the bug in my refactor.\n * Corrected the check_only and stop_at_first_file logic in the last patch\n and added a big comment explaining how/why it all works. Also stopped\n disabling part of the untracked cache in the same patch, and undid all\n the changes to t7063 in that patch.\n\nStuff still missing from v4:\n\n * I didn't make the DIR_KEEP_UNTRACKED_CONTENTS changes I mentioned in \n https://lore.kernel.org/git/CABPp-BEQ5s=+6Rnb-A+pdEaoPXxfo-hMSegSe1eai=RE74A3Og@mail.gmail.com/ \n which I think would make the code cleaner & clearer. I guess I'm leaving\n that for future work.\n\nAs per the commit message of the final patch, this series has some risk.\nExtra eyes would be greatly appreciated; one pair already helped me find one\nbug. Also, we should probably merge it early in some cycle, either this one\nor a later one.\n\nDerrick Stolee (1):\n dir: refactor treat_directory to clarify control flow\n\nElijah Newren (6):\n t7063: more thorough status checking\n dir: fix simple typo in comment\n dir: consolidate treat_path() and treat_one_path()\n dir: fix broken comment\n dir: fix confusion based on variable tense\n dir: replace exponential algorithm with a linear one\n\n dir.c | 349 ++++++++++++++++++------------\n t/t7063-status-untracked-cache.sh | 52 +++++\n 2 files changed, 258 insertions(+), 143 deletions(-)\n\n\nbase-commit: 0cbb60574e741e8255ba457606c4c90898cfc755\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-700%2Fnewren%2Ffill-directory-exponential-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-700/newren/fill-directory-exponential-v4\nPull-Request: https://github.com/git/git/pull/700\n\nRange-diff vs v3:\n\n 1: d4fe5d33577 ! 1: 752403e339b t7063: correct broken test expectation\n @@ -1,61 +1,23 @@\n Author: Elijah Newren \n \n - t7063: correct broken test expectation\n + t7063: more thorough status checking\n \n - The untracked cache is caching wrong information, resulting in commands\n - like `git status --porcelain` producing erroneous answers. The tests in\n - t7063 actually have a wide enough test to catch a relevant case, in\n - particular surrounding the directory 'dthree/', but it appears the\n - answers were not checked quite closely enough and the tests were coded\n - with the wrong expectation. Once the wrong info got into the cache in\n - an early test, since later tests built on it, many others have a wrong\n - expectation as well. This affects just over a third of the tests in\n - t7063.\n + It turns out the t7063 has some testcases that even without using the\n + untracked cache cover situations that nothing else in the testsuite\n + handles. Checking the results of\n + git status --porcelain\n + both with and without the untracked cache, and comparing both against\n + our expected results helped uncover a critical bug in some dir.c\n + restructuring.\n \n - The error can be seen starting at t7063.12 (the first one switched from\n - expect_success to expect_failure in this patch). That test runs in a\n - directory with the following files present:\n - done/one\n - dthree/three\n - dtwo/two\n - four\n - .gitignore\n - one\n - three\n - two\n + Unfortunately, it's not easy to run status and tell it to ignore the\n + untracked cache; the only knob we have it to instruct it to *delete*\n + (and ignore) the untracked cache.\n \n - Of those files, the following files are tracked:\n - done/one\n - one\n - two\n -\n - and the contents of .gitignore are:\n - four\n -\n - and the contents of .git/info/exclude are:\n - three\n -\n - And there is no core.excludesfile. Therefore, the following should be\n - untracked:\n - .gitignore\n - dthree/\n - dtwo/\n - Indeed, these three paths are reported if you run\n - git ls-files -o --directory --exclude-standard\n - within this directory. However, 'git status --porcelain' was reporting\n - for this test:\n - A done/one\n - A one\n - A two\n - ?? .gitignore\n - ?? dtwo/\n - which was clearly wrong -- dthree/ should also be listed as untracked.\n - This appears to have been broken since the test was introduced with\n - commit a3ddcefd97 (\"t7063: tests for untracked cache\", 2015-03-08).\n - Correct the test to expect the right output, marking the test as failed\n - for now. Make the same change throughout the remainder of the testsuite\n - to reflect that dthree/ remains an untracked directory throughout and\n - should be recognized as such.\n + Create a simple helper that will create a clone of the index that is\n + missing the untracked cache bits, and use it to compare that the results\n + with the untracked cache match the results we get without the untracked\n + cache.\n \n Signed-off-by: Elijah Newren \n \n @@ -63,279 +25,230 @@\n --- a/t/t7063-status-untracked-cache.sh\n +++ b/t/t7063-status-untracked-cache.sh\n @@\n - \ttest_cmp ../expect ../actual\n - '\n - \n --test_expect_success 'new info/exclude invalidates everything' '\n -+test_expect_failure 'new info/exclude invalidates everything' '\n - \tavoid_racy &&\n - \techo three >>.git/info/exclude &&\n + \ttest_must_be_empty ../status.actual\n + }\n + \n ++# Ignore_Untracked_Cache, abbreviated to 3 letters because then people can\n ++# compare commands side-by-side, e.g.\n ++# iuc status --porcelain >expect &&\n ++# git status --porcelain >actual &&\n ++# test_cmp expect actual\n ++iuc() {\n ++\tgit ls-files -s >../current-index-entries\n ++\tgit ls-files -t | grep ^S | sed -e s/^S.// >../current-sparse-entries\n ++\n ++\tGIT_INDEX_FILE=.git/tmp_index\n ++\texport GIT_INDEX_FILE\n ++\tgit update-index --index-info <../current-index-entries\n ++\tgit update-index --skip-worktree $(cat ../current-sparse-entries)\n ++\n ++\tgit -c core.untrackedCache=false \"$@\"\n ++\tret=$?\n ++\n ++\trm ../current-index-entries\n ++\trm $GIT_INDEX_FILE\n ++\tunset GIT_INDEX_FILE\n ++\n ++\treturn $ret\n ++}\n ++\n + test_lazy_prereq UNTRACKED_CACHE '\n + \t{ git update-index --test-untracked-cache; ret=$?; } &&\n + \ttest $ret -ne 1\n +@@\n \t: >../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n ++\ttest_cmp ../status.expect ../status.iuc &&\n + \ttest_cmp ../status.expect ../actual &&\n + \tcat >../trace.expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n ++\ttest_cmp ../status.expect ../status.iuc &&\n + \ttest_cmp ../status.expect ../actual &&\n + \tcat >../trace.expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n + \tcat >../status.expect <../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n + \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n -@@\n ++\tiuc status --porcelain >../status.iuc &&\n + \tcat >../status.expect <../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n + \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n + \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n -@@\n ++\tiuc status --porcelain >../status.iuc &&\n + \tcat >../status.expect <../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n \tavoid_racy &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n -@@\n + \tgit status --porcelain >../status.actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n + \tcat >../status.expect <../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n -@@\n + \tgit status --porcelain >../status.actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n + \tcat >../status.expect <../trace.expect < done/sub/sub/file\n - '\n - \n --test_expect_success 'test sparse status with untracked cache and subdir' '\n -+test_expect_failure 'test sparse status with untracked cache and subdir' '\n - \tavoid_racy &&\n \t: >../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n -@@\n + \tgit status --porcelain >../status.actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n + \tcat >../status.expect <../trace.expect <../actual &&\n - \tcat >../expect-from-test-dump <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n + \tgit status --porcelain >../status.actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n ++\ttest_cmp ../status.expect ../status.iuc &&\n + \ttest_cmp ../status.expect ../status.actual &&\n + \tcat >../trace.expect <../status.actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../status.actual &&\n ++\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../actual &&\n - \ttest_cmp ../expect-no-uc ../actual &&\n -@@\n - \ttest_cmp ../expect-empty ../actual\n '\n \n --test_expect_success 'setting core.untrackedCache to keep' '\n -+test_expect_failure 'setting core.untrackedCache to keep' '\n - \tgit config core.untrackedCache keep &&\n - \tgit update-index --untracked-cache &&\n - \ttest-tool dump-untracked-cache >../actual &&\n 2: b20bc3b9afd = 2: a4287d690be dir: fix simple typo in comment\n 3: fa9035949e0 = 3: 48f37e5b114 dir: consolidate treat_path() and treat_one_path()\n 4: 02e652d1869 = 4: b5ad1939379 dir: fix broken comment\n 5: 705c008d993 = 5: 2603c1a9d13 dir: fix confusion based on variable tense\n 6: f5d69102946 = 6: 576f364329d dir: refactor treat_directory to clarify control flow\n 7: 6cfca619e2c ! 7: e20525429e5 dir: replace exponential algorithm with a linear one, fix untracked cache\n @@ -1,6 +1,6 @@\n Author: Elijah Newren \n \n - dir: replace exponential algorithm with a linear one, fix untracked cache\n + dir: replace exponential algorithm with a linear one\n \n dir's read_directory_recursive() naturally operates recursively in order\n to walk the directory tree. Treating of directories is sometimes weird\n @@ -161,28 +161,6 @@\n to have completed the 240 nested directories case. It's not often\n that you get to speed something up by a factor of 3*10^69.\n \n - Finally, this also fixes the untracked cache, as noted by the test fixes\n - in t7063. Unfortunately, it does so by passing stop_at_first_file to\n - close_cached_dir() in order to disable the caching of whether\n - directories were empty (this caching was only relevant for directories\n - that we knew we didn't need to walk all the entries under but just\n - needed to know whether the directory had any entries within it in order\n - to know if the directory itself should be marked as path_none or\n - path_untracked). I'm not convinced that disabling the is-the-dir-empty\n - check is necessary; there is probably some way to still cache that and\n - not get erroneous results. However, I have not figured out how to do\n - so. If I revert the change to close_cached_dir() in this patch (thus\n - continuing to cache cases where stop_at_first_file is true meaning we\n - continue to cache whether directories are empty), then the untracked\n - cache breakage in t7063 becomes more prevalant. With my change to\n - close_cached_dir() and the other changes to avoid traversing directories\n - 2^n times in this patch, I not only avoid making the untracked_cache\n - breakage in t7063 worse but actually fix the existing breakage. Update\n - the test results in t7063 to no longer expect check_only cache entries,\n - to reflect that we have to do a bit more work in terms of how many\n - directories we have to open, and to reflect that we fixed the 1/3 of\n - tests that were broken in that testsuite.\n -\n Signed-off-by: Elijah Newren \n \n diff --git a/dir.c b/dir.c\n @@ -199,7 +177,7 @@\n +\t * you CAN'T DO BOTH.\n +\t */\n +\tenum path_treatment state;\n -+\tint nested_repo = 0, old_ignored_nr, stop_early;\n ++\tint nested_repo = 0, old_ignored_nr, check_only, stop_early;\n \t/* The \"len-1\" is to strip the final '/' */\n \tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n @@ -220,16 +198,32 @@\n \t\treturn excluded ? path_excluded : path_untracked;\n \n +\t/*\n -+\t * If we only want to determine if dirname is empty, then we can\n -+\t * stop at the first file we find underneath that directory rather\n -+\t * than continuing to recurse beyond it. If DIR_SHOW_IGNORED_TOO\n -+\t * is set, then we want MORE than just determining if dirname is\n -+\t * empty.\n ++\t * If we have we don't want to know the all the paths under an\n ++\t * untracked or ignored directory, we still need to go into the\n ++\t * directory to determine if it is empty (because an empty directory\n ++\t * should be path_none instead of path_excluded or path_untracked).\n +\t */\n -+\tstop_early = ((dir->flags & DIR_HIDE_EMPTY_DIRECTORIES) &&\n ++\tcheck_only = ((dir->flags & DIR_HIDE_EMPTY_DIRECTORIES) &&\n +\t\t !(dir->flags & DIR_SHOW_IGNORED_TOO));\n +\n +\t/*\n ++\t * However, there's another optimization possible as a subset of\n ++\t * check_only, based on the cases we have to consider:\n ++\t * A) Directory matches no exclude patterns:\n ++\t * * Directory is empty => path_none\n ++\t * * Directory has an untracked file under it => path_untracked\n ++\t * * Directory has only ignored files under it => path_excluded\n ++\t * B) Directory matches an exclude pattern:\n ++\t * * Directory is empty => path_none\n ++\t * * Directory has an untracked file under it => path_excluded\n ++\t * * Directory has only ignored files under it => path_excluded\n ++\t * In case A, we can exit as soon as we've found an untracked\n ++\t * file but otherwise have to walk all files. In case B, though,\n ++\t * we can stop at the first file we find under the directory.\n ++\t */\n ++\tstop_early = check_only && excluded;\n ++\n ++\t/*\n +\t * If /every/ file within an untracked directory is ignored, then\n +\t * we want to treat the directory as ignored (for e.g. status\n +\t * --porcelain), without listing the individual ignored files\n @@ -243,7 +237,7 @@\n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n +\tstate = read_directory_recursive(dir, istate, dirname, len, untracked,\n -+\t\t\t\t\t stop_early, stop_early, pathspec);\n ++\t\t\t\t\t check_only, stop_early, pathspec);\n +\n +\t/* There are a variety of reasons we may need to fixup the state... */\n +\tif (state == path_excluded) {\n @@ -281,25 +275,25 @@\n +\t\t\tdir->ignored_nr = old_ignored_nr;\n +\t\t}\n +\t}\n - \n - \t/*\n --\t * If this is an excluded directory, then we only need to check if\n --\t * the directory contains any files.\n ++\n ++\t/*\n +\t * If there is nothing under the current directory and we are not\n +\t * hiding empty directories, then we need to report on the\n +\t * untracked or ignored status of the directory itself.\n - \t */\n --\treturn read_directory_recursive(dir, istate, dirname, len,\n --\t\t\t\t\tuntracked, 1, excluded, pathspec);\n ++\t */\n +\tif (state == path_none && !(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n +\t\tstate = excluded ? path_excluded : path_untracked;\n -+\n -+\t/*\n + \n + \t/*\n +-\t * If this is an excluded directory, then we only need to check if\n +-\t * the directory contains any files.\n +\t * We can recurse into untracked directories that don't match any\n +\t * of the given pathspecs when some file underneath the directory\n +\t * might match one of the pathspecs. If so, we should make sure\n +\t * to note that the directory itself did not match.\n -+\t */\n + \t */\n +-\treturn read_directory_recursive(dir, istate, dirname, len,\n +-\t\t\t\t\tuntracked, 1, excluded, pathspec);\n +\tif (pathspec &&\n +\t !match_pathspec(istate, pathspec, dirname, len,\n +\t\t\t 0 /* prefix */, NULL,\n @@ -322,24 +316,6 @@\n \tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n \t\tstrbuf_addstr(path, cdir->file);\n -@@\n - \treturn -1;\n - }\n - \n --static void close_cached_dir(struct cached_dir *cdir)\n -+static void close_cached_dir(struct cached_dir *cdir, int stop_at_first_file)\n - {\n - \tif (cdir->fdir)\n - \t\tclosedir(cdir->fdir);\n -@@\n - \t * We have gone through this directory and found no untracked\n - \t * entries. Mark it valid.\n - \t */\n --\tif (cdir->untracked) {\n -+\tif (!stop_at_first_file && cdir->untracked) {\n - \t\tcdir->untracked->valid = 1;\n - \t\tcdir->untracked->recurse = 1;\n - \t}\n @@\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n @@ -373,15 +349,6 @@\n \t\t\tstruct untracked_cache_dir *ud;\n \t\t\tud = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t\t path.buf + baselen,\n -@@\n - \t\t\t\t\t\t istate, &path, baselen,\n - \t\t\t\t\t\t pathspec, state);\n - \t}\n --\tclose_cached_dir(&cdir);\n -+\tclose_cached_dir(&cdir, stop_at_first_file);\n - out:\n - \tstrbuf_release(&path);\n - \n @@\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n @@ -423,342 +390,3 @@\n \n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n -\n - diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\n - --- a/t/t7063-status-untracked-cache.sh\n - +++ b/t/t7063-status-untracked-cache.sh\n -@@\n - dtwo/\n - three\n - /done/ 0000000000000000000000000000000000000000 recurse valid\n --/dthree/ 0000000000000000000000000000000000000000 recurse check_only valid\n --three\n --/dtwo/ 0000000000000000000000000000000000000000 recurse check_only valid\n --two\n - EOF\n - \n - test_expect_success 'status first time (empty cache)' '\n -@@\n - EOF\n - \ttest_cmp ../status.expect ../actual &&\n - \tcat >../trace.expect <../trace.expect <>.git/info/exclude &&\n - \t: >../trace &&\n -@@\n - EOF\n - \ttest_cmp ../status.expect ../actual &&\n - \tcat >../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n - \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n - \tgit status --porcelain >../actual &&\n -@@\n - EOF\n - \ttest_cmp ../status.expect ../actual &&\n - \tcat >../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n - \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n - \tgit status --porcelain >../actual &&\n -@@\n - EOF\n - \ttest_cmp ../status.expect ../actual &&\n - \tcat >../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n - \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n - \tgit status --porcelain >../actual &&\n -@@\n - EOF\n - \ttest_cmp ../status.expect ../actual &&\n - \tcat >../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n - \tavoid_racy &&\n - \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n -@@\n - EOF\n - \ttest_cmp ../status.expect ../status.actual &&\n - \tcat >../trace.expect <../actual &&\n - \tcat >../expect <../trace &&\n - \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n -@@\n - \techo \"sub\" > done/sub/sub/file\n - '\n - \n --test_expect_failure 'test sparse status with untracked cache and subdir' '\n -+test_expect_success 'test sparse status with untracked cache and subdir' '\n - \tavoid_racy &&\n - \t: >../trace &&\n - \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n -@@\n - \ttest_cmp ../trace.expect ../trace\n - '\n - \n --test_expect_failure 'verify untracked cache dump (sparse/subdirs)' '\n -+test_expect_success 'verify untracked cache dump (sparse/subdirs)' '\n - \ttest-tool dump-untracked-cache >../actual &&\n - \tcat >../expect-from-test-dump <../trace &&\n - \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n -@@\n - \ttest_cmp ../trace.expect ../trace\n - '\n - \n --test_expect_failure 'move entry in subdir from untracked to cached' '\n -+test_expect_success 'move entry in subdir from untracked to cached' '\n - \tgit add dtwo/two &&\n - \tgit status --porcelain >../status.actual &&\n - \tcat >../status.expect <../status.actual &&\n - \tcat >../status.expect <../actual &&\n - \ttest_cmp ../expect-no-uc ../actual &&\n -@@\n - \ttest_cmp ../expect-empty ../actual\n - '\n - \n --test_expect_failure 'setting core.untrackedCache to keep' '\n -+test_expect_success 'setting core.untrackedCache to keep' '\n - \tgit config core.untrackedCache keep &&\n - \tgit update-index --untracked-cache &&\n - \ttest-tool dump-untracked-cache >../actual &&\n\n-- \ngitgitgadget\n"},{"id":"394119","messageId":"2603c1a9d13a2ff8c6d719a0f5858211116ef007.1585258061.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v4.git.git.1585258061.gitgitgadget@gmail.com","subject":"[PATCH v4 5/7] dir: fix confusion based on variable tense","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-26T21:27:38Z","receivedAt":"2020-03-26T21:27:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nDespite having contributed several fixes in this area, I have for months\n(years?) assumed that the \"exclude\" variable was a directive; this\ncaused me to think of it as a different mode we operate in and left me\nconfused as I tried to build up a mental model around why we'd need such\na directive. I mostly tried to ignore it while focusing on the pieces I\nwas trying to understand.\n\nThen I finally traced this variable all back to a call to is_excluded(),\nmeaning it was actually functioning as an adjective. In particular, it\nwas a checked property (\"Does this path match a rule in .gitignore?\"),\nrather than a mode passed in from the caller. Change the variable name\nto match the part of speech used by the function called to define it,\nwhich will hopefully make these bits of code slightly clearer to the\nnext reader.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 26 +++++++++++++-------------\n 1 file changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3a367683661..8074e651e6f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1656,7 +1656,7 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n static enum path_treatment treat_directory(struct dir_struct *dir,\n \tstruct index_state *istate,\n \tstruct untracked_cache_dir *untracked,\n-\tconst char *dirname, int len, int baselen, int exclude,\n+\tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n@@ -1679,13 +1679,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\t}\n \t\tif (nested_repo)\n \t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(exclude ? path_excluded : path_untracked));\n+\t\t\t\t(excluded ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n \t\t\tbreak;\n-\t\tif (exclude &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n+\t\tif (excluded &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n \n \t\t\t/*\n \t\t\t * This is an excluded directory and we are\n@@ -1713,7 +1713,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* This is the \"show_other_directories\" case */\n \n \tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n@@ -1723,7 +1723,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t * the directory contains any files.\n \t */\n \treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, exclude, pathspec);\n+\t\t\t\t\tuntracked, 1, excluded, pathspec);\n }\n \n /*\n@@ -1904,7 +1904,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t int baselen,\n \t\t\t\t const struct pathspec *pathspec)\n {\n-\tint has_path_in_index, dtype, exclude;\n+\tint has_path_in_index, dtype, excluded;\n \tenum path_treatment path_treatment;\n \n \tif (!cdir->d_name)\n@@ -1949,13 +1949,13 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t (directory_exists_in_index(istate, path->buf, path->len) == index_nonexistent))\n \t\treturn path_none;\n \n-\texclude = is_excluded(dir, istate, path->buf, &dtype);\n+\texcluded = is_excluded(dir, istate, path->buf, &dtype);\n \n \t/*\n \t * Excluded? If we don't explicitly want to show\n \t * ignored files, ignore it\n \t */\n-\tif (exclude && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n+\tif (excluded && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n \t\treturn path_excluded;\n \n \tswitch (dtype) {\n@@ -1965,7 +1965,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\tstrbuf_addch(path, '/');\n \t\tpath_treatment = treat_directory(dir, istate, untracked,\n \t\t\t\t\t\t path->buf, path->len,\n-\t\t\t\t\t\t baselen, exclude, pathspec);\n+\t\t\t\t\t\t baselen, excluded, pathspec);\n \t\t/*\n \t\t * If 1) we only want to return directories that\n \t\t * match an exclude pattern and 2) this directory does\n@@ -1974,7 +1974,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t * recurse into this directory (instead of marking the\n \t\t * directory itself as an ignored path).\n \t\t */\n-\t\tif (!exclude &&\n+\t\tif (!excluded &&\n \t\t path_treatment == path_excluded &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n@@ -1982,7 +1982,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\treturn path_treatment;\n \tcase DT_REG:\n \tcase DT_LNK:\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"394121","messageId":"576f364329dba507e9976b775a4b48dd1257b077.1585258061.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v4.git.git.1585258061.gitgitgadget@gmail.com","subject":"[PATCH v4 6/7] dir: refactor treat_directory to clarify control flow","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-26T21:27:39Z","receivedAt":"2020-03-26T21:27:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe logic in treat_directory() is handled by a multi-case\nswitch statement, but this switch is very asymmetrical, as\nthe first two cases are simple but the third is more\ncomplicated than the rest of the method. In fact, the third\ncase includes a \"break\" statement that leads to the block\nof code outside the switch statement. That is the only way\nto reach that block, as the switch handles all possible\nvalues from directory_exists_in_index();\n\nExtract the switch statement into a series of \"if\" statements.\nThis simplifies the trivial cases, while clarifying how to\nreach the \"show_other_directories\" case. This is particularly\nimportant as the \"show_other_directories\" case will expand\nin a later change.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n dir.c | 35 +++++++++++++++++------------------\n 1 file changed, 17 insertions(+), 18 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 8074e651e6f..d9bcb7e19b6 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1660,29 +1660,28 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n-\n \t/* The \"len-1\" is to strip the final '/' */\n-\tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n-\tcase index_directory:\n-\t\treturn path_recurse;\n+\tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n-\tcase index_gitdir:\n+\tif (status == index_directory)\n+\t\treturn path_recurse;\n+\tif (status == index_gitdir)\n \t\treturn path_none;\n+\tif (status != index_nonexistent)\n+\t\tBUG(\"Unhandled value for directory_exists_in_index: %d\\n\", status);\n \n-\tcase index_nonexistent:\n-\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n-\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n-\t\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\t\tstrbuf_addstr(&sb, dirname);\n-\t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n-\t\t\tstrbuf_release(&sb);\n-\t\t}\n-\t\tif (nested_repo)\n-\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(excluded ? path_excluded : path_untracked));\n+\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n+\t\t!(dir->flags & DIR_NO_GITLINKS)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tstrbuf_addstr(&sb, dirname);\n+\t\tnested_repo = is_nonbare_repository_dir(&sb);\n+\t\tstrbuf_release(&sb);\n+\t}\n+\tif (nested_repo)\n+\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n+\t\t\t(excluded ? path_excluded : path_untracked));\n \n-\t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n-\t\t\tbreak;\n+\tif (!(dir->flags & DIR_SHOW_OTHER_DIRECTORIES)) {\n \t\tif (excluded &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n-- \ngitgitgadget\n\n"},{"id":"394122","messageId":"e20525429e56b9a2b7fae43cd41eece740084b8a.1585258061.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v4.git.git.1585258061.gitgitgadget@gmail.com","subject":"[PATCH v4 7/7] dir: replace exponential algorithm with a linear one","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-03-26T21:27:40Z","receivedAt":"2020-03-26T21:27:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ndir's read_directory_recursive() naturally operates recursively in order\nto walk the directory tree. Treating of directories is sometimes weird\nbecause there are so many different permutations about how to handle\ndirectories. Some examples:\n\n * 'git ls-files -o --directory' only needs to know that a directory\n itself is untracked; it doesn't need to recurse into it to see what\n is underneath.\n\n * 'git status' needs to recurse into an untracked directory, but only\n to determine whether or not it is empty. If there are no files\n underneath, the directory itself will be omitted from the output.\n If it is not empty, only the directory will be listed.\n\n * 'git status --ignored' needs to recurse into untracked directories\n and report all the ignored entries and then report the directory as\n untracked -- UNLESS all the entries under the directory are\n ignored, in which case we don't print any of the entries under the\n directory and just report the directory itself as ignored. (Note\n that although this forces us to walk all untracked files underneath\n the directory as well, we strip them from the output, except for\n users like 'git clean' who also set DIR_KEEP_TRACKED_CONTENTS.)\n\n * For 'git clean', we may need to recurse into a directory that\n doesn't match any specified pathspecs, if it's possible that there\n is an entry underneath the directory that can match one of the\n pathspecs. In such a case, we need to be careful to omit the\n directory itself from the list of paths (see commit 404ebceda01c\n (\"dir: also check directories for matching pathspecs\", 2019-09-17))\n\nPart of the tension noted above is that the treatment of a directory can\nchange based on the files within it, and based on the various settings\nin dir->flags. Trying to keep this in mind while reading over the code,\nit is easy to think in terms of \"treat_directory() tells us what to do\nwith a directory, and read_directory_recursive() is the thing that\nrecurses\". Since we need to look into a directory to know how to treat\nit, though, it is quite easy to decide to (also) recurse into the\ndirectory from treat_directory() by adding a read_directory_recursive()\ncall. Adding such a call is actually fine, IF we make sure that\nread_directory_recursive() does not also recurse into that same\ndirectory.\n\nUnfortunately, commit df5bcdf83aeb (\"dir: recurse into untracked dirs\nfor ignored files\", 2017-05-18), added exactly such a case to the code,\nmeaning we'd have two calls to read_directory_recursive() for an\nuntracked directory. So, if we had a file named\n one/two/three/four/five/somefile.txt\nand nothing in one/ was tracked, then 'git status --ignored' would\ncall read_directory_recursive() twice on the directory 'one/', and\neach of those would call read_directory_recursive() twice on the\ndirectory 'one/two/', and so on until read_directory_recursive() was\ncalled 2^5 times for 'one/two/three/four/five/'.\n\nAvoid calling read_directory_recursive() twice per level by moving a\nlot of the special logic into treat_directory().\n\nSince dir.c is somewhat complex, extra cruft built up around this over\ntime. While trying to unravel it, I noticed several instances where the\nfirst call to read_directory_recursive() would return e.g.\npath_untracked for some directory and a later one would return e.g.\npath_none, despite the fact that the directory clearly should have been\nconsidered untracked. The code happened to work due to the side-effect\nfrom the first invocation of adding untracked entries to dir->entries;\nthis allowed it to get the correct output despite the supposed override\nin return value by the later call.\n\nI am somewhat concerned that there are still bugs and maybe even\ntestcases with the wrong expectation. I have tried to carefully\ndocument treat_directory() since it becomes more complex after this\nchange (though much of this complexity came from elsewhere that probably\ndeserved better comments to begin with). However, much of my work felt\nmore like a game of whackamole while attempting to make the code match\nthe existing regression tests than an attempt to create an\nimplementation that matched some clear design. That seems wrong to me,\nbut the rules of existing behavior had so many special cases that I had\na hard time coming up with some overarching rules about what correct\nbehavior is for all cases, forcing me to hope that the regression tests\nare correct and sufficient. Such a hope seems likely to be ill-founded,\ngiven my experience with dir.c-related testcases in the last few months:\n\n Examples where the documentation was hard to parse or even just wrong:\n * 3aca58045f4f (git-clean.txt: do not claim we will delete files with\n -n/--dry-run, 2019-09-17)\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n Examples where testcases were declared wrong and changed:\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n * a2b13367fe55 (Revert \"dir.c: make 'git-status --ignored' work within\n leading directories\", 2019-12-10)\n Examples where testcases were clearly inadequate:\n * 502c386ff944 (t7300-clean: demonstrate deleting nested repo with an\n ignored file breakage, 2019-08-25)\n * 7541cc530239 (t7300: add testcases showing failure to clean specified\n pathspecs, 2019-09-17)\n * a5e916c7453b (dir: fix off-by-one error in match_pathspec_item,\n 2019-09-17)\n * 404ebceda01c (dir: also check directories for matching pathspecs,\n 2019-09-17)\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n * 452efd11fbf6 (t3011: demonstrate directory traversal failures,\n 2019-12-10)\n * b9670c1f5e6b (dir: fix checks on common prefix directory, 2019-12-19)\n Examples where \"correct behavior\" was unclear to everyone:\n https://lore.kernel.org/git/20190905154735.29784-1-newren@gmail.com/\n Other commits of note:\n * 902b90cf42bc (clean: fix theoretical path corruption, 2019-09-17)\n\nHowever, on the positive side, it does make the code much faster. For\nthe following simple shell loop in an empty repository:\n\n for depth in $(seq 10 25)\n do\n dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n rm -rf dir\n mkdir -p $dirs\n >$dirs/untracked-file\n /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n done\n\nI saw the following timings, in seconds (note that the numbers are a\nlittle noisy from run-to-run, but the trend is very clear with every\nrun):\n\n 10: 0.03\n 11: 0.05\n 12: 0.08\n 13: 0.19\n 14: 0.29\n 15: 0.50\n 16: 1.05\n 17: 2.11\n 18: 4.11\n 19: 8.60\n 20: 17.55\n 21: 33.87\n 22: 68.71\n 23: 140.05\n 24: 274.45\n 25: 551.15\n\nFor the above run, using strace I can look for the number of untracked\ndirectories opened and can verify that it matches the expected\n2^($depth+1)-2 (the sum of 2^1 + 2^2 + 2^3 + ... + 2^$depth).\n\nAfter this fix, with strace I can verify that the number of untracked\ndirectories that are opened drops to just $depth, and the timings all\ndrop to 0.00. In fact, it isn't until a depth of 190 nested directories\nthat it sometimes starts reporting a time of 0.01 seconds and doesn't\nconsistently report 0.01 seconds until there are 240 nested directories.\nThe previous code would have taken\n 17.55 * 2^220 / (60*60*24*365) = 9.4 * 10^59 YEARS\nto have completed the 240 nested directories case. It's not often\nthat you get to speed something up by a factor of 3*10^69.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 167 ++++++++++++++++++++++++++++++++++++++++++----------------\n 1 file changed, 121 insertions(+), 46 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex d9bcb7e19b6..29283fc2588 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1659,7 +1659,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n-\tint nested_repo = 0;\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n+\tenum path_treatment state;\n+\tint nested_repo = 0, old_ignored_nr, check_only, stop_early;\n \t/* The \"len-1\" is to strip the final '/' */\n \tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n@@ -1711,18 +1717,117 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/* This is the \"show_other_directories\" case */\n \n-\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t/*\n+\t * We only need to recurse into untracked/ignored directories if\n+\t * either of the following bits is set:\n+\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n+\t * there are ignored directories below)\n+\t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n+\t * the directory is empty)\n+\t */\n+\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n \t\treturn excluded ? path_excluded : path_untracked;\n \n+\t/*\n+\t * If we have we don't want to know the all the paths under an\n+\t * untracked or ignored directory, we still need to go into the\n+\t * directory to determine if it is empty (because an empty directory\n+\t * should be path_none instead of path_excluded or path_untracked).\n+\t */\n+\tcheck_only = ((dir->flags & DIR_HIDE_EMPTY_DIRECTORIES) &&\n+\t\t !(dir->flags & DIR_SHOW_IGNORED_TOO));\n+\n+\t/*\n+\t * However, there's another optimization possible as a subset of\n+\t * check_only, based on the cases we have to consider:\n+\t * A) Directory matches no exclude patterns:\n+\t * * Directory is empty => path_none\n+\t * * Directory has an untracked file under it => path_untracked\n+\t * * Directory has only ignored files under it => path_excluded\n+\t * B) Directory matches an exclude pattern:\n+\t * * Directory is empty => path_none\n+\t * * Directory has an untracked file under it => path_excluded\n+\t * * Directory has only ignored files under it => path_excluded\n+\t * In case A, we can exit as soon as we've found an untracked\n+\t * file but otherwise have to walk all files. In case B, though,\n+\t * we can stop at the first file we find under the directory.\n+\t */\n+\tstop_early = check_only && excluded;\n+\n+\t/*\n+\t * If /every/ file within an untracked directory is ignored, then\n+\t * we want to treat the directory as ignored (for e.g. status\n+\t * --porcelain), without listing the individual ignored files\n+\t * underneath. To do so, we'll save the current ignored_nr, and\n+\t * pop all the ones added after it if it turns out the entire\n+\t * directory is ignored.\n+\t */\n+\told_ignored_nr = dir->ignored_nr;\n+\n+\t/* Actually recurse into dirname now, we'll fixup the state later. */\n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n+\tstate = read_directory_recursive(dir, istate, dirname, len, untracked,\n+\t\t\t\t\t check_only, stop_early, pathspec);\n+\n+\t/* There are a variety of reasons we may need to fixup the state... */\n+\tif (state == path_excluded) {\n+\t\tint i;\n+\n+\t\t/*\n+\t\t * When stop_early is set, read_directory_recursive() will\n+\t\t * never return path_untracked regardless of whether\n+\t\t * underlying paths were untracked or ignored (because\n+\t\t * returning early means it excluded some paths, or\n+\t\t * something like that -- see commit 5aaa7fd39aaf (\"Improve\n+\t\t * performance of git status --ignored\", 2017-09-18)).\n+\t\t * However, we're not really concerned with the status of\n+\t\t * files under the directory, we just wanted to know\n+\t\t * whether the directory was empty (state == path_none) or\n+\t\t * not (state == path_excluded), and if not, we'd return\n+\t\t * our original status based on whether the untracked\n+\t\t * directory matched an exclusion pattern.\n+\t\t */\n+\t\tif (stop_early)\n+\t\t\tstate = excluded ? path_excluded : path_untracked;\n+\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * When\n+\t\t\t * !stop_early && state == path_excluded\n+\t\t\t * then all paths under dirname were ignored. For\n+\t\t\t * this case, git status --porcelain wants to just\n+\t\t\t * list the directory itself as ignored and not\n+\t\t\t * list the individual paths underneath. Remove\n+\t\t\t * the individual paths underneath.\n+\t\t\t */\n+\t\t\tfor (i = old_ignored_nr + 1; iignored_nr; ++i)\n+\t\t\t\tfree(dir->ignored[i]);\n+\t\t\tdir->ignored_nr = old_ignored_nr;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * If there is nothing under the current directory and we are not\n+\t * hiding empty directories, then we need to report on the\n+\t * untracked or ignored status of the directory itself.\n+\t */\n+\tif (state == path_none && !(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\tstate = excluded ? path_excluded : path_untracked;\n \n \t/*\n-\t * If this is an excluded directory, then we only need to check if\n-\t * the directory contains any files.\n+\t * We can recurse into untracked directories that don't match any\n+\t * of the given pathspecs when some file underneath the directory\n+\t * might match one of the pathspecs. If so, we should make sure\n+\t * to note that the directory itself did not match.\n \t */\n-\treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, excluded, pathspec);\n+\tif (pathspec &&\n+\t !match_pathspec(istate, pathspec, dirname, len,\n+\t\t\t 0 /* prefix */, NULL,\n+\t\t\t 0 /* do NOT special case dirs */))\n+\t\tstate = path_none;\n+\n+\treturn state;\n }\n \n /*\n@@ -1870,6 +1975,11 @@ static enum path_treatment treat_path_fast(struct dir_struct *dir,\n \t\t\t\t\t int baselen,\n \t\t\t\t\t const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n \tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n \t\tstrbuf_addstr(path, cdir->file);\n@@ -2175,14 +2285,10 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n \t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in treat_leading_path(). See the commit message for the\n-\t * commit adding this warning as well as the commit preceding it\n-\t * for details.\n+\t * WARNING: Do NOT recurse unless path_recurse is returned from\n+\t * treat_path(). Recursing on any other return value\n+\t * can result in exponential slowdown.\n \t */\n-\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -2204,13 +2310,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tdir_state = state;\n \n \t\t/* recurse into subdir if instructed by treat_path */\n-\t\tif ((state == path_recurse) ||\n-\t\t\t((state == path_untracked) &&\n-\t\t\t (resolve_dtype(cdir.d_type, istate, path.buf, path.len) == DT_DIR) &&\n-\t\t\t ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t (pathspec &&\n-\t\t\t do_match_pathspec(istate, pathspec, path.buf, path.len,\n-\t\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {\n+\t\tif (state == path_recurse) {\n \t\t\tstruct untracked_cache_dir *ud;\n \t\t\tud = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t\t path.buf + baselen,\n@@ -2294,15 +2394,6 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n {\n-\t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in read_directory_recursive(). See 777b420347 (dir:\n-\t * synchronize treat_leading_path() and read_directory_recursive(),\n-\t * 2019-12-19) and its parent commit for details.\n-\t */\n-\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct strbuf subdir = STRBUF_INIT;\n \tint prevlen, baselen;\n@@ -2353,23 +2444,7 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tstrbuf_reset(&subdir);\n \t\tstrbuf_add(&subdir, path+prevlen, baselen-prevlen);\n \t\tcdir.d_name = subdir.buf;\n-\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n-\t\t\t\t pathspec);\n-\t\tif (state == path_untracked &&\n-\t\t resolve_dtype(cdir.d_type, istate, sb.buf, sb.len) == DT_DIR &&\n-\t\t (dir->flags & DIR_SHOW_IGNORED_TOO ||\n-\t\t do_match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)) {\n-\t\t\tif (!match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t\t 0 /* prefix */, NULL,\n-\t\t\t\t\t 0 /* do NOT special case dirs */))\n-\t\t\t\tstate = path_none;\n-\t\t\tadd_path_to_appropriate_result_list(dir, NULL, &cdir,\n-\t\t\t\t\t\t\t istate,\n-\t\t\t\t\t\t\t &sb, baselen,\n-\t\t\t\t\t\t\t pathspec, state);\n-\t\t\tstate = path_recurse;\n-\t\t}\n+\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen, pathspec);\n \n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n-- \ngitgitgadget\n"},{"id":"394167","messageId":"ffec2a50-e746-370c-1957-3615950bb6fb@gmail.com","threadId":"52713","inReplyTo":"752403e339bae098104f41c541c2b0e684586c1f.1585258061.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 1/7] t7063: more thorough status checking","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-27T13:09:19Z","receivedAt":"2020-03-27T13:09:27Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/26/2020 5:27 PM, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> It turns out the t7063 has some testcases that even without using the\n> untracked cache cover situations that nothing else in the testsuite\n> handles. Checking the results of\n> git status --porcelain\n> both with and without the untracked cache, and comparing both against\n> our expected results helped uncover a critical bug in some dir.c\n> restructuring.\n> \n> Unfortunately, it's not easy to run status and tell it to ignore the\n> untracked cache; the only knob we have it to instruct it to *delete*\n> (and ignore) the untracked cache.\n> \n> Create a simple helper that will create a clone of the index that is\n> missing the untracked cache bits, and use it to compare that the results\n> with the untracked cache match the results we get without the untracked\n> cache.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> t/t7063-status-untracked-cache.sh | 52 +++++++++++++++++++++++++++++++\n> 1 file changed, 52 insertions(+)\n> \n> diff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\n> index 190ae149cf3..156d06c34e8 100755\n> --- a/t/t7063-status-untracked-cache.sh\n> +++ b/t/t7063-status-untracked-cache.sh\n> @@ -30,6 +30,30 @@ status_is_clean() {\n> \ttest_must_be_empty ../status.actual\n> }\n> \n> +# Ignore_Untracked_Cache, abbreviated to 3 letters because then people can\n> +# compare commands side-by-side, e.g.\n> +# iuc status --porcelain >expect &&\n> +# git status --porcelain >actual &&\n> +# test_cmp expect actual\n> +iuc() {\n> +\tgit ls-files -s >../current-index-entries\n> +\tgit ls-files -t | grep ^S | sed -e s/^S.// >../current-sparse-entries\n> +\n> +\tGIT_INDEX_FILE=.git/tmp_index\n> +\texport GIT_INDEX_FILE\n> +\tgit update-index --index-info <../current-index-entries\n> +\tgit update-index --skip-worktree $(cat ../current-sparse-entries)\n> +\n> +\tgit -c core.untrackedCache=false \"$@\"\n> +\tret=$?\n> +\n> +\trm ../current-index-entries\n> +\trm $GIT_INDEX_FILE\n> +\tunset GIT_INDEX_FILE\n> +\n> +\treturn $ret\n> +}\n\nThis is a clever way to get around the untracked cache deletion.\n\nThanks for adding these extra comparisons! It really does help guarantee\nthat we are doing the right thing in each case.\n\n-Stolee\n\n"},{"id":"394168","messageId":"ad7c7cd7-2f2e-d406-6a1f-9dd7f155dc10@gmail.com","threadId":"52713","inReplyTo":"pull.700.v4.git.git.1585258061.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 0/7] Avoid multiple recursive calls for same path in read_directory_recursive()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-03-27T13:13:51Z","receivedAt":"2020-03-27T13:13:57Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 3/26/2020 5:27 PM, Elijah Newren via GitGitGadget wrote:\n> This series provides some \"modest\" speedups (see last commit message), and\n> should allow 'git status --ignored' to complete in a more reasonable\n> timeframe for Martin Melka (see \n> https://lore.kernel.org/git/CANt4O2L_DZnMqVxZzTBMvr=BTWqB6L0uyORkoN_yMHLmUX7yHw@mail.gmail.com/\n> ).\n> \n> Changes since v3:\n> \n> * Turns out I was wrong about the untracked cache stuff and had some bugs\n> around untracked directories with nothing bug ignored sub-entries.\n> * First patch now is no longer a change of expectation of the untracked\n> cache, but some more thorough testing/verification in that test that\n> helped explain my misunderstanding and uncover the bug in my refactor.\n> * Corrected the check_only and stop_at_first_file logic in the last patch\n> and added a big comment explaining how/why it all works. Also stopped\n> disabling part of the untracked cache in the same patch, and undid all\n> the changes to t7063 in that patch.\n> \n> Stuff still missing from v4:\n> \n> * I didn't make the DIR_KEEP_UNTRACKED_CONTENTS changes I mentioned in \n> https://lore.kernel.org/git/CABPp-BEQ5s=+6Rnb-A+pdEaoPXxfo-hMSegSe1eai=RE74A3Og@mail.gmail.com/ \n> which I think would make the code cleaner & clearer. I guess I'm leaving\n> that for future work.\n> \n> As per the commit message of the final patch, this series has some risk.\n> Extra eyes would be greatly appreciated; one pair already helped me find one\n> bug.\n\nI'm glad that I could help you discover mixed expectations. This pair of eyes\nis now satisfied with this series to the extent I can check it.\n\nAdding the previous patch to our microsoft/git fork pass the functional tests\nin Scalar and VFS for Git, for what it's worth:\n\n[1] https://github.com/microsoft/scalar/pull/358\n[2] https://github.com/microsoft/VFSForGit/pull/1646\n\nThanks,\n-Stolee\n\n"},{"id":"394246","messageId":"CABPp-BFXWx_MPL1Z0-X5hHf0N1T_13TEbAhVTKh6-atO1+g90Q@mail.gmail.com","threadId":"52713","inReplyTo":"ad7c7cd7-2f2e-d406-6a1f-9dd7f155dc10@gmail.com","subject":"Re: [PATCH v4 0/7] Avoid multiple recursive calls for same path in read_directory_recursive()","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-28T17:33:10Z","receivedAt":"2020-03-28T17:33:23Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Fri, Mar 27, 2020 at 6:13 AM Derrick Stolee wrote:\n>\n> On 3/26/2020 5:27 PM, Elijah Newren via GitGitGadget wrote:\n> > This series provides some \"modest\" speedups (see last commit message), and\n> > should allow 'git status --ignored' to complete in a more reasonable\n> > timeframe for Martin Melka (see\n> > https://lore.kernel.org/git/CANt4O2L_DZnMqVxZzTBMvr=BTWqB6L0uyORkoN_yMHLmUX7yHw@mail.gmail.com/\n> > ).\n> >\n> > Changes since v3:\n> >\n> > * Turns out I was wrong about the untracked cache stuff and had some bugs\n> > around untracked directories with nothing bug ignored sub-entries.\n> > * First patch now is no longer a change of expectation of the untracked\n> > cache, but some more thorough testing/verification in that test that\n> > helped explain my misunderstanding and uncover the bug in my refactor.\n> > * Corrected the check_only and stop_at_first_file logic in the last patch\n> > and added a big comment explaining how/why it all works. Also stopped\n> > disabling part of the untracked cache in the same patch, and undid all\n> > the changes to t7063 in that patch.\n> >\n> > Stuff still missing from v4:\n> >\n> > * I didn't make the DIR_KEEP_UNTRACKED_CONTENTS changes I mentioned in\n> > https://lore.kernel.org/git/CABPp-BEQ5s=+6Rnb-A+pdEaoPXxfo-hMSegSe1eai=RE74A3Og@mail.gmail.com/\n> > which I think would make the code cleaner & clearer. I guess I'm leaving\n> > that for future work.\n> >\n> > As per the commit message of the final patch, this series has some risk.\n> > Extra eyes would be greatly appreciated; one pair already helped me find one\n> > bug.\n>\n> I'm glad that I could help you discover mixed expectations. This pair of eyes\n> is now satisfied with this series to the extent I can check it.\n>\n> Adding the previous patch to our microsoft/git fork pass the functional tests\n> in Scalar and VFS for Git, for what it's worth:\n>\n> [1] https://github.com/microsoft/scalar/pull/358\n> [2] https://github.com/microsoft/VFSForGit/pull/1646\n\nThanks, that helps.\n\nAn update of my own for this series: Based on Felipe's reported\nbash-completion issue I was modifying commands to try out a number of\nother things and discovered some cases that can trigger the die(\"git\nls-files: internal error - directory entry not superset of prefix\")\nmessage from ls-files so there's still some fixes I need to make.\nWill send an update when I've got it.\n"},{"id":"394293","messageId":"xmqq369raut5.fsf@gitster.c.googlers.com","threadId":"52713","inReplyTo":"ffec2a50-e746-370c-1957-3615950bb6fb@gmail.com","subject":"Re: [PATCH v4 1/7] t7063: more thorough status checking","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-29T18:18:46Z","receivedAt":"2020-03-29T18:18:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n>> +# Ignore_Untracked_Cache, abbreviated to 3 letters because then people can\n>> +# compare commands side-by-side, e.g.\n>> +# iuc status --porcelain >expect &&\n>> +# git status --porcelain >actual &&\n>> +# test_cmp expect actual\n\n;-)\n\n>> +iuc() {\n\nMissing SP after \"iuc\".\n\n>> +\tgit ls-files -s >../current-index-entries\n>> +\tgit ls-files -t | grep ^S | sed -e s/^S.// >../current-sparse-entries\n\nWhen you see yourself piping grep output to sed, think twice to see\nif you can lose one of them. sed -ne 's/^S.//p' perhaps?\n\n>> +\n>> +\tGIT_INDEX_FILE=.git/tmp_index\n>> +\texport GIT_INDEX_FILE\n>> +\tgit update-index --index-info <../current-index-entries\n>> +\tgit update-index --skip-worktree $(cat ../current-sparse-entries)\n\nAre the dances with ls-files and update-index to prepare us for a\npossible future in which we do not use .git/index as the index file,\nor something? IOW, would \n\n\texport GIT_INDEX_FILE=.git/tmp_index &&\n\tcp .git/index \"$GIT_INDEX_FILE &&\n\nbe insufficient?\n\n>> +\n>> +\tgit -c core.untrackedCache=false \"$@\"\n>> +\tret=$?\n>> +\n>> +\trm ../current-index-entries\n>> +\trm $GIT_INDEX_FILE\n>> +\tunset GIT_INDEX_FILE\n>> +\n>> +\treturn $ret\n>> +}\n>\n> This is a clever way to get around the untracked cache deletion.\n>\n> Thanks for adding these extra comparisons! It really does help guarantee\n> that we are doing the right thing in each case.\n\nYes, I think it is a great idea to see tested commands behave the\nsame way with or without the untracked cache.\n\nThanks.\n"},{"id":"394294","messageId":"xmqqy2rj9g5x.fsf@gitster.c.googlers.com","threadId":"52713","inReplyTo":"CABPp-BFXWx_MPL1Z0-X5hHf0N1T_13TEbAhVTKh6-atO1+g90Q@mail.gmail.com","subject":"Re: [PATCH v4 0/7] Avoid multiple recursive calls for same path in read_directory_recursive()","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-03-29T18:20:26Z","receivedAt":"2020-03-29T18:20:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> An update of my own for this series: Based on Felipe's reported\n> bash-completion issue I was modifying commands to try out a number of\n> other things and discovered some cases that can trigger the die(\"git\n> ls-files: internal error - directory entry not superset of prefix\")\n> message from ls-files so there's still some fixes I need to make.\n> Will send an update when I've got it.\n\nThanks. This is uncomfortably exciting ;-)\n\n"},{"id":"394437","messageId":"CABPp-BHQB=cfRtS+xCAZEpk7Y32X=nVr1heAeo+np60054snWg@mail.gmail.com","threadId":"52713","inReplyTo":"xmqq369raut5.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v4 1/7] t7063: more thorough status checking","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-03-31T20:15:49Z","receivedAt":"2020-03-31T20:16:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Mar 29, 2020 at 11:18 AM Junio C Hamano wrote:\n>\n> Derrick Stolee writes:\n>\n> >> +# Ignore_Untracked_Cache, abbreviated to 3 letters because then people can\n> >> +# compare commands side-by-side, e.g.\n> >> +# iuc status --porcelain >expect &&\n> >> +# git status --porcelain >actual &&\n> >> +# test_cmp expect actual\n>\n> ;-)\n>\n> >> +iuc() {\n>\n> Missing SP after \"iuc\".\n\nWill fix.\n\n> >> + git ls-files -s >../current-index-entries\n> >> + git ls-files -t | grep ^S | sed -e s/^S.// >../current-sparse-entries\n>\n> When you see yourself piping grep output to sed, think twice to see\n> if you can lose one of them. sed -ne 's/^S.//p' perhaps?\n\nOoh, thanks. I have to admit that I don't know sed very well. In\nfact, 'sed -e s/pattern/replacement/' was the _only_ piece of sed I\nknew. But the -n flag and p modifier look handy; I think I ran across\nthem in perl before as well.\n\n> >> +\n> >> + GIT_INDEX_FILE=.git/tmp_index\n> >> + export GIT_INDEX_FILE\n> >> + git update-index --index-info <../current-index-entries\n> >> + git update-index --skip-worktree $(cat ../current-sparse-entries)\n>\n> Are the dances with ls-files and update-index to prepare us for a\n> possible future in which we do not use .git/index as the index file,\n> or something? IOW, would\n>\n> export GIT_INDEX_FILE=.git/tmp_index &&\n> cp .git/index \"$GIT_INDEX_FILE &&\n>\n> be insufficient?\n\nI guess it's a matter of perspective. Do we want to compare to how\ngit behaves when there is no untracked cache (as I was trying to\nimplement), or compare to how git behaves when there is an untracked\ncache and git is told to remove it? (The documentation for\ncore.untrackedCache doesn't actually say when\ncore.untrackedCache=false that git will ignore it, just that it will\ndelete the untracked cache when that option is set. Perhaps if we do\ngo the route with your alternative, we at least need to update the\ndocumentation as well and perhaps also audit the code to make sure it\nignores the untracked cache as I'd expect? Or maybe we just need to\nrun two operations, one to delete the untracked cache, and then the\nsecond that we are actually comparing to?)\n\n> >> +\n> >> + git -c core.untrackedCache=false \"$@\"\n> >> + ret=$?\n> >> +\n> >> + rm ../current-index-entries\n> >> + rm $GIT_INDEX_FILE\n> >> + unset GIT_INDEX_FILE\n> >> +\n> >> + return $ret\n> >> +}\n> >\n> > This is a clever way to get around the untracked cache deletion.\n> >\n> > Thanks for adding these extra comparisons! It really does help guarantee\n> > that we are doing the right thing in each case.\n>\n> Yes, I think it is a great idea to see tested commands behave the\n> same way with or without the untracked cache.\n\nThanks.\n"},{"id":"394452","messageId":"38d4d5a46b1ee4cabe0fafc0eea12542577adf49.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 03/12] dir: fix simple typo in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:37Z","receivedAt":"2020-04-01T04:17:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex b460211e614..b505ba747bb 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2174,7 +2174,7 @@ static void add_path_to_appropriate_result_list(struct dir_struct *dir,\n * If 'stop_at_first_file' is specified, 'path_excluded' is returned\n * to signal that a file was found. This is the least significant value that\n * indicates that a file was encountered that does not depend on the order of\n- * whether an untracked or exluded path was encountered first.\n+ * whether an untracked or excluded path was encountered first.\n *\n * Returns the most significant path_treatment value encountered in the scan.\n * If 'stop_at_first_file' is specified, `path_excluded` is the most\n-- \ngitgitgadget\n\n"},{"id":"394453","messageId":"88e9d5d5dbd86123a8af274b44d4d6c4992f37f8.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 02/12] t3000: add more testcases testing a variety of ls-files issues","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:36Z","receivedAt":"2020-04-01T04:17:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThis adds seven new ls-files tests. While currently all seven test\npass, my earlier rounds of restructuring dir.c to replace an exponential\nalgorithm with a linear one passed all the tests in the testsuite but\nfailed six of these seven new tests. Add these tests to increase our\ncase coverage.\n\nSigned-off-by: Elijah Newren \n---\n t/t3000-ls-files-others.sh | 121 +++++++++++++++++++++++++++++++++++++\n 1 file changed, 121 insertions(+)\n\ndiff --git a/t/t3000-ls-files-others.sh b/t/t3000-ls-files-others.sh\nindex 0aefadacb05..ffdfb16f580 100755\n--- a/t/t3000-ls-files-others.sh\n+++ b/t/t3000-ls-files-others.sh\n@@ -91,4 +91,125 @@ test_expect_success SYMLINKS 'ls-files --others with symlinked submodule' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'setup nested pathspec search' '\n+\ttest_create_repo nested &&\n+\t(\n+\t\tcd nested &&\n+\n+\t\tmkdir -p partially_tracked/untracked_dir &&\n+\t\t> partially_tracked/content &&\n+\t\t> partially_tracked/untracked_dir/file &&\n+\n+\t\tmkdir -p untracked/deep &&\n+\t\t> untracked/deep/path &&\n+\t\t> untracked/deep/foo.c &&\n+\n+\t\tgit add partially_tracked/content\n+\t)\n+'\n+\n+test_expect_success 'ls-files -o --directory with single deep dir pathspec' '\n+\t(\n+\t\tcd nested &&\n+\n+\t\tgit ls-files -o --directory untracked/deep/ >actual &&\n+\n+\t\tcat <<-EOF >expect &&\n+\t\tuntracked/deep/\n+\t\tEOF\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ls-files -o --directory with multiple dir pathspecs' '\n+\t(\n+\t\tcd nested &&\n+\n+\t\tgit ls-files -o --directory partially_tracked/ untracked/ >actual &&\n+\n+\t\tcat <<-EOF >expect &&\n+\t\tpartially_tracked/untracked_dir/\n+\t\tuntracked/\n+\t\tEOF\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ls-files -o --directory with mix dir/file pathspecs' '\n+\t(\n+\t\tcd nested &&\n+\n+\t\tgit ls-files -o --directory partially_tracked/ untracked/deep/path >actual &&\n+\n+\t\tcat <<-EOF >expect &&\n+\t\tpartially_tracked/untracked_dir/\n+\t\tuntracked/deep/path\n+\t\tEOF\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ls-files --o --directory with glob filetype match' '\n+\t(\n+\t\tcd nested &&\n+\n+\t\t# globs kinda defeat --directory, but only for that pathspec\n+\t\tgit ls-files --others --directory partially_tracked \"untracked/*.c\" >actual &&\n+\n+\t\tcat <<-EOF >expect &&\n+\t\tpartially_tracked/untracked_dir/\n+\t\tuntracked/deep/foo.c\n+\t\tEOF\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ls-files --o --directory with mix of tracked states' '\n+\t(\n+\t\tcd nested &&\n+\n+\t\t# globs kinda defeat --directory, but only for that pathspec\n+\t\tgit ls-files --others --directory partially_tracked/ \"untracked/?*\" >actual &&\n+\n+\t\tcat <<-EOF >expect &&\n+\t\tpartially_tracked/untracked_dir/\n+\t\tuntracked/deep/\n+\t\tEOF\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ls-files --o --directory with glob filetype match only' '\n+\t(\n+\t\tcd nested &&\n+\n+\t\tgit ls-files --others --directory \"untracked/*.c\" >actual &&\n+\n+\t\tcat <<-EOF >expect &&\n+\t\tuntracked/deep/foo.c\n+\t\tEOF\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'ls-files --o --directory to get immediate paths under one dir only' '\n+\t(\n+\t\tcd nested &&\n+\n+\t\tgit ls-files --others --directory \"untracked/?*\" >actual &&\n+\n+\t\tcat <<-EOF >expect &&\n+\t\tuntracked/deep/\n+\t\tEOF\n+\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \ngitgitgadget\n\n"},{"id":"394455","messageId":"e2704245854fb6f9a2daac9f2bd2ed97ad05b227.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 01/12] t7063: more thorough status checking","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:35Z","receivedAt":"2020-04-01T04:17:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nIt turns out the t7063 has some testcases that even without using the\nuntracked cache cover situations that nothing else in the testsuite\nhandles. Checking the results of\n git status --porcelain\nboth with and without the untracked cache, and comparing both against\nour expected results helped uncover a critical bug in some dir.c\nrestructuring.\n\nUnfortunately, it's not easy to run status and tell it to ignore the\nuntracked cache; the only knob we have is core.untrackedCache=false,\nwhich is used to instruct git to *delete* the untracked cache (which\nmight also ignore the untracked cache when it operates, but that isn't\nspecified in the docs).\n\nCreate a simple helper that will create a clone of the index that is\nmissing the untracked cache bits, and use it to compare that the results\nwith the untracked cache match the results we get without the untracked\ncache.\n\nSigned-off-by: Elijah Newren \n---\n t/t7063-status-untracked-cache.sh | 52 +++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 190ae149cf3..69c39ff2e49 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -30,6 +30,30 @@ status_is_clean() {\n \ttest_must_be_empty ../status.actual\n }\n \n+# Ignore_Untracked_Cache, abbreviated to 3 letters because then people can\n+# compare commands side-by-side, e.g.\n+# iuc status --porcelain >expect &&\n+# git status --porcelain >actual &&\n+# test_cmp expect actual\n+iuc () {\n+\tgit ls-files -s >../current-index-entries\n+\tgit ls-files -t | sed -ne s/^S.//p >../current-sparse-entries\n+\n+\tGIT_INDEX_FILE=.git/tmp_index\n+\texport GIT_INDEX_FILE\n+\tgit update-index --index-info <../current-index-entries\n+\tgit update-index --skip-worktree $(cat ../current-sparse-entries)\n+\n+\tgit -c core.untrackedCache=false \"$@\"\n+\tret=$?\n+\n+\trm ../current-index-entries\n+\trm $GIT_INDEX_FILE\n+\tunset GIT_INDEX_FILE\n+\n+\treturn $ret\n+}\n+\n test_lazy_prereq UNTRACKED_CACHE '\n \t{ git update-index --test-untracked-cache; ret=$?; } &&\n \ttest $ret -ne 1\n@@ -95,6 +119,8 @@ test_expect_success 'status first time (empty cache)' '\n \t: >../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n+\ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n+\ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../actual &&\n \tcat >../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../trace.expect <../trace &&\n \tGIT_TRACE_UNTRACKED_STATS=\"$TRASH_DIRECTORY/trace\" \\\n \tgit status --porcelain >../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n+\ttest_cmp ../status.expect ../status.iuc &&\n \ttest_cmp ../status.expect ../status.actual &&\n \tcat >../trace.expect <../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <../status.actual &&\n+\tiuc status --porcelain >../status.iuc &&\n \tcat >../status.expect <\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex d0f3d660850..3a367683661 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2259,7 +2259,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n \t\t\t\tbreak;\n \t\t\t}\n-\t\t\t/* skip the dir_add_* part */\n+\t\t\t/* skip the add_path_to_appropriate_result_list() */\n \t\t\tcontinue;\n \t\t}\n \n-- \ngitgitgadget\n\n"},{"id":"394456","messageId":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v4.git.git.1585258061.gitgitgadget@gmail.com","subject":"[PATCH v5 00/12] Avoid multiple recursive calls for same path in read_directory_recursive()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:34Z","receivedAt":"2020-04-01T04:17:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series provides some \"modest\" speedups (see commit message for patch\n8), and should allow 'git status --ignored' to complete in a more reasonable\ntimeframe for Martin Melka (see \nhttps://lore.kernel.org/git/CANt4O2L_DZnMqVxZzTBMvr=BTWqB6L0uyORkoN_yMHLmUX7yHw@mail.gmail.com/\n). It also cleans up the fill_directory() code and API, and fixes\nbash-completion for 'git add untracked-dir/'.\n\nChanges since v4:\n\n * cleanups suggested by Junio (patch 1)\n * new testcases that would have displayed multiple bugs with v4 (patch 2)\n * fixed the bugs with v4 (look for LEADING_PATHSPEC in patch 8)\n * fixed ANOTHER exponential slowdown codepath (look for MODE_MATCHING in\n patch 8)\n * make DIR_KEEP_UNTRACKED_CONTENTS less of a weird one-off (patch 9)\n * reduce number of calls to [do_]match_pathspec() (patch 10)\n * fix error-proneness of fill_directory() API (patch 11)\n * fix bash-completion results for 'git add' on an untracked dir (patch 12)\n\nThis is one of those rare patchsets that is absolutely perfect and\nrisk-free. That's right, bask in their glory and the ease of conscience from\nusing such solid stuff. Using this series will even innoculate you from bugs\noutside of dir.c, and ones external to git, and even bugs external to your\ncomputer. It's just that good. Pay no attention to the man behind the\ncurtain, er, I mean the huge warnings in patch 8, er...I mean what warnings?\nThere's no warnings to view, this stuff is solid as can be.\n\nBut if an extra pair of eyes wants to look at commit message in patch 8, or\nat the new patches (2 and 9-12) and opine on how perfect everything looks\nand feels, be my guest.\n\nDerrick Stolee (1):\n dir: refactor treat_directory to clarify control flow\n\nElijah Newren (11):\n t7063: more thorough status checking\n t3000: add more testcases testing a variety of ls-files issues\n dir: fix simple typo in comment\n dir: consolidate treat_path() and treat_one_path()\n dir: fix broken comment\n dir: fix confusion based on variable tense\n dir: replace exponential algorithm with a linear one\n dir: include DIR_KEEP_UNTRACKED_CONTENTS handling in treat_directory()\n dir: replace double pathspec matching with single in treat_directory()\n Fix error-prone fill_directory() API; make it only return matches\n completion: fix 'git add' on paths under an untracked directory\n\n builtin/clean.c | 6 -\n builtin/grep.c | 2 -\n builtin/ls-files.c | 5 +-\n builtin/stash.c | 17 +-\n contrib/completion/git-completion.bash | 2 +-\n dir.c | 422 +++++++++++++++----------\n t/t3000-ls-files-others.sh | 121 +++++++\n t/t7063-status-untracked-cache.sh | 52 +++\n t/t9902-completion.sh | 5 +\n wt-status.c | 6 +-\n 10 files changed, 437 insertions(+), 201 deletions(-)\n\n\nbase-commit: 0cbb60574e741e8255ba457606c4c90898cfc755\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-700%2Fnewren%2Ffill-directory-exponential-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-700/newren/fill-directory-exponential-v5\nPull-Request: https://github.com/git/git/pull/700\n\nRange-diff vs v4:\n\n 1: 752403e339b ! 1: e2704245854 t7063: more thorough status checking\n @@ -11,8 +11,10 @@\n restructuring.\n \n Unfortunately, it's not easy to run status and tell it to ignore the\n - untracked cache; the only knob we have it to instruct it to *delete*\n - (and ignore) the untracked cache.\n + untracked cache; the only knob we have is core.untrackedCache=false,\n + which is used to instruct git to *delete* the untracked cache (which\n + might also ignore the untracked cache when it operates, but that isn't\n + specified in the docs).\n \n Create a simple helper that will create a clone of the index that is\n missing the untracked cache bits, and use it to compare that the results\n @@ -33,9 +35,9 @@\n +# iuc status --porcelain >expect &&\n +# git status --porcelain >actual &&\n +# test_cmp expect actual\n -+iuc() {\n ++iuc () {\n +\tgit ls-files -s >../current-index-entries\n -+\tgit ls-files -t | grep ^S | sed -e s/^S.// >../current-sparse-entries\n ++\tgit ls-files -t | sed -ne s/^S.//p >../current-sparse-entries\n +\n +\tGIT_INDEX_FILE=.git/tmp_index\n +\texport GIT_INDEX_FILE\n -: ----------- > 2: 88e9d5d5dbd t3000: add more testcases testing a variety of ls-files issues\n 2: a4287d690be = 3: 38d4d5a46b1 dir: fix simple typo in comment\n 3: 48f37e5b114 = 4: eeb38a25f3a dir: consolidate treat_path() and treat_one_path()\n 4: b5ad1939379 = 5: 6e29f1f6aec dir: fix broken comment\n 5: 2603c1a9d13 = 6: 62dae938c8f dir: fix confusion based on variable tense\n 6: 576f364329d = 7: 25921cb792e dir: refactor treat_directory to clarify control flow\n 7: e20525429e5 ! 8: b2caa426790 dir: replace exponential algorithm with a linear one\n @@ -187,8 +187,23 @@\n \n -\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n +\t/*\n -+\t * We only need to recurse into untracked/ignored directories if\n -+\t * either of the following bits is set:\n ++\t * If we have a pathspec which could match something _below_ this\n ++\t * directory (e.g. when checking 'subdir/' having a pathspec like\n ++\t * 'subdir/some/deep/path/file' or 'subdir/widget-*.c'), then we\n ++\t * need to recurse.\n ++\t */\n ++\tif (pathspec) {\n ++\t\tint ret = do_match_pathspec(istate, pathspec, dirname, len,\n ++\t\t\t\t\t 0 /* prefix */, NULL /* seen */,\n ++\t\t\t\t\t DO_MATCH_LEADING_PATHSPEC);\n ++\t\tif (ret == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n ++\t\t\treturn path_recurse;\n ++\t}\n ++\n ++\t/*\n ++\t * Other than the path_recurse case immediately above, we only need\n ++\t * to recurse into untracked/ignored directories if either of the\n ++\t * following bits is set:\n +\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n +\t * there are ignored directories below)\n +\t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n @@ -197,6 +212,16 @@\n +\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n \t\treturn excluded ? path_excluded : path_untracked;\n \n ++\t/*\n ++\t * ...and even if DIR_SHOW_IGNORED_TOO is set, we can still avoid\n ++\t * recursing into ignored directories if the path is excluded and\n ++\t * DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n ++\t */\n ++\tif (excluded &&\n ++\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n ++\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n ++\t\treturn path_excluded;\n ++\n +\t/*\n +\t * If we have we don't want to know the all the paths under an\n +\t * untracked or ignored directory, we still need to go into the\n @@ -241,59 +266,52 @@\n +\n +\t/* There are a variety of reasons we may need to fixup the state... */\n +\tif (state == path_excluded) {\n -+\t\tint i;\n -+\n -+\t\t/*\n -+\t\t * When stop_early is set, read_directory_recursive() will\n -+\t\t * never return path_untracked regardless of whether\n -+\t\t * underlying paths were untracked or ignored (because\n -+\t\t * returning early means it excluded some paths, or\n -+\t\t * something like that -- see commit 5aaa7fd39aaf (\"Improve\n -+\t\t * performance of git status --ignored\", 2017-09-18)).\n -+\t\t * However, we're not really concerned with the status of\n -+\t\t * files under the directory, we just wanted to know\n -+\t\t * whether the directory was empty (state == path_none) or\n -+\t\t * not (state == path_excluded), and if not, we'd return\n -+\t\t * our original status based on whether the untracked\n -+\t\t * directory matched an exclusion pattern.\n ++\t\t/* state == path_excluded implies all paths under\n ++\t\t * dirname were ignored...\n ++\t\t *\n ++\t\t * if running e.g. `git status --porcelain --ignored=matching`,\n ++\t\t * then we want to see the subpaths that are ignored.\n ++\t\t *\n ++\t\t * if running e.g. just `git status --porcelain`, then\n ++\t\t * we just want the directory itself to be listed as ignored\n ++\t\t * and not the individual paths underneath.\n +\t\t */\n -+\t\tif (stop_early)\n -+\t\t\tstate = excluded ? path_excluded : path_untracked;\n ++\t\tint want_ignored_subpaths =\n ++\t\t\t((dir->flags & DIR_SHOW_IGNORED_TOO) &&\n ++\t\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING));\n +\n -+\t\telse {\n ++\t\tif (want_ignored_subpaths) {\n +\t\t\t/*\n -+\t\t\t * When\n -+\t\t\t * !stop_early && state == path_excluded\n -+\t\t\t * then all paths under dirname were ignored. For\n -+\t\t\t * this case, git status --porcelain wants to just\n -+\t\t\t * list the directory itself as ignored and not\n -+\t\t\t * list the individual paths underneath. Remove\n -+\t\t\t * the individual paths underneath.\n ++\t\t\t * with --ignored=matching, we want the subpaths\n ++\t\t\t * INSTEAD of the directory itself.\n +\t\t\t */\n ++\t\t\tstate = path_none;\n ++\t\t} else {\n ++\t\t\tint i;\n +\t\t\tfor (i = old_ignored_nr + 1; iignored_nr; ++i)\n -+\t\t\t\tfree(dir->ignored[i]);\n ++\t\t\t\tFREE_AND_NULL(dir->ignored[i]);\n +\t\t\tdir->ignored_nr = old_ignored_nr;\n +\t\t}\n +\t}\n -+\n -+\t/*\n + \n + \t/*\n +-\t * If this is an excluded directory, then we only need to check if\n +-\t * the directory contains any files.\n +\t * If there is nothing under the current directory and we are not\n +\t * hiding empty directories, then we need to report on the\n +\t * untracked or ignored status of the directory itself.\n -+\t */\n + \t */\n +-\treturn read_directory_recursive(dir, istate, dirname, len,\n +-\t\t\t\t\tuntracked, 1, excluded, pathspec);\n +\tif (state == path_none && !(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n +\t\tstate = excluded ? path_excluded : path_untracked;\n - \n - \t/*\n --\t * If this is an excluded directory, then we only need to check if\n --\t * the directory contains any files.\n ++\n ++\t/*\n +\t * We can recurse into untracked directories that don't match any\n +\t * of the given pathspecs when some file underneath the directory\n +\t * might match one of the pathspecs. If so, we should make sure\n +\t * to note that the directory itself did not match.\n - \t */\n --\treturn read_directory_recursive(dir, istate, dirname, len,\n --\t\t\t\t\tuntracked, 1, excluded, pathspec);\n ++\t */\n +\tif (pathspec &&\n +\t !match_pathspec(istate, pathspec, dirname, len,\n +\t\t\t 0 /* prefix */, NULL,\n @@ -316,6 +334,47 @@\n \tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n \t\tstrbuf_addstr(path, cdir->file);\n +@@\n + \t\t\t\t const struct pathspec *pathspec)\n + {\n + \tint has_path_in_index, dtype, excluded;\n +-\tenum path_treatment path_treatment;\n + \n + \tif (!cdir->d_name)\n + \t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n +@@\n + \tdefault:\n + \t\treturn path_none;\n + \tcase DT_DIR:\n +-\t\tstrbuf_addch(path, '/');\n +-\t\tpath_treatment = treat_directory(dir, istate, untracked,\n +-\t\t\t\t\t\t path->buf, path->len,\n +-\t\t\t\t\t\t baselen, excluded, pathspec);\n + \t\t/*\n +-\t\t * If 1) we only want to return directories that\n +-\t\t * match an exclude pattern and 2) this directory does\n +-\t\t * not match an exclude pattern but all of its\n +-\t\t * contents are excluded, then indicate that we should\n +-\t\t * recurse into this directory (instead of marking the\n +-\t\t * directory itself as an ignored path).\n ++\t\t * WARNING: Do not ignore/amend the return value from\n ++\t\t * treat_directory(), and especially do not change it to return\n ++\t\t * path_recurse as that can cause exponential slowdown.\n ++\t\t * Instead, modify treat_directory() to return the right value.\n + \t\t */\n +-\t\tif (!excluded &&\n +-\t\t path_treatment == path_excluded &&\n +-\t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n +-\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n +-\t\t\treturn path_recurse;\n +-\t\treturn path_treatment;\n ++\t\tstrbuf_addch(path, '/');\n ++\t\treturn treat_directory(dir, istate, untracked,\n ++\t\t\t\t path->buf, path->len,\n ++\t\t\t\t baselen, excluded, pathspec);\n + \tcase DT_REG:\n + \tcase DT_LNK:\n + \t\treturn excluded ? path_excluded : path_untracked;\n @@\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n -: ----------- > 9: 08a10869816 dir: include DIR_KEEP_UNTRACKED_CONTENTS handling in treat_directory()\n -: ----------- > 10: cee74871e43 dir: replace double pathspec matching with single in treat_directory()\n -: ----------- > 11: 61d9c9d758e Fix error-prone fill_directory() API; make it only return matches\n -: ----------- > 12: 725adf0a9b8 completion: fix 'git add' on paths under an untracked directory\n\n-- \ngitgitgadget\n"},{"id":"394457","messageId":"eeb38a25f3a14a3032d59ef9c0a60ac585177556.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 04/12] dir: consolidate treat_path() and treat_one_path()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:38Z","receivedAt":"2020-04-01T04:17:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCommit 16e2cfa90993 (\"read_directory(): further split treat_path()\",\n2010-01-08) split treat_one_path() out of treat_path(), because\ntreat_leading_path() would not have access to a dirent but wanted to\nre-use as much of treat_path() as possible. Not re-using all of\ntreat_path() caused other bugs, as noted in commit b9670c1f5e6b (\"dir:\nfix checks on common prefix directory\", 2019-12-19). Finally, in commit\nad6f2157f951 (\"dir: restructure in a way to avoid passing around a\nstruct dirent\", 2020-01-16), dirents were removed from treat_path() and\nother functions entirely.\n\nSince the only reason for splitting these functions was the lack of a\ndirent -- which no longer applies to either function -- and since the\nsplit caused problems in the past resulting in us not using\ntreat_one_path() separately anymore, just undo the split.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 121 ++++++++++++++++++++++++++--------------------------------\n 1 file changed, 55 insertions(+), 66 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex b505ba747bb..d0f3d660850 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1863,21 +1863,65 @@ static int resolve_dtype(int dtype, struct index_state *istate,\n \treturn dtype;\n }\n \n-static enum path_treatment treat_one_path(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec,\n-\t\t\t\t\t int dtype)\n-{\n-\tint exclude;\n-\tint has_path_in_index = !!index_file_exists(istate, path->buf, path->len, ignore_case);\n+static enum path_treatment treat_path_fast(struct dir_struct *dir,\n+\t\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t\t struct index_state *istate,\n+\t\t\t\t\t struct strbuf *path,\n+\t\t\t\t\t int baselen,\n+\t\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tstrbuf_setlen(path, baselen);\n+\tif (!cdir->ucd) {\n+\t\tstrbuf_addstr(path, cdir->file);\n+\t\treturn path_untracked;\n+\t}\n+\tstrbuf_addstr(path, cdir->ucd->name);\n+\t/* treat_one_path() does this before it calls treat_directory() */\n+\tstrbuf_complete(path, '/');\n+\tif (cdir->ucd->check_only)\n+\t\t/*\n+\t\t * check_only is set as a result of treat_directory() getting\n+\t\t * to its bottom. Verify again the same set of directories\n+\t\t * with check_only set.\n+\t\t */\n+\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n+\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n+\t/*\n+\t * We get path_recurse in the first run when\n+\t * directory_exists_in_index() returns index_nonexistent. We\n+\t * are sure that new changes in the index does not impact the\n+\t * outcome. Return now.\n+\t */\n+\treturn path_recurse;\n+}\n+\n+static enum path_treatment treat_path(struct dir_struct *dir,\n+\t\t\t\t struct untracked_cache_dir *untracked,\n+\t\t\t\t struct cached_dir *cdir,\n+\t\t\t\t struct index_state *istate,\n+\t\t\t\t struct strbuf *path,\n+\t\t\t\t int baselen,\n+\t\t\t\t const struct pathspec *pathspec)\n+{\n+\tint has_path_in_index, dtype, exclude;\n \tenum path_treatment path_treatment;\n \n-\tdtype = resolve_dtype(dtype, istate, path->buf, path->len);\n+\tif (!cdir->d_name)\n+\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n+\t\t\t\t baselen, pathspec);\n+\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n+\t\treturn path_none;\n+\tstrbuf_setlen(path, baselen);\n+\tstrbuf_addstr(path, cdir->d_name);\n+\tif (simplify_away(path->buf, path->len, pathspec))\n+\t\treturn path_none;\n+\n+\tdtype = resolve_dtype(cdir->d_type, istate, path->buf, path->len);\n \n \t/* Always exclude indexed files */\n+\thas_path_in_index = !!index_file_exists(istate, path->buf, path->len,\n+\t\t\t\t\t\tignore_case);\n \tif (dtype != DT_DIR && has_path_in_index)\n \t\treturn path_none;\n \n@@ -1942,61 +1986,6 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t}\n }\n \n-static enum path_treatment treat_path_fast(struct dir_struct *dir,\n-\t\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t\t struct index_state *istate,\n-\t\t\t\t\t struct strbuf *path,\n-\t\t\t\t\t int baselen,\n-\t\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tstrbuf_setlen(path, baselen);\n-\tif (!cdir->ucd) {\n-\t\tstrbuf_addstr(path, cdir->file);\n-\t\treturn path_untracked;\n-\t}\n-\tstrbuf_addstr(path, cdir->ucd->name);\n-\t/* treat_one_path() does this before it calls treat_directory() */\n-\tstrbuf_complete(path, '/');\n-\tif (cdir->ucd->check_only)\n-\t\t/*\n-\t\t * check_only is set as a result of treat_directory() getting\n-\t\t * to its bottom. Verify again the same set of directories\n-\t\t * with check_only set.\n-\t\t */\n-\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n-\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n-\t/*\n-\t * We get path_recurse in the first run when\n-\t * directory_exists_in_index() returns index_nonexistent. We\n-\t * are sure that new changes in the index does not impact the\n-\t * outcome. Return now.\n-\t */\n-\treturn path_recurse;\n-}\n-\n-static enum path_treatment treat_path(struct dir_struct *dir,\n-\t\t\t\t struct untracked_cache_dir *untracked,\n-\t\t\t\t struct cached_dir *cdir,\n-\t\t\t\t struct index_state *istate,\n-\t\t\t\t struct strbuf *path,\n-\t\t\t\t int baselen,\n-\t\t\t\t const struct pathspec *pathspec)\n-{\n-\tif (!cdir->d_name)\n-\t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n-\t\t\t\t baselen, pathspec);\n-\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n-\t\treturn path_none;\n-\tstrbuf_setlen(path, baselen);\n-\tstrbuf_addstr(path, cdir->d_name);\n-\tif (simplify_away(path->buf, path->len, pathspec))\n-\t\treturn path_none;\n-\n-\treturn treat_one_path(dir, untracked, istate, path, baselen, pathspec,\n-\t\t\t cdir->d_type);\n-}\n-\n static void add_untracked(struct untracked_cache_dir *dir, const char *name)\n {\n \tif (!dir)\n-- \ngitgitgadget\n\n"},{"id":"394458","messageId":"62dae938c8f21bc01c2e79052827920d99189422.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 06/12] dir: fix confusion based on variable tense","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:40Z","receivedAt":"2020-04-01T04:18:02Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nDespite having contributed several fixes in this area, I have for months\n(years?) assumed that the \"exclude\" variable was a directive; this\ncaused me to think of it as a different mode we operate in and left me\nconfused as I tried to build up a mental model around why we'd need such\na directive. I mostly tried to ignore it while focusing on the pieces I\nwas trying to understand.\n\nThen I finally traced this variable all back to a call to is_excluded(),\nmeaning it was actually functioning as an adjective. In particular, it\nwas a checked property (\"Does this path match a rule in .gitignore?\"),\nrather than a mode passed in from the caller. Change the variable name\nto match the part of speech used by the function called to define it,\nwhich will hopefully make these bits of code slightly clearer to the\nnext reader.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 26 +++++++++++++-------------\n 1 file changed, 13 insertions(+), 13 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 3a367683661..8074e651e6f 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1656,7 +1656,7 @@ static enum exist_status directory_exists_in_index(struct index_state *istate,\n static enum path_treatment treat_directory(struct dir_struct *dir,\n \tstruct index_state *istate,\n \tstruct untracked_cache_dir *untracked,\n-\tconst char *dirname, int len, int baselen, int exclude,\n+\tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n@@ -1679,13 +1679,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\t}\n \t\tif (nested_repo)\n \t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(exclude ? path_excluded : path_untracked));\n+\t\t\t\t(excluded ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n \t\t\tbreak;\n-\t\tif (exclude &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t\t\t(dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n+\t\tif (excluded &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n \n \t\t\t/*\n \t\t\t * This is an excluded directory and we are\n@@ -1713,7 +1713,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t/* This is the \"show_other_directories\" case */\n \n \tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n@@ -1723,7 +1723,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t * the directory contains any files.\n \t */\n \treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, exclude, pathspec);\n+\t\t\t\t\tuntracked, 1, excluded, pathspec);\n }\n \n /*\n@@ -1904,7 +1904,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t int baselen,\n \t\t\t\t const struct pathspec *pathspec)\n {\n-\tint has_path_in_index, dtype, exclude;\n+\tint has_path_in_index, dtype, excluded;\n \tenum path_treatment path_treatment;\n \n \tif (!cdir->d_name)\n@@ -1949,13 +1949,13 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t (directory_exists_in_index(istate, path->buf, path->len) == index_nonexistent))\n \t\treturn path_none;\n \n-\texclude = is_excluded(dir, istate, path->buf, &dtype);\n+\texcluded = is_excluded(dir, istate, path->buf, &dtype);\n \n \t/*\n \t * Excluded? If we don't explicitly want to show\n \t * ignored files, ignore it\n \t */\n-\tif (exclude && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n+\tif (excluded && !(dir->flags & (DIR_SHOW_IGNORED|DIR_SHOW_IGNORED_TOO)))\n \t\treturn path_excluded;\n \n \tswitch (dtype) {\n@@ -1965,7 +1965,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\tstrbuf_addch(path, '/');\n \t\tpath_treatment = treat_directory(dir, istate, untracked,\n \t\t\t\t\t\t path->buf, path->len,\n-\t\t\t\t\t\t baselen, exclude, pathspec);\n+\t\t\t\t\t\t baselen, excluded, pathspec);\n \t\t/*\n \t\t * If 1) we only want to return directories that\n \t\t * match an exclude pattern and 2) this directory does\n@@ -1974,7 +1974,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t * recurse into this directory (instead of marking the\n \t\t * directory itself as an ignored path).\n \t\t */\n-\t\tif (!exclude &&\n+\t\tif (!excluded &&\n \t\t path_treatment == path_excluded &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n@@ -1982,7 +1982,7 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\treturn path_treatment;\n \tcase DT_REG:\n \tcase DT_LNK:\n-\t\treturn exclude ? path_excluded : path_untracked;\n+\t\treturn excluded ? path_excluded : path_untracked;\n \t}\n }\n \n-- \ngitgitgadget\n\n"},{"id":"394459","messageId":"25921cb792eaeece5a20b8e2ff684b25951dc1d5.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 07/12] dir: refactor treat_directory to clarify control flow","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:41Z","receivedAt":"2020-04-01T04:18:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe logic in treat_directory() is handled by a multi-case\nswitch statement, but this switch is very asymmetrical, as\nthe first two cases are simple but the third is more\ncomplicated than the rest of the method. In fact, the third\ncase includes a \"break\" statement that leads to the block\nof code outside the switch statement. That is the only way\nto reach that block, as the switch handles all possible\nvalues from directory_exists_in_index();\n\nExtract the switch statement into a series of \"if\" statements.\nThis simplifies the trivial cases, while clarifying how to\nreach the \"show_other_directories\" case. This is particularly\nimportant as the \"show_other_directories\" case will expand\nin a later change.\n\nHelped-by: Elijah Newren \nSigned-off-by: Derrick Stolee \nSigned-off-by: Elijah Newren \n---\n dir.c | 35 +++++++++++++++++------------------\n 1 file changed, 17 insertions(+), 18 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 8074e651e6f..d9bcb7e19b6 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1660,29 +1660,28 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst struct pathspec *pathspec)\n {\n \tint nested_repo = 0;\n-\n \t/* The \"len-1\" is to strip the final '/' */\n-\tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n-\tcase index_directory:\n-\t\treturn path_recurse;\n+\tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n-\tcase index_gitdir:\n+\tif (status == index_directory)\n+\t\treturn path_recurse;\n+\tif (status == index_gitdir)\n \t\treturn path_none;\n+\tif (status != index_nonexistent)\n+\t\tBUG(\"Unhandled value for directory_exists_in_index: %d\\n\", status);\n \n-\tcase index_nonexistent:\n-\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n-\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n-\t\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\t\tstrbuf_addstr(&sb, dirname);\n-\t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n-\t\t\tstrbuf_release(&sb);\n-\t\t}\n-\t\tif (nested_repo)\n-\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n-\t\t\t\t(excluded ? path_excluded : path_untracked));\n+\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n+\t\t!(dir->flags & DIR_NO_GITLINKS)) {\n+\t\tstruct strbuf sb = STRBUF_INIT;\n+\t\tstrbuf_addstr(&sb, dirname);\n+\t\tnested_repo = is_nonbare_repository_dir(&sb);\n+\t\tstrbuf_release(&sb);\n+\t}\n+\tif (nested_repo)\n+\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n+\t\t\t(excluded ? path_excluded : path_untracked));\n \n-\t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n-\t\t\tbreak;\n+\tif (!(dir->flags & DIR_SHOW_OTHER_DIRECTORIES)) {\n \t\tif (excluded &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n \t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING)) {\n-- \ngitgitgadget\n\n"},{"id":"394460","messageId":"08a108698169e0cad5ea27761b9476515eca1410.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 09/12] dir: include DIR_KEEP_UNTRACKED_CONTENTS handling in treat_directory()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:43Z","receivedAt":"2020-04-01T04:18:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nHandling DIR_KEEP_UNTRACKED_CONTENTS within treat_directory() instead of\nas a post-processing step in read_directory():\n * allows us to directly access and remove the relevant entries instead\n of needing to calculate which ones need to be removed\n * keeps the logic for directory handling in one location (and puts it\n closer the the logic for stripping out extra ignored entries, which\n seems logical).\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 43 +++++++++++++++++++------------------------\n 1 file changed, 19 insertions(+), 24 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 1b3c095b5a4..8be31df58c2 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1665,7 +1665,8 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t * you CAN'T DO BOTH.\n \t */\n \tenum path_treatment state;\n-\tint nested_repo = 0, old_ignored_nr, check_only, stop_early;\n+\tint nested_repo = 0, check_only, stop_early;\n+\tint old_ignored_nr, old_untracked_nr;\n \t/* The \"len-1\" is to strip the final '/' */\n \tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n@@ -1785,9 +1786,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t * --porcelain), without listing the individual ignored files\n \t * underneath. To do so, we'll save the current ignored_nr, and\n \t * pop all the ones added after it if it turns out the entire\n-\t * directory is ignored.\n+\t * directory is ignored. Also, when DIR_SHOW_IGNORED_TOO and\n+\t * !DIR_KEEP_UNTRACKED_CONTENTS then we don't want to show\n+\t * untracked paths so will need to pop all those off the last\n+\t * after we traverse.\n \t */\n \told_ignored_nr = dir->ignored_nr;\n+\told_untracked_nr = dir->nr;\n \n \t/* Actually recurse into dirname now, we'll fixup the state later. */\n \tuntracked = lookup_untracked(dir->untracked, untracked,\n@@ -1825,6 +1830,18 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\t}\n \t}\n \n+\t/*\n+\t * We may need to ignore some of the untracked paths we found while\n+\t * traversing subdirectories.\n+\t */\n+\tif ((dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t !(dir->flags & DIR_KEEP_UNTRACKED_CONTENTS)) {\n+\t\tint i;\n+\t\tfor (i = old_untracked_nr + 1; inr; ++i)\n+\t\t\tFREE_AND_NULL(dir->entries[i]);\n+\t\tdir->nr = old_untracked_nr;\n+\t}\n+\n \t/*\n \t * If there is nothing under the current directory and we are not\n \t * hiding empty directories, then we need to report on the\n@@ -2653,28 +2670,6 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \tQSORT(dir->entries, dir->nr, cmp_dir_entry);\n \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n \n-\t/*\n-\t * If DIR_SHOW_IGNORED_TOO is set, read_directory_recursive() will\n-\t * also pick up untracked contents of untracked dirs; by default\n-\t * we discard these, but given DIR_KEEP_UNTRACKED_CONTENTS we do not.\n-\t */\n-\tif ((dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t\t !(dir->flags & DIR_KEEP_UNTRACKED_CONTENTS)) {\n-\t\tint i, j;\n-\n-\t\t/* remove from dir->entries untracked contents of untracked dirs */\n-\t\tfor (i = j = 0; j < dir->nr; j++) {\n-\t\t\tif (i &&\n-\t\t\t check_dir_entry_contains(dir->entries[i - 1], dir->entries[j])) {\n-\t\t\t\tFREE_AND_NULL(dir->entries[j]);\n-\t\t\t} else {\n-\t\t\t\tdir->entries[i++] = dir->entries[j];\n-\t\t\t}\n-\t\t}\n-\n-\t\tdir->nr = i;\n-\t}\n-\n \ttrace_performance_leave(\"read directory %.*s\", len, path);\n \tif (dir->untracked) {\n \t\tstatic int force_untracked_cache = -1;\n-- \ngitgitgadget\n\n"},{"id":"394461","messageId":"cee74871e43444ee4b85499f52dd58ed0cf07f0f.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 10/12] dir: replace double pathspec matching with single in treat_directory()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:44Z","receivedAt":"2020-04-01T04:18:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ntreat_directory() had a call to both do_match_pathspec() and\nmatch_pathspec(). These calls have migrated through the code somewhat\nsince their introduction, but we don't actually need both. Replace the\ntwo calls with one, and while at it, move the check earlier in order to\nreduce the need for callers of fill_directory() to do post-filtering of\nresults.\n\nThe next patch will address post-filtering more forcefully and provide\nmore relevant history and context.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 38 +++++++++++++++++++-------------------\n 1 file changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 8be31df58c2..a67930dcff6 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1665,6 +1665,7 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t * you CAN'T DO BOTH.\n \t */\n \tenum path_treatment state;\n+\tint matches_how = 0;\n \tint nested_repo = 0, check_only, stop_early;\n \tint old_ignored_nr, old_untracked_nr;\n \t/* The \"len-1\" is to strip the final '/' */\n@@ -1677,6 +1678,22 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tif (status != index_nonexistent)\n \t\tBUG(\"Unhandled value for directory_exists_in_index: %d\\n\", status);\n \n+\t/*\n+\t * We don't want to descend into paths that don't match the necessary\n+\t * patterns. Clearly, if we don't have a pathspec, then we can't check\n+\t * for matching patterns. Also, if (excluded) then we know we matched\n+\t * the exclusion patterns so as an optimization we can skip checking\n+\t * for matching patterns.\n+\t */\n+\tif (pathspec && !excluded) {\n+\t\tmatches_how = do_match_pathspec(istate, pathspec, dirname, len,\n+\t\t\t\t\t\t0 /* prefix */, NULL /* seen */,\n+\t\t\t\t\t\tDO_MATCH_LEADING_PATHSPEC);\n+\t\tif (!matches_how)\n+\t\t\treturn path_none;\n+\t}\n+\n+\n \tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n \t\t!(dir->flags & DIR_NO_GITLINKS)) {\n \t\tstruct strbuf sb = STRBUF_INIT;\n@@ -1724,13 +1741,8 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t * 'subdir/some/deep/path/file' or 'subdir/widget-*.c'), then we\n \t * need to recurse.\n \t */\n-\tif (pathspec) {\n-\t\tint ret = do_match_pathspec(istate, pathspec, dirname, len,\n-\t\t\t\t\t 0 /* prefix */, NULL /* seen */,\n-\t\t\t\t\t DO_MATCH_LEADING_PATHSPEC);\n-\t\tif (ret == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n-\t\t\treturn path_recurse;\n-\t}\n+\tif (matches_how == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n+\t\treturn path_recurse;\n \n \t/*\n \t * Other than the path_recurse case immediately above, we only need\n@@ -1850,18 +1862,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tif (state == path_none && !(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n \t\tstate = excluded ? path_excluded : path_untracked;\n \n-\t/*\n-\t * We can recurse into untracked directories that don't match any\n-\t * of the given pathspecs when some file underneath the directory\n-\t * might match one of the pathspecs. If so, we should make sure\n-\t * to note that the directory itself did not match.\n-\t */\n-\tif (pathspec &&\n-\t !match_pathspec(istate, pathspec, dirname, len,\n-\t\t\t 0 /* prefix */, NULL,\n-\t\t\t 0 /* do NOT special case dirs */))\n-\t\tstate = path_none;\n-\n \treturn state;\n }\n \n-- \ngitgitgadget\n\n"},{"id":"394462","messageId":"61d9c9d758e90c6afdaa1605b191f367650d18f0.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 11/12] Fix error-prone fill_directory() API; make it only return matches","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:45Z","receivedAt":"2020-04-01T04:18:03Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nTraditionally, the expected calling convention for the dir.c API was:\n\n fill_directory(&dir, ..., pathspec)\n foreach entry in dir->entries:\n if (dir_path_match(entry, pathspec))\n process_or_display(entry)\n\nThis may have made sense once upon a time, because the fill_directory() call\ncould use cheap checks to avoid doing full pathspec matching, and an external\ncaller may have wanted to do other post-processing of the results anyway.\nHowever:\n\n * this structure makes it easy for users of the API to get it wrong\n\n * this structure actually makes it harder to understand\n fill_directory() and the functions it uses internally. It has\n tripped me up several times while trying to fix bugs and\n restructure things.\n\n * relying on post-filtering was already found to produce wrong\n results; pathspec matching had to be added internally for multiple\n cases in order to get the right results (see commits 404ebceda01c\n (dir: also check directories for matching pathspecs, 2019-09-17)\n and 89a1f4aaf765 (dir: if our pathspec might match files under a\n dir, recurse into it, 2019-09-17))\n\n * it's bad for performance: fill_directory() already has to do lots\n of checks and knows the subset of cases where it still needs to do\n more checks. Forcing external callers to do full pathspec\n matching means they must re-check _every_ path.\n\nSo, add the pathspec matching within the fill_directory() internals, and\nremove it from external callers.\n\nSigned-off-by: Elijah Newren \n---\n builtin/clean.c | 6 ------\n builtin/grep.c | 2 --\n builtin/ls-files.c | 5 +++--\n builtin/stash.c | 17 +++++------------\n dir.c | 9 ++++++++-\n wt-status.c | 6 ++----\n 6 files changed, 18 insertions(+), 27 deletions(-)\n\ndiff --git a/builtin/clean.c b/builtin/clean.c\nindex 5abf087e7c4..b189b7b4ea0 100644\n--- a/builtin/clean.c\n+++ b/builtin/clean.c\n@@ -989,12 +989,6 @@ int cmd_clean(int argc, const char **argv, const char *prefix)\n \t\tif (!cache_name_is_other(ent->name, ent->len))\n \t\t\tcontinue;\n \n-\t\tif (pathspec.nr)\n-\t\t\tmatches = dir_path_match(&the_index, ent, &pathspec, 0, NULL);\n-\n-\t\tif (pathspec.nr && !matches)\n-\t\t\tcontinue;\n-\n \t\tif (lstat(ent->name, &st))\n \t\t\tdie_errno(\"Cannot lstat '%s'\", ent->name);\n \ndiff --git a/builtin/grep.c b/builtin/grep.c\nindex 50ce8d94612..f3425102999 100644\n--- a/builtin/grep.c\n+++ b/builtin/grep.c\n@@ -691,8 +691,6 @@ static int grep_directory(struct grep_opt *opt, const struct pathspec *pathspec,\n \n \tfill_directory(&dir, opt->repo->index, pathspec);\n \tfor (i = 0; i < dir.nr; i++) {\n-\t\tif (!dir_path_match(opt->repo->index, dir.entries[i], pathspec, 0, NULL))\n-\t\t\tcontinue;\n \t\thit |= grep_file(opt, dir.entries[i]->name);\n \t\tif (hit && opt->status_only)\n \t\t\tbreak;\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex f069a028cea..b87c22ac240 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -128,8 +128,9 @@ static void show_dir_entry(const struct index_state *istate,\n \tif (len > ent->len)\n \t\tdie(\"git ls-files: internal error - directory entry not superset of prefix\");\n \n-\tif (!dir_path_match(istate, ent, &pathspec, len, ps_matched))\n-\t\treturn;\n+\t/* If ps_matches is non-NULL, figure out which pathspec(s) match. */\n+\tif (ps_matched)\n+\t\tdir_path_match(istate, ent, &pathspec, len, ps_matched);\n \n \tfputs(tag, stdout);\n \twrite_eolinfo(istate, NULL, ent->name);\ndiff --git a/builtin/stash.c b/builtin/stash.c\nindex 4ad3adf4ba5..704740b245c 100644\n--- a/builtin/stash.c\n+++ b/builtin/stash.c\n@@ -856,30 +856,23 @@ static int get_untracked_files(const struct pathspec *ps, int include_untracked,\n \t\t\t struct strbuf *untracked_files)\n {\n \tint i;\n-\tint max_len;\n \tint found = 0;\n-\tchar *seen;\n \tstruct dir_struct dir;\n \n \tmemset(&dir, 0, sizeof(dir));\n \tif (include_untracked != INCLUDE_ALL_FILES)\n \t\tsetup_standard_excludes(&dir);\n \n-\tseen = xcalloc(ps->nr, 1);\n-\n-\tmax_len = fill_directory(&dir, the_repository->index, ps);\n+\tfill_directory(&dir, the_repository->index, ps);\n \tfor (i = 0; i < dir.nr; i++) {\n \t\tstruct dir_entry *ent = dir.entries[i];\n-\t\tif (dir_path_match(&the_index, ent, ps, max_len, seen)) {\n-\t\t\tfound++;\n-\t\t\tstrbuf_addstr(untracked_files, ent->name);\n-\t\t\t/* NUL-terminate: will be fed to update-index -z */\n-\t\t\tstrbuf_addch(untracked_files, '\\0');\n-\t\t}\n+\t\tfound++;\n+\t\tstrbuf_addstr(untracked_files, ent->name);\n+\t\t/* NUL-terminate: will be fed to update-index -z */\n+\t\tstrbuf_addch(untracked_files, '\\0');\n \t\tfree(ent);\n \t}\n \n-\tfree(seen);\n \tfree(dir.entries);\n \tfree(dir.ignored);\n \tclear_directory(&dir);\ndiff --git a/dir.c b/dir.c\nindex a67930dcff6..2de64910401 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2117,7 +2117,14 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t baselen, excluded, pathspec);\n \tcase DT_REG:\n \tcase DT_LNK:\n-\t\treturn excluded ? path_excluded : path_untracked;\n+\t\tif (excluded)\n+\t\t\treturn path_excluded;\n+\t\tif (pathspec &&\n+\t\t !do_match_pathspec(istate, pathspec, path->buf, path->len,\n+\t\t\t\t 0 /* prefix */, NULL /* seen */,\n+\t\t\t\t 0 /* flags */))\n+\t\t\treturn path_none;\n+\t\treturn path_untracked;\n \t}\n }\n \ndiff --git a/wt-status.c b/wt-status.c\nindex cc6f94504d9..98dfa6f73f9 100644\n--- a/wt-status.c\n+++ b/wt-status.c\n@@ -722,16 +722,14 @@ static void wt_status_collect_untracked(struct wt_status *s)\n \n \tfor (i = 0; i < dir.nr; i++) {\n \t\tstruct dir_entry *ent = dir.entries[i];\n-\t\tif (index_name_is_other(istate, ent->name, ent->len) &&\n-\t\t dir_path_match(istate, ent, &s->pathspec, 0, NULL))\n+\t\tif (index_name_is_other(istate, ent->name, ent->len))\n \t\t\tstring_list_insert(&s->untracked, ent->name);\n \t\tfree(ent);\n \t}\n \n \tfor (i = 0; i < dir.ignored_nr; i++) {\n \t\tstruct dir_entry *ent = dir.ignored[i];\n-\t\tif (index_name_is_other(istate, ent->name, ent->len) &&\n-\t\t dir_path_match(istate, ent, &s->pathspec, 0, NULL))\n+\t\tif (index_name_is_other(istate, ent->name, ent->len))\n \t\t\tstring_list_insert(&s->ignored, ent->name);\n \t\tfree(ent);\n \t}\n-- \ngitgitgadget\n\n"},{"id":"394464","messageId":"b2caa426790a2cd6a510e5ebf2d182b018f12185.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 08/12] dir: replace exponential algorithm with a linear one","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:42Z","receivedAt":"2020-04-01T04:18:05Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\ndir's read_directory_recursive() naturally operates recursively in order\nto walk the directory tree. Treating of directories is sometimes weird\nbecause there are so many different permutations about how to handle\ndirectories. Some examples:\n\n * 'git ls-files -o --directory' only needs to know that a directory\n itself is untracked; it doesn't need to recurse into it to see what\n is underneath.\n\n * 'git status' needs to recurse into an untracked directory, but only\n to determine whether or not it is empty. If there are no files\n underneath, the directory itself will be omitted from the output.\n If it is not empty, only the directory will be listed.\n\n * 'git status --ignored' needs to recurse into untracked directories\n and report all the ignored entries and then report the directory as\n untracked -- UNLESS all the entries under the directory are\n ignored, in which case we don't print any of the entries under the\n directory and just report the directory itself as ignored. (Note\n that although this forces us to walk all untracked files underneath\n the directory as well, we strip them from the output, except for\n users like 'git clean' who also set DIR_KEEP_TRACKED_CONTENTS.)\n\n * For 'git clean', we may need to recurse into a directory that\n doesn't match any specified pathspecs, if it's possible that there\n is an entry underneath the directory that can match one of the\n pathspecs. In such a case, we need to be careful to omit the\n directory itself from the list of paths (see commit 404ebceda01c\n (\"dir: also check directories for matching pathspecs\", 2019-09-17))\n\nPart of the tension noted above is that the treatment of a directory can\nchange based on the files within it, and based on the various settings\nin dir->flags. Trying to keep this in mind while reading over the code,\nit is easy to think in terms of \"treat_directory() tells us what to do\nwith a directory, and read_directory_recursive() is the thing that\nrecurses\". Since we need to look into a directory to know how to treat\nit, though, it is quite easy to decide to (also) recurse into the\ndirectory from treat_directory() by adding a read_directory_recursive()\ncall. Adding such a call is actually fine, IF we make sure that\nread_directory_recursive() does not also recurse into that same\ndirectory.\n\nUnfortunately, commit df5bcdf83aeb (\"dir: recurse into untracked dirs\nfor ignored files\", 2017-05-18), added exactly such a case to the code,\nmeaning we'd have two calls to read_directory_recursive() for an\nuntracked directory. So, if we had a file named\n one/two/three/four/five/somefile.txt\nand nothing in one/ was tracked, then 'git status --ignored' would\ncall read_directory_recursive() twice on the directory 'one/', and\neach of those would call read_directory_recursive() twice on the\ndirectory 'one/two/', and so on until read_directory_recursive() was\ncalled 2^5 times for 'one/two/three/four/five/'.\n\nAvoid calling read_directory_recursive() twice per level by moving a\nlot of the special logic into treat_directory().\n\nSince dir.c is somewhat complex, extra cruft built up around this over\ntime. While trying to unravel it, I noticed several instances where the\nfirst call to read_directory_recursive() would return e.g.\npath_untracked for some directory and a later one would return e.g.\npath_none, despite the fact that the directory clearly should have been\nconsidered untracked. The code happened to work due to the side-effect\nfrom the first invocation of adding untracked entries to dir->entries;\nthis allowed it to get the correct output despite the supposed override\nin return value by the later call.\n\nI am somewhat concerned that there are still bugs and maybe even\ntestcases with the wrong expectation. I have tried to carefully\ndocument treat_directory() since it becomes more complex after this\nchange (though much of this complexity came from elsewhere that probably\ndeserved better comments to begin with). However, much of my work felt\nmore like a game of whackamole while attempting to make the code match\nthe existing regression tests than an attempt to create an\nimplementation that matched some clear design. That seems wrong to me,\nbut the rules of existing behavior had so many special cases that I had\na hard time coming up with some overarching rules about what correct\nbehavior is for all cases, forcing me to hope that the regression tests\nare correct and sufficient. Such a hope seems likely to be ill-founded,\ngiven my experience with dir.c-related testcases in the last few months:\n\n Examples where the documentation was hard to parse or even just wrong:\n * 3aca58045f4f (git-clean.txt: do not claim we will delete files with\n -n/--dry-run, 2019-09-17)\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n Examples where testcases were declared wrong and changed:\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n * a2b13367fe55 (Revert \"dir.c: make 'git-status --ignored' work within\n leading directories\", 2019-12-10)\n Examples where testcases were clearly inadequate:\n * 502c386ff944 (t7300-clean: demonstrate deleting nested repo with an\n ignored file breakage, 2019-08-25)\n * 7541cc530239 (t7300: add testcases showing failure to clean specified\n pathspecs, 2019-09-17)\n * a5e916c7453b (dir: fix off-by-one error in match_pathspec_item,\n 2019-09-17)\n * 404ebceda01c (dir: also check directories for matching pathspecs,\n 2019-09-17)\n * 09487f2cbad3 (clean: avoid removing untracked files in a nested git\n repository, 2019-09-17)\n * e86bbcf987fa (clean: disambiguate the definition of -d, 2019-09-17)\n * 452efd11fbf6 (t3011: demonstrate directory traversal failures,\n 2019-12-10)\n * b9670c1f5e6b (dir: fix checks on common prefix directory, 2019-12-19)\n Examples where \"correct behavior\" was unclear to everyone:\n https://lore.kernel.org/git/20190905154735.29784-1-newren@gmail.com/\n Other commits of note:\n * 902b90cf42bc (clean: fix theoretical path corruption, 2019-09-17)\n\nHowever, on the positive side, it does make the code much faster. For\nthe following simple shell loop in an empty repository:\n\n for depth in $(seq 10 25)\n do\n dirs=$(for i in $(seq 1 $depth) ; do printf 'dir/' ; done)\n rm -rf dir\n mkdir -p $dirs\n >$dirs/untracked-file\n /usr/bin/time --format=\"$depth: %e\" git status --ignored >/dev/null\n done\n\nI saw the following timings, in seconds (note that the numbers are a\nlittle noisy from run-to-run, but the trend is very clear with every\nrun):\n\n 10: 0.03\n 11: 0.05\n 12: 0.08\n 13: 0.19\n 14: 0.29\n 15: 0.50\n 16: 1.05\n 17: 2.11\n 18: 4.11\n 19: 8.60\n 20: 17.55\n 21: 33.87\n 22: 68.71\n 23: 140.05\n 24: 274.45\n 25: 551.15\n\nFor the above run, using strace I can look for the number of untracked\ndirectories opened and can verify that it matches the expected\n2^($depth+1)-2 (the sum of 2^1 + 2^2 + 2^3 + ... + 2^$depth).\n\nAfter this fix, with strace I can verify that the number of untracked\ndirectories that are opened drops to just $depth, and the timings all\ndrop to 0.00. In fact, it isn't until a depth of 190 nested directories\nthat it sometimes starts reporting a time of 0.01 seconds and doesn't\nconsistently report 0.01 seconds until there are 240 nested directories.\nThe previous code would have taken\n 17.55 * 2^220 / (60*60*24*365) = 9.4 * 10^59 YEARS\nto have completed the 240 nested directories case. It's not often\nthat you get to speed something up by a factor of 3*10^69.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 210 ++++++++++++++++++++++++++++++++++++++++------------------\n 1 file changed, 147 insertions(+), 63 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex d9bcb7e19b6..1b3c095b5a4 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1659,7 +1659,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int excluded,\n \tconst struct pathspec *pathspec)\n {\n-\tint nested_repo = 0;\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n+\tenum path_treatment state;\n+\tint nested_repo = 0, old_ignored_nr, check_only, stop_early;\n \t/* The \"len-1\" is to strip the final '/' */\n \tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n \n@@ -1711,18 +1717,135 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t/* This is the \"show_other_directories\" case */\n \n-\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t/*\n+\t * If we have a pathspec which could match something _below_ this\n+\t * directory (e.g. when checking 'subdir/' having a pathspec like\n+\t * 'subdir/some/deep/path/file' or 'subdir/widget-*.c'), then we\n+\t * need to recurse.\n+\t */\n+\tif (pathspec) {\n+\t\tint ret = do_match_pathspec(istate, pathspec, dirname, len,\n+\t\t\t\t\t 0 /* prefix */, NULL /* seen */,\n+\t\t\t\t\t DO_MATCH_LEADING_PATHSPEC);\n+\t\tif (ret == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n+\t\t\treturn path_recurse;\n+\t}\n+\n+\t/*\n+\t * Other than the path_recurse case immediately above, we only need\n+\t * to recurse into untracked/ignored directories if either of the\n+\t * following bits is set:\n+\t * - DIR_SHOW_IGNORED_TOO (because then we need to determine if\n+\t * there are ignored directories below)\n+\t * - DIR_HIDE_EMPTY_DIRECTORIES (because we have to determine if\n+\t * the directory is empty)\n+\t */\n+\tif (!(dir->flags & (DIR_SHOW_IGNORED_TOO | DIR_HIDE_EMPTY_DIRECTORIES)))\n \t\treturn excluded ? path_excluded : path_untracked;\n \n+\t/*\n+\t * ...and even if DIR_SHOW_IGNORED_TOO is set, we can still avoid\n+\t * recursing into ignored directories if the path is excluded and\n+\t * DIR_SHOW_IGNORED_TOO_MODE_MATCHING is also set.\n+\t */\n+\tif (excluded &&\n+\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n+\t\treturn path_excluded;\n+\n+\t/*\n+\t * If we have we don't want to know the all the paths under an\n+\t * untracked or ignored directory, we still need to go into the\n+\t * directory to determine if it is empty (because an empty directory\n+\t * should be path_none instead of path_excluded or path_untracked).\n+\t */\n+\tcheck_only = ((dir->flags & DIR_HIDE_EMPTY_DIRECTORIES) &&\n+\t\t !(dir->flags & DIR_SHOW_IGNORED_TOO));\n+\n+\t/*\n+\t * However, there's another optimization possible as a subset of\n+\t * check_only, based on the cases we have to consider:\n+\t * A) Directory matches no exclude patterns:\n+\t * * Directory is empty => path_none\n+\t * * Directory has an untracked file under it => path_untracked\n+\t * * Directory has only ignored files under it => path_excluded\n+\t * B) Directory matches an exclude pattern:\n+\t * * Directory is empty => path_none\n+\t * * Directory has an untracked file under it => path_excluded\n+\t * * Directory has only ignored files under it => path_excluded\n+\t * In case A, we can exit as soon as we've found an untracked\n+\t * file but otherwise have to walk all files. In case B, though,\n+\t * we can stop at the first file we find under the directory.\n+\t */\n+\tstop_early = check_only && excluded;\n+\n+\t/*\n+\t * If /every/ file within an untracked directory is ignored, then\n+\t * we want to treat the directory as ignored (for e.g. status\n+\t * --porcelain), without listing the individual ignored files\n+\t * underneath. To do so, we'll save the current ignored_nr, and\n+\t * pop all the ones added after it if it turns out the entire\n+\t * directory is ignored.\n+\t */\n+\told_ignored_nr = dir->ignored_nr;\n+\n+\t/* Actually recurse into dirname now, we'll fixup the state later. */\n \tuntracked = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t dirname + baselen, len - baselen);\n+\tstate = read_directory_recursive(dir, istate, dirname, len, untracked,\n+\t\t\t\t\t check_only, stop_early, pathspec);\n+\n+\t/* There are a variety of reasons we may need to fixup the state... */\n+\tif (state == path_excluded) {\n+\t\t/* state == path_excluded implies all paths under\n+\t\t * dirname were ignored...\n+\t\t *\n+\t\t * if running e.g. `git status --porcelain --ignored=matching`,\n+\t\t * then we want to see the subpaths that are ignored.\n+\t\t *\n+\t\t * if running e.g. just `git status --porcelain`, then\n+\t\t * we just want the directory itself to be listed as ignored\n+\t\t * and not the individual paths underneath.\n+\t\t */\n+\t\tint want_ignored_subpaths =\n+\t\t\t((dir->flags & DIR_SHOW_IGNORED_TOO) &&\n+\t\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING));\n+\n+\t\tif (want_ignored_subpaths) {\n+\t\t\t/*\n+\t\t\t * with --ignored=matching, we want the subpaths\n+\t\t\t * INSTEAD of the directory itself.\n+\t\t\t */\n+\t\t\tstate = path_none;\n+\t\t} else {\n+\t\t\tint i;\n+\t\t\tfor (i = old_ignored_nr + 1; iignored_nr; ++i)\n+\t\t\t\tFREE_AND_NULL(dir->ignored[i]);\n+\t\t\tdir->ignored_nr = old_ignored_nr;\n+\t\t}\n+\t}\n \n \t/*\n-\t * If this is an excluded directory, then we only need to check if\n-\t * the directory contains any files.\n+\t * If there is nothing under the current directory and we are not\n+\t * hiding empty directories, then we need to report on the\n+\t * untracked or ignored status of the directory itself.\n \t */\n-\treturn read_directory_recursive(dir, istate, dirname, len,\n-\t\t\t\t\tuntracked, 1, excluded, pathspec);\n+\tif (state == path_none && !(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\tstate = excluded ? path_excluded : path_untracked;\n+\n+\t/*\n+\t * We can recurse into untracked directories that don't match any\n+\t * of the given pathspecs when some file underneath the directory\n+\t * might match one of the pathspecs. If so, we should make sure\n+\t * to note that the directory itself did not match.\n+\t */\n+\tif (pathspec &&\n+\t !match_pathspec(istate, pathspec, dirname, len,\n+\t\t\t 0 /* prefix */, NULL,\n+\t\t\t 0 /* do NOT special case dirs */))\n+\t\tstate = path_none;\n+\n+\treturn state;\n }\n \n /*\n@@ -1870,6 +1993,11 @@ static enum path_treatment treat_path_fast(struct dir_struct *dir,\n \t\t\t\t\t int baselen,\n \t\t\t\t\t const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING: From this function, you can return path_recurse or you\n+\t * can call read_directory_recursive() (or neither), but\n+\t * you CAN'T DO BOTH.\n+\t */\n \tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n \t\tstrbuf_addstr(path, cdir->file);\n@@ -1904,7 +2032,6 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t const struct pathspec *pathspec)\n {\n \tint has_path_in_index, dtype, excluded;\n-\tenum path_treatment path_treatment;\n \n \tif (!cdir->d_name)\n \t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n@@ -1961,24 +2088,16 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \tdefault:\n \t\treturn path_none;\n \tcase DT_DIR:\n-\t\tstrbuf_addch(path, '/');\n-\t\tpath_treatment = treat_directory(dir, istate, untracked,\n-\t\t\t\t\t\t path->buf, path->len,\n-\t\t\t\t\t\t baselen, excluded, pathspec);\n \t\t/*\n-\t\t * If 1) we only want to return directories that\n-\t\t * match an exclude pattern and 2) this directory does\n-\t\t * not match an exclude pattern but all of its\n-\t\t * contents are excluded, then indicate that we should\n-\t\t * recurse into this directory (instead of marking the\n-\t\t * directory itself as an ignored path).\n+\t\t * WARNING: Do not ignore/amend the return value from\n+\t\t * treat_directory(), and especially do not change it to return\n+\t\t * path_recurse as that can cause exponential slowdown.\n+\t\t * Instead, modify treat_directory() to return the right value.\n \t\t */\n-\t\tif (!excluded &&\n-\t\t path_treatment == path_excluded &&\n-\t\t (dir->flags & DIR_SHOW_IGNORED_TOO) &&\n-\t\t (dir->flags & DIR_SHOW_IGNORED_TOO_MODE_MATCHING))\n-\t\t\treturn path_recurse;\n-\t\treturn path_treatment;\n+\t\tstrbuf_addch(path, '/');\n+\t\treturn treat_directory(dir, istate, untracked,\n+\t\t\t\t path->buf, path->len,\n+\t\t\t\t baselen, excluded, pathspec);\n \tcase DT_REG:\n \tcase DT_LNK:\n \t\treturn excluded ? path_excluded : path_untracked;\n@@ -2175,14 +2294,10 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n \t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in treat_leading_path(). See the commit message for the\n-\t * commit adding this warning as well as the commit preceding it\n-\t * for details.\n+\t * WARNING: Do NOT recurse unless path_recurse is returned from\n+\t * treat_path(). Recursing on any other return value\n+\t * can result in exponential slowdown.\n \t */\n-\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -2204,13 +2319,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tdir_state = state;\n \n \t\t/* recurse into subdir if instructed by treat_path */\n-\t\tif ((state == path_recurse) ||\n-\t\t\t((state == path_untracked) &&\n-\t\t\t (resolve_dtype(cdir.d_type, istate, path.buf, path.len) == DT_DIR) &&\n-\t\t\t ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t (pathspec &&\n-\t\t\t do_match_pathspec(istate, pathspec, path.buf, path.len,\n-\t\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)))) {\n+\t\tif (state == path_recurse) {\n \t\t\tstruct untracked_cache_dir *ud;\n \t\t\tud = lookup_untracked(dir->untracked, untracked,\n \t\t\t\t\t path.buf + baselen,\n@@ -2294,15 +2403,6 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n {\n-\t/*\n-\t * WARNING WARNING WARNING:\n-\t *\n-\t * Any updates to the traversal logic here may need corresponding\n-\t * updates in read_directory_recursive(). See 777b420347 (dir:\n-\t * synchronize treat_leading_path() and read_directory_recursive(),\n-\t * 2019-12-19) and its parent commit for details.\n-\t */\n-\n \tstruct strbuf sb = STRBUF_INIT;\n \tstruct strbuf subdir = STRBUF_INIT;\n \tint prevlen, baselen;\n@@ -2353,23 +2453,7 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tstrbuf_reset(&subdir);\n \t\tstrbuf_add(&subdir, path+prevlen, baselen-prevlen);\n \t\tcdir.d_name = subdir.buf;\n-\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n-\t\t\t\t pathspec);\n-\t\tif (state == path_untracked &&\n-\t\t resolve_dtype(cdir.d_type, istate, sb.buf, sb.len) == DT_DIR &&\n-\t\t (dir->flags & DIR_SHOW_IGNORED_TOO ||\n-\t\t do_match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)) {\n-\t\t\tif (!match_pathspec(istate, pathspec, sb.buf, sb.len,\n-\t\t\t\t\t 0 /* prefix */, NULL,\n-\t\t\t\t\t 0 /* do NOT special case dirs */))\n-\t\t\t\tstate = path_none;\n-\t\t\tadd_path_to_appropriate_result_list(dir, NULL, &cdir,\n-\t\t\t\t\t\t\t istate,\n-\t\t\t\t\t\t\t &sb, baselen,\n-\t\t\t\t\t\t\t pathspec, state);\n-\t\t\tstate = path_recurse;\n-\t\t}\n+\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen, pathspec);\n \n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n-- \ngitgitgadget\n\n"},{"id":"394463","messageId":"725adf0a9b89a5426579b73cc5a3a35999eac9a1.1585714667.git.gitgitgadget@gmail.com","threadId":"52713","inReplyTo":"pull.700.v5.git.git.1585714667.gitgitgadget@gmail.com","subject":"[PATCH v5 12/12] completion: fix 'git add' on paths under an untracked directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-04-01T04:17:46Z","receivedAt":"2020-04-01T04:18:06Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAs reported on the git mailing list, since git-2.25,\n git add untracked-dir/\nhas been tab completing to\n git add untracked-dir/./\n\nThe cause for this was that with commit b9670c1f5e (dir: fix checks on\ncommon prefix directory, 2019-12-19),\n git ls-files -o --directory untracked-dir/\n(or the equivalent `git -C untracked-dir ls-files -o --directory`) began\nreporting\n untracked-dir/\ninstead of listing paths underneath that directory. It may also be\nworth noting that the real command in question was\n git -C untracked-dir ls-files -o --directory '*'\nwhich is equivalent to\n git ls-files -o --directory 'untracked-dir/*'\nwhich behaves the same for the purposes of this issue (the '*' can match\nthe empty string), but becomes relevant for the proposed fix.\n\nAt first, based on the report, I decided to try to view this as a\nregression and tried to find a way to recover the old behavior without\nbreaking other stuff, or at least breaking as little as possible.\nHowever, in the end, I couldn't figure out a way to do it that wouldn't\njust cause lots more problems than it solved. The old behavior was a\nbug:\n * Although older git would avoid cleaning anything with `git clean -f\n .git`, it would wipe out everything under that direcotry with `git\n clean -f .git/`. Despite the difference in command used, this is\n relevant because the exact same change that fixed clean changed the\n behavior of ls-files.\n * Older git would report different results based solely on presence or\n absence of a trailing slash for $SUBDIR in the command `git ls-files\n -o --directory $SUBDIR`.\n * Older git violated the documented behavior of not recursing into\n directories that matched the pathspec when --directory was\n specified.\n * And, after all, commit b9670c1f5e (dir: fix checks on common prefix\n directory, 2019-12-19) didn't overlook this issue; it explicitly\n stated that the behavior of the command was being changed to bring\n it inline with the docs.\n\n(Also, if it helps, despite that commit being merged during the 2.25\nseries, this bug was not reported during the 2.25 cycle, nor even during\nmost of the 2.26 cycle -- it was reported a day before 2.26 was\nreleased. So the impact of the change is at least somewhat small.)\n\nInstead of relying on a bug of ls-files in reporting the wrong content,\nchange the invocation of ls-files used by git-completion to make it grab\npaths one depth deeper. Do this by changing '$DIR/*' (match $DIR/ plus\n0 or more characters) into '$DIR/?*' (match $DIR/ plus 1 or more\ncharacters). Note that the '?' character should not be added when\ntrying to complete a filename (e.g. 'git ls-files -o --directory\n\"merge.c?*\"' would not correctly return \"merge.c\" when such a file\nexists), so we have to make sure to add the '?' character only in cases\nwhere the path specified so far is a directory.\n\nSigned-off-by: Elijah Newren \n---\n contrib/completion/git-completion.bash | 2 +-\n t/t9902-completion.sh | 5 +++++\n 2 files changed, 6 insertions(+), 1 deletion(-)\n\ndiff --git a/contrib/completion/git-completion.bash b/contrib/completion/git-completion.bash\nindex e4d9ff4a95c..1032b642297 100644\n--- a/contrib/completion/git-completion.bash\n+++ b/contrib/completion/git-completion.bash\n@@ -504,7 +504,7 @@ __git_index_files ()\n {\n \tlocal root=\"$2\" match=\"$3\"\n \n-\t__git_ls_files_helper \"$root\" \"$1\" \"$match\" |\n+\t__git_ls_files_helper \"$root\" \"$1\" \"${match:-?}\" |\n \tawk -F / -v pfx=\"${2//\\\\/\\\\\\\\}\" '{\n \t\tpaths[$1] = 1\n \t}\ndiff --git a/t/t9902-completion.sh b/t/t9902-completion.sh\nindex 93877ba9cd6..d9a6425671f 100755\n--- a/t/t9902-completion.sh\n+++ b/t/t9902-completion.sh\n@@ -1581,6 +1581,11 @@ test_expect_success 'complete files' '\n \techo modify > modified &&\n \ttest_completion \"git add \" \"modified\" &&\n \n+\tmkdir -p some/deep &&\n+\ttouch some/deep/path &&\n+\ttest_completion \"git add some/\" \"some/deep\" &&\n+\tgit clean -f some &&\n+\n \ttouch untracked &&\n \n \t: TODO .gitignore should not be here &&\n-- \ngitgitgadget\n"},{"id":"394493","messageId":"977213ee-3d5f-150a-2982-e1083557bce3@gmail.com","threadId":"52713","inReplyTo":"b2caa426790a2cd6a510e5ebf2d182b018f12185.1585714667.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 08/12] dir: replace exponential algorithm with a linear one","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-04-01T13:57:47Z","receivedAt":"2020-04-01T13:57:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 4/1/2020 12:17 AM, Elijah Newren via GitGitGadget wrote:\n> @@ -1659,7 +1659,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> \tconst char *dirname, int len, int baselen, int excluded,\n> \tconst struct pathspec *pathspec)\n> {\n> -\tint nested_repo = 0;\n> +\t/*\n> +\t * WARNING: From this function, you can return path_recurse or you\n> +\t * can call read_directory_recursive() (or neither), but\n> +\t * you CAN'T DO BOTH.\n> +\t */\n> +\tenum path_treatment state;\n> +\tint nested_repo = 0, old_ignored_nr, check_only, stop_early;\n> \t/* The \"len-1\" is to strip the final '/' */\n> \tenum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n> \n> @@ -1711,18 +1717,135 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> \n> \t/* This is the \"show_other_directories\" case */\n> \n> -\tif (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n> +\t/*\n> +\t * If we have a pathspec which could match something _below_ this\n> +\t * directory (e.g. when checking 'subdir/' having a pathspec like\n> +\t * 'subdir/some/deep/path/file' or 'subdir/widget-*.c'), then we\n> +\t * need to recurse.\n\nI was extremely skeptical about this approach due to leading wildcards\nlike \"*.c\" or \"sub*/*.h\" but found this comment inside math_pathspec_item():\n\n\t\t/*\n\t\t * Here is where we would perform a wildmatch to check if\n\t\t * \"name\" can be matched as a directory (or a prefix) against\n\t\t * the pathspec. Since wildmatch doesn't have this capability\n\t\t * at the present we have to punt and say that it is a match,\n\t\t * potentially returning a false positive\n\t\t * The submodules themselves will be able to perform more\n\t\t * accurate matching to determine if the pathspec matches.\n\t\t */\n\t\treturn MATCHED_RECURSIVELY_LEADING_PATHSPEC;\n\nSo it looks like it will return MATCHED_RECURSIVELY_LEADING_PATHSPEC as\nexpected by this block below:\n\n> +\t */\n> +\tif (pathspec) {\n> +\t\tint ret = do_match_pathspec(istate, pathspec, dirname, len,\n> +\t\t\t\t\t 0 /* prefix */, NULL /* seen */,\n> +\t\t\t\t\t DO_MATCH_LEADING_PATHSPEC);\n> +\t\tif (ret == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n> +\t\t\treturn path_recurse;\n> +\t}\n\nI can't say that I fully understand the change to this patch yet, but at\nleast my initial \"THAT CAN'T POSSIBLY WORK!\" reaction has been tempered.\n\nThanks,\n-Stolee\n"},{"id":"394503","messageId":"CABPp-BHOoM+xDJA3pCaymuRrAwPJKOPxU__c7uJGZaZNPc7eaA@mail.gmail.com","threadId":"52713","inReplyTo":"977213ee-3d5f-150a-2982-e1083557bce3@gmail.com","subject":"Re: [PATCH v5 08/12] dir: replace exponential algorithm with a linear one","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-04-01T15:59:02Z","receivedAt":"2020-04-01T15:59:16Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Apr 1, 2020 at 6:57 AM Derrick Stolee wrote:\n>\n> On 4/1/2020 12:17 AM, Elijah Newren via GitGitGadget wrote:\n> > @@ -1659,7 +1659,13 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> > const char *dirname, int len, int baselen, int excluded,\n> > const struct pathspec *pathspec)\n> > {\n> > - int nested_repo = 0;\n> > + /*\n> > + * WARNING: From this function, you can return path_recurse or you\n> > + * can call read_directory_recursive() (or neither), but\n> > + * you CAN'T DO BOTH.\n> > + */\n> > + enum path_treatment state;\n> > + int nested_repo = 0, old_ignored_nr, check_only, stop_early;\n> > /* The \"len-1\" is to strip the final '/' */\n> > enum exist_status status = directory_exists_in_index(istate, dirname, len-1);\n> >\n> > @@ -1711,18 +1717,135 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n> >\n> > /* This is the \"show_other_directories\" case */\n> >\n> > - if (!(dir->flags & DIR_HIDE_EMPTY_DIRECTORIES))\n> > + /*\n> > + * If we have a pathspec which could match something _below_ this\n> > + * directory (e.g. when checking 'subdir/' having a pathspec like\n> > + * 'subdir/some/deep/path/file' or 'subdir/widget-*.c'), then we\n> > + * need to recurse.\n>\n> I was extremely skeptical about this approach due to leading wildcards\n> like \"*.c\" or \"sub*/*.h\" but found this comment inside math_pathspec_item():\n>\n> /*\n> * Here is where we would perform a wildmatch to check if\n> * \"name\" can be matched as a directory (or a prefix) against\n> * the pathspec. Since wildmatch doesn't have this capability\n> * at the present we have to punt and say that it is a match,\n> * potentially returning a false positive\n> * The submodules themselves will be able to perform more\n> * accurate matching to determine if the pathspec matches.\n> */\n> return MATCHED_RECURSIVELY_LEADING_PATHSPEC;\n>\n> So it looks like it will return MATCHED_RECURSIVELY_LEADING_PATHSPEC as\n> expected by this block below:\n>\n> > + */\n> > + if (pathspec) {\n> > + int ret = do_match_pathspec(istate, pathspec, dirname, len,\n> > + 0 /* prefix */, NULL /* seen */,\n> > + DO_MATCH_LEADING_PATHSPEC);\n> > + if (ret == MATCHED_RECURSIVELY_LEADING_PATHSPEC)\n> > + return path_recurse;\n> > + }\n>\n> I can't say that I fully understand the change to this patch yet, but at\n> least my initial \"THAT CAN'T POSSIBLY WORK!\" reaction has been tempered.\n\nI don't know if it helps you feel better about this block or not, but\nit existed (in just slightly modified form) in dir.c before patch 7; I\njust missed it when I was restructuring and thus didn't have it in my\nfirst four rounds of this series. (Funnily enough, I was the one who\nadded this LEADING_PATHSPEC logic to dir.c a while back, and you'd\nthink if I was going to overlook some code when I was restructuring,\nthat it surely couldn't be bits that I had added myself.) So, that\nbasically means that dir.c has been relying on this logic for some\ntime, and I just needed to make sure to include it in this\nrestructuring.\n\nElijah\n"},{"id":"401755","messageId":"87lfjg6mkn.fsf@linux-m68k.org","threadId":"52713","inReplyTo":"61d9c9d758e90c6afdaa1605b191f367650d18f0.1585714667.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v5 11/12] Fix error-prone fill_directory() API; make it only return matches","fromName":"Andreas Schwab","fromEmail":"schwab@linux-m68k.org","sentAt":"2020-07-19T06:33:28Z","receivedAt":"2020-07-19T06:35:31Z","isPatch":true,"sender":{"key":"schwab@linux-m68k.org","avatar":"https://avatars.githubusercontent.com/u/2175493?v=4"},"body":"This breaks git status --ignored.\n\n$ ./git status --porcelain --ignored -- a\n!! abspath.o\n!! add-interactive.o\n!! add-patch.o\n!! advice.o\n!! alias.o\n!! alloc.o\n!! apply.o\n!! archive-tar.o\n!! archive-zip.o\n!! archive.o\n!! argv-array.o\n!! attr.o\n\nAndreas.\n\n-- \nAndreas Schwab, schwab@linux-m68k.org\nGPG Key fingerprint = 7578 EB47 D4E5 4D69 2510 2552 DF73 E780 A9DA AEC1\n\"And now for something completely different.\"\n"},{"id":"401758","messageId":"20200719123919.1802-1-martin.agren@gmail.com","threadId":"52713","inReplyTo":"87lfjg6mkn.fsf@linux-m68k.org","subject":"Re: [PATCH v5 11/12] Fix error-prone fill_directory() API; make it only return matches","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-07-19T12:39:19Z","receivedAt":"2020-07-19T12:40:02Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"\nOn Sun, 19 Jul 2020 at 08:37, Andreas Schwab wrote:\n>\n> This breaks git status --ignored.\n>\n> $ ./git status --porcelain --ignored -- a\n> !! abspath.o\n> !! add-interactive.o\n...\n> !! attr.o\n\nThanks for bisecting. This is 95c11ecc73 (\"Fix error-prone\nfill_directory() API; make it only return matches\", 2020-04-01).\n\nI wonder if the below makes any sense. It seems to fix this usage and\nthe tests pass, but I have no idea what else this might be breaking...\n\nMaybe Elijah has an idea whether this is roughly the right approach?\nLooking at the commit in question (95c11ecc73), there must have been\nsome reason that it injected the pathspec check between the\n\"path_excluded\" and the \"path_untracked\" cases. The diff below\nbasically undoes that split, so I have a feeling I'm missing something.\n\nMartin\n\ndiff --git a/dir.c b/dir.c\nindex 1045cc9c6f..fe64be30ed 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2209,13 +2209,13 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t baselen, excluded, pathspec);\n \tcase DT_REG:\n \tcase DT_LNK:\n-\t\tif (excluded)\n-\t\t\treturn path_excluded;\n \t\tif (pathspec &&\n \t\t !match_pathspec(istate, pathspec, path->buf, path->len,\n \t\t\t\t 0 /* prefix */, NULL /* seen */,\n \t\t\t\t 0 /* is_dir */))\n \t\t\treturn path_none;\n+\t\tif (excluded)\n+\t\t\treturn path_excluded;\n \t\treturn path_untracked;\n \t}\n }\ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex e4cf5484f9..2f9bea9793 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -30,6 +30,31 @@ test_expect_success 'same with gitignore starting with BOM' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'status untracked files --ignored with pathspec (no match)' '\n+\tgit status --porcelain --ignored -- untracked/i >actual &&\n+\ttest_must_be_empty actual &&\n+\tgit status --porcelain --ignored -- untracked/u >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success 'status untracked files --ignored with pathspec (literal match)' '\n+\tgit status --porcelain --ignored -- untracked/ignored >actual &&\n+\techo \"!! untracked/ignored\" >expected &&\n+\ttest_cmp expected actual &&\n+\tgit status --porcelain --ignored -- untracked/uncommitted >actual &&\n+\techo \"?? untracked/uncommitted\" >expected &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'status untracked files --ignored with pathspec (glob match)' '\n+\tgit status --porcelain --ignored -- untracked/i\\* >actual &&\n+\techo \"!! untracked/ignored\" >expected &&\n+\ttest_cmp expected actual &&\n+\tgit status --porcelain --ignored -- untracked/u\\* >actual &&\n+\techo \"?? untracked/uncommitted\" >expected &&\n+\ttest_cmp expected actual\n+'\n+\n cat >expected <<\\EOF\n ?? .gitignore\n ?? actual\n-- \n2.28.0.rc1.7.g31f2d237fa\n\n"},{"id":"401766","messageId":"CABPp-BH6xYb8a5hkV_68vbXhH4kApCxw0WG9oveXOk5zzU==1A@mail.gmail.com","threadId":"52713","inReplyTo":"20200719123919.1802-1-martin.agren@gmail.com","subject":"Re: [PATCH v5 11/12] Fix error-prone fill_directory() API; make it only return matches","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-07-20T15:25:20Z","receivedAt":"2020-07-20T15:25:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi,\n\nOn Sun, Jul 19, 2020 at 5:39 AM Martin Ågren wrote:\n>\n> On Sun, 19 Jul 2020 at 08:37, Andreas Schwab wrote:\n> >\n> > This breaks git status --ignored.\n> >\n> > $ ./git status --porcelain --ignored -- a\n> > !! abspath.o\n> > !! add-interactive.o\n> ...\n> > !! attr.o\n>\n> Thanks for bisecting. This is 95c11ecc73 (\"Fix error-prone\n> fill_directory() API; make it only return matches\", 2020-04-01).\n>\n> I wonder if the below makes any sense. It seems to fix this usage and\n> the tests pass, but I have no idea what else this might be breaking...\n>\n> Maybe Elijah has an idea whether this is roughly the right approach?\n> Looking at the commit in question (95c11ecc73), there must have been\n> some reason that it injected the pathspec check between the\n> \"path_excluded\" and the \"path_untracked\" cases. The diff below\n> basically undoes that split, so I have a feeling I'm missing something.\n\nAwesome, thanks Andreas for the bisected report and Martin for finding\nand fixing the bug. As for the reason that the old patch injected the\npathspec check between the path_excluded and the path_untracked cases,\nthat appears to me to just be \"I'm good at making boneheaded\nmistakes\". Your changes here are the right fix. As a separate\noptimization, we could maybe make simplify_away() a bit more involved\nand have it exclude a few more paths so that fewer make it to this\nfinal check, but that's just optimization work that is separate from\nyour fix here.\n\nReviewed-by: Elijah Newren \n\n> Martin\n>\n> diff --git a/dir.c b/dir.c\n> index 1045cc9c6f..fe64be30ed 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -2209,13 +2209,13 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n> baselen, excluded, pathspec);\n> case DT_REG:\n> case DT_LNK:\n> - if (excluded)\n> - return path_excluded;\n> if (pathspec &&\n> !match_pathspec(istate, pathspec, path->buf, path->len,\n> 0 /* prefix */, NULL /* seen */,\n> 0 /* is_dir */))\n> return path_none;\n> + if (excluded)\n> + return path_excluded;\n> return path_untracked;\n> }\n> }\n> diff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\n> index e4cf5484f9..2f9bea9793 100755\n> --- a/t/t7061-wtstatus-ignore.sh\n> +++ b/t/t7061-wtstatus-ignore.sh\n> @@ -30,6 +30,31 @@ test_expect_success 'same with gitignore starting with BOM' '\n> test_cmp expected actual\n> '\n>\n> +test_expect_success 'status untracked files --ignored with pathspec (no match)' '\n> + git status --porcelain --ignored -- untracked/i >actual &&\n> + test_must_be_empty actual &&\n> + git status --porcelain --ignored -- untracked/u >actual &&\n> + test_must_be_empty actual\n> +'\n> +\n> +test_expect_success 'status untracked files --ignored with pathspec (literal match)' '\n> + git status --porcelain --ignored -- untracked/ignored >actual &&\n> + echo \"!! untracked/ignored\" >expected &&\n> + test_cmp expected actual &&\n> + git status --porcelain --ignored -- untracked/uncommitted >actual &&\n> + echo \"?? untracked/uncommitted\" >expected &&\n> + test_cmp expected actual\n> +'\n> +\n> +test_expect_success 'status untracked files --ignored with pathspec (glob match)' '\n> + git status --porcelain --ignored -- untracked/i\\* >actual &&\n> + echo \"!! untracked/ignored\" >expected &&\n> + test_cmp expected actual &&\n> + git status --porcelain --ignored -- untracked/u\\* >actual &&\n> + echo \"?? untracked/uncommitted\" >expected &&\n> + test_cmp expected actual\n> +'\n> +\n> cat >expected <<\\EOF\n> ?? .gitignore\n> ?? actual\n> --\n> 2.28.0.rc1.7.g31f2d237fa\n>\n"},{"id":"401777","messageId":"20200720184529.22449-1-martin.agren@gmail.com","threadId":"52713","inReplyTo":"CABPp-BH6xYb8a5hkV_68vbXhH4kApCxw0WG9oveXOk5zzU==1A@mail.gmail.com","subject":"[PATCH] dir: check pathspecs before returning `path_excluded`","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-07-20T18:45:29Z","receivedAt":"2020-07-20T18:46:07Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Mon, 20 Jul 2020 at 17:25, Elijah Newren wrote:\n>\n> Awesome, thanks Andreas for the bisected report and Martin for finding\n> and fixing the bug. As for the reason that the old patch injected the\n> pathspec check between the path_excluded and the path_untracked cases,\n> that appears to me to just be \"I'm good at making boneheaded\n> mistakes\". Your changes here are the right fix.\n\nOk, here it is as a proper patch.\n\n> Reviewed-by: Elijah Newren \n\nThanks. I've included your reviewed-by below. The log message is\nobviously new, but the diff is identical to what I posted earlier.\n\nBTW, this bug first appeared in v2.27.0, so this is not a regression\nduring the v2.28.0 cycle.\n\nMartin\n\n-- >8 --\nIn 95c11ecc73 (\"Fix error-prone fill_directory() API; make it only\nreturn matches\", 2020-04-01), we taught `fill_directory()`, or more\nspecifically `treat_path()`, to check against any pathspecs so that we\ncould simplify the callers.\n\nBut in doing so, we added a slightly-to-early return for the \"excluded\"\ncase. We end up not checking the pathspecs, meaning we return\n`path_excluded` when maybe we should return `path_none`. As a result,\n`git status --ignored -- pathspec` might show paths that don't actually\nmatch \"pathspec\".\n\nMove the \"excluded\" check down to after we've checked any pathspecs.\n\nReported-by: Andreas Schwab \nReviewed-by: Elijah Newren \nSigned-off-by: Martin Ågren \n---\n dir.c | 4 ++--\n t/t7061-wtstatus-ignore.sh | 25 +++++++++++++++++++++++++\n 2 files changed, 27 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 1045cc9c6f..fe64be30ed 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2209,13 +2209,13 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t baselen, excluded, pathspec);\n \tcase DT_REG:\n \tcase DT_LNK:\n-\t\tif (excluded)\n-\t\t\treturn path_excluded;\n \t\tif (pathspec &&\n \t\t !match_pathspec(istate, pathspec, path->buf, path->len,\n \t\t\t\t 0 /* prefix */, NULL /* seen */,\n \t\t\t\t 0 /* is_dir */))\n \t\t\treturn path_none;\n+\t\tif (excluded)\n+\t\t\treturn path_excluded;\n \t\treturn path_untracked;\n \t}\n }\ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex e4cf5484f9..2f9bea9793 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -30,6 +30,31 @@ test_expect_success 'same with gitignore starting with BOM' '\n \ttest_cmp expected actual\n '\n \n+test_expect_success 'status untracked files --ignored with pathspec (no match)' '\n+\tgit status --porcelain --ignored -- untracked/i >actual &&\n+\ttest_must_be_empty actual &&\n+\tgit status --porcelain --ignored -- untracked/u >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success 'status untracked files --ignored with pathspec (literal match)' '\n+\tgit status --porcelain --ignored -- untracked/ignored >actual &&\n+\techo \"!! untracked/ignored\" >expected &&\n+\ttest_cmp expected actual &&\n+\tgit status --porcelain --ignored -- untracked/uncommitted >actual &&\n+\techo \"?? untracked/uncommitted\" >expected &&\n+\ttest_cmp expected actual\n+'\n+\n+test_expect_success 'status untracked files --ignored with pathspec (glob match)' '\n+\tgit status --porcelain --ignored -- untracked/i\\* >actual &&\n+\techo \"!! untracked/ignored\" >expected &&\n+\ttest_cmp expected actual &&\n+\tgit status --porcelain --ignored -- untracked/u\\* >actual &&\n+\techo \"?? untracked/uncommitted\" >expected &&\n+\ttest_cmp expected actual\n+'\n+\n cat >expected <<\\EOF\n ?? .gitignore\n ?? actual\n-- \n2.28.0.rc1\n\n"},{"id":"401778","messageId":"CABPp-BESf35g2zyCUZgBdbz4RacbToX+g7N4kwSQUCLq0h0ACQ@mail.gmail.com","threadId":"52713","inReplyTo":"20200720184529.22449-1-martin.agren@gmail.com","subject":"Re: [PATCH] dir: check pathspecs before returning `path_excluded`","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2020-07-20T18:49:13Z","receivedAt":"2020-07-20T18:49:27Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Jul 20, 2020 at 11:46 AM Martin Ågren wrote:\n>\n> On Mon, 20 Jul 2020 at 17:25, Elijah Newren wrote:\n> >\n> > Awesome, thanks Andreas for the bisected report and Martin for finding\n> > and fixing the bug. As for the reason that the old patch injected the\n> > pathspec check between the path_excluded and the path_untracked cases,\n> > that appears to me to just be \"I'm good at making boneheaded\n> > mistakes\". Your changes here are the right fix.\n>\n> Ok, here it is as a proper patch.\n>\n> > Reviewed-by: Elijah Newren \n>\n> Thanks. I've included your reviewed-by below. The log message is\n> obviously new, but the diff is identical to what I posted earlier.\n>\n> BTW, this bug first appeared in v2.27.0, so this is not a regression\n> during the v2.28.0 cycle.\n\nLooks good other than a minor typo in the new commit message.\n\n> Martin\n>\n> -- >8 --\n> In 95c11ecc73 (\"Fix error-prone fill_directory() API; make it only\n> return matches\", 2020-04-01), we taught `fill_directory()`, or more\n> specifically `treat_path()`, to check against any pathspecs so that we\n> could simplify the callers.\n>\n> But in doing so, we added a slightly-to-early return for the \"excluded\"\n\ns/to/too/\n\n> case. We end up not checking the pathspecs, meaning we return\n> `path_excluded` when maybe we should return `path_none`. As a result,\n> `git status --ignored -- pathspec` might show paths that don't actually\n> match \"pathspec\".\n>\n> Move the \"excluded\" check down to after we've checked any pathspecs.\n>\n> Reported-by: Andreas Schwab \n> Reviewed-by: Elijah Newren \n> Signed-off-by: Martin Ågren \n> ---\n> dir.c | 4 ++--\n> t/t7061-wtstatus-ignore.sh | 25 +++++++++++++++++++++++++\n> 2 files changed, 27 insertions(+), 2 deletions(-)\n>\n> diff --git a/dir.c b/dir.c\n> index 1045cc9c6f..fe64be30ed 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -2209,13 +2209,13 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n> baselen, excluded, pathspec);\n> case DT_REG:\n> case DT_LNK:\n> - if (excluded)\n> - return path_excluded;\n> if (pathspec &&\n> !match_pathspec(istate, pathspec, path->buf, path->len,\n> 0 /* prefix */, NULL /* seen */,\n> 0 /* is_dir */))\n> return path_none;\n> + if (excluded)\n> + return path_excluded;\n> return path_untracked;\n> }\n> }\n> diff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\n> index e4cf5484f9..2f9bea9793 100755\n> --- a/t/t7061-wtstatus-ignore.sh\n> +++ b/t/t7061-wtstatus-ignore.sh\n> @@ -30,6 +30,31 @@ test_expect_success 'same with gitignore starting with BOM' '\n> test_cmp expected actual\n> '\n>\n> +test_expect_success 'status untracked files --ignored with pathspec (no match)' '\n> + git status --porcelain --ignored -- untracked/i >actual &&\n> + test_must_be_empty actual &&\n> + git status --porcelain --ignored -- untracked/u >actual &&\n> + test_must_be_empty actual\n> +'\n> +\n> +test_expect_success 'status untracked files --ignored with pathspec (literal match)' '\n> + git status --porcelain --ignored -- untracked/ignored >actual &&\n> + echo \"!! untracked/ignored\" >expected &&\n> + test_cmp expected actual &&\n> + git status --porcelain --ignored -- untracked/uncommitted >actual &&\n> + echo \"?? untracked/uncommitted\" >expected &&\n> + test_cmp expected actual\n> +'\n> +\n> +test_expect_success 'status untracked files --ignored with pathspec (glob match)' '\n> + git status --porcelain --ignored -- untracked/i\\* >actual &&\n> + echo \"!! untracked/ignored\" >expected &&\n> + test_cmp expected actual &&\n> + git status --porcelain --ignored -- untracked/u\\* >actual &&\n> + echo \"?? untracked/uncommitted\" >expected &&\n> + test_cmp expected actual\n> +'\n> +\n> cat >expected <<\\EOF\n> ?? .gitignore\n> ?? actual\n> --\n> 2.28.0.rc1\n>\n"},{"id":"401780","messageId":"CAN0heSo25utsXJj1qQgqz2ByB45xu_5O4uSVOV=TZq=3WU+iPQ@mail.gmail.com","threadId":"52713","inReplyTo":"CABPp-BESf35g2zyCUZgBdbz4RacbToX+g7N4kwSQUCLq0h0ACQ@mail.gmail.com","subject":"Re: [PATCH] dir: check pathspecs before returning `path_excluded`","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-07-20T18:51:26Z","receivedAt":"2020-07-20T18:51:41Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Mon, 20 Jul 2020 at 20:49, Elijah Newren wrote:\n>\n> On Mon, Jul 20, 2020 at 11:46 AM Martin Ågren wrote:\n>\n> Looks good other than a minor typo in the new commit message.\n>\n> > But in doing so, we added a slightly-to-early return for the \"excluded\"\n>\n> s/to/too/\n\nThanks!\n\nMartin\n"},{"id":"401782","messageId":"xmqq365m9fom.fsf@gitster.c.googlers.com","threadId":"52713","inReplyTo":"CABPp-BH6xYb8a5hkV_68vbXhH4kApCxw0WG9oveXOk5zzU==1A@mail.gmail.com","subject":"Re: [PATCH v5 11/12] Fix error-prone fill_directory() API; make it only return matches","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-20T18:58:33Z","receivedAt":"2020-07-20T18:58:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n>> Looking at the commit in question (95c11ecc73), there must have been\n>> some reason that it injected the pathspec check between the\n>> \"path_excluded\" and the \"path_untracked\" cases. The diff below\n>> basically undoes that split, so I have a feeling I'm missing something.\n>\n> Awesome, thanks Andreas for the bisected report and Martin for finding\n> and fixing the bug. As for the reason that the old patch injected the\n> pathspec check between the path_excluded and the path_untracked cases,\n> that appears to me to just be \"I'm good at making boneheaded\n> mistakes\". Your changes here are the right fix. As a separate\n> optimization, we could maybe make simplify_away() a bit more involved\n> and have it exclude a few more paths so that fewer make it to this\n> final check, but that's just optimization work that is separate from\n> your fix here.\n>\n> Reviewed-by: Elijah Newren \n\nThis is in 2.27-rc0, so it is not ultra-urgent to fix it at the tip\nof the current development track, but let's make sure we have a fix\nwith a proper log messasge in a mergeable state (meaning \"have\nalready been cooked in 'next' for a week or two) early in the next\ncycle.\n\nThank you, Andreas, Martin and Elijah, as always.\n"},{"id":"401788","messageId":"xmqqtuy27x2q.fsf@gitster.c.googlers.com","threadId":"52713","inReplyTo":"20200720184529.22449-1-martin.agren@gmail.com","subject":"Re: [PATCH] dir: check pathspecs before returning `path_excluded`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-07-20T20:25:49Z","receivedAt":"2020-07-20T20:25:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Ågren writes:\n\n> In 95c11ecc73 (\"Fix error-prone fill_directory() API; make it only\n> return matches\", 2020-04-01), we taught `fill_directory()`, or more\n> specifically `treat_path()`, to check against any pathspecs so that we\n> could simplify the callers.\n>\n> But in doing so, we added a slightly-to-early return for the \"excluded\"\n> case. We end up not checking the pathspecs, meaning we return\n> `path_excluded` when maybe we should return `path_none`. As a result,\n> `git status --ignored -- pathspec` might show paths that don't actually\n> match \"pathspec\".\n>\n> Move the \"excluded\" check down to after we've checked any pathspecs.\n>\n> Reported-by: Andreas Schwab \n> Reviewed-by: Elijah Newren \n> Signed-off-by: Martin Ågren \n> ---\n\nMakes sense. Thanks. Will queue.\n"}]}