{"thread":{"id":"52419","subject":"[PATCH 0/8] Directory traversal bugs","startedAt":"2019-12-09T20:47:50Z","lastAt":"2019-12-21T22:06:24Z","messageCount":69,"participants":["Elijah Newren via GitGitGadget","Denton Liu","Elijah Newren","Eric Sunshine","Johannes Schindelin","Junio C Hamano","Jeff King"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"387833","messageId":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":null,"subject":"[PATCH 0/8] Directory traversal bugs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-09T20:47:37Z","receivedAt":"2019-12-09T20:47:50Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series fixes multiple fill_directory() bugs, one of them new to 2.24.0\ncoming from en/clean-nested-with-ignored-topic, the rest having been around\nin versions of git going back up to a decade. There is also one testcase it\ndocuments but does not fix; I tracked the code for that testcase far enough\nto determine that fill_directory() and its callees were not at fault; rather\npost-fill_directory() processing done by ls-files eventually calls\ngit_fnmatch() and it incorrectly filters out one of the returned paths. I\nsuspect there's a whole can of wildmatch() worms to be found following that\nthread, so I just opted to document my findings next to that testcase.\n\nSee https://lore.kernel.org/git/87fti15agv.fsf@kyleam.com/ for the report\nspawning this series.\n\nSome comments about notable items in this series:\n\n * Patch 2: Revert a previous patch which fixed status --ignore behavior\n incorrectly and which complicates code that we will need to significantly\n restructure in order to fix all the issues we want to address (patches 6\n & 7 provide the right fix)\n * Patch 4: a fix to my en/clean-nested-with-ignored-topic, fixing the new\n bugs\n * Patches 6&7: the fixes to the old issues (Other patches were adding\n testcases, code cleanups, comment cleanups, etc.)\n\nCC: blees@dcon.de, gitster@pobox.com, kyle@kyleam.com, sxlijin@gmail.com\n\nElijah Newren (8):\n t3011: demonstrate directory traversal failures\n Revert \"dir.c: make 'git-status --ignored' work within leading\n directories\"\n dir: remove stray quote character in comment\n dir: exit before wildcard fall-through if there is no wildcard\n dir: break part of read_directory_recursive() out for reuse\n dir: fix checks on common prefix directory\n dir: synchronize treat_leading_path() and read_directory_recursive()\n dir: consolidate similar code in treat_directory()\n\n dir.c | 174 +++++++++++-----\n ...common-prefixes-and-directory-traversal.sh | 193 ++++++++++++++++++\n t/t7061-wtstatus-ignore.sh | 7 +-\n 3 files changed, 323 insertions(+), 51 deletions(-)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\n\nbase-commit: da72936f544fec5a335e66432610e4cef4430991\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-676%2Fnewren%2Fls-files-bug-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-676/newren/ls-files-bug-v1\nPull-Request: https://github.com/git/git/pull/676\n-- \ngitgitgadget\n"},{"id":"387834","messageId":"bfaf7592ee611f7e9f6fbcf7b1e4ae32f8307548.1575924465.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","subject":"[PATCH 2/8] Revert \"dir.c: make 'git-status --ignored' work within leading directories\"","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-09T20:47:39Z","receivedAt":"2019-12-09T20:47:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCommit be8a84c52669 (\"dir.c: make 'git-status --ignored' work within\nleading directories\", 2013-04-15) noted that\n git status --ignored \nwould not list ignored files and directories within if\n was untracked, and modified the behavior to make it show\nthem. However, it did so via a hack that broke consistency; it would\nshow paths under differently than a simple\n git status --ignored | grep \nwould show them. A correct fix is slightly more involved, and\ncomplicated slightly by this hack, so we revert this commit (but keep\ncorrected versions of the testcases) and will later fix the original\nbug with a subsequent patch.\n\nSome history may be helpful:\n\nA very, very similar case to the commit we are reverting was raised in\ncommit 48ffef966c76 (\"ls-files: fix overeager pathspec optimization\",\n2010-01-08); but it actually went in somewhat the opposite direction. In\nthat commit, it mentioned how\n git ls-files -o --exclude-standard t/\nused to show untracked files under t/ even when t/ was ignored, and then\nchanged the behavior to stop showing untracked files under an ignored\ndirectory. More importantly, this commit considered keeping this\nbehavior but noted that it would be inconsistent with the behavior when\nmultiple pathspecs were specified and thus rejected it.\n\nThe reason for this whole inconsistency when one pathspec is specified\nversus zero or two is because common prefixes of pathspecs are sent\nthrough a different set of checks (in treat_leading_path()) than normal\nfile/directory traversal (those go through read_directory_recursive()\nand treat_path()). As such, for consistency, one needs to check that\nboth codepaths produce the same result.\n\nRevert commit be8a84c526691667fc04a8241d93a3de1de298ab, except instead\nof removing the testcase it added, modify it to check for correct and\nconsistent behavior. A subsequent patch in this series will fix the\ntestcase.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 3 ---\n t/t7061-wtstatus-ignore.sh | 7 +++++--\n 2 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 61f559f980..0dd5266629 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2083,14 +2083,12 @@ static int treat_leading_path(struct dir_struct *dir,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint baselen, rc = 0;\n \tconst char *cp;\n-\tint old_flags = dir->flags;\n \n \twhile (len && path[len - 1] == '/')\n \t\tlen--;\n \tif (!len)\n \t\treturn 1;\n \tbaselen = 0;\n-\tdir->flags &= ~DIR_SHOW_OTHER_DIRECTORIES;\n \twhile (1) {\n \t\tcp = path + baselen + !!baselen;\n \t\tcp = memchr(cp, '/', path + len - cp);\n@@ -2113,7 +2111,6 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t}\n \t}\n \tstrbuf_release(&sb);\n-\tdir->flags = old_flags;\n \treturn rc;\n }\n \ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex 0c394cf995..ded7f97181 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -43,11 +43,14 @@ test_expect_success 'status untracked directory with --ignored -u' '\n \ttest_cmp expected actual\n '\n cat >expected <<\\EOF\n-?? untracked/uncommitted\n+?? untracked/\n !! untracked/ignored\n EOF\n \n-test_expect_success 'status prefixed untracked directory with --ignored' '\n+test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n+\tgit status --porcelain --ignored | grep untracked/ >actual &&\n+\ttest_cmp expected actual &&\n+\n \tgit status --porcelain --ignored untracked/ >actual &&\n \ttest_cmp expected actual\n '\n-- \ngitgitgadget\n\n"},{"id":"387835","messageId":"ea2588e87cd4f42f685eef0aa41da83c8fd7333f.1575924465.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","subject":"[PATCH 3/8] dir: remove stray quote character in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-09T20:47:40Z","receivedAt":"2019-12-09T20:47:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 0dd5266629..5dacacd469 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -373,7 +373,7 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t !ps_strncmp(item, match, name, namelen))\n \t\t\treturn MATCHED_RECURSIVELY_LEADING_PATHSPEC;\n \n-\t\t/* name\" doesn't match up to the first wild character */\n+\t\t/* name doesn't match up to the first wild character */\n \t\tif (item->nowildcard_len < item->len &&\n \t\t ps_strncmp(item, match, name,\n \t\t\t item->nowildcard_len - prefix))\n-- \ngitgitgadget\n\n"},{"id":"387836","messageId":"738d9ae4c9ca0e54502b9ef606e2fc8dba9a94e1.1575924465.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","subject":"[PATCH 5/8] dir: break part of read_directory_recursive() out for reuse","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-09T20:47:42Z","receivedAt":"2019-12-09T20:47:54Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCreate an add_path_to_appropriate_result_list() function from the code\nat the end of read_directory_recursive() so we can use it elsewhere.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 60 ++++++++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 37 insertions(+), 23 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 517a569e10..645b44ea64 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1932,6 +1932,40 @@ static void close_cached_dir(struct cached_dir *cdir)\n \t}\n }\n \n+static void add_path_to_appropriate_result_list(struct dir_struct *dir,\n+\tstruct untracked_cache_dir *untracked,\n+\tstruct cached_dir *cdir,\n+\tstruct index_state *istate,\n+\tstruct strbuf *path,\n+\tint baselen,\n+\tconst struct pathspec *pathspec,\n+\tenum path_treatment state)\n+{\n+\t/* add the path to the appropriate result list */\n+\tswitch (state) {\n+\tcase path_excluded:\n+\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\tdir_add_name(dir, istate, path->buf, path->len);\n+\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n+\t\t\t((dir->flags & DIR_COLLECT_IGNORED) &&\n+\t\t\texclude_matches_pathspec(path->buf, path->len,\n+\t\t\t\t\t\t pathspec)))\n+\t\t\tdir_add_ignored(dir, istate, path->buf, path->len);\n+\t\tbreak;\n+\n+\tcase path_untracked:\n+\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\tbreak;\n+\t\tdir_add_name(dir, istate, path->buf, path->len);\n+\t\tif (cdir->fdir)\n+\t\t\tadd_untracked(untracked, path->buf + baselen);\n+\t\tbreak;\n+\n+\tdefault:\n+\t\tbreak;\n+\t}\n+}\n+\n /*\n * Read a directory tree. We currently ignore anything but\n * directories, regular files and symlinks. That's because git\n@@ -2035,29 +2069,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tcontinue;\n \t\t}\n \n-\t\t/* add the path to the appropriate result list */\n-\t\tswitch (state) {\n-\t\tcase path_excluded:\n-\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n-\t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n-\t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t\t((dir->flags & DIR_COLLECT_IGNORED) &&\n-\t\t\t\texclude_matches_pathspec(path.buf, path.len,\n-\t\t\t\t\t\t\t pathspec)))\n-\t\t\t\tdir_add_ignored(dir, istate, path.buf, path.len);\n-\t\t\tbreak;\n-\n-\t\tcase path_untracked:\n-\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n-\t\t\t\tbreak;\n-\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n-\t\t\tif (cdir.fdir)\n-\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n-\t\t\tbreak;\n-\n-\t\tdefault:\n-\t\t\tbreak;\n-\t\t}\n+\t\tadd_path_to_appropriate_result_list(dir, untracked, &cdir,\n+\t\t\t\t\t\t istate, &path, baselen,\n+\t\t\t\t\t\t pathspec, state);\n \t}\n \tclose_cached_dir(&cdir);\n out:\n-- \ngitgitgadget\n\n"},{"id":"387837","messageId":"4f8bf05d265bdf6c9242e21feeda14ad86dd0a4c.1575924466.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","subject":"[PATCH 7/8] dir: synchronize treat_leading_path() and read_directory_recursive()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-09T20:47:44Z","receivedAt":"2019-12-09T20:47:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nOur optimization to avoid calling into read_directory_recursive() when\nall pathspecs have a common leading directory mean that we need to match\nthe logic that read_directory_recursive() would use if we had just\ncalled it from the root. Since it does more than call treat_path() we\nneed to copy that same logic.\n\nAlternatively, we could try to change treat_path to return path_recurse\nfor an untracked directory under the given special circumstances that\nthis logic checks for, but a simple switch results in many test failures\nsuch as 'git clean -d' not wiping out untracked but empty directories.\nTo work around that, we'd need the caller of treat_path to check for\npath_recurse and sometimes special case it into path_untracked. In\nother words, we'd still have extra logic in both places.\n\nNeeding to duplicate logic like this means it is guaranteed someone will\neventually need to make further changes and forget to update both\nlocations. It is tempting to just nuke the leading_directory special\ncasing to avoid such bugs and simplify the code, but unpack_trees'\nverify_clean_subdirectory() also calls read_directory() and does so with\na non-empty leading path, so I'm hesitant to try to restructure further.\nAdd obnoxious warnings to treat_leading_path() and\nread_directory_recursive() to try to warn people of such problems.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 30 +++++++++++++++++++\n ...common-prefixes-and-directory-traversal.sh | 2 +-\n t/t7061-wtstatus-ignore.sh | 2 +-\n 3 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 9c71a9ac21..bb6e481909 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1990,6 +1990,15 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tstruct untracked_cache_dir *untracked, int check_only,\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING WARNING WARNING:\n+\t *\n+\t * Any updates to the traversal logic here may need corresponding\n+\t * updates in treat_leading_path(). See the commit message for the\n+\t * commit adding this warning as well as the commit preceding it\n+\t * for details.\n+\t */\n+\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -2101,6 +2110,15 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING WARNING WARNING:\n+\t *\n+\t * Any updates to the traversal logic here may need corresponding\n+\t * updates in treat_leading_path(). See the commit message for the\n+\t * commit adding this warning as well as the commit preceding it\n+\t * for details.\n+\t */\n+\n \tstruct strbuf sb = STRBUF_INIT;\n \tint prevlen, baselen;\n \tconst char *cp;\n@@ -2154,6 +2172,18 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tde.d_name[baselen-prevlen] = '\\0';\n \t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n \t\t\t\t pathspec);\n+\t\tif (state == path_untracked &&\n+\t\t get_dtype(cdir.de, istate, sb.buf, sb.len) == DT_DIR &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO ||\n+\t\t do_match_pathspec(istate, pathspec, sb.buf, sb.len,\n+\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)) {\n+\t\t\tadd_path_to_appropriate_result_list(dir, NULL, &cdir,\n+\t\t\t\t\t\t\t istate,\n+\t\t\t\t\t\t\t &sb, baselen,\n+\t\t\t\t\t\t\t pathspec, state);\n+\t\t\tstate = path_recurse;\n+\t\t}\n+\n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n \t\tif (len <= baselen)\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex 0151ea8b6d..d48ee1a320 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -179,7 +179,7 @@ test_expect_success 'git ls-files -o consistent between one or two dirs' '\n \n # ls-files doesn't have a way to request showing both untracked and ignored\n # files at the same time, so use `git status --ignored`\n-test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n+test_expect_success 'git status --ignored shows same files under dir with or without pathspec' '\n \tcat <<-EOF >expect &&\n \t?? an_untracked_dir/\n \t!! an_untracked_dir/ignored\ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex ded7f97181..d8060a42e4 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -47,7 +47,7 @@ cat >expected <<\\EOF\n !! untracked/ignored\n EOF\n \n-test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n+test_expect_success 'status of untracked directory with --ignored works with or without prefix' '\n \tgit status --porcelain --ignored | grep untracked/ >actual &&\n \ttest_cmp expected actual &&\n \n-- \ngitgitgadget\n\n"},{"id":"387838","messageId":"2200bf144af2eab98919dd6ed6313d408b1cba1e.1575924466.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","subject":"[PATCH 8/8] dir: consolidate similar code in treat_directory()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-09T20:47:45Z","receivedAt":"2019-12-09T20:47:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nBoth the DIR_SKIP_NESTED_GIT and DIR_NO_GITLINKS cases were checking for\nwhether a path was actually a nonbare repository. That code could be\nshared, with just the result of how to act differing between the two\ncases.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 18 +++++++-----------\n 1 file changed, 7 insertions(+), 11 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex bb6e481909..04541b798b 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1461,6 +1461,8 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int exclude,\n \tconst struct pathspec *pathspec)\n {\n+\tint nested_repo = 0;\n+\n \t/* The \"len-1\" is to strip the final '/' */\n \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n \tcase index_directory:\n@@ -1470,15 +1472,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_none;\n \n \tcase index_nonexistent:\n-\t\tif (dir->flags & DIR_SKIP_NESTED_GIT) {\n-\t\t\tint nested_repo;\n+\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n+\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n \t\t\tstruct strbuf sb = STRBUF_INIT;\n \t\t\tstrbuf_addstr(&sb, dirname);\n \t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n \t\t\tstrbuf_release(&sb);\n-\t\t\tif (nested_repo)\n-\t\t\t\treturn path_none;\n \t\t}\n+\t\tif (nested_repo)\n+\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n+\t\t\t\t(exclude ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n \t\t\tbreak;\n@@ -1506,13 +1509,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t\t\treturn path_none;\n \t\t}\n-\t\tif (!(dir->flags & DIR_NO_GITLINKS)) {\n-\t\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\t\tstrbuf_addstr(&sb, dirname);\n-\t\t\tif (is_nonbare_repository_dir(&sb))\n-\t\t\t\treturn exclude ? path_excluded : path_untracked;\n-\t\t\tstrbuf_release(&sb);\n-\t\t}\n \t\treturn path_recurse;\n \t}\n \n-- \ngitgitgadget\n"},{"id":"387839","messageId":"b897095136920965111f22840373e03524a8c0ae.1575924465.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","subject":"[PATCH 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-09T20:47:43Z","receivedAt":"2019-12-09T20:47:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany years ago, the directory traversing logic had an optimization that\nwould always recurse into any directory that was a common prefix of all\nthe pathspecs without walking the leading directories to get down to\nthe desired directory. Thus,\n git ls-files -o .git/ # case A\nwould notice that .git/ was a common prefix of all pathspecs (since\nit is the only pathspec listed), and then traverse into it and start\nshowing unknown files under that directory. Unfortunately, .git/ is not\na directory we should be traversing into, which made this optimization\nproblematic. This also affected cases like\n git ls-files -o --exclude-standard t/ # case B\nwhere t/ was in the .gitignore file and thus isn't interesting and\nshouldn't be recursed into. It also affected cases like\n git ls-files -o --directory untracked_dir/ # case C\nwhere untracked_dir/ is indeed untracked and thus interesting, but the\n--directory flag means we only want to show the directory itself, not\nrecurse into it and start listing untracked files below it.\n\nThe case B class of bugs were noted and fixed in commits 16e2cfa90993\n(\"read_directory(): further split treat_path()\", 2010-01-08) and\n48ffef966c76 (\"ls-files: fix overeager pathspec optimization\",\n2010-01-08), with the idea being that we first wanted to check whether\nthe common prefix was interesting. The former patch noted that\ntreat_path() couldn't be used when checking the common prefix because\ntreat_path() requires a dir_entry() and we haven't read any directories\nat the point we are checking the common prefix. So, that patch split\ntreat_one_path() out of treat_path(). The latter patch then created a\nnew treat_leading_path() which duplicated by hand the bits of\ntreat_path() that couldn't be broken out and then called\ntreat_one_path() for the remainder. There were three problems with this\napproach:\n\n * The duplicated logic in treat_leading_path() accidentally missed the\n check for special paths (such as is_dot_or_dotdot and matching\n \".git\"), causing case A types of bugs to continue to be an issue.\n * The treat_leading_path() logic assumed we should traverse into\n anything where path_treatment was not path_none, i.e. it perpetuated\n class C types of bugs.\n * It meant we had split logic that needed to kept in sync, running the\n risk that people introduced new inconsistencies (such as in commit\n be8a84c52669, which we reverted earlier in this series, or in commit\n df5bcdf83ae which we'll fix in a subsequent commit)\n\nFix most these problems by making treat_leading_path() not only loop\nover each leading path component, but calling treat_path() directly on\neach. To do so, we have to create a synthetic dir_entry, but that only\ntakes a few lines. Then, pay attention to the path_treatment result we\nget from treat_path() and don't treat path_excluded, path_untracked, and\npath_recurse all the same as path_recurse.\n\nThis leaves one remaining problem, the new inconsistency from commit\ndf5bcdf83ae. That will be addressed in a subsequent commit.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 54 +++++++++++++++----\n ...common-prefixes-and-directory-traversal.sh | 6 +--\n 2 files changed, 46 insertions(+), 14 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 645b44ea64..9c71a9ac21 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2102,37 +2102,69 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const struct pathspec *pathspec)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint baselen, rc = 0;\n+\tint prevlen, baselen;\n \tconst char *cp;\n+\tstruct cached_dir cdir;\n+\tstruct dirent de;\n+\tenum path_treatment state = path_none;\n+\n+\t/*\n+\t * For each directory component of path, we are going to check whether\n+\t * that path is relevant given the pathspec. For example, if path is\n+\t * foo/bar/baz/\n+\t * then we will ask treat_path() whether we should go into foo, then\n+\t * whether we should go into bar, then whether baz is relevant.\n+\t * Checking each is important because e.g. if path is\n+\t * .git/info/\n+\t * then we need to check .git to know we shouldn't traverse it.\n+\t * If the return from treat_path() is:\n+\t * * path_none, for any path, we return false.\n+\t * * path_recurse, for all path components, we return true\n+\t * * for some intermediate component, we make sure\n+\t * to add that path to the relevant list but return false\n+\t * signifying that we shouldn't recurse into it.\n+\t */\n \n \twhile (len && path[len - 1] == '/')\n \t\tlen--;\n \tif (!len)\n \t\treturn 1;\n+\n+\tmemset(&cdir, 0, sizeof(cdir));\n+\tmemset(&de, 0, sizeof(de));\n+\tcdir.de = &de;\n+\tde.d_type = DT_DIR;\n \tbaselen = 0;\n+\tprevlen = 0;\n \twhile (1) {\n-\t\tcp = path + baselen + !!baselen;\n+\t\tprevlen = baselen + !!baselen;\n+\t\tcp = path + prevlen;\n \t\tcp = memchr(cp, '/', path + len - cp);\n \t\tif (!cp)\n \t\t\tbaselen = len;\n \t\telse\n \t\t\tbaselen = cp - path;\n-\t\tstrbuf_setlen(&sb, 0);\n+\t\tstrbuf_reset(&sb);\n \t\tstrbuf_add(&sb, path, baselen);\n \t\tif (!is_directory(sb.buf))\n \t\t\tbreak;\n-\t\tif (simplify_away(sb.buf, sb.len, pathspec))\n-\t\t\tbreak;\n-\t\tif (treat_one_path(dir, NULL, istate, &sb, baselen, pathspec,\n-\t\t\t\t DT_DIR, NULL) == path_none)\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_add(&sb, path, prevlen);\n+\t\tmemcpy(de.d_name, path+prevlen, baselen-prevlen);\n+\t\tde.d_name[baselen-prevlen] = '\\0';\n+\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n+\t\t\t\t pathspec);\n+\t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n-\t\tif (len <= baselen) {\n-\t\t\trc = 1;\n+\t\tif (len <= baselen)\n \t\t\tbreak; /* finished checking */\n-\t\t}\n \t}\n+\tadd_path_to_appropriate_result_list(dir, NULL, &cdir, istate,\n+\t\t\t\t\t &sb, baselen, pathspec,\n+\t\t\t\t\t state);\n+\n \tstrbuf_release(&sb);\n-\treturn rc;\n+\treturn state == path_recurse;\n }\n \n static const char *get_ident_string(void)\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex d4c06fcd76..0151ea8b6d 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -72,7 +72,7 @@ test_expect_success 'git ls-files -o --directory untracked_dir does not recurse'\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n+test_expect_success 'git ls-files -o --directory untracked_dir/ does not recurse' '\n \techo untracked_dir/ >expect &&\n \tgit ls-files -o --directory untracked_dir/ >actual &&\n \ttest_cmp expect actual\n@@ -84,7 +84,7 @@ test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n+test_expect_success 'git ls-files -o untracked_repo/ does not recurse' '\n \techo untracked_repo/ >expect &&\n \tgit ls-files -o untracked_repo/ >actual &&\n \ttest_cmp expect actual\n@@ -124,7 +124,7 @@ test_expect_success 'git ls-files -o .git shows nothing' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o .git/ shows nothing' '\n+test_expect_success 'git ls-files -o .git/ shows nothing' '\n \t>expect &&\n \tgit ls-files -o .git/ >actual &&\n \ttest_cmp expect actual\n-- \ngitgitgadget\n\n"},{"id":"387840","messageId":"c3220758ab2393ea07f403de04951e32404408b2.1575924465.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","subject":"[PATCH 4/8] dir: exit before wildcard fall-through if there is no wildcard","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-09T20:47:41Z","receivedAt":"2019-12-09T20:47:58Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe DO_MATCH_LEADING_PATHSPEC had a fall-through case for if there was a\nwildcard, noting that we don't yet have enough information to determine\nif a further paths under the current directory might match due to the\npresence of wildcards. But if we have no wildcards in our pathspec,\nthen we shouldn't get to that fall-through case.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 7 +++++++\n t/t3011-common-prefixes-and-directory-traversal.sh | 4 ++--\n 2 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 5dacacd469..517a569e10 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -379,6 +379,13 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t\t item->nowildcard_len - prefix))\n \t\t\treturn 0;\n \n+\t\t/*\n+\t\t * name has no wildcard, and it didn't match as a leading\n+\t\t * pathspec so return.\n+\t\t */\n+\t\tif (item->nowildcard_len == item->len)\n+\t\t\treturn 0;\n+\n \t\t/*\n \t\t * Here is where we would perform a wildmatch to check if\n \t\t * \"name\" can be matched as a directory (or a prefix) against\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex 773d6038d1..d4c06fcd76 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -90,7 +90,7 @@ test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+test_expect_success 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n \techo untracked_dir/empty >expect &&\n \techo untracked_repo/ >>expect &&\n \tgit ls-files -o untracked_dir untracked_repo >actual &&\n@@ -104,7 +104,7 @@ test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses int\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+test_expect_success 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n \techo untracked_dir/ >expect &&\n \techo untracked_repo/ >>expect &&\n \tgit ls-files -o --directory untracked_dir untracked_repo >actual &&\n-- \ngitgitgadget\n\n"},{"id":"387841","messageId":"4b24ba9966d41558cc2382d81e6e76b4375b555c.1575924465.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","subject":"[PATCH 1/8] t3011: demonstrate directory traversal failures","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-09T20:47:38Z","receivedAt":"2019-12-09T20:47:59Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd several tests demonstrating directory traversal failures of various\nsorts in dir.c (and one similar looking test that turns out to be a\ngit_fnmatch bug). A lot of these tests look like near duplicates of\neach other, but an optimization path in dir.c to pre-descend into a\ncommon prefix and the specialized treatment of trailing slashes in dir.c\nmean the tiny differences are sometimes important and potentially cause\ndifferent codepaths to be explored.\n\nOf the 7 failing tests, 2 are new to git-2.24.0 (tweaked by side effects\nof the en/clean-nested-with-ignored-topic); the other 5 also failed\nunder git-2.23.0 and earlier.\n\nSigned-off-by: Elijah Newren \n---\n ...common-prefixes-and-directory-traversal.sh | 193 ++++++++++++++++++\n 1 file changed, 193 insertions(+)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nnew file mode 100755\nindex 0000000000..773d6038d1\n--- /dev/null\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -0,0 +1,193 @@\n+#!/bin/sh\n+\n+test_description='directory traversal handling, especially with common prefixes'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit hello &&\n+\n+\t>empty &&\n+\tmkdir untracked_dir &&\n+\t>untracked_dir/empty &&\n+\tgit init untracked_repo &&\n+\t>untracked_repo/empty &&\n+\n+\techo ignored >.gitignore &&\n+\techo an_ignored_dir/ >>.gitignore &&\n+\tmkdir an_ignored_dir &&\n+\tmkdir an_untracked_dir &&\n+\t>an_ignored_dir/ignored &&\n+\t>an_ignored_dir/untracked &&\n+\t>an_untracked_dir/ignored &&\n+\t>an_untracked_dir/untracked\n+'\n+\n+test_expect_success 'git ls-files -o shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_ignored_dir/ignored\n+\tan_ignored_dir/untracked\n+\tan_untracked_dir/ignored\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --exclude-standard shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --exclude-standard >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+\techo untracked_dir/empty >expect &&\n+\techo untracked_repo/ >>expect &&\n+\tgit ls-files -o untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses into untracked_dir only' '\n+\techo untracked_dir/empty >expect &&\n+\techo untracked_repo/ >>expect &&\n+\tgit ls-files -o untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\techo untracked_repo/ >>expect &&\n+\tgit ls-files -o --directory untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir/ untracked_repo/ does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\techo untracked_repo/ >>expect &&\n+\tgit ls-files -o --directory untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o .git shows nothing' '\n+\t>expect &&\n+\tgit ls-files -o .git >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o .git/ shows nothing' '\n+\t>expect &&\n+\tgit ls-files -o .git/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o untracked_* recurses appropriately' '\n+\tmkdir \"untracked_*\" &&\n+\t>\"untracked_*/empty\" &&\n+\n+\techo \"untracked_*/empty\" >expect &&\n+\techo untracked_dir/empty >>expect &&\n+\techo untracked_repo/ >>expect &&\n+\tgit ls-files -o \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# It turns out fill_directory returns the right paths, but ls-files' post-call\n+# filtering in show_dir_entry() via calling dir_path_match() which ends up\n+# in git_fnmatch() has logic for PATHSPEC_ONESTAR that assumes the pathspec\n+# must match the full path; it doesn't check it for matching a leading\n+# directory.\n+test_expect_failure FUNNYNAMES 'git ls-files -o untracked_*/ recurses appropriately' '\n+\techo \"untracked_*/empty\" >expect &&\n+\techo untracked_dir/empty >>expect &&\n+\techo untracked_repo/ >>expect &&\n+\tgit ls-files -o \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_* does not recurse' '\n+\techo \"untracked_*/\" >expect &&\n+\techo untracked_dir/ >>expect &&\n+\techo untracked_repo/ >>expect &&\n+\tgit ls-files -o --directory \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_*/ does not recurse' '\n+\techo \"untracked_*/\" >expect &&\n+\techo untracked_dir/ >>expect &&\n+\techo untracked_repo/ >>expect &&\n+\tgit ls-files -o --directory \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o consistent between one or two dirs' '\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ an_untracked_dir/ >tmp &&\n+\t! grep ^an_ignored_dir/ tmp >expect &&\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# ls-files doesn't have a way to request showing both untracked and ignored\n+# files at the same time, so use `git status --ignored`\n+test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n+\tcat <<-EOF >expect &&\n+\t?? an_untracked_dir/\n+\t!! an_untracked_dir/ignored\n+\tEOF\n+\tgit status --porcelain --ignored >output &&\n+\tgrep an_untracked_dir output >expect &&\n+\tgit status --porcelain --ignored an_untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"387843","messageId":"20191209210608.GA39599@generichostname","threadId":"52419","inReplyTo":"4b24ba9966d41558cc2382d81e6e76b4375b555c.1575924465.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/8] t3011: demonstrate directory traversal failures","fromName":"Denton Liu","fromEmail":"liu.denton@gmail.com","sentAt":"2019-12-09T21:06:08Z","receivedAt":"2019-12-09T21:05:47Z","isPatch":true,"sender":{"key":"liu.denton@gmail.com","avatar":"https://avatars.githubusercontent.com/u/9620836?v=4"},"body":"Hi Elijah,\n\nOn Mon, Dec 09, 2019 at 08:47:38PM +0000, Elijah Newren via GitGitGadget wrote:\n> From: Elijah Newren \n> \n> Add several tests demonstrating directory traversal failures of various\n> sorts in dir.c (and one similar looking test that turns out to be a\n> git_fnmatch bug). A lot of these tests look like near duplicates of\n> each other, but an optimization path in dir.c to pre-descend into a\n> common prefix and the specialized treatment of trailing slashes in dir.c\n> mean the tiny differences are sometimes important and potentially cause\n> different codepaths to be explored.\n> \n> Of the 7 failing tests, 2 are new to git-2.24.0 (tweaked by side effects\n> of the en/clean-nested-with-ignored-topic); the other 5 also failed\n> under git-2.23.0 and earlier.\n> \n> Signed-off-by: Elijah Newren \n> ---\n> ...common-prefixes-and-directory-traversal.sh | 193 ++++++++++++++++++\n> 1 file changed, 193 insertions(+)\n> create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n> \n> diff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\n> new file mode 100755\n> index 0000000000..773d6038d1\n> --- /dev/null\n> +++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n> @@ -0,0 +1,193 @@\n> +#!/bin/sh\n> +\n> +test_description='directory traversal handling, especially with common prefixes'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'setup' '\n> +\ttest_commit hello &&\n> +\n> +\t>empty &&\n> +\tmkdir untracked_dir &&\n> +\t>untracked_dir/empty &&\n> +\tgit init untracked_repo &&\n> +\t>untracked_repo/empty &&\n> +\n> +\techo ignored >.gitignore &&\n> +\techo an_ignored_dir/ >>.gitignore &&\n\nCould we just do\n\n\tcat <<-EOF >expect\n\tignored\n\tan_ignored_dir/\n\tEOF\n\ninstead? Alternatively, we can use test_write_lines() but since we have\nhere-doc usages below already, let's keep using here-docs here to be\nconsistent. Same comment applies to anywhere in this patch where we have\na >>.\n\n> +\tmkdir an_ignored_dir &&\n> +\tmkdir an_untracked_dir &&\n> +\t>an_ignored_dir/ignored &&\n> +\t>an_ignored_dir/untracked &&\n> +\t>an_untracked_dir/ignored &&\n> +\t>an_untracked_dir/untracked\n> +'\n> +\n> +test_expect_success 'git ls-files -o shows the right entries' '\n> +\tcat <<-EOF >expect &&\n> +\t.gitignore\n> +\tactual\n> +\tan_ignored_dir/ignored\n> +\tan_ignored_dir/untracked\n> +\tan_untracked_dir/ignored\n> +\tan_untracked_dir/untracked\n> +\tempty\n> +\texpect\n> +\tuntracked_dir/empty\n> +\tuntracked_repo/\n> +\tEOF\n> +\tgit ls-files -o >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git ls-files -o --exclude-standard shows the right entries' '\n> +\tcat <<-EOF >expect &&\n> +\t.gitignore\n> +\tactual\n> +\tan_untracked_dir/untracked\n> +\tempty\n> +\texpect\n> +\tuntracked_dir/empty\n> +\tuntracked_repo/\n> +\tEOF\n> +\tgit ls-files -o --exclude-standard >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git ls-files -o untracked_dir recurses' '\n> +\techo untracked_dir/empty >expect &&\n> +\tgit ls-files -o untracked_dir >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git ls-files -o untracked_dir/ recurses' '\n> +\techo untracked_dir/empty >expect &&\n> +\tgit ls-files -o untracked_dir/ >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git ls-files -o --directory untracked_dir does not recurse' '\n> +\techo untracked_dir/ >expect &&\n> +\tgit ls-files -o --directory untracked_dir >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n> +\techo untracked_dir/ >expect &&\n> +\tgit ls-files -o --directory untracked_dir/ >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n> +\techo untracked_repo/ >expect &&\n> +\tgit ls-files -o untracked_repo >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n> +\techo untracked_repo/ >expect &&\n> +\tgit ls-files -o untracked_repo/ >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n> +\techo untracked_dir/empty >expect &&\n> +\techo untracked_repo/ >>expect &&\n> +\tgit ls-files -o untracked_dir untracked_repo >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses into untracked_dir only' '\n> +\techo untracked_dir/empty >expect &&\n> +\techo untracked_repo/ >>expect &&\n> +\tgit ls-files -o untracked_dir/ untracked_repo/ >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n> +\techo untracked_dir/ >expect &&\n> +\techo untracked_repo/ >>expect &&\n> +\tgit ls-files -o --directory untracked_dir untracked_repo >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git ls-files -o --directory untracked_dir/ untracked_repo/ does not recurse' '\n> +\techo untracked_dir/ >expect &&\n> +\techo untracked_repo/ >>expect &&\n> +\tgit ls-files -o --directory untracked_dir/ untracked_repo/ >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git ls-files -o .git shows nothing' '\n> +\t>expect &&\n> +\tgit ls-files -o .git >actual &&\n> +\ttest_cmp expect actual\n> +'\n\nCould we use test_must_be_empty here instead?\n\n> +\n> +test_expect_failure 'git ls-files -o .git/ shows nothing' '\n> +\t>expect &&\n> +\tgit ls-files -o .git/ >actual &&\n> +\ttest_cmp expect actual\n> +'\n\ntest_must_be_empty?\n\nThanks,\n\nDenton\n\n> +\n> +test_expect_success FUNNYNAMES 'git ls-files -o untracked_* recurses appropriately' '\n> +\tmkdir \"untracked_*\" &&\n> +\t>\"untracked_*/empty\" &&\n> +\n> +\techo \"untracked_*/empty\" >expect &&\n> +\techo untracked_dir/empty >>expect &&\n> +\techo untracked_repo/ >>expect &&\n> +\tgit ls-files -o \"untracked_*\" >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +# It turns out fill_directory returns the right paths, but ls-files' post-call\n> +# filtering in show_dir_entry() via calling dir_path_match() which ends up\n> +# in git_fnmatch() has logic for PATHSPEC_ONESTAR that assumes the pathspec\n> +# must match the full path; it doesn't check it for matching a leading\n> +# directory.\n> +test_expect_failure FUNNYNAMES 'git ls-files -o untracked_*/ recurses appropriately' '\n> +\techo \"untracked_*/empty\" >expect &&\n> +\techo untracked_dir/empty >>expect &&\n> +\techo untracked_repo/ >>expect &&\n> +\tgit ls-files -o \"untracked_*/\" >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_* does not recurse' '\n> +\techo \"untracked_*/\" >expect &&\n> +\techo untracked_dir/ >>expect &&\n> +\techo untracked_repo/ >>expect &&\n> +\tgit ls-files -o --directory \"untracked_*\" >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_*/ does not recurse' '\n> +\techo \"untracked_*/\" >expect &&\n> +\techo untracked_dir/ >>expect &&\n> +\techo untracked_repo/ >>expect &&\n> +\tgit ls-files -o --directory \"untracked_*/\" >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_expect_success 'git ls-files -o consistent between one or two dirs' '\n> +\tgit ls-files -o --exclude-standard an_ignored_dir/ an_untracked_dir/ >tmp &&\n> +\t! grep ^an_ignored_dir/ tmp >expect &&\n> +\tgit ls-files -o --exclude-standard an_ignored_dir/ >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +# ls-files doesn't have a way to request showing both untracked and ignored\n> +# files at the same time, so use `git status --ignored`\n> +test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n> +\tcat <<-EOF >expect &&\n> +\t?? an_untracked_dir/\n> +\t!! an_untracked_dir/ignored\n> +\tEOF\n> +\tgit status --porcelain --ignored >output &&\n> +\tgrep an_untracked_dir output >expect &&\n> +\tgit status --porcelain --ignored an_untracked_dir/ >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> +test_done\n> -- \n> gitgitgadget\n> \n"},{"id":"387847","messageId":"20191209213246.GA41680@generichostname","threadId":"52419","inReplyTo":"bfaf7592ee611f7e9f6fbcf7b1e4ae32f8307548.1575924465.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/8] Revert \"dir.c: make 'git-status --ignored' work within leading directories\"","fromName":"Denton Liu","fromEmail":"liu.denton@gmail.com","sentAt":"2019-12-09T21:32:46Z","receivedAt":"2019-12-09T21:32:24Z","isPatch":true,"sender":{"key":"liu.denton@gmail.com","avatar":"https://avatars.githubusercontent.com/u/9620836?v=4"},"body":"Hi Elijah,\n\nOn Mon, Dec 09, 2019 at 08:47:39PM +0000, Elijah Newren via GitGitGadget wrote:\n> diff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\n> index 0c394cf995..ded7f97181 100755\n> --- a/t/t7061-wtstatus-ignore.sh\n> +++ b/t/t7061-wtstatus-ignore.sh\n> @@ -43,11 +43,14 @@ test_expect_success 'status untracked directory with --ignored -u' '\n> \ttest_cmp expected actual\n> '\n> cat >expected <<\\EOF\n> -?? untracked/uncommitted\n> +?? untracked/\n> !! untracked/ignored\n> EOF\n> \n> -test_expect_success 'status prefixed untracked directory with --ignored' '\n> +test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n> +\tgit status --porcelain --ignored | grep untracked/ >actual &&\n\nCan we break this pipe up into two invocations so that we don't have a\ngit command in the upstream of a pipe?\n\nThanks,\n\nDenton\n\nP.S. Perhaps in the future, we (I) could try to extend chainlint so that\nit catches this and git commands in non-assignment command\nsubstitutions... I think that would be pretty nice.\n\n> +\ttest_cmp expected actual &&\n> +\n> \tgit status --porcelain --ignored untracked/ >actual &&\n> \ttest_cmp expected actual\n> '\n> -- \n> gitgitgadget\n> \n"},{"id":"387850","messageId":"CABPp-BGdUjQ8gg-VH+udmwBmW-cfhiE7wX=qkQ2jnCgS6eQJiw@mail.gmail.com","threadId":"52419","inReplyTo":"20191209213246.GA41680@generichostname","subject":"Re: [PATCH 2/8] Revert \"dir.c: make 'git-status --ignored' work within leading directories\"","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-12-09T21:51:27Z","receivedAt":"2019-12-09T21:51:40Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Dec 9, 2019 at 1:32 PM Denton Liu wrote:\n>\n> Hi Elijah,\n>\n> On Mon, Dec 09, 2019 at 08:47:39PM +0000, Elijah Newren via GitGitGadget wrote:\n> > diff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\n> > index 0c394cf995..ded7f97181 100755\n> > --- a/t/t7061-wtstatus-ignore.sh\n> > +++ b/t/t7061-wtstatus-ignore.sh\n> > @@ -43,11 +43,14 @@ test_expect_success 'status untracked directory with --ignored -u' '\n> > test_cmp expected actual\n> > '\n> > cat >expected <<\\EOF\n> > -?? untracked/uncommitted\n> > +?? untracked/\n> > !! untracked/ignored\n> > EOF\n> >\n> > -test_expect_success 'status prefixed untracked directory with --ignored' '\n> > +test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n> > + git status --porcelain --ignored | grep untracked/ >actual &&\n>\n> Can we break this pipe up into two invocations so that we don't have a\n> git command in the upstream of a pipe?\n\nSigh...yeah, I keep doing this. And I'll probably keep doing it if\nsomeone can't chainlint (or pipefail) it. I'll fix it up.\n"},{"id":"387853","messageId":"CAPig+cTfnNRPzYm+TnZ4GJPi52MdOLb3VhH-Sgmu=aA_Mq7fzA@mail.gmail.com","threadId":"52419","inReplyTo":"20191209213246.GA41680@generichostname","subject":"Re: [PATCH 2/8] Revert \"dir.c: make 'git-status --ignored' work within leading directories\"","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2019-12-09T22:09:10Z","receivedAt":"2019-12-09T22:09:25Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Mon, Dec 9, 2019 at 4:32 PM Denton Liu wrote:\n> On Mon, Dec 09, 2019 at 08:47:39PM +0000, Elijah Newren via GitGitGadget wrote:\n> > +test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n> > + git status --porcelain --ignored | grep untracked/ >actual &&\n>\n> Can we break this pipe up into two invocations so that we don't have a\n> git command in the upstream of a pipe?\n>\n> P.S. Perhaps in the future, we (I) could try to extend chainlint so that\n> it catches this and git commands in non-assignment command\n> substitutions... I think that would be pretty nice.\n\nRather than getting mired down in chainlint (which could make your\neyeballs melt), an easier way to catch this sort of thing would be to\nintroduce a new script which checks test scripts for Git\nbest-practices non-conformity, similar to how\nt/check-non-portable-shell.pl checks for non-portable shell\nconstructs. (You could even extend check-non-portable-shell.pl with\nthe functionality, but then the script would no longer be specific to\n\"non-portable shell\", so either renaming it or making a new new script\nis warranted.)\n\nBy the way, I have considered adding a best-practices linting script\nlike this, but it (at least at first) would need to have some sort of\nopt-in or opt-out feature since there (likely) are still so many\ninstances of tests which don't follow best-practices, and it could\ntake a while to \"fix\" them all (and eat up a lot of reviewer time, so\nit should be done in small batches).\n"},{"id":"387906","messageId":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.git.git.1575924465.gitgitgadget@gmail.com","subject":"[PATCH v2 0/8] Directory traversal bugs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-10T20:00:19Z","receivedAt":"2019-12-10T20:00:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series fixes multiple fill_directory() bugs, one of them new to 2.24.0\ncoming from en/clean-nested-with-ignored-topic, the rest having been around\nin versions of git going back up to a decade.\nSee https://lore.kernel.org/git/87fti15agv.fsf@kyleam.com/ for the report\nspawning this series.\n\nChanges since v1:\n\n * Testcase cleanups and tweaks suggested by Denton\n * A tweak to this cover letter so that gitgitgadget will hopefully pick up\n the cc-list. (It apparently needs to be 'Cc' now, instead of 'CC')\n\nElijah Newren (8):\n t3011: demonstrate directory traversal failures\n Revert \"dir.c: make 'git-status --ignored' work within leading\n directories\"\n dir: remove stray quote character in comment\n dir: exit before wildcard fall-through if there is no wildcard\n dir: break part of read_directory_recursive() out for reuse\n dir: fix checks on common prefix directory\n dir: synchronize treat_leading_path() and read_directory_recursive()\n dir: consolidate similar code in treat_directory()\n\n dir.c | 174 +++++++++++----\n ...common-prefixes-and-directory-traversal.sh | 209 ++++++++++++++++++\n t/t7061-wtstatus-ignore.sh | 9 +-\n 3 files changed, 341 insertions(+), 51 deletions(-)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\n\nbase-commit: da72936f544fec5a335e66432610e4cef4430991\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-676%2Fnewren%2Fls-files-bug-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-676/newren/ls-files-bug-v2\nPull-Request: https://github.com/git/git/pull/676\n\nRange-diff vs v1:\n\n 1: 4b24ba9966 ! 1: 6d659b2302 t3011: demonstrate directory traversal failures\n @@ -36,8 +36,10 @@\n +\tgit init untracked_repo &&\n +\t>untracked_repo/empty &&\n +\n -+\techo ignored >.gitignore &&\n -+\techo an_ignored_dir/ >>.gitignore &&\n ++\tcat <<-EOF >.gitignore &&\n ++\tignored\n ++\tan_ignored_dir/\n ++\tEOF\n +\tmkdir an_ignored_dir &&\n +\tmkdir an_untracked_dir &&\n +\t>an_ignored_dir/ignored &&\n @@ -114,52 +116,60 @@\n +'\n +\n +test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n -+\techo untracked_dir/empty >expect &&\n -+\techo untracked_repo/ >>expect &&\n ++\tcat <<-EOF >expect &&\n ++\tuntracked_dir/empty\n ++\tuntracked_repo/\n ++\tEOF\n +\tgit ls-files -o untracked_dir untracked_repo >actual &&\n +\ttest_cmp expect actual\n +'\n +\n +test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses into untracked_dir only' '\n -+\techo untracked_dir/empty >expect &&\n -+\techo untracked_repo/ >>expect &&\n ++\tcat <<-EOF >expect &&\n ++\tuntracked_dir/empty\n ++\tuntracked_repo/\n ++\tEOF\n +\tgit ls-files -o untracked_dir/ untracked_repo/ >actual &&\n +\ttest_cmp expect actual\n +'\n +\n +test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n -+\techo untracked_dir/ >expect &&\n -+\techo untracked_repo/ >>expect &&\n ++\tcat <<-EOF >expect &&\n ++\tuntracked_dir/\n ++\tuntracked_repo/\n ++\tEOF\n +\tgit ls-files -o --directory untracked_dir untracked_repo >actual &&\n +\ttest_cmp expect actual\n +'\n +\n +test_expect_success 'git ls-files -o --directory untracked_dir/ untracked_repo/ does not recurse' '\n -+\techo untracked_dir/ >expect &&\n -+\techo untracked_repo/ >>expect &&\n ++\tcat <<-EOF >expect &&\n ++\tuntracked_dir/\n ++\tuntracked_repo/\n ++\tEOF\n +\tgit ls-files -o --directory untracked_dir/ untracked_repo/ >actual &&\n +\ttest_cmp expect actual\n +'\n +\n +test_expect_success 'git ls-files -o .git shows nothing' '\n -+\t>expect &&\n +\tgit ls-files -o .git >actual &&\n -+\ttest_cmp expect actual\n ++\ttest_must_be_empty actual\n +'\n +\n +test_expect_failure 'git ls-files -o .git/ shows nothing' '\n -+\t>expect &&\n +\tgit ls-files -o .git/ >actual &&\n -+\ttest_cmp expect actual\n ++\ttest_must_be_empty actual\n +'\n +\n +test_expect_success FUNNYNAMES 'git ls-files -o untracked_* recurses appropriately' '\n +\tmkdir \"untracked_*\" &&\n +\t>\"untracked_*/empty\" &&\n +\n -+\techo \"untracked_*/empty\" >expect &&\n -+\techo untracked_dir/empty >>expect &&\n -+\techo untracked_repo/ >>expect &&\n ++\tcat <<-EOF >expect &&\n ++\tuntracked_*/empty\n ++\tuntracked_dir/empty\n ++\tuntracked_repo/\n ++\tEOF\n +\tgit ls-files -o \"untracked_*\" >actual &&\n +\ttest_cmp expect actual\n +'\n @@ -170,25 +180,31 @@\n +# must match the full path; it doesn't check it for matching a leading\n +# directory.\n +test_expect_failure FUNNYNAMES 'git ls-files -o untracked_*/ recurses appropriately' '\n -+\techo \"untracked_*/empty\" >expect &&\n -+\techo untracked_dir/empty >>expect &&\n -+\techo untracked_repo/ >>expect &&\n ++\tcat <<-EOF >expect &&\n ++\tuntracked_*/empty\n ++\tuntracked_dir/empty\n ++\tuntracked_repo/\n ++\tEOF\n +\tgit ls-files -o \"untracked_*/\" >actual &&\n +\ttest_cmp expect actual\n +'\n +\n +test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_* does not recurse' '\n -+\techo \"untracked_*/\" >expect &&\n -+\techo untracked_dir/ >>expect &&\n -+\techo untracked_repo/ >>expect &&\n ++\tcat <<-EOF >expect &&\n ++\tuntracked_*/\n ++\tuntracked_dir/\n ++\tuntracked_repo/\n ++\tEOF\n +\tgit ls-files -o --directory \"untracked_*\" >actual &&\n +\ttest_cmp expect actual\n +'\n +\n +test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_*/ does not recurse' '\n -+\techo \"untracked_*/\" >expect &&\n -+\techo untracked_dir/ >>expect &&\n -+\techo untracked_repo/ >>expect &&\n ++\tcat <<-EOF >expect &&\n ++\tuntracked_*/\n ++\tuntracked_dir/\n ++\tuntracked_repo/\n ++\tEOF\n +\tgit ls-files -o --directory \"untracked_*/\" >actual &&\n +\ttest_cmp expect actual\n +'\n 2: bfaf7592ee ! 2: 79f2b56174 Revert \"dir.c: make 'git-status --ignored' work within leading directories\"\n @@ -83,7 +83,9 @@\n \n -test_expect_success 'status prefixed untracked directory with --ignored' '\n +test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n -+\tgit status --porcelain --ignored | grep untracked/ >actual &&\n ++\tgit status --porcelain --ignored >tmp &&\n ++\tgrep untracked/ tmp >actual &&\n ++\trm tmp &&\n +\ttest_cmp expected actual &&\n +\n \tgit status --porcelain --ignored untracked/ >actual &&\n 3: ea2588e87c = 3: d6f858cab1 dir: remove stray quote character in comment\n 4: c3220758ab ! 4: 8d2d98eec3 dir: exit before wildcard fall-through if there is no wildcard\n @@ -37,15 +37,15 @@\n \n -test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n +test_expect_success 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n - \techo untracked_dir/empty >expect &&\n - \techo untracked_repo/ >>expect &&\n - \tgit ls-files -o untracked_dir untracked_repo >actual &&\n + \tcat <<-EOF >expect &&\n + \tuntracked_dir/empty\n + \tuntracked_repo/\n @@\n \ttest_cmp expect actual\n '\n \n -test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n +test_expect_success 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n - \techo untracked_dir/ >expect &&\n - \techo untracked_repo/ >>expect &&\n - \tgit ls-files -o --directory untracked_dir untracked_repo >actual &&\n + \tcat <<-EOF >expect &&\n + \tuntracked_dir/\n + \tuntracked_repo/\n 5: 738d9ae4c9 = 5: d2f5623bd7 dir: break part of read_directory_recursive() out for reuse\n 6: b897095136 ! 6: 9839aca00a dir: fix checks on common prefix directory\n @@ -164,11 +164,11 @@\n \tgit ls-files -o untracked_repo/ >actual &&\n \ttest_cmp expect actual\n @@\n - \ttest_cmp expect actual\n + \ttest_must_be_empty actual\n '\n \n -test_expect_failure 'git ls-files -o .git/ shows nothing' '\n +test_expect_success 'git ls-files -o .git/ shows nothing' '\n - \t>expect &&\n \tgit ls-files -o .git/ >actual &&\n - \ttest_cmp expect actual\n + \ttest_must_be_empty actual\n + '\n 7: 4f8bf05d26 ! 7: df7f08886a dir: synchronize treat_leading_path() and read_directory_recursive()\n @@ -104,6 +104,6 @@\n \n -test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n +test_expect_success 'status of untracked directory with --ignored works with or without prefix' '\n - \tgit status --porcelain --ignored | grep untracked/ >actual &&\n - \ttest_cmp expected actual &&\n - \n + \tgit status --porcelain --ignored >tmp &&\n + \tgrep untracked/ tmp >actual &&\n + \trm tmp &&\n 8: 2200bf144a = 8: 77b57e44fd dir: consolidate similar code in treat_directory()\n\n-- \ngitgitgadget\n"},{"id":"387907","messageId":"79f2b561742372234a52dc7194dd72026c6e96ec.1576008027.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","subject":"[PATCH v2 2/8] Revert \"dir.c: make 'git-status --ignored' work within leading directories\"","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-10T20:00:21Z","receivedAt":"2019-12-10T20:00:34Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCommit be8a84c52669 (\"dir.c: make 'git-status --ignored' work within\nleading directories\", 2013-04-15) noted that\n git status --ignored \nwould not list ignored files and directories within if\n was untracked, and modified the behavior to make it show\nthem. However, it did so via a hack that broke consistency; it would\nshow paths under differently than a simple\n git status --ignored | grep \nwould show them. A correct fix is slightly more involved, and\ncomplicated slightly by this hack, so we revert this commit (but keep\ncorrected versions of the testcases) and will later fix the original\nbug with a subsequent patch.\n\nSome history may be helpful:\n\nA very, very similar case to the commit we are reverting was raised in\ncommit 48ffef966c76 (\"ls-files: fix overeager pathspec optimization\",\n2010-01-08); but it actually went in somewhat the opposite direction. In\nthat commit, it mentioned how\n git ls-files -o --exclude-standard t/\nused to show untracked files under t/ even when t/ was ignored, and then\nchanged the behavior to stop showing untracked files under an ignored\ndirectory. More importantly, this commit considered keeping this\nbehavior but noted that it would be inconsistent with the behavior when\nmultiple pathspecs were specified and thus rejected it.\n\nThe reason for this whole inconsistency when one pathspec is specified\nversus zero or two is because common prefixes of pathspecs are sent\nthrough a different set of checks (in treat_leading_path()) than normal\nfile/directory traversal (those go through read_directory_recursive()\nand treat_path()). As such, for consistency, one needs to check that\nboth codepaths produce the same result.\n\nRevert commit be8a84c526691667fc04a8241d93a3de1de298ab, except instead\nof removing the testcase it added, modify it to check for correct and\nconsistent behavior. A subsequent patch in this series will fix the\ntestcase.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 3 ---\n t/t7061-wtstatus-ignore.sh | 9 +++++++--\n 2 files changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 61f559f980..0dd5266629 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2083,14 +2083,12 @@ static int treat_leading_path(struct dir_struct *dir,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint baselen, rc = 0;\n \tconst char *cp;\n-\tint old_flags = dir->flags;\n \n \twhile (len && path[len - 1] == '/')\n \t\tlen--;\n \tif (!len)\n \t\treturn 1;\n \tbaselen = 0;\n-\tdir->flags &= ~DIR_SHOW_OTHER_DIRECTORIES;\n \twhile (1) {\n \t\tcp = path + baselen + !!baselen;\n \t\tcp = memchr(cp, '/', path + len - cp);\n@@ -2113,7 +2111,6 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t}\n \t}\n \tstrbuf_release(&sb);\n-\tdir->flags = old_flags;\n \treturn rc;\n }\n \ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex 0c394cf995..84366050da 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -43,11 +43,16 @@ test_expect_success 'status untracked directory with --ignored -u' '\n \ttest_cmp expected actual\n '\n cat >expected <<\\EOF\n-?? untracked/uncommitted\n+?? untracked/\n !! untracked/ignored\n EOF\n \n-test_expect_success 'status prefixed untracked directory with --ignored' '\n+test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n+\tgit status --porcelain --ignored >tmp &&\n+\tgrep untracked/ tmp >actual &&\n+\trm tmp &&\n+\ttest_cmp expected actual &&\n+\n \tgit status --porcelain --ignored untracked/ >actual &&\n \ttest_cmp expected actual\n '\n-- \ngitgitgadget\n\n"},{"id":"387908","messageId":"6d659b230239b530c90fbcc15c9a869f8d5dd5c5.1576008027.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","subject":"[PATCH v2 1/8] t3011: demonstrate directory traversal failures","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-10T20:00:20Z","receivedAt":"2019-12-10T20:00:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd several tests demonstrating directory traversal failures of various\nsorts in dir.c (and one similar looking test that turns out to be a\ngit_fnmatch bug). A lot of these tests look like near duplicates of\neach other, but an optimization path in dir.c to pre-descend into a\ncommon prefix and the specialized treatment of trailing slashes in dir.c\nmean the tiny differences are sometimes important and potentially cause\ndifferent codepaths to be explored.\n\nOf the 7 failing tests, 2 are new to git-2.24.0 (tweaked by side effects\nof the en/clean-nested-with-ignored-topic); the other 5 also failed\nunder git-2.23.0 and earlier.\n\nSigned-off-by: Elijah Newren \n---\n ...common-prefixes-and-directory-traversal.sh | 209 ++++++++++++++++++\n 1 file changed, 209 insertions(+)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nnew file mode 100755\nindex 0000000000..54f80c62b8\n--- /dev/null\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -0,0 +1,209 @@\n+#!/bin/sh\n+\n+test_description='directory traversal handling, especially with common prefixes'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit hello &&\n+\n+\t>empty &&\n+\tmkdir untracked_dir &&\n+\t>untracked_dir/empty &&\n+\tgit init untracked_repo &&\n+\t>untracked_repo/empty &&\n+\n+\tcat <<-EOF >.gitignore &&\n+\tignored\n+\tan_ignored_dir/\n+\tEOF\n+\tmkdir an_ignored_dir &&\n+\tmkdir an_untracked_dir &&\n+\t>an_ignored_dir/ignored &&\n+\t>an_ignored_dir/untracked &&\n+\t>an_untracked_dir/ignored &&\n+\t>an_untracked_dir/untracked\n+'\n+\n+test_expect_success 'git ls-files -o shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_ignored_dir/ignored\n+\tan_ignored_dir/untracked\n+\tan_untracked_dir/ignored\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --exclude-standard shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --exclude-standard >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses into untracked_dir only' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir/ untracked_repo/ does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o .git shows nothing' '\n+\tgit ls-files -o .git >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_failure 'git ls-files -o .git/ shows nothing' '\n+\tgit ls-files -o .git/ >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o untracked_* recurses appropriately' '\n+\tmkdir \"untracked_*\" &&\n+\t>\"untracked_*/empty\" &&\n+\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/empty\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# It turns out fill_directory returns the right paths, but ls-files' post-call\n+# filtering in show_dir_entry() via calling dir_path_match() which ends up\n+# in git_fnmatch() has logic for PATHSPEC_ONESTAR that assumes the pathspec\n+# must match the full path; it doesn't check it for matching a leading\n+# directory.\n+test_expect_failure FUNNYNAMES 'git ls-files -o untracked_*/ recurses appropriately' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/empty\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_* does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_*/ does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o consistent between one or two dirs' '\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ an_untracked_dir/ >tmp &&\n+\t! grep ^an_ignored_dir/ tmp >expect &&\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# ls-files doesn't have a way to request showing both untracked and ignored\n+# files at the same time, so use `git status --ignored`\n+test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n+\tcat <<-EOF >expect &&\n+\t?? an_untracked_dir/\n+\t!! an_untracked_dir/ignored\n+\tEOF\n+\tgit status --porcelain --ignored >output &&\n+\tgrep an_untracked_dir output >expect &&\n+\tgit status --porcelain --ignored an_untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"387909","messageId":"9839aca00a10b16d96c47db631ac025281ffc864.1576008027.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","subject":"[PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-10T20:00:25Z","receivedAt":"2019-12-10T20:00:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany years ago, the directory traversing logic had an optimization that\nwould always recurse into any directory that was a common prefix of all\nthe pathspecs without walking the leading directories to get down to\nthe desired directory. Thus,\n git ls-files -o .git/ # case A\nwould notice that .git/ was a common prefix of all pathspecs (since\nit is the only pathspec listed), and then traverse into it and start\nshowing unknown files under that directory. Unfortunately, .git/ is not\na directory we should be traversing into, which made this optimization\nproblematic. This also affected cases like\n git ls-files -o --exclude-standard t/ # case B\nwhere t/ was in the .gitignore file and thus isn't interesting and\nshouldn't be recursed into. It also affected cases like\n git ls-files -o --directory untracked_dir/ # case C\nwhere untracked_dir/ is indeed untracked and thus interesting, but the\n--directory flag means we only want to show the directory itself, not\nrecurse into it and start listing untracked files below it.\n\nThe case B class of bugs were noted and fixed in commits 16e2cfa90993\n(\"read_directory(): further split treat_path()\", 2010-01-08) and\n48ffef966c76 (\"ls-files: fix overeager pathspec optimization\",\n2010-01-08), with the idea being that we first wanted to check whether\nthe common prefix was interesting. The former patch noted that\ntreat_path() couldn't be used when checking the common prefix because\ntreat_path() requires a dir_entry() and we haven't read any directories\nat the point we are checking the common prefix. So, that patch split\ntreat_one_path() out of treat_path(). The latter patch then created a\nnew treat_leading_path() which duplicated by hand the bits of\ntreat_path() that couldn't be broken out and then called\ntreat_one_path() for the remainder. There were three problems with this\napproach:\n\n * The duplicated logic in treat_leading_path() accidentally missed the\n check for special paths (such as is_dot_or_dotdot and matching\n \".git\"), causing case A types of bugs to continue to be an issue.\n * The treat_leading_path() logic assumed we should traverse into\n anything where path_treatment was not path_none, i.e. it perpetuated\n class C types of bugs.\n * It meant we had split logic that needed to kept in sync, running the\n risk that people introduced new inconsistencies (such as in commit\n be8a84c52669, which we reverted earlier in this series, or in commit\n df5bcdf83ae which we'll fix in a subsequent commit)\n\nFix most these problems by making treat_leading_path() not only loop\nover each leading path component, but calling treat_path() directly on\neach. To do so, we have to create a synthetic dir_entry, but that only\ntakes a few lines. Then, pay attention to the path_treatment result we\nget from treat_path() and don't treat path_excluded, path_untracked, and\npath_recurse all the same as path_recurse.\n\nThis leaves one remaining problem, the new inconsistency from commit\ndf5bcdf83ae. That will be addressed in a subsequent commit.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 54 +++++++++++++++----\n ...common-prefixes-and-directory-traversal.sh | 6 +--\n 2 files changed, 46 insertions(+), 14 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 645b44ea64..9c71a9ac21 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2102,37 +2102,69 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const struct pathspec *pathspec)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint baselen, rc = 0;\n+\tint prevlen, baselen;\n \tconst char *cp;\n+\tstruct cached_dir cdir;\n+\tstruct dirent de;\n+\tenum path_treatment state = path_none;\n+\n+\t/*\n+\t * For each directory component of path, we are going to check whether\n+\t * that path is relevant given the pathspec. For example, if path is\n+\t * foo/bar/baz/\n+\t * then we will ask treat_path() whether we should go into foo, then\n+\t * whether we should go into bar, then whether baz is relevant.\n+\t * Checking each is important because e.g. if path is\n+\t * .git/info/\n+\t * then we need to check .git to know we shouldn't traverse it.\n+\t * If the return from treat_path() is:\n+\t * * path_none, for any path, we return false.\n+\t * * path_recurse, for all path components, we return true\n+\t * * for some intermediate component, we make sure\n+\t * to add that path to the relevant list but return false\n+\t * signifying that we shouldn't recurse into it.\n+\t */\n \n \twhile (len && path[len - 1] == '/')\n \t\tlen--;\n \tif (!len)\n \t\treturn 1;\n+\n+\tmemset(&cdir, 0, sizeof(cdir));\n+\tmemset(&de, 0, sizeof(de));\n+\tcdir.de = &de;\n+\tde.d_type = DT_DIR;\n \tbaselen = 0;\n+\tprevlen = 0;\n \twhile (1) {\n-\t\tcp = path + baselen + !!baselen;\n+\t\tprevlen = baselen + !!baselen;\n+\t\tcp = path + prevlen;\n \t\tcp = memchr(cp, '/', path + len - cp);\n \t\tif (!cp)\n \t\t\tbaselen = len;\n \t\telse\n \t\t\tbaselen = cp - path;\n-\t\tstrbuf_setlen(&sb, 0);\n+\t\tstrbuf_reset(&sb);\n \t\tstrbuf_add(&sb, path, baselen);\n \t\tif (!is_directory(sb.buf))\n \t\t\tbreak;\n-\t\tif (simplify_away(sb.buf, sb.len, pathspec))\n-\t\t\tbreak;\n-\t\tif (treat_one_path(dir, NULL, istate, &sb, baselen, pathspec,\n-\t\t\t\t DT_DIR, NULL) == path_none)\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_add(&sb, path, prevlen);\n+\t\tmemcpy(de.d_name, path+prevlen, baselen-prevlen);\n+\t\tde.d_name[baselen-prevlen] = '\\0';\n+\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n+\t\t\t\t pathspec);\n+\t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n-\t\tif (len <= baselen) {\n-\t\t\trc = 1;\n+\t\tif (len <= baselen)\n \t\t\tbreak; /* finished checking */\n-\t\t}\n \t}\n+\tadd_path_to_appropriate_result_list(dir, NULL, &cdir, istate,\n+\t\t\t\t\t &sb, baselen, pathspec,\n+\t\t\t\t\t state);\n+\n \tstrbuf_release(&sb);\n-\treturn rc;\n+\treturn state == path_recurse;\n }\n \n static const char *get_ident_string(void)\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex d6e161ddd8..098fddc75b 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -74,7 +74,7 @@ test_expect_success 'git ls-files -o --directory untracked_dir does not recurse'\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n+test_expect_success 'git ls-files -o --directory untracked_dir/ does not recurse' '\n \techo untracked_dir/ >expect &&\n \tgit ls-files -o --directory untracked_dir/ >actual &&\n \ttest_cmp expect actual\n@@ -86,7 +86,7 @@ test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n+test_expect_success 'git ls-files -o untracked_repo/ does not recurse' '\n \techo untracked_repo/ >expect &&\n \tgit ls-files -o untracked_repo/ >actual &&\n \ttest_cmp expect actual\n@@ -133,7 +133,7 @@ test_expect_success 'git ls-files -o .git shows nothing' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_failure 'git ls-files -o .git/ shows nothing' '\n+test_expect_success 'git ls-files -o .git/ shows nothing' '\n \tgit ls-files -o .git/ >actual &&\n \ttest_must_be_empty actual\n '\n-- \ngitgitgadget\n\n"},{"id":"387910","messageId":"d6f858cab122869425f5801d98713bcbb6d00334.1576008027.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","subject":"[PATCH v2 3/8] dir: remove stray quote character in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-10T20:00:22Z","receivedAt":"2019-12-10T20:00:40Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 0dd5266629..5dacacd469 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -373,7 +373,7 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t !ps_strncmp(item, match, name, namelen))\n \t\t\treturn MATCHED_RECURSIVELY_LEADING_PATHSPEC;\n \n-\t\t/* name\" doesn't match up to the first wild character */\n+\t\t/* name doesn't match up to the first wild character */\n \t\tif (item->nowildcard_len < item->len &&\n \t\t ps_strncmp(item, match, name,\n \t\t\t item->nowildcard_len - prefix))\n-- \ngitgitgadget\n\n"},{"id":"387911","messageId":"77b57e44fdd819536c9be37a6190a79792efbccb.1576008027.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","subject":"[PATCH v2 8/8] dir: consolidate similar code in treat_directory()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-10T20:00:27Z","receivedAt":"2019-12-10T20:00:44Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nBoth the DIR_SKIP_NESTED_GIT and DIR_NO_GITLINKS cases were checking for\nwhether a path was actually a nonbare repository. That code could be\nshared, with just the result of how to act differing between the two\ncases.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 18 +++++++-----------\n 1 file changed, 7 insertions(+), 11 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex bb6e481909..04541b798b 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1461,6 +1461,8 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int exclude,\n \tconst struct pathspec *pathspec)\n {\n+\tint nested_repo = 0;\n+\n \t/* The \"len-1\" is to strip the final '/' */\n \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n \tcase index_directory:\n@@ -1470,15 +1472,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_none;\n \n \tcase index_nonexistent:\n-\t\tif (dir->flags & DIR_SKIP_NESTED_GIT) {\n-\t\t\tint nested_repo;\n+\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n+\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n \t\t\tstruct strbuf sb = STRBUF_INIT;\n \t\t\tstrbuf_addstr(&sb, dirname);\n \t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n \t\t\tstrbuf_release(&sb);\n-\t\t\tif (nested_repo)\n-\t\t\t\treturn path_none;\n \t\t}\n+\t\tif (nested_repo)\n+\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n+\t\t\t\t(exclude ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n \t\t\tbreak;\n@@ -1506,13 +1509,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t\t\treturn path_none;\n \t\t}\n-\t\tif (!(dir->flags & DIR_NO_GITLINKS)) {\n-\t\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\t\tstrbuf_addstr(&sb, dirname);\n-\t\t\tif (is_nonbare_repository_dir(&sb))\n-\t\t\t\treturn exclude ? path_excluded : path_untracked;\n-\t\t\tstrbuf_release(&sb);\n-\t\t}\n \t\treturn path_recurse;\n \t}\n \n-- \ngitgitgadget\n"},{"id":"387912","messageId":"d2f5623bd71352f3adbf12f211dbae5820b33666.1576008027.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","subject":"[PATCH v2 5/8] dir: break part of read_directory_recursive() out for reuse","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-10T20:00:24Z","receivedAt":"2019-12-10T20:00:46Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCreate an add_path_to_appropriate_result_list() function from the code\nat the end of read_directory_recursive() so we can use it elsewhere.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 60 ++++++++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 37 insertions(+), 23 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 517a569e10..645b44ea64 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1932,6 +1932,40 @@ static void close_cached_dir(struct cached_dir *cdir)\n \t}\n }\n \n+static void add_path_to_appropriate_result_list(struct dir_struct *dir,\n+\tstruct untracked_cache_dir *untracked,\n+\tstruct cached_dir *cdir,\n+\tstruct index_state *istate,\n+\tstruct strbuf *path,\n+\tint baselen,\n+\tconst struct pathspec *pathspec,\n+\tenum path_treatment state)\n+{\n+\t/* add the path to the appropriate result list */\n+\tswitch (state) {\n+\tcase path_excluded:\n+\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\tdir_add_name(dir, istate, path->buf, path->len);\n+\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n+\t\t\t((dir->flags & DIR_COLLECT_IGNORED) &&\n+\t\t\texclude_matches_pathspec(path->buf, path->len,\n+\t\t\t\t\t\t pathspec)))\n+\t\t\tdir_add_ignored(dir, istate, path->buf, path->len);\n+\t\tbreak;\n+\n+\tcase path_untracked:\n+\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\tbreak;\n+\t\tdir_add_name(dir, istate, path->buf, path->len);\n+\t\tif (cdir->fdir)\n+\t\t\tadd_untracked(untracked, path->buf + baselen);\n+\t\tbreak;\n+\n+\tdefault:\n+\t\tbreak;\n+\t}\n+}\n+\n /*\n * Read a directory tree. We currently ignore anything but\n * directories, regular files and symlinks. That's because git\n@@ -2035,29 +2069,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tcontinue;\n \t\t}\n \n-\t\t/* add the path to the appropriate result list */\n-\t\tswitch (state) {\n-\t\tcase path_excluded:\n-\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n-\t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n-\t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t\t((dir->flags & DIR_COLLECT_IGNORED) &&\n-\t\t\t\texclude_matches_pathspec(path.buf, path.len,\n-\t\t\t\t\t\t\t pathspec)))\n-\t\t\t\tdir_add_ignored(dir, istate, path.buf, path.len);\n-\t\t\tbreak;\n-\n-\t\tcase path_untracked:\n-\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n-\t\t\t\tbreak;\n-\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n-\t\t\tif (cdir.fdir)\n-\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n-\t\t\tbreak;\n-\n-\t\tdefault:\n-\t\t\tbreak;\n-\t\t}\n+\t\tadd_path_to_appropriate_result_list(dir, untracked, &cdir,\n+\t\t\t\t\t\t istate, &path, baselen,\n+\t\t\t\t\t\t pathspec, state);\n \t}\n \tclose_cached_dir(&cdir);\n out:\n-- \ngitgitgadget\n\n"},{"id":"387914","messageId":"8d2d98eec3c620c55100a322087ef83e4f51c8cf.1576008027.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","subject":"[PATCH v2 4/8] dir: exit before wildcard fall-through if there is no wildcard","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-10T20:00:23Z","receivedAt":"2019-12-10T20:00:47Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe DO_MATCH_LEADING_PATHSPEC had a fall-through case for if there was a\nwildcard, noting that we don't yet have enough information to determine\nif a further paths under the current directory might match due to the\npresence of wildcards. But if we have no wildcards in our pathspec,\nthen we shouldn't get to that fall-through case.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 7 +++++++\n t/t3011-common-prefixes-and-directory-traversal.sh | 4 ++--\n 2 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 5dacacd469..517a569e10 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -379,6 +379,13 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t\t item->nowildcard_len - prefix))\n \t\t\treturn 0;\n \n+\t\t/*\n+\t\t * name has no wildcard, and it didn't match as a leading\n+\t\t * pathspec so return.\n+\t\t */\n+\t\tif (item->nowildcard_len == item->len)\n+\t\t\treturn 0;\n+\n \t\t/*\n \t\t * Here is where we would perform a wildmatch to check if\n \t\t * \"name\" can be matched as a directory (or a prefix) against\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex 54f80c62b8..d6e161ddd8 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -92,7 +92,7 @@ test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+test_expect_success 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n \tcat <<-EOF >expect &&\n \tuntracked_dir/empty\n \tuntracked_repo/\n@@ -110,7 +110,7 @@ test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses int\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+test_expect_success 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n \tcat <<-EOF >expect &&\n \tuntracked_dir/\n \tuntracked_repo/\n-- \ngitgitgadget\n\n"},{"id":"387913","messageId":"df7f08886a73f24095376c5d1a5454794f329dfa.1576008027.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","subject":"[PATCH v2 7/8] dir: synchronize treat_leading_path() and read_directory_recursive()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-10T20:00:26Z","receivedAt":"2019-12-10T20:00:49Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nOur optimization to avoid calling into read_directory_recursive() when\nall pathspecs have a common leading directory mean that we need to match\nthe logic that read_directory_recursive() would use if we had just\ncalled it from the root. Since it does more than call treat_path() we\nneed to copy that same logic.\n\nAlternatively, we could try to change treat_path to return path_recurse\nfor an untracked directory under the given special circumstances that\nthis logic checks for, but a simple switch results in many test failures\nsuch as 'git clean -d' not wiping out untracked but empty directories.\nTo work around that, we'd need the caller of treat_path to check for\npath_recurse and sometimes special case it into path_untracked. In\nother words, we'd still have extra logic in both places.\n\nNeeding to duplicate logic like this means it is guaranteed someone will\neventually need to make further changes and forget to update both\nlocations. It is tempting to just nuke the leading_directory special\ncasing to avoid such bugs and simplify the code, but unpack_trees'\nverify_clean_subdirectory() also calls read_directory() and does so with\na non-empty leading path, so I'm hesitant to try to restructure further.\nAdd obnoxious warnings to treat_leading_path() and\nread_directory_recursive() to try to warn people of such problems.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 30 +++++++++++++++++++\n ...common-prefixes-and-directory-traversal.sh | 2 +-\n t/t7061-wtstatus-ignore.sh | 2 +-\n 3 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 9c71a9ac21..bb6e481909 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1990,6 +1990,15 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tstruct untracked_cache_dir *untracked, int check_only,\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING WARNING WARNING:\n+\t *\n+\t * Any updates to the traversal logic here may need corresponding\n+\t * updates in treat_leading_path(). See the commit message for the\n+\t * commit adding this warning as well as the commit preceding it\n+\t * for details.\n+\t */\n+\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -2101,6 +2110,15 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING WARNING WARNING:\n+\t *\n+\t * Any updates to the traversal logic here may need corresponding\n+\t * updates in treat_leading_path(). See the commit message for the\n+\t * commit adding this warning as well as the commit preceding it\n+\t * for details.\n+\t */\n+\n \tstruct strbuf sb = STRBUF_INIT;\n \tint prevlen, baselen;\n \tconst char *cp;\n@@ -2154,6 +2172,18 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tde.d_name[baselen-prevlen] = '\\0';\n \t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n \t\t\t\t pathspec);\n+\t\tif (state == path_untracked &&\n+\t\t get_dtype(cdir.de, istate, sb.buf, sb.len) == DT_DIR &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO ||\n+\t\t do_match_pathspec(istate, pathspec, sb.buf, sb.len,\n+\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)) {\n+\t\t\tadd_path_to_appropriate_result_list(dir, NULL, &cdir,\n+\t\t\t\t\t\t\t istate,\n+\t\t\t\t\t\t\t &sb, baselen,\n+\t\t\t\t\t\t\t pathspec, state);\n+\t\t\tstate = path_recurse;\n+\t\t}\n+\n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n \t\tif (len <= baselen)\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex 098fddc75b..3da5b2b6e7 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -195,7 +195,7 @@ test_expect_success 'git ls-files -o consistent between one or two dirs' '\n \n # ls-files doesn't have a way to request showing both untracked and ignored\n # files at the same time, so use `git status --ignored`\n-test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n+test_expect_success 'git status --ignored shows same files under dir with or without pathspec' '\n \tcat <<-EOF >expect &&\n \t?? an_untracked_dir/\n \t!! an_untracked_dir/ignored\ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex 84366050da..e4cf5484f9 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -47,7 +47,7 @@ cat >expected <<\\EOF\n !! untracked/ignored\n EOF\n \n-test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n+test_expect_success 'status of untracked directory with --ignored works with or without prefix' '\n \tgit status --porcelain --ignored >tmp &&\n \tgrep untracked/ tmp >actual &&\n \trm tmp &&\n-- \ngitgitgadget\n\n"},{"id":"388227","messageId":"nycvar.QRO.7.76.6.1912151126030.46@tvgsbejvaqbjf.bet","threadId":"52419","inReplyTo":"9839aca00a10b16d96c47db631ac025281ffc864.1576008027.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-15T10:29:32Z","receivedAt":"2019-12-15T10:29:58Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nI have not had time to dive deeply into this, but I know that it _does_\ncause a ton of segmentation faults in the `shears/pu` branch (where all of\nGit for Windows' patches are rebased on top of `pu`):\n\nOn Tue, 10 Dec 2019, Elijah Newren via GitGitGadget wrote:\n\n> diff --git a/dir.c b/dir.c\n> index 645b44ea64..9c71a9ac21 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -2102,37 +2102,69 @@ static int treat_leading_path(struct dir_struct *dir,\n> \t\t\t const struct pathspec *pathspec)\n> {\n> \tstruct strbuf sb = STRBUF_INIT;\n> -\tint baselen, rc = 0;\n> +\tint prevlen, baselen;\n> \tconst char *cp;\n> +\tstruct cached_dir cdir;\n> +\tstruct dirent de;\n> +\tenum path_treatment state = path_none;\n> +\n> +\t/*\n> +\t * For each directory component of path, we are going to check whether\n> +\t * that path is relevant given the pathspec. For example, if path is\n> +\t * foo/bar/baz/\n> +\t * then we will ask treat_path() whether we should go into foo, then\n> +\t * whether we should go into bar, then whether baz is relevant.\n> +\t * Checking each is important because e.g. if path is\n> +\t * .git/info/\n> +\t * then we need to check .git to know we shouldn't traverse it.\n> +\t * If the return from treat_path() is:\n> +\t * * path_none, for any path, we return false.\n> +\t * * path_recurse, for all path components, we return true\n> +\t * * for some intermediate component, we make sure\n> +\t * to add that path to the relevant list but return false\n> +\t * signifying that we shouldn't recurse into it.\n> +\t */\n>\n> \twhile (len && path[len - 1] == '/')\n> \t\tlen--;\n> \tif (!len)\n> \t\treturn 1;\n> +\n> +\tmemset(&cdir, 0, sizeof(cdir));\n> +\tmemset(&de, 0, sizeof(de));\n> +\tcdir.de = &de;\n> +\tde.d_type = DT_DIR;\n\nSo here, `de` is zeroed out, and therefore `de.d_name` is `NULL`.\n\n> \tbaselen = 0;\n> +\tprevlen = 0;\n> \twhile (1) {\n> -\t\tcp = path + baselen + !!baselen;\n> +\t\tprevlen = baselen + !!baselen;\n> +\t\tcp = path + prevlen;\n> \t\tcp = memchr(cp, '/', path + len - cp);\n> \t\tif (!cp)\n> \t\t\tbaselen = len;\n> \t\telse\n> \t\t\tbaselen = cp - path;\n> -\t\tstrbuf_setlen(&sb, 0);\n> +\t\tstrbuf_reset(&sb);\n> \t\tstrbuf_add(&sb, path, baselen);\n> \t\tif (!is_directory(sb.buf))\n> \t\t\tbreak;\n> -\t\tif (simplify_away(sb.buf, sb.len, pathspec))\n> -\t\t\tbreak;\n> -\t\tif (treat_one_path(dir, NULL, istate, &sb, baselen, pathspec,\n> -\t\t\t\t DT_DIR, NULL) == path_none)\n> +\t\tstrbuf_reset(&sb);\n> +\t\tstrbuf_add(&sb, path, prevlen);\n> +\t\tmemcpy(de.d_name, path+prevlen, baselen-prevlen);\n\nBut here we try to copy a path into that `de.d_name`, which is still\n`NULL`?\n\nThat can't be right, can it?\n\nThanks for your help,\nDscho\n\n> +\t\tde.d_name[baselen-prevlen] = '\\0';\n> +\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n> +\t\t\t\t pathspec);\n> +\t\tif (state != path_recurse)\n> \t\t\tbreak; /* do not recurse into it */\n> -\t\tif (len <= baselen) {\n> -\t\t\trc = 1;\n> +\t\tif (len <= baselen)\n> \t\t\tbreak; /* finished checking */\n> -\t\t}\n> \t}\n> +\tadd_path_to_appropriate_result_list(dir, NULL, &cdir, istate,\n> +\t\t\t\t\t &sb, baselen, pathspec,\n> +\t\t\t\t\t state);\n> +\n> \tstrbuf_release(&sb);\n> -\treturn rc;\n> +\treturn state == path_recurse;\n> }\n>\n> static const char *get_ident_string(void)\n> diff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\n> index d6e161ddd8..098fddc75b 100755\n> --- a/t/t3011-common-prefixes-and-directory-traversal.sh\n> +++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n> @@ -74,7 +74,7 @@ test_expect_success 'git ls-files -o --directory untracked_dir does not recurse'\n> \ttest_cmp expect actual\n> '\n>\n> -test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n> +test_expect_success 'git ls-files -o --directory untracked_dir/ does not recurse' '\n> \techo untracked_dir/ >expect &&\n> \tgit ls-files -o --directory untracked_dir/ >actual &&\n> \ttest_cmp expect actual\n> @@ -86,7 +86,7 @@ test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n> \ttest_cmp expect actual\n> '\n>\n> -test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n> +test_expect_success 'git ls-files -o untracked_repo/ does not recurse' '\n> \techo untracked_repo/ >expect &&\n> \tgit ls-files -o untracked_repo/ >actual &&\n> \ttest_cmp expect actual\n> @@ -133,7 +133,7 @@ test_expect_success 'git ls-files -o .git shows nothing' '\n> \ttest_must_be_empty actual\n> '\n>\n> -test_expect_failure 'git ls-files -o .git/ shows nothing' '\n> +test_expect_success 'git ls-files -o .git/ shows nothing' '\n> \tgit ls-files -o .git/ >actual &&\n> \ttest_must_be_empty actual\n> '\n> --\n> gitgitgadget\n>\n>\n>\n"},{"id":"388251","messageId":"CABPp-BE04=A9wx1VfWsghn6scy8aaVFoENxV6YiW2AxgM2jhjQ@mail.gmail.com","threadId":"52419","inReplyTo":"nycvar.QRO.7.76.6.1912151126030.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-12-16T13:51:30Z","receivedAt":"2019-12-16T13:51:45Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Sun, Dec 15, 2019 at 2:29 AM Johannes Schindelin\n wrote:\n>\n> Hi Elijah,\n>\n> I have not had time to dive deeply into this, but I know that it _does_\n> cause a ton of segmentation faults in the `shears/pu` branch (where all of\n> Git for Windows' patches are rebased on top of `pu`):\n\nWeird. If it's going to cause segmentation faults at all, it would\ncertainly do it all over the place, but I tested the patches on the\nmajor platforms using your Azure Pipelines setup on git.git so it\nshould be good on all the platforms. Did your shears/pu branch make\nsome other changes to the setup?\n\n> On Tue, 10 Dec 2019, Elijah Newren via GitGitGadget wrote:\n>\n> > diff --git a/dir.c b/dir.c\n> > index 645b44ea64..9c71a9ac21 100644\n> > --- a/dir.c\n> > +++ b/dir.c\n> > @@ -2102,37 +2102,69 @@ static int treat_leading_path(struct dir_struct *dir,\n> > const struct pathspec *pathspec)\n> > {\n> > struct strbuf sb = STRBUF_INIT;\n> > - int baselen, rc = 0;\n> > + int prevlen, baselen;\n> > const char *cp;\n> > + struct cached_dir cdir;\n> > + struct dirent de;\n> > + enum path_treatment state = path_none;\n> > +\n> > + /*\n> > + * For each directory component of path, we are going to check whether\n> > + * that path is relevant given the pathspec. For example, if path is\n> > + * foo/bar/baz/\n> > + * then we will ask treat_path() whether we should go into foo, then\n> > + * whether we should go into bar, then whether baz is relevant.\n> > + * Checking each is important because e.g. if path is\n> > + * .git/info/\n> > + * then we need to check .git to know we shouldn't traverse it.\n> > + * If the return from treat_path() is:\n> > + * * path_none, for any path, we return false.\n> > + * * path_recurse, for all path components, we return true\n> > + * * for some intermediate component, we make sure\n> > + * to add that path to the relevant list but return false\n> > + * signifying that we shouldn't recurse into it.\n> > + */\n> >\n> > while (len && path[len - 1] == '/')\n> > len--;\n> > if (!len)\n> > return 1;\n> > +\n> > + memset(&cdir, 0, sizeof(cdir));\n> > + memset(&de, 0, sizeof(de));\n> > + cdir.de = &de;\n> > + de.d_type = DT_DIR;\n>\n> So here, `de` is zeroed out, and therefore `de.d_name` is `NULL`.\n\nUm, yeah...didn't I have an allocation of de.d_name here? It will\nalways have a subset of path copied into it, so an allocation of len+1\nis plenty long enough.\n\n> > baselen = 0;\n> > + prevlen = 0;\n> > while (1) {\n> > - cp = path + baselen + !!baselen;\n> > + prevlen = baselen + !!baselen;\n> > + cp = path + prevlen;\n> > cp = memchr(cp, '/', path + len - cp);\n> > if (!cp)\n> > baselen = len;\n> > else\n> > baselen = cp - path;\n> > - strbuf_setlen(&sb, 0);\n> > + strbuf_reset(&sb);\n> > strbuf_add(&sb, path, baselen);\n> > if (!is_directory(sb.buf))\n> > break;\n> > - if (simplify_away(sb.buf, sb.len, pathspec))\n> > - break;\n> > - if (treat_one_path(dir, NULL, istate, &sb, baselen, pathspec,\n> > - DT_DIR, NULL) == path_none)\n> > + strbuf_reset(&sb);\n> > + strbuf_add(&sb, path, prevlen);\n> > + memcpy(de.d_name, path+prevlen, baselen-prevlen);\n>\n> But here we try to copy a path into that `de.d_name`, which is still\n> `NULL`?\n>\n> That can't be right, can it?\n\nYes, it can't be right. How did this possibly pass on any platform\nlet alone all of them?\n(https://dev.azure.com/git/git/_build/results?buildId=1462&view=results).\nThis is absolutely an important codepath that is hit; otherwise it\ncouldn't fix the three tests from failure to success. Further, the\nsubsequent patch added code within this if-block after this memcpy and\nfixed a few tests from failures to success. So it had to hit this\ncode path as well. How could it not have segfaulted? I'm very\nconfused...\n"},{"id":"388283","messageId":"CABPp-BGoC_D6LzzMNyf30wFssTU2WA1kTLmFvJ2Do+Tfg4+YQA@mail.gmail.com","threadId":"52419","inReplyTo":"CABPp-BE04=A9wx1VfWsghn6scy8aaVFoENxV6YiW2AxgM2jhjQ@mail.gmail.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-12-16T16:00:56Z","receivedAt":"2019-12-16T16:01:10Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Dec 16, 2019 at 5:51 AM Elijah Newren wrote:\n>\n> On Sun, Dec 15, 2019 at 2:29 AM Johannes Schindelin\n> wrote:\n> >\n> > Hi Elijah,\n> >\n> > I have not had time to dive deeply into this, but I know that it _does_\n> > cause a ton of segmentation faults in the `shears/pu` branch (where all of\n> > Git for Windows' patches are rebased on top of `pu`):\n>\n> Weird. If it's going to cause segmentation faults at all, it would\n> certainly do it all over the place, but I tested the patches on the\n> major platforms using your Azure Pipelines setup on git.git so it\n> should be good on all the platforms. Did your shears/pu branch make\n> some other changes to the setup?\n>\n> > On Tue, 10 Dec 2019, Elijah Newren via GitGitGadget wrote:\n> >\n> > > diff --git a/dir.c b/dir.c\n> > > index 645b44ea64..9c71a9ac21 100644\n> > > --- a/dir.c\n> > > +++ b/dir.c\n> > > @@ -2102,37 +2102,69 @@ static int treat_leading_path(struct dir_struct *dir,\n> > > const struct pathspec *pathspec)\n> > > {\n> > > struct strbuf sb = STRBUF_INIT;\n> > > - int baselen, rc = 0;\n> > > + int prevlen, baselen;\n> > > const char *cp;\n> > > + struct cached_dir cdir;\n> > > + struct dirent de;\n> > > + enum path_treatment state = path_none;\n> > > +\n> > > + /*\n> > > + * For each directory component of path, we are going to check whether\n> > > + * that path is relevant given the pathspec. For example, if path is\n> > > + * foo/bar/baz/\n> > > + * then we will ask treat_path() whether we should go into foo, then\n> > > + * whether we should go into bar, then whether baz is relevant.\n> > > + * Checking each is important because e.g. if path is\n> > > + * .git/info/\n> > > + * then we need to check .git to know we shouldn't traverse it.\n> > > + * If the return from treat_path() is:\n> > > + * * path_none, for any path, we return false.\n> > > + * * path_recurse, for all path components, we return true\n> > > + * * for some intermediate component, we make sure\n> > > + * to add that path to the relevant list but return false\n> > > + * signifying that we shouldn't recurse into it.\n> > > + */\n> > >\n> > > while (len && path[len - 1] == '/')\n> > > len--;\n> > > if (!len)\n> > > return 1;\n> > > +\n> > > + memset(&cdir, 0, sizeof(cdir));\n> > > + memset(&de, 0, sizeof(de));\n> > > + cdir.de = &de;\n> > > + de.d_type = DT_DIR;\n> >\n> > So here, `de` is zeroed out, and therefore `de.d_name` is `NULL`.\n>\n> Um, yeah...didn't I have an allocation of de.d_name here? It will\n> always have a subset of path copied into it, so an allocation of len+1\n> is plenty long enough.\n\nActually, it looks like I looked up the definition of dirent\npreviously and forgot by the time you emailed. On linux, from\n/usr/include/bits/dirent.h:\n\nstruct dirent\n {\n ....\n unsigned char d_type;\n char d_name[256]; /* We must not include limits.h! */\n };\n\nand from compat/win32/dirent.h defines it as:\n\nstruct dirent {\n unsigned char d_type; /* file type to prevent lstat after\nreaddir */\n char d_name[MAX_PATH * 3]; /* file name (* 3 for UTF-8 conversion) */\n};\n\nand 'man dirent' on Mac OS X says it's defined as:\n\nstruct dirent {\n ...\n _uint8_t d_type;\n _unit8_t d_namlen; /* length of string in d_name */\n char d_name[255+1]; /* name must be no longer than this */\n}\n\nso, allocating it would be incorrect and my memset would just fill\nd_name with nul characters.\n\n\nBut the raises the question...what kind of segfaults are you getting?\nCan you link to any builds or post any stack traces? Can I duplicate\nwith some copy of git-for-windows on linux?\n"},{"id":"388286","messageId":"xmqqwoawjg09.fsf@gitster-ct.c.googlers.com","threadId":"52419","inReplyTo":"CABPp-BGoC_D6LzzMNyf30wFssTU2WA1kTLmFvJ2Do+Tfg4+YQA@mail.gmail.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-12-16T18:13:10Z","receivedAt":"2019-12-16T18:30:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n>> > > + memset(&cdir, 0, sizeof(cdir));\n>> > > + memset(&de, 0, sizeof(de));\n>> > > + cdir.de = &de;\n>> > > + de.d_type = DT_DIR;\n>> >\n>> > So here, `de` is zeroed out, and therefore `de.d_name` is `NULL`.\n>>\n>> Um, yeah...didn't I have an allocation of de.d_name here? It will\n>> always have a subset of path copied into it, so an allocation of len+1\n>> is plenty long enough.\n>\n> Actually, it looks like I looked up the definition of dirent\n> previously and forgot by the time you emailed. On linux, from\n> /usr/include/bits/dirent.h:\n>\n> struct dirent\n> {\n> ....\n> unsigned char d_type;\n> char d_name[256]; /* We must not include limits.h! */\n> };\n>\n> ...\n\nUh, oh. The size of \"struct dirent\" is unspecified and it is asking\nfor trouble to allocate one yourself (iow, treat it pretty much as\nsomething you can only get a pointer to an instance from readdir()).\nFor example, a dirent that comes back readdir() may have a lot\nlonger name than the sizeof(.d_name[]) above may imply.\n\nDo you really need to manufacture a dirent yourself, or can you use\na more concrete type you invent yourself?\n"},{"id":"388298","messageId":"CABPp-BF2UL5m4_f2tETfZvq=011KsKheGA=axFwYhC+kJXi0JQ@mail.gmail.com","threadId":"52419","inReplyTo":"xmqqwoawjg09.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-12-16T21:08:44Z","receivedAt":"2019-12-16T21:08:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Dec 16, 2019 at 10:13 AM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> >> > > + memset(&cdir, 0, sizeof(cdir));\n> >> > > + memset(&de, 0, sizeof(de));\n> >> > > + cdir.de = &de;\n> >> > > + de.d_type = DT_DIR;\n> >> >\n> >> > So here, `de` is zeroed out, and therefore `de.d_name` is `NULL`.\n> >>\n> >> Um, yeah...didn't I have an allocation of de.d_name here? It will\n> >> always have a subset of path copied into it, so an allocation of len+1\n> >> is plenty long enough.\n> >\n> > Actually, it looks like I looked up the definition of dirent\n> > previously and forgot by the time you emailed. On linux, from\n> > /usr/include/bits/dirent.h:\n> >\n> > struct dirent\n> > {\n> > ....\n> > unsigned char d_type;\n> > char d_name[256]; /* We must not include limits.h! */\n> > };\n> >\n> > ...\n>\n> Uh, oh. The size of \"struct dirent\" is unspecified and it is asking\n> for trouble to allocate one yourself (iow, treat it pretty much as\n> something you can only get a pointer to an instance from readdir()).\n> For example, a dirent that comes back readdir() may have a lot\n> longer name than the sizeof(.d_name[]) above may imply.\n>\n> Do you really need to manufacture a dirent yourself, or can you use\n> a more concrete type you invent yourself?\n\nI need to manufacture a dirent myself; short of that, the most likely\nalternative is to drop patches 2 & 5-8 of this series and throw my\nhands in the air and give up. That probably deserves an\nexplanation...\n\nYears ago someone noticed that if a user ran \"git ls-files -o\nfoo/bar/one foo/bar/two\", that we could try to optimize by noticing\nthat we won't be interested in anything until we get to foo/bar/. So,\nthey tried to short-circuit the read_directory_recursive() and\nreaddir() calls, but couldn't reuse the same treat_path() logic to\ncheck that we should even go into foo/bar/ at all. So there was some\ncopy & paste from treat_path() into a new treat_leading_path()...and\nthat both missed some important parts and the logic further diverged\nover time.\n\nThis patch was about categorizing the suite of bugs that arose from\nnot using treat_path() for checks from both codepaths, and tried to\ncorrect those problems. treat_path() takes a dirent, and several of\nthe functions it calls all take a dirent. It'd be an awful lot of\nwork to rip it out. So I manufactured a dirent myself so that we\ncould use the same codepaths and not only fix all these bugs but\nprevent future ones as well. If we can't manufacture a dirent, then\nunless someone else has some bright ideas about something clever we\ncan do, then I think this problem blows up in complexity to a level\nwhere I don't think it's worth addressing.\n\nI almost ripped the optimization out altogether (just how much do we\nreally save by not looking into the leading two directories?), except\nthat unpack_trees() calls into the same code with a leading path and I\ndidn't want to mess with that.\n\nAny bright ideas about what to do here?\n"},{"id":"388299","messageId":"xmqqzhfshsk2.fsf@gitster-ct.c.googlers.com","threadId":"52419","inReplyTo":"CABPp-BF2UL5m4_f2tETfZvq=011KsKheGA=axFwYhC+kJXi0JQ@mail.gmail.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-12-16T21:25:01Z","receivedAt":"2019-12-16T21:25:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Elijah Newren writes:\n\n> Any bright ideas about what to do here?\n\nRestructuring the code so that we do not use \"struct dirent\" in the\nfirst place, even in the original code that used only those obtained\nfrom readdir(), perhaps? Then the codepath that would take the _thing_\nthat describes the diretory entry would expect to see the data in\nthe struct you define (not \"struct dirent\" from the system), and you\ncan safely manufacture ones out of thin air.\n\n\n"},{"id":"388307","messageId":"CABPp-BHETKt1rTT76u9phGqjN+RU6B-ydzM=S0tzE6To_N4PQw@mail.gmail.com","threadId":"52419","inReplyTo":"xmqqzhfshsk2.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-12-16T22:39:34Z","receivedAt":"2019-12-16T22:39:48Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Mon, Dec 16, 2019 at 1:25 PM Junio C Hamano wrote:\n>\n> Elijah Newren writes:\n>\n> > Any bright ideas about what to do here?\n>\n> Restructuring the code so that we do not use \"struct dirent\" in the\n> first place, even in the original code that used only those obtained\n> from readdir(), perhaps? Then the codepath that would take the _thing_\n> that describes the diretory entry would expect to see the data in\n> the struct you define (not \"struct dirent\" from the system), and you\n> can safely manufacture ones out of thin air.\n\nOkay, I'll submit a new series dropping most the patches, but note\nthis thread in the commit message of the new testcases in case someone\n(maybe my future self?) wants to dig further.\n"},{"id":"388316","messageId":"nycvar.QRO.7.76.6.1912170101230.46@tvgsbejvaqbjf.bet","threadId":"52419","inReplyTo":"CABPp-BGoC_D6LzzMNyf30wFssTU2WA1kTLmFvJ2Do+Tfg4+YQA@mail.gmail.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-17T00:04:36Z","receivedAt":"2019-12-17T00:05:08Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Mon, 16 Dec 2019, Elijah Newren wrote:\n\n> On Mon, Dec 16, 2019 at 5:51 AM Elijah Newren wrote:\n> >\n> > On Sun, Dec 15, 2019 at 2:29 AM Johannes Schindelin\n> > wrote:\n> > >\n> > > Hi Elijah,\n> > >\n> > > I have not had time to dive deeply into this, but I know that it _does_\n> > > cause a ton of segmentation faults in the `shears/pu` branch (where all of\n> > > Git for Windows' patches are rebased on top of `pu`):\n> >\n> > Weird. If it's going to cause segmentation faults at all, it would\n> > certainly do it all over the place, but I tested the patches on the\n> > major platforms using your Azure Pipelines setup on git.git so it\n> > should be good on all the platforms. Did your shears/pu branch make\n> > some other changes to the setup?\n\nNot really.\n\n> > > On Tue, 10 Dec 2019, Elijah Newren via GitGitGadget wrote:\n> > >\n> > > > diff --git a/dir.c b/dir.c\n> > > > index 645b44ea64..9c71a9ac21 100644\n> > > > --- a/dir.c\n> > > > +++ b/dir.c\n> > > > @@ -2102,37 +2102,69 @@ static int treat_leading_path(struct dir_struct *dir,\n> > > > const struct pathspec *pathspec)\n> > > > {\n> > > > struct strbuf sb = STRBUF_INIT;\n> > > > - int baselen, rc = 0;\n> > > > + int prevlen, baselen;\n> > > > const char *cp;\n> > > > + struct cached_dir cdir;\n> > > > + struct dirent de;\n> > > > + enum path_treatment state = path_none;\n> > > > +\n> > > > + /*\n> > > > + * For each directory component of path, we are going to check whether\n> > > > + * that path is relevant given the pathspec. For example, if path is\n> > > > + * foo/bar/baz/\n> > > > + * then we will ask treat_path() whether we should go into foo, then\n> > > > + * whether we should go into bar, then whether baz is relevant.\n> > > > + * Checking each is important because e.g. if path is\n> > > > + * .git/info/\n> > > > + * then we need to check .git to know we shouldn't traverse it.\n> > > > + * If the return from treat_path() is:\n> > > > + * * path_none, for any path, we return false.\n> > > > + * * path_recurse, for all path components, we return true\n> > > > + * * for some intermediate component, we make sure\n> > > > + * to add that path to the relevant list but return false\n> > > > + * signifying that we shouldn't recurse into it.\n> > > > + */\n> > > >\n> > > > while (len && path[len - 1] == '/')\n> > > > len--;\n> > > > if (!len)\n> > > > return 1;\n> > > > +\n> > > > + memset(&cdir, 0, sizeof(cdir));\n> > > > + memset(&de, 0, sizeof(de));\n> > > > + cdir.de = &de;\n> > > > + de.d_type = DT_DIR;\n> > >\n> > > So here, `de` is zeroed out, and therefore `de.d_name` is `NULL`.\n> >\n> > Um, yeah...didn't I have an allocation of de.d_name here? It will\n> > always have a subset of path copied into it, so an allocation of len+1\n> > is plenty long enough.\n>\n> Actually, it looks like I looked up the definition of dirent\n> previously and forgot by the time you emailed. On linux, from\n> /usr/include/bits/dirent.h:\n>\n> struct dirent\n> {\n> ....\n> unsigned char d_type;\n> char d_name[256]; /* We must not include limits.h! */\n> };\n>\n> and from compat/win32/dirent.h defines it as:\n>\n> struct dirent {\n> unsigned char d_type; /* file type to prevent lstat after\n> readdir */\n> char d_name[MAX_PATH * 3]; /* file name (* 3 for UTF-8 conversion) */\n> };\n>\n> and 'man dirent' on Mac OS X says it's defined as:\n>\n> struct dirent {\n> ...\n> _uint8_t d_type;\n> _unit8_t d_namlen; /* length of string in d_name */\n> char d_name[255+1]; /* name must be no longer than this */\n> }\n>\n> so, allocating it would be incorrect and my memset would just fill\n> d_name with nul characters.\n>\n>\n> But the raises the question...what kind of segfaults are you getting?\n> Can you link to any builds or post any stack traces? Can I duplicate\n> with some copy of git-for-windows on linux?\n\nIf you care to look at our very own `compat/win32/dirent.h`, you will see\nthis:\n\nstruct dirent {\n unsigned char d_type; /* file type to prevent lstat after readdir */\n char *d_name; /* file name */\n};\n\nAnd looking at\nhttps://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html, I\ndo not see any guarantee of that `[256]` at all:\n\nThe header shall [...] define the structure dirent which shall\ninclude the following members:\n\n[XSI][Option Start]\nino_t d_ino File serial number.\n[Option End]\nchar d_name[] Filename string of entry.\n\nYou will notice that not even `d_type` is guaranteed.\n\nCiao,\nDscho\n"},{"id":"388317","messageId":"xmqqr213iz9z.fsf@gitster-ct.c.googlers.com","threadId":"52419","inReplyTo":"nycvar.QRO.7.76.6.1912170101230.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-12-17T00:14:32Z","receivedAt":"2019-12-17T00:14:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> If you care to look at our very own `compat/win32/dirent.h`, you will see\n> this:\n>\n> struct dirent {\n> unsigned char d_type; /* file type to prevent lstat after readdir */\n> char *d_name; /* file name */\n> };\n>\n> And looking at\n> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html, I\n> do not see any guarantee of that `[256]` at all:\n>\n> The header shall [...] define the structure dirent which shall\n> include the following members:\n>\n> [XSI][Option Start]\n> ino_t d_ino File serial number.\n> [Option End]\n> char d_name[] Filename string of entry.\n>\n> You will notice that not even `d_type` is guaranteed.\n\nI am reasonably sure that the code (without Elijah's patches anyway)\ntakes the possibility of missing d_type into account already.\n\nDoesn't the above mean d_name[] has to be an in-place array of some\nsize (i.e. even a flex-array is OK)? It does not look to me that it\nallows for it to be a pointer pointing at elsewhere (possibly on\nheap), which may be asking for trouble.\n"},{"id":"388328","messageId":"CABPp-BGFRMNAgeyTvDQ3F5nH36ERn+ndjrwaXuLUE-Uto_RBdQ@mail.gmail.com","threadId":"52419","inReplyTo":"nycvar.QRO.7.76.6.1912170101230.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-12-17T05:26:27Z","receivedAt":"2019-12-17T05:26:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Dscho,\n\nOn Mon, Dec 16, 2019 at 4:04 PM Johannes Schindelin\n wrote:\n> On Mon, 16 Dec 2019, Elijah Newren wrote:\n> > On Mon, Dec 16, 2019 at 5:51 AM Elijah Newren wrote:\n> > >\n> > > On Sun, Dec 15, 2019 at 2:29 AM Johannes Schindelin\n> > > wrote:\n> > > >\n> > > > Hi Elijah,\n> > > >\n> > > > I have not had time to dive deeply into this, but I know that it _does_\n> > > > cause a ton of segmentation faults in the `shears/pu` branch (where all of\n> > > > Git for Windows' patches are rebased on top of `pu`):\n> > >\n> > > Weird. If it's going to cause segmentation faults at all, it would\n> > > certainly do it all over the place, but I tested the patches on the\n> > > major platforms using your Azure Pipelines setup on git.git so it\n> > > should be good on all the platforms. Did your shears/pu branch make\n> > > some other changes to the setup?\n>\n> Not really.\n>\n> >\n> > Actually, it looks like I looked up the definition of dirent\n> > previously and forgot by the time you emailed. On linux, from\n> > /usr/include/bits/dirent.h:\n...\n> > and from compat/win32/dirent.h defines it as:\n> >\n> > struct dirent {\n> > unsigned char d_type; /* file type to prevent lstat after\n> > readdir */\n> > char d_name[MAX_PATH * 3]; /* file name (* 3 for UTF-8 conversion) */\n> > };\n...\n>\n> If you care to look at our very own `compat/win32/dirent.h`, you will see\n> this:\n\nInteresting, we both brought up compat/win32/dirent.h and quoted from\nit in our emails...\n\n> struct dirent {\n> unsigned char d_type; /* file type to prevent lstat after readdir */\n> char *d_name; /* file name */\n> };\n\n...but the contents were different? Looks like git-for-windows forked\ncompat/win32/dirent.h, possibly in a way that violates POSIX as\npointed out by Junio. Any reason those changes weren't sent back\nupstream, by chance? Feels odd having a compat/win32/ directory that\nour downstream windows users aren't actually using. It also means the\ntesting I'm getting from gitgitgadget and your Azure setup (which all\nis really, really nice by the way), is far less reassuring and helpful\nthan I hoped.\n\n> And looking at\n> https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html, I\n> do not see any guarantee of that `[256]` at all:\n>\n> The header shall [...] define the structure dirent which shall\n> include the following members:\n>\n> [XSI][Option Start]\n> ino_t d_ino File serial number.\n> [Option End]\n> char d_name[] Filename string of entry.\n>\n> You will notice that not even `d_type` is guaranteed.\n\nDoh, yeah, I messed that up too.\n\nAnyway, as I mentioned to Junio, I'll resubmit after gutting the\nseries. I'll still include a fix for the issue that a real world user\nreported, but all the other ancillary bugs I found that have been\naround for over a decade aren't important enough to merit a major\nrefactor, IMO.\n\nElijah\n"},{"id":"388333","messageId":"pull.676.v3.git.git.1576571586.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v2.git.git.1576008027.gitgitgadget@gmail.com","subject":"[PATCH v3 0/3] Directory traversal bugs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-17T08:33:03Z","receivedAt":"2019-12-17T08:33:11Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series documents multiple fill_directory() bugs, and fixes the one that\nis new to 2.24.0 coming from en/clean-nested-with-ignored-topic, the rest\nhaving been around in versions of git going back up to a decade. \n\nChanges since v2:\n\n * gutted the series of most the fixes, dropping the patch count from 8 to\n 3, due to incompatibility with git-for-windows (which interestingly has a\n different compat/win32/dirent.h than git.git does). The only bugs\n reported by a user are fixed by patch 3, and fixing the remaining bugs\n (which I found while investigating the one fixed bug) would require a\n major refactor that I don't have the time for currently.\n\nElijah Newren (3):\n t3011: demonstrate directory traversal failures\n dir: remove stray quote character in comment\n dir: exit before wildcard fall-through if there is no wildcard\n\n dir.c | 9 +-\n ...common-prefixes-and-directory-traversal.sh | 209 ++++++++++++++++++\n 2 files changed, 217 insertions(+), 1 deletion(-)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\n\nbase-commit: da72936f544fec5a335e66432610e4cef4430991\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-676%2Fnewren%2Fls-files-bug-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-676/newren/ls-files-bug-v3\nPull-Request: https://github.com/git/git/pull/676\n\nRange-diff vs v2:\n\n 1: 6d659b2302 ! 1: 61d303d8bd t3011: demonstrate directory traversal failures\n @@ -14,6 +14,18 @@\n of the en/clean-nested-with-ignored-topic); the other 5 also failed\n under git-2.23.0 and earlier.\n \n + The old failing tests can be traced down to the common prefix\n + optimization in dir.c handling paths differently than\n + read_directory_recursive() and treat_path() would, due to incomplete\n + duplication of logic into treat_leading_path() and having that\n + function call treat_one_path() rather than treat_path(). Fixing\n + that problem would require restructuring treat_path() and its full\n + call hierarchy to stop taking a dirent; see\n + https://lore.kernel.org/git/xmqqzhfshsk2.fsf@gitster-ct.c.googlers.com/\n + and the thread surrounding it for details.\n +\n + For now, simply document the breakages.\n +\n Signed-off-by: Elijah Newren \n \n diff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\n 2: 79f2b56174 < -: ---------- Revert \"dir.c: make 'git-status --ignored' work within leading directories\"\n 3: d6f858cab1 = 2: 49b0b628db dir: remove stray quote character in comment\n 4: 8d2d98eec3 = 3: 47814640e4 dir: exit before wildcard fall-through if there is no wildcard\n 5: d2f5623bd7 < -: ---------- dir: break part of read_directory_recursive() out for reuse\n 6: 9839aca00a < -: ---------- dir: fix checks on common prefix directory\n 7: df7f08886a < -: ---------- dir: synchronize treat_leading_path() and read_directory_recursive()\n 8: 77b57e44fd < -: ---------- dir: consolidate similar code in treat_directory()\n\n-- \ngitgitgadget\n"},{"id":"388334","messageId":"49b0b628dbc898fc2dd93603b4bc23d41037e569.1576571586.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v3.git.git.1576571586.gitgitgadget@gmail.com","subject":"[PATCH v3 2/3] dir: remove stray quote character in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-17T08:33:05Z","receivedAt":"2019-12-17T08:33:12Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 61f559f980..13c0c2333c 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -373,7 +373,7 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t !ps_strncmp(item, match, name, namelen))\n \t\t\treturn MATCHED_RECURSIVELY_LEADING_PATHSPEC;\n \n-\t\t/* name\" doesn't match up to the first wild character */\n+\t\t/* name doesn't match up to the first wild character */\n \t\tif (item->nowildcard_len < item->len &&\n \t\t ps_strncmp(item, match, name,\n \t\t\t item->nowildcard_len - prefix))\n-- \ngitgitgadget\n\n"},{"id":"388335","messageId":"47814640e411b3fda19e5035234513e7d238f75f.1576571586.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v3.git.git.1576571586.gitgitgadget@gmail.com","subject":"[PATCH v3 3/3] dir: exit before wildcard fall-through if there is no wildcard","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-17T08:33:06Z","receivedAt":"2019-12-17T08:33:14Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe DO_MATCH_LEADING_PATHSPEC had a fall-through case for if there was a\nwildcard, noting that we don't yet have enough information to determine\nif a further paths under the current directory might match due to the\npresence of wildcards. But if we have no wildcards in our pathspec,\nthen we shouldn't get to that fall-through case.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 7 +++++++\n t/t3011-common-prefixes-and-directory-traversal.sh | 4 ++--\n 2 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 13c0c2333c..f08bb073ef 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -379,6 +379,13 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t\t item->nowildcard_len - prefix))\n \t\t\treturn 0;\n \n+\t\t/*\n+\t\t * name has no wildcard, and it didn't match as a leading\n+\t\t * pathspec so return.\n+\t\t */\n+\t\tif (item->nowildcard_len == item->len)\n+\t\t\treturn 0;\n+\n \t\t/*\n \t\t * Here is where we would perform a wildmatch to check if\n \t\t * \"name\" can be matched as a directory (or a prefix) against\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex 54f80c62b8..d6e161ddd8 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -92,7 +92,7 @@ test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+test_expect_success 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n \tcat <<-EOF >expect &&\n \tuntracked_dir/empty\n \tuntracked_repo/\n@@ -110,7 +110,7 @@ test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses int\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+test_expect_success 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n \tcat <<-EOF >expect &&\n \tuntracked_dir/\n \tuntracked_repo/\n-- \ngitgitgadget\n"},{"id":"388336","messageId":"61d303d8bdd3287996a1037982d0d27ded295ec0.1576571586.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v3.git.git.1576571586.gitgitgadget@gmail.com","subject":"[PATCH v3 1/3] t3011: demonstrate directory traversal failures","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-17T08:33:04Z","receivedAt":"2019-12-17T08:33:15Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd several tests demonstrating directory traversal failures of various\nsorts in dir.c (and one similar looking test that turns out to be a\ngit_fnmatch bug). A lot of these tests look like near duplicates of\neach other, but an optimization path in dir.c to pre-descend into a\ncommon prefix and the specialized treatment of trailing slashes in dir.c\nmean the tiny differences are sometimes important and potentially cause\ndifferent codepaths to be explored.\n\nOf the 7 failing tests, 2 are new to git-2.24.0 (tweaked by side effects\nof the en/clean-nested-with-ignored-topic); the other 5 also failed\nunder git-2.23.0 and earlier.\n\nThe old failing tests can be traced down to the common prefix\noptimization in dir.c handling paths differently than\nread_directory_recursive() and treat_path() would, due to incomplete\nduplication of logic into treat_leading_path() and having that\nfunction call treat_one_path() rather than treat_path(). Fixing\nthat problem would require restructuring treat_path() and its full\ncall hierarchy to stop taking a dirent; see\n https://lore.kernel.org/git/xmqqzhfshsk2.fsf@gitster-ct.c.googlers.com/\nand the thread surrounding it for details.\n\nFor now, simply document the breakages.\n\nSigned-off-by: Elijah Newren \n---\n ...common-prefixes-and-directory-traversal.sh | 209 ++++++++++++++++++\n 1 file changed, 209 insertions(+)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nnew file mode 100755\nindex 0000000000..54f80c62b8\n--- /dev/null\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -0,0 +1,209 @@\n+#!/bin/sh\n+\n+test_description='directory traversal handling, especially with common prefixes'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit hello &&\n+\n+\t>empty &&\n+\tmkdir untracked_dir &&\n+\t>untracked_dir/empty &&\n+\tgit init untracked_repo &&\n+\t>untracked_repo/empty &&\n+\n+\tcat <<-EOF >.gitignore &&\n+\tignored\n+\tan_ignored_dir/\n+\tEOF\n+\tmkdir an_ignored_dir &&\n+\tmkdir an_untracked_dir &&\n+\t>an_ignored_dir/ignored &&\n+\t>an_ignored_dir/untracked &&\n+\t>an_untracked_dir/ignored &&\n+\t>an_untracked_dir/untracked\n+'\n+\n+test_expect_success 'git ls-files -o shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_ignored_dir/ignored\n+\tan_ignored_dir/untracked\n+\tan_untracked_dir/ignored\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --exclude-standard shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --exclude-standard >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses into untracked_dir only' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir/ untracked_repo/ does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o .git shows nothing' '\n+\tgit ls-files -o .git >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_failure 'git ls-files -o .git/ shows nothing' '\n+\tgit ls-files -o .git/ >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o untracked_* recurses appropriately' '\n+\tmkdir \"untracked_*\" &&\n+\t>\"untracked_*/empty\" &&\n+\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/empty\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# It turns out fill_directory returns the right paths, but ls-files' post-call\n+# filtering in show_dir_entry() via calling dir_path_match() which ends up\n+# in git_fnmatch() has logic for PATHSPEC_ONESTAR that assumes the pathspec\n+# must match the full path; it doesn't check it for matching a leading\n+# directory.\n+test_expect_failure FUNNYNAMES 'git ls-files -o untracked_*/ recurses appropriately' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/empty\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_* does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_*/ does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o consistent between one or two dirs' '\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ an_untracked_dir/ >tmp &&\n+\t! grep ^an_ignored_dir/ tmp >expect &&\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# ls-files doesn't have a way to request showing both untracked and ignored\n+# files at the same time, so use `git status --ignored`\n+test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n+\tcat <<-EOF >expect &&\n+\t?? an_untracked_dir/\n+\t!! an_untracked_dir/ignored\n+\tEOF\n+\tgit status --porcelain --ignored >output &&\n+\tgrep an_untracked_dir output >expect &&\n+\tgit status --porcelain --ignored an_untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"388350","messageId":"nycvar.QRO.7.76.6.1912171157140.46@tvgsbejvaqbjf.bet","threadId":"52419","inReplyTo":"xmqqr213iz9z.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-17T11:08:12Z","receivedAt":"2019-12-17T11:08:41Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 16 Dec 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > If you care to look at our very own `compat/win32/dirent.h`, you will see\n> > this:\n> >\n> > struct dirent {\n> > unsigned char d_type; /* file type to prevent lstat after readdir */\n> > char *d_name; /* file name */\n> > };\n> >\n> > And looking at\n> > https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html, I\n> > do not see any guarantee of that `[256]` at all:\n> >\n> > The header shall [...] define the structure dirent which shall\n> > include the following members:\n> >\n> > [XSI][Option Start]\n> > ino_t d_ino File serial number.\n> > [Option End]\n> > char d_name[] Filename string of entry.\n> >\n> > You will notice that not even `d_type` is guaranteed.\n>\n> I am reasonably sure that the code (without Elijah's patches anyway)\n> takes the possibility of missing d_type into account already.\n>\n> Doesn't the above mean d_name[] has to be an in-place array of some\n> size (i.e. even a flex-array is OK)? It does not look to me that it\n> allows for it to be a pointer pointing at elsewhere (possibly on\n> heap), which may be asking for trouble.\n\nYou are right, of course.\n\nI also was not _quite_ spot on, as I had looked at Git for Windows'\n`shears/pu` branch, not at the `pu` branch. Alas, we have patches in Git\nfor Windows that allow for switching to a faster, caching way to access\nthe stat() and readdir() data (it is called the \"FSCache\" and it is\nresponsible for some rather dramatic speed-ups). And these patches change\n`struct dirent` to the form that is quoted above, to allow switching back\nand forth between two _different_ backends, storing the actual `d_name`\nnot in `struct dirent` but in `DIR`.\n\nIs this compliant with POSIX? I guess not. Does it work? Yes, it does.\n\nI can't know for sure that it makes a dent, but FSCache is designed for\nspeed, and it actually does not even store the `d_name` in the `DIR`, but\ndirectly in the cache structure, avoiding copying at all.\n\nIn short: if we can allow FSCache to keep operating that way (i.e. keep\n`d_name` as a pointer), then that would be helpful to keep the performance\non Windows somewhat within acceptable boundaries.\n\nCiao,\nDscho\n"},{"id":"388352","messageId":"nycvar.QRO.7.76.6.1912171209170.46@tvgsbejvaqbjf.bet","threadId":"52419","inReplyTo":"CABPp-BGFRMNAgeyTvDQ3F5nH36ERn+ndjrwaXuLUE-Uto_RBdQ@mail.gmail.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-17T11:15:43Z","receivedAt":"2019-12-17T11:16:14Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Mon, 16 Dec 2019, Elijah Newren wrote:\n\n> On Mon, Dec 16, 2019 at 4:04 PM Johannes Schindelin\n> wrote:\n> > On Mon, 16 Dec 2019, Elijah Newren wrote:\n> > > On Mon, Dec 16, 2019 at 5:51 AM Elijah Newren wrote:\n> > > >\n> > > > On Sun, Dec 15, 2019 at 2:29 AM Johannes Schindelin\n> > > > wrote:\n> > > > >\n> > > > > Hi Elijah,\n> > > > >\n> > > > > I have not had time to dive deeply into this, but I know that it _does_\n> > > > > cause a ton of segmentation faults in the `shears/pu` branch (where all of\n> > > > > Git for Windows' patches are rebased on top of `pu`):\n> > > >\n> > > > Weird. If it's going to cause segmentation faults at all, it would\n> > > > certainly do it all over the place, but I tested the patches on the\n> > > > major platforms using your Azure Pipelines setup on git.git so it\n> > > > should be good on all the platforms. Did your shears/pu branch make\n> > > > some other changes to the setup?\n> >\n> > Not really.\n> >\n> > >\n> > > Actually, it looks like I looked up the definition of dirent\n> > > previously and forgot by the time you emailed. On linux, from\n> > > /usr/include/bits/dirent.h:\n> ...\n> > > and from compat/win32/dirent.h defines it as:\n> > >\n> > > struct dirent {\n> > > unsigned char d_type; /* file type to prevent lstat after\n> > > readdir */\n> > > char d_name[MAX_PATH * 3]; /* file name (* 3 for UTF-8 conversion) */\n> > > };\n> ...\n> >\n> > If you care to look at our very own `compat/win32/dirent.h`, you will see\n> > this:\n>\n> Interesting, we both brought up compat/win32/dirent.h and quoted from\n> it in our emails...\n>\n> > struct dirent {\n> > unsigned char d_type; /* file type to prevent lstat after readdir */\n> > char *d_name; /* file name */\n> > };\n>\n> ...but the contents were different? Looks like git-for-windows forked\n> compat/win32/dirent.h, possibly in a way that violates POSIX as\n> pointed out by Junio.\n\nYep, I messed that up, sorry.\n\n> Any reason those changes weren't sent back upstream, by chance? Feels\n> odd having a compat/win32/ directory that our downstream windows users\n> aren't actually using. It also means the testing I'm getting from\n> gitgitgadget and your Azure setup (which all is really, really nice by\n> the way), is far less reassuring and helpful than I hoped.\n\nYes. I was ready to submit the FSCache feature to the Git mailing list for\nreview some 2.5 years ago when along came Ben Peart, finding ways to speed\nup FSCache even further. That is the reason why I held off, and I still\nhave to condense the patches (which currently form a topology of 17 patch\nseries!!!) into a nice small patch series that does not reflect the\nmeandering history of the FSCache history, but instead presents one neat\nstory.\n\n> > And looking at\n> > https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html, I\n> > do not see any guarantee of that `[256]` at all:\n> >\n> > The header shall [...] define the structure dirent which shall\n> > include the following members:\n> >\n> > [XSI][Option Start]\n> > ino_t d_ino File serial number.\n> > [Option End]\n> > char d_name[] Filename string of entry.\n> >\n> > You will notice that not even `d_type` is guaranteed.\n>\n> Doh, yeah, I messed that up too.\n>\n> Anyway, as I mentioned to Junio, I'll resubmit after gutting the\n> series. I'll still include a fix for the issue that a real world user\n> reported, but all the other ancillary bugs I found that have been\n> around for over a decade aren't important enough to merit a major\n> refactor, IMO.\n\nHmm. I am really sorry that I nudged you to go down this route. Quite\nhonestly, I'd rather add an ugly work-around that is Windows-only just so\nthat you can fix those ancillary bugs.\n\nI might even go so far as to adjust the FSCache's internal data structure\nto _store_ `struct dirent` items, then the fast `readdir()` implementation\ncould be even faster by just pointing at those items.\n\nWhat do you think? Can we strike a deal to keep those bug fixes?\n\nCiao,\nDscho\n"},{"id":"388353","messageId":"nycvar.QRO.7.76.6.1912171217000.46@tvgsbejvaqbjf.bet","threadId":"52419","inReplyTo":"pull.676.v3.git.git.1576571586.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 0/3] Directory traversal bugs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-17T11:18:12Z","receivedAt":"2019-12-17T11:18:40Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Elijah,\n\nOn Tue, 17 Dec 2019, Elijah Newren via GitGitGadget wrote:\n\n> This series documents multiple fill_directory() bugs, and fixes the one that\n> is new to 2.24.0 coming from en/clean-nested-with-ignored-topic, the rest\n> having been around in versions of git going back up to a decade.\n>\n> Changes since v2:\n>\n> * gutted the series of most the fixes, dropping the patch count from 8 to\n> 3, due to incompatibility with git-for-windows (which interestingly has a\n> different compat/win32/dirent.h than git.git does). The only bugs\n> reported by a user are fixed by patch 3, and fixing the remaining bugs\n> (which I found while investigating the one fixed bug) would require a\n> major refactor that I don't have the time for currently.\n\nI am really sorry that I caused you so much work.\n\nAs I said elsewhere, if Git for Windows' FSCache hack is the only thing\nthat is broken by this patch series, in light of the bugs that it _does_\nfix I would rather adjust the FSCache patches to accommodate v2.\n\nWhat do you think?\n\nCiao,\nDscho\n\n> Elijah Newren (3):\n> t3011: demonstrate directory traversal failures\n> dir: remove stray quote character in comment\n> dir: exit before wildcard fall-through if there is no wildcard\n>\n> dir.c | 9 +-\n> ...common-prefixes-and-directory-traversal.sh | 209 ++++++++++++++++++\n> 2 files changed, 217 insertions(+), 1 deletion(-)\n> create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n>\n>\n> base-commit: da72936f544fec5a335e66432610e4cef4430991\n> Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-676%2Fnewren%2Fls-files-bug-v3\n> Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-676/newren/ls-files-bug-v3\n> Pull-Request: https://github.com/git/git/pull/676\n>\n> Range-diff vs v2:\n>\n> 1: 6d659b2302 ! 1: 61d303d8bd t3011: demonstrate directory traversal failures\n> @@ -14,6 +14,18 @@\n> of the en/clean-nested-with-ignored-topic); the other 5 also failed\n> under git-2.23.0 and earlier.\n>\n> + The old failing tests can be traced down to the common prefix\n> + optimization in dir.c handling paths differently than\n> + read_directory_recursive() and treat_path() would, due to incomplete\n> + duplication of logic into treat_leading_path() and having that\n> + function call treat_one_path() rather than treat_path(). Fixing\n> + that problem would require restructuring treat_path() and its full\n> + call hierarchy to stop taking a dirent; see\n> + https://lore.kernel.org/git/xmqqzhfshsk2.fsf@gitster-ct.c.googlers.com/\n> + and the thread surrounding it for details.\n> +\n> + For now, simply document the breakages.\n> +\n> Signed-off-by: Elijah Newren \n>\n> diff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\n> 2: 79f2b56174 < -: ---------- Revert \"dir.c: make 'git-status --ignored' work within leading directories\"\n> 3: d6f858cab1 = 2: 49b0b628db dir: remove stray quote character in comment\n> 4: 8d2d98eec3 = 3: 47814640e4 dir: exit before wildcard fall-through if there is no wildcard\n> 5: d2f5623bd7 < -: ---------- dir: break part of read_directory_recursive() out for reuse\n> 6: 9839aca00a < -: ---------- dir: fix checks on common prefix directory\n> 7: df7f08886a < -: ---------- dir: synchronize treat_leading_path() and read_directory_recursive()\n> 8: 77b57e44fd < -: ---------- dir: consolidate similar code in treat_directory()\n>\n> --\n> gitgitgadget\n>\n"},{"id":"388378","messageId":"CABPp-BEkX9cH1=r3dJ4WLzcJKVcF-KpGUkshL34MMp3Xhhhpuw@mail.gmail.com","threadId":"52419","inReplyTo":"nycvar.QRO.7.76.6.1912171209170.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-12-17T16:58:49Z","receivedAt":"2019-12-17T16:59:04Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Dscho,\n\nOn Tue, Dec 17, 2019 at 3:16 AM Johannes Schindelin\n wrote:\n>\n> Hi Elijah,\n>\n> On Mon, 16 Dec 2019, Elijah Newren wrote:\n>\n> > On Mon, Dec 16, 2019 at 4:04 PM Johannes Schindelin\n> > wrote:\n> > > On Mon, 16 Dec 2019, Elijah Newren wrote:\n> > > > On Mon, Dec 16, 2019 at 5:51 AM Elijah Newren wrote:\n> > > > >\n> > > > > On Sun, Dec 15, 2019 at 2:29 AM Johannes Schindelin\n> > > > > wrote:\n> > > > > >\n> > > > > > Hi Elijah,\n> > > > > >\n> > > > > > I have not had time to dive deeply into this, but I know that it _does_\n> > > > > > cause a ton of segmentation faults in the `shears/pu` branch (where all of\n> > > > > > Git for Windows' patches are rebased on top of `pu`):\n> > > > >\n> > > > > Weird. If it's going to cause segmentation faults at all, it would\n> > > > > certainly do it all over the place, but I tested the patches on the\n> > > > > major platforms using your Azure Pipelines setup on git.git so it\n> > > > > should be good on all the platforms. Did your shears/pu branch make\n> > > > > some other changes to the setup?\n> > >\n> > > Not really.\n> > >\n> > > >\n> > > > Actually, it looks like I looked up the definition of dirent\n> > > > previously and forgot by the time you emailed. On linux, from\n> > > > /usr/include/bits/dirent.h:\n> > ...\n> > > > and from compat/win32/dirent.h defines it as:\n> > > >\n> > > > struct dirent {\n> > > > unsigned char d_type; /* file type to prevent lstat after\n> > > > readdir */\n> > > > char d_name[MAX_PATH * 3]; /* file name (* 3 for UTF-8 conversion) */\n> > > > };\n> > ...\n> > >\n> > > If you care to look at our very own `compat/win32/dirent.h`, you will see\n> > > this:\n> >\n> > Interesting, we both brought up compat/win32/dirent.h and quoted from\n> > it in our emails...\n> >\n> > > struct dirent {\n> > > unsigned char d_type; /* file type to prevent lstat after readdir */\n> > > char *d_name; /* file name */\n> > > };\n> >\n> > ...but the contents were different? Looks like git-for-windows forked\n> > compat/win32/dirent.h, possibly in a way that violates POSIX as\n> > pointed out by Junio.\n>\n> Yep, I messed that up, sorry.\n>\n> > Any reason those changes weren't sent back upstream, by chance? Feels\n> > odd having a compat/win32/ directory that our downstream windows users\n> > aren't actually using. It also means the testing I'm getting from\n> > gitgitgadget and your Azure setup (which all is really, really nice by\n> > the way), is far less reassuring and helpful than I hoped.\n>\n> Yes. I was ready to submit the FSCache feature to the Git mailing list for\n> review some 2.5 years ago when along came Ben Peart, finding ways to speed\n> up FSCache even further. That is the reason why I held off, and I still\n> have to condense the patches (which currently form a topology of 17 patch\n> series!!!) into a nice small patch series that does not reflect the\n> meandering history of the FSCache history, but instead presents one neat\n> story.\n>\n> > > And looking at\n> > > https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/dirent.h.html, I\n> > > do not see any guarantee of that `[256]` at all:\n> > >\n> > > The header shall [...] define the structure dirent which shall\n> > > include the following members:\n> > >\n> > > [XSI][Option Start]\n> > > ino_t d_ino File serial number.\n> > > [Option End]\n> > > char d_name[] Filename string of entry.\n> > >\n> > > You will notice that not even `d_type` is guaranteed.\n> >\n> > Doh, yeah, I messed that up too.\n> >\n> > Anyway, as I mentioned to Junio, I'll resubmit after gutting the\n> > series. I'll still include a fix for the issue that a real world user\n> > reported, but all the other ancillary bugs I found that have been\n> > around for over a decade aren't important enough to merit a major\n> > refactor, IMO.\n>\n> Hmm. I am really sorry that I nudged you to go down this route. Quite\n> honestly, I'd rather add an ugly work-around that is Windows-only just so\n> that you can fix those ancillary bugs.\n\nYou brought up issues; that's what you're supposed to do. You\nshouldn't feel bad about that. Besides, the d_type one is real, and\nmeans the patches at least need a\n #if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)\nsurrounding my explicit setting of d_type. The problem wasn't what\nyou brought up or how you brought it up, it's massive fatigue on my\nend from dir.c, from before even submitting this series[*]. I'm not\ngiving up on these changes or trying to discourage anyone else from\npicking them up and extending them, I just don't want to touch them\nright now and would rather put them on the shelf for a while.\n\nElijah\n\n\n[*] If you're really curious...I got involved in dir.c because of a\nsimple bug report nearly two years ago[1], and found myself working on\na foundation that was error-prone by design[2], with ambiguous or even\nwrong documentation[3] about not just what the code does but the\nintent. Further, it was a place where not only is the correct fix\nunclear, and not only is the \"right\" behavior unclear, but the cases\nin question affect so few people that pinging the list periodically\nover more than a year can't generate enough interest for anyone else\nto hazard a guess as to what \"correct\" behavior is[4]. Stack on that\nthe fact that every time I touch this area, I think I'm really close\nto having a fix, only to find I never, ever am. There's always\none-more-thing before I can finally get back to something I really\nwanted to work on instead. Speaking of which, I've only managed to\nwork on my new merge strategy like once every 3-6 months for a small\namount of time each time. Yes, part of that's my fault with\ngit-filter-repo (another case of perpetually thinking I'm close to\ndone), rebase changes, and whatnot. But this series arose right when\nI had my calendar nearly cleared so that I could work on the merge\nstrategy again (and of course the rebase bug report came in about the\nsame time too). But at least git-filter-repo and rebase are generally\nuseful; dir.c at most generates \"meh, this seems annoying\" reports.\nAnd I've already fixed all of those, the remaining fixes are stuff\nthat it appears I'm the only one to have reported, and I only reported\nit because I was digging into the other \"meh, seems annoying\" reports.\nI'm usually happy when I have a patch series ready to submit to git;\nit means I think I'll make things better for others. I didn't feel\nthat way with this series; I kind of wanted to just drop it entirely\nand not even turn it in. But I figured I should to at least document\nmy findings, so I pushed myself to submit and hoped no one would\nrespond. Then this issue arose and when I mentioned in my\npossibilities of fixing it that ripping the usage of dirent out would\nbe a lot of work and would probably cause me to give up and asked for\nideas, Junio responded that we should rip out dirent. I think he's\nright, and it's important the he defend code quality and point out the\nright way to do things, it's just that I want out of this rabbit hole\nright now.\n\n[1] https://lore.kernel.org/git/20180405173446.32372-1-newren@gmail.com/\n[2] https://lore.kernel.org/git/xmqqefjp6sko.fsf@gitster-ct.c.googlers.com/\n[3] e.g. https://lore.kernel.org/git/20190905154735.29784-10-newren@gmail.com/\n[4] https://lore.kernel.org/git/20190905154735.29784-1-newren@gmail.com/\nand links referenced therein\n"},{"id":"388379","messageId":"xmqqk16uj1qa.fsf@gitster-ct.c.googlers.com","threadId":"52419","inReplyTo":"nycvar.QRO.7.76.6.1912171157140.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-12-17T17:33:49Z","receivedAt":"2019-12-17T17:33:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n>> > [XSI][Option Start]\n>> > ino_t d_ino File serial number.\n>> > [Option End]\n>> > char d_name[] Filename string of entry.\n>> >\n>> > You will notice that not even `d_type` is guaranteed.\n>>\n>> I am reasonably sure that the code (without Elijah's patches anyway)\n>> takes the possibility of missing d_type into account already.\n>>\n>> Doesn't the above mean d_name[] has to be an in-place array of some\n>> size (i.e. even a flex-array is OK)? It does not look to me that it\n>> allows for it to be a pointer pointing at elsewhere (possibly on\n>> heap), which may be asking for trouble.\n>\n> You are right, of course.\n>\n> ...\n>\n> Is this compliant with POSIX? I guess not. Does it work? Yes, it does.\n\nI actually would not throw it into \"it works\" category. The obvious\nimplication is that a program like this:\n\n\tstatic struct dirent *fabricate(const char *name)\n\t{\n \t/* over-allocate as we do not know how long the\td_name[] is */\n\t\tstruct dirent *ent = calloc(1, sizeof(*ent) + strlen(name) + 1);\n\t\tstrcpy(ent->d_name, name);\n\t\treturn ent;\n\t}\n\n\tstatic void show_name(const struct dirent *ent)\n\t{\n\t\tprintf(\"%s\\n\", ent->d_name);\n\t}\n\n\tint main(int ac, char **av)\n\t{\n\t\tstruct dirent *mine = fabricate(\"mine\");\n show_name(mine);\n\t\tfree(mine);\n\t\treturn 0;\n\t}\n\nwould be broken if you do not have d_name as an array.\n\nI would not be surprised if the segfaults you saw with Elijah's\nseries all were caused by your d_name not being an array, and if\nthat is the case, I'd rather see it fixed on your end than fixes\nwithdrawn.\n\nThanks.\n"},{"id":"388383","messageId":"xmqq1rt2ized.fsf@gitster-ct.c.googlers.com","threadId":"52419","inReplyTo":"nycvar.QRO.7.76.6.1912171217000.46@tvgsbejvaqbjf.bet","subject":"Re: [PATCH v3 0/3] Directory traversal bugs","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-12-17T18:24:10Z","receivedAt":"2019-12-17T18:24:19Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin writes:\n\n> As I said elsewhere, if Git for Windows' FSCache hack is the only thing\n> that is broken by this patch series, in light of the bugs that it _does_\n> fix I would rather adjust the FSCache patches to accommodate v2.\n\nWith \"FSCache hack\", do you refer to the \"d_name is a pointer to\nelsewhere\" thing? If so, I too very much appreciate the direction\nyou are suggesting. Seeing that these three patches essentially are\nthe same as three (1/8, 3/8 and 4/8) from the v2, I'd keep all the 8\npatches from v2 in my tree for now.\n\nThanks, both.\n"},{"id":"388395","messageId":"nycvar.QRO.7.76.6.1912172031490.46@tvgsbejvaqbjf.bet","threadId":"52419","inReplyTo":"xmqqk16uj1qa.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v2 6/8] dir: fix checks on common prefix directory","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-17T19:32:04Z","receivedAt":"2019-12-17T19:32:31Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 17 Dec 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> >> > [XSI][Option Start]\n> >> > ino_t d_ino File serial number.\n> >> > [Option End]\n> >> > char d_name[] Filename string of entry.\n> >> >\n> >> > You will notice that not even `d_type` is guaranteed.\n> >>\n> >> I am reasonably sure that the code (without Elijah's patches anyway)\n> >> takes the possibility of missing d_type into account already.\n> >>\n> >> Doesn't the above mean d_name[] has to be an in-place array of some\n> >> size (i.e. even a flex-array is OK)? It does not look to me that it\n> >> allows for it to be a pointer pointing at elsewhere (possibly on\n> >> heap), which may be asking for trouble.\n> >\n> > You are right, of course.\n> >\n> > ...\n> >\n> > Is this compliant with POSIX? I guess not. Does it work? Yes, it does.\n>\n> I actually would not throw it into \"it works\" category. The obvious\n> implication is that a program like this:\n>\n> \tstatic struct dirent *fabricate(const char *name)\n> \t{\n> \t/* over-allocate as we do not know how long the\td_name[] is */\n> \t\tstruct dirent *ent = calloc(1, sizeof(*ent) + strlen(name) + 1);\n> \t\tstrcpy(ent->d_name, name);\n> \t\treturn ent;\n> \t}\n>\n> \tstatic void show_name(const struct dirent *ent)\n> \t{\n> \t\tprintf(\"%s\\n\", ent->d_name);\n> \t}\n>\n> \tint main(int ac, char **av)\n> \t{\n> \t\tstruct dirent *mine = fabricate(\"mine\");\n> show_name(mine);\n> \t\tfree(mine);\n> \t\treturn 0;\n> \t}\n>\n> would be broken if you do not have d_name as an array.\n>\n> I would not be surprised if the segfaults you saw with Elijah's\n> series all were caused by your d_name not being an array, and if\n> that is the case, I'd rather see it fixed on your end than fixes\n> withdrawn.\n\nI agree with this reasoning.\n\nCiao,\nDscho\n"},{"id":"388459","messageId":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v3.git.git.1576571586.gitgitgadget@gmail.com","subject":"[PATCH v4 0/8] Directory traversal bugs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-18T19:29:38Z","receivedAt":"2019-12-18T19:29:51Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series documents multiple fill_directory() bugs, and fixes the one that\nis new to 2.24.0 coming from en/clean-nested-with-ignored-topic, the rest\nhaving been around in versions of git going back up to a decade. \n\nChanges since v2 (v3 was sent earlier, but this series is closer to v2):\n\n * protected access to d_type with a !defined(NO_D_TYPE_IN_DIRENT) and made\n sure to allocate a dirent on the heap with some extra space for d_name\n rather than allocating the dirent on the stack.\n\nElijah Newren (8):\n t3011: demonstrate directory traversal failures\n Revert \"dir.c: make 'git-status --ignored' work within leading\n directories\"\n dir: remove stray quote character in comment\n dir: exit before wildcard fall-through if there is no wildcard\n dir: break part of read_directory_recursive() out for reuse\n dir: fix checks on common prefix directory\n dir: synchronize treat_leading_path() and read_directory_recursive()\n dir: consolidate similar code in treat_directory()\n\n dir.c | 177 +++++++++++----\n ...common-prefixes-and-directory-traversal.sh | 209 ++++++++++++++++++\n t/t7061-wtstatus-ignore.sh | 9 +-\n 3 files changed, 344 insertions(+), 51 deletions(-)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\n\nbase-commit: da72936f544fec5a335e66432610e4cef4430991\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-676%2Fnewren%2Fls-files-bug-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-676/newren/ls-files-bug-v4\nPull-Request: https://github.com/git/git/pull/676\n\nRange-diff vs v3:\n\n 1: 61d303d8bd ! 1: 6d659b2302 t3011: demonstrate directory traversal failures\n @@ -14,18 +14,6 @@\n of the en/clean-nested-with-ignored-topic); the other 5 also failed\n under git-2.23.0 and earlier.\n \n - The old failing tests can be traced down to the common prefix\n - optimization in dir.c handling paths differently than\n - read_directory_recursive() and treat_path() would, due to incomplete\n - duplication of logic into treat_leading_path() and having that\n - function call treat_one_path() rather than treat_path(). Fixing\n - that problem would require restructuring treat_path() and its full\n - call hierarchy to stop taking a dirent; see\n - https://lore.kernel.org/git/xmqqzhfshsk2.fsf@gitster-ct.c.googlers.com/\n - and the thread surrounding it for details.\n -\n - For now, simply document the breakages.\n -\n Signed-off-by: Elijah Newren \n \n diff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\n -: ---------- > 2: 79f2b56174 Revert \"dir.c: make 'git-status --ignored' work within leading directories\"\n 2: 49b0b628db = 3: d6f858cab1 dir: remove stray quote character in comment\n 3: 47814640e4 = 4: 8d2d98eec3 dir: exit before wildcard fall-through if there is no wildcard\n -: ---------- > 5: d2f5623bd7 dir: break part of read_directory_recursive() out for reuse\n -: ---------- > 6: 1f3978aa46 dir: fix checks on common prefix directory\n -: ---------- > 7: 542c6e5792 dir: synchronize treat_leading_path() and read_directory_recursive()\n -: ---------- > 8: 31079dc1cf dir: consolidate similar code in treat_directory()\n\n-- \ngitgitgadget\n"},{"id":"388460","messageId":"79f2b561742372234a52dc7194dd72026c6e96ec.1576697386.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","subject":"[PATCH v4 2/8] Revert \"dir.c: make 'git-status --ignored' work within leading directories\"","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-18T19:29:40Z","receivedAt":"2019-12-18T19:29:52Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCommit be8a84c52669 (\"dir.c: make 'git-status --ignored' work within\nleading directories\", 2013-04-15) noted that\n git status --ignored \nwould not list ignored files and directories within if\n was untracked, and modified the behavior to make it show\nthem. However, it did so via a hack that broke consistency; it would\nshow paths under differently than a simple\n git status --ignored | grep \nwould show them. A correct fix is slightly more involved, and\ncomplicated slightly by this hack, so we revert this commit (but keep\ncorrected versions of the testcases) and will later fix the original\nbug with a subsequent patch.\n\nSome history may be helpful:\n\nA very, very similar case to the commit we are reverting was raised in\ncommit 48ffef966c76 (\"ls-files: fix overeager pathspec optimization\",\n2010-01-08); but it actually went in somewhat the opposite direction. In\nthat commit, it mentioned how\n git ls-files -o --exclude-standard t/\nused to show untracked files under t/ even when t/ was ignored, and then\nchanged the behavior to stop showing untracked files under an ignored\ndirectory. More importantly, this commit considered keeping this\nbehavior but noted that it would be inconsistent with the behavior when\nmultiple pathspecs were specified and thus rejected it.\n\nThe reason for this whole inconsistency when one pathspec is specified\nversus zero or two is because common prefixes of pathspecs are sent\nthrough a different set of checks (in treat_leading_path()) than normal\nfile/directory traversal (those go through read_directory_recursive()\nand treat_path()). As such, for consistency, one needs to check that\nboth codepaths produce the same result.\n\nRevert commit be8a84c526691667fc04a8241d93a3de1de298ab, except instead\nof removing the testcase it added, modify it to check for correct and\nconsistent behavior. A subsequent patch in this series will fix the\ntestcase.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 3 ---\n t/t7061-wtstatus-ignore.sh | 9 +++++++--\n 2 files changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 61f559f980..0dd5266629 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2083,14 +2083,12 @@ static int treat_leading_path(struct dir_struct *dir,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint baselen, rc = 0;\n \tconst char *cp;\n-\tint old_flags = dir->flags;\n \n \twhile (len && path[len - 1] == '/')\n \t\tlen--;\n \tif (!len)\n \t\treturn 1;\n \tbaselen = 0;\n-\tdir->flags &= ~DIR_SHOW_OTHER_DIRECTORIES;\n \twhile (1) {\n \t\tcp = path + baselen + !!baselen;\n \t\tcp = memchr(cp, '/', path + len - cp);\n@@ -2113,7 +2111,6 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t}\n \t}\n \tstrbuf_release(&sb);\n-\tdir->flags = old_flags;\n \treturn rc;\n }\n \ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex 0c394cf995..84366050da 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -43,11 +43,16 @@ test_expect_success 'status untracked directory with --ignored -u' '\n \ttest_cmp expected actual\n '\n cat >expected <<\\EOF\n-?? untracked/uncommitted\n+?? untracked/\n !! untracked/ignored\n EOF\n \n-test_expect_success 'status prefixed untracked directory with --ignored' '\n+test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n+\tgit status --porcelain --ignored >tmp &&\n+\tgrep untracked/ tmp >actual &&\n+\trm tmp &&\n+\ttest_cmp expected actual &&\n+\n \tgit status --porcelain --ignored untracked/ >actual &&\n \ttest_cmp expected actual\n '\n-- \ngitgitgadget\n\n"},{"id":"388461","messageId":"6d659b230239b530c90fbcc15c9a869f8d5dd5c5.1576697386.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","subject":"[PATCH v4 1/8] t3011: demonstrate directory traversal failures","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-18T19:29:39Z","receivedAt":"2019-12-18T19:29:53Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd several tests demonstrating directory traversal failures of various\nsorts in dir.c (and one similar looking test that turns out to be a\ngit_fnmatch bug). A lot of these tests look like near duplicates of\neach other, but an optimization path in dir.c to pre-descend into a\ncommon prefix and the specialized treatment of trailing slashes in dir.c\nmean the tiny differences are sometimes important and potentially cause\ndifferent codepaths to be explored.\n\nOf the 7 failing tests, 2 are new to git-2.24.0 (tweaked by side effects\nof the en/clean-nested-with-ignored-topic); the other 5 also failed\nunder git-2.23.0 and earlier.\n\nSigned-off-by: Elijah Newren \n---\n ...common-prefixes-and-directory-traversal.sh | 209 ++++++++++++++++++\n 1 file changed, 209 insertions(+)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nnew file mode 100755\nindex 0000000000..54f80c62b8\n--- /dev/null\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -0,0 +1,209 @@\n+#!/bin/sh\n+\n+test_description='directory traversal handling, especially with common prefixes'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit hello &&\n+\n+\t>empty &&\n+\tmkdir untracked_dir &&\n+\t>untracked_dir/empty &&\n+\tgit init untracked_repo &&\n+\t>untracked_repo/empty &&\n+\n+\tcat <<-EOF >.gitignore &&\n+\tignored\n+\tan_ignored_dir/\n+\tEOF\n+\tmkdir an_ignored_dir &&\n+\tmkdir an_untracked_dir &&\n+\t>an_ignored_dir/ignored &&\n+\t>an_ignored_dir/untracked &&\n+\t>an_untracked_dir/ignored &&\n+\t>an_untracked_dir/untracked\n+'\n+\n+test_expect_success 'git ls-files -o shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_ignored_dir/ignored\n+\tan_ignored_dir/untracked\n+\tan_untracked_dir/ignored\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --exclude-standard shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --exclude-standard >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses into untracked_dir only' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir/ untracked_repo/ does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o .git shows nothing' '\n+\tgit ls-files -o .git >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_failure 'git ls-files -o .git/ shows nothing' '\n+\tgit ls-files -o .git/ >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o untracked_* recurses appropriately' '\n+\tmkdir \"untracked_*\" &&\n+\t>\"untracked_*/empty\" &&\n+\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/empty\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# It turns out fill_directory returns the right paths, but ls-files' post-call\n+# filtering in show_dir_entry() via calling dir_path_match() which ends up\n+# in git_fnmatch() has logic for PATHSPEC_ONESTAR that assumes the pathspec\n+# must match the full path; it doesn't check it for matching a leading\n+# directory.\n+test_expect_failure FUNNYNAMES 'git ls-files -o untracked_*/ recurses appropriately' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/empty\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_* does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_*/ does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o consistent between one or two dirs' '\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ an_untracked_dir/ >tmp &&\n+\t! grep ^an_ignored_dir/ tmp >expect &&\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# ls-files doesn't have a way to request showing both untracked and ignored\n+# files at the same time, so use `git status --ignored`\n+test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n+\tcat <<-EOF >expect &&\n+\t?? an_untracked_dir/\n+\t!! an_untracked_dir/ignored\n+\tEOF\n+\tgit status --porcelain --ignored >output &&\n+\tgrep an_untracked_dir output >expect &&\n+\tgit status --porcelain --ignored an_untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"388462","messageId":"d6f858cab122869425f5801d98713bcbb6d00334.1576697386.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","subject":"[PATCH v4 3/8] dir: remove stray quote character in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-18T19:29:41Z","receivedAt":"2019-12-18T19:29:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 0dd5266629..5dacacd469 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -373,7 +373,7 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t !ps_strncmp(item, match, name, namelen))\n \t\t\treturn MATCHED_RECURSIVELY_LEADING_PATHSPEC;\n \n-\t\t/* name\" doesn't match up to the first wild character */\n+\t\t/* name doesn't match up to the first wild character */\n \t\tif (item->nowildcard_len < item->len &&\n \t\t ps_strncmp(item, match, name,\n \t\t\t item->nowildcard_len - prefix))\n-- \ngitgitgadget\n\n"},{"id":"388463","messageId":"8d2d98eec3c620c55100a322087ef83e4f51c8cf.1576697386.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","subject":"[PATCH v4 4/8] dir: exit before wildcard fall-through if there is no wildcard","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-18T19:29:42Z","receivedAt":"2019-12-18T19:29:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe DO_MATCH_LEADING_PATHSPEC had a fall-through case for if there was a\nwildcard, noting that we don't yet have enough information to determine\nif a further paths under the current directory might match due to the\npresence of wildcards. But if we have no wildcards in our pathspec,\nthen we shouldn't get to that fall-through case.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 7 +++++++\n t/t3011-common-prefixes-and-directory-traversal.sh | 4 ++--\n 2 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 5dacacd469..517a569e10 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -379,6 +379,13 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t\t item->nowildcard_len - prefix))\n \t\t\treturn 0;\n \n+\t\t/*\n+\t\t * name has no wildcard, and it didn't match as a leading\n+\t\t * pathspec so return.\n+\t\t */\n+\t\tif (item->nowildcard_len == item->len)\n+\t\t\treturn 0;\n+\n \t\t/*\n \t\t * Here is where we would perform a wildmatch to check if\n \t\t * \"name\" can be matched as a directory (or a prefix) against\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex 54f80c62b8..d6e161ddd8 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -92,7 +92,7 @@ test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+test_expect_success 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n \tcat <<-EOF >expect &&\n \tuntracked_dir/empty\n \tuntracked_repo/\n@@ -110,7 +110,7 @@ test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses int\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+test_expect_success 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n \tcat <<-EOF >expect &&\n \tuntracked_dir/\n \tuntracked_repo/\n-- \ngitgitgadget\n\n"},{"id":"388464","messageId":"d2f5623bd71352f3adbf12f211dbae5820b33666.1576697386.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","subject":"[PATCH v4 5/8] dir: break part of read_directory_recursive() out for reuse","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-18T19:29:43Z","receivedAt":"2019-12-18T19:29:56Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCreate an add_path_to_appropriate_result_list() function from the code\nat the end of read_directory_recursive() so we can use it elsewhere.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 60 ++++++++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 37 insertions(+), 23 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 517a569e10..645b44ea64 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1932,6 +1932,40 @@ static void close_cached_dir(struct cached_dir *cdir)\n \t}\n }\n \n+static void add_path_to_appropriate_result_list(struct dir_struct *dir,\n+\tstruct untracked_cache_dir *untracked,\n+\tstruct cached_dir *cdir,\n+\tstruct index_state *istate,\n+\tstruct strbuf *path,\n+\tint baselen,\n+\tconst struct pathspec *pathspec,\n+\tenum path_treatment state)\n+{\n+\t/* add the path to the appropriate result list */\n+\tswitch (state) {\n+\tcase path_excluded:\n+\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\tdir_add_name(dir, istate, path->buf, path->len);\n+\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n+\t\t\t((dir->flags & DIR_COLLECT_IGNORED) &&\n+\t\t\texclude_matches_pathspec(path->buf, path->len,\n+\t\t\t\t\t\t pathspec)))\n+\t\t\tdir_add_ignored(dir, istate, path->buf, path->len);\n+\t\tbreak;\n+\n+\tcase path_untracked:\n+\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\tbreak;\n+\t\tdir_add_name(dir, istate, path->buf, path->len);\n+\t\tif (cdir->fdir)\n+\t\t\tadd_untracked(untracked, path->buf + baselen);\n+\t\tbreak;\n+\n+\tdefault:\n+\t\tbreak;\n+\t}\n+}\n+\n /*\n * Read a directory tree. We currently ignore anything but\n * directories, regular files and symlinks. That's because git\n@@ -2035,29 +2069,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tcontinue;\n \t\t}\n \n-\t\t/* add the path to the appropriate result list */\n-\t\tswitch (state) {\n-\t\tcase path_excluded:\n-\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n-\t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n-\t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t\t((dir->flags & DIR_COLLECT_IGNORED) &&\n-\t\t\t\texclude_matches_pathspec(path.buf, path.len,\n-\t\t\t\t\t\t\t pathspec)))\n-\t\t\t\tdir_add_ignored(dir, istate, path.buf, path.len);\n-\t\t\tbreak;\n-\n-\t\tcase path_untracked:\n-\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n-\t\t\t\tbreak;\n-\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n-\t\t\tif (cdir.fdir)\n-\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n-\t\t\tbreak;\n-\n-\t\tdefault:\n-\t\t\tbreak;\n-\t\t}\n+\t\tadd_path_to_appropriate_result_list(dir, untracked, &cdir,\n+\t\t\t\t\t\t istate, &path, baselen,\n+\t\t\t\t\t\t pathspec, state);\n \t}\n \tclose_cached_dir(&cdir);\n out:\n-- \ngitgitgadget\n\n"},{"id":"388465","messageId":"1f3978aa461929923eeb5037e69be6569f0ba331.1576697386.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","subject":"[PATCH v4 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-18T19:29:44Z","receivedAt":"2019-12-18T19:29:57Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany years ago, the directory traversing logic had an optimization that\nwould always recurse into any directory that was a common prefix of all\nthe pathspecs without walking the leading directories to get down to\nthe desired directory. Thus,\n git ls-files -o .git/ # case A\nwould notice that .git/ was a common prefix of all pathspecs (since\nit is the only pathspec listed), and then traverse into it and start\nshowing unknown files under that directory. Unfortunately, .git/ is not\na directory we should be traversing into, which made this optimization\nproblematic. This also affected cases like\n git ls-files -o --exclude-standard t/ # case B\nwhere t/ was in the .gitignore file and thus isn't interesting and\nshouldn't be recursed into. It also affected cases like\n git ls-files -o --directory untracked_dir/ # case C\nwhere untracked_dir/ is indeed untracked and thus interesting, but the\n--directory flag means we only want to show the directory itself, not\nrecurse into it and start listing untracked files below it.\n\nThe case B class of bugs were noted and fixed in commits 16e2cfa90993\n(\"read_directory(): further split treat_path()\", 2010-01-08) and\n48ffef966c76 (\"ls-files: fix overeager pathspec optimization\",\n2010-01-08), with the idea being that we first wanted to check whether\nthe common prefix was interesting. The former patch noted that\ntreat_path() couldn't be used when checking the common prefix because\ntreat_path() requires a dir_entry() and we haven't read any directories\nat the point we are checking the common prefix. So, that patch split\ntreat_one_path() out of treat_path(). The latter patch then created a\nnew treat_leading_path() which duplicated by hand the bits of\ntreat_path() that couldn't be broken out and then called\ntreat_one_path() for the remainder. There were three problems with this\napproach:\n\n * The duplicated logic in treat_leading_path() accidentally missed the\n check for special paths (such as is_dot_or_dotdot and matching\n \".git\"), causing case A types of bugs to continue to be an issue.\n * The treat_leading_path() logic assumed we should traverse into\n anything where path_treatment was not path_none, i.e. it perpetuated\n class C types of bugs.\n * It meant we had split logic that needed to kept in sync, running the\n risk that people introduced new inconsistencies (such as in commit\n be8a84c52669, which we reverted earlier in this series, or in commit\n df5bcdf83ae which we'll fix in a subsequent commit)\n\nFix most these problems by making treat_leading_path() not only loop\nover each leading path component, but calling treat_path() directly on\neach. To do so, we have to create a synthetic dir_entry, but that only\ntakes a few lines. Then, pay attention to the path_treatment result we\nget from treat_path() and don't treat path_excluded, path_untracked, and\npath_recurse all the same as path_recurse.\n\nThis leaves one remaining problem, the new inconsistency from commit\ndf5bcdf83ae. That will be addressed in a subsequent commit.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 57 +++++++++++++++----\n ...common-prefixes-and-directory-traversal.sh | 6 +-\n 2 files changed, 49 insertions(+), 14 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 645b44ea64..1de5d7ad33 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2102,37 +2102,72 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const struct pathspec *pathspec)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint baselen, rc = 0;\n+\tint prevlen, baselen;\n \tconst char *cp;\n+\tstruct cached_dir cdir;\n+\tstruct dirent *de;\n+\tenum path_treatment state = path_none;\n+\n+\t/*\n+\t * For each directory component of path, we are going to check whether\n+\t * that path is relevant given the pathspec. For example, if path is\n+\t * foo/bar/baz/\n+\t * then we will ask treat_path() whether we should go into foo, then\n+\t * whether we should go into bar, then whether baz is relevant.\n+\t * Checking each is important because e.g. if path is\n+\t * .git/info/\n+\t * then we need to check .git to know we shouldn't traverse it.\n+\t * If the return from treat_path() is:\n+\t * * path_none, for any path, we return false.\n+\t * * path_recurse, for all path components, we return true\n+\t * * for some intermediate component, we make sure\n+\t * to add that path to the relevant list but return false\n+\t * signifying that we shouldn't recurse into it.\n+\t */\n \n \twhile (len && path[len - 1] == '/')\n \t\tlen--;\n \tif (!len)\n \t\treturn 1;\n+\n+\tde = xcalloc(1, sizeof(struct dirent)+len+1);\n+\tmemset(&cdir, 0, sizeof(cdir));\n+\tcdir.de = de;\n+#if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)\n+\tde->d_type = DT_DIR;\n+#endif\n \tbaselen = 0;\n+\tprevlen = 0;\n \twhile (1) {\n-\t\tcp = path + baselen + !!baselen;\n+\t\tprevlen = baselen + !!baselen;\n+\t\tcp = path + prevlen;\n \t\tcp = memchr(cp, '/', path + len - cp);\n \t\tif (!cp)\n \t\t\tbaselen = len;\n \t\telse\n \t\t\tbaselen = cp - path;\n-\t\tstrbuf_setlen(&sb, 0);\n+\t\tstrbuf_reset(&sb);\n \t\tstrbuf_add(&sb, path, baselen);\n \t\tif (!is_directory(sb.buf))\n \t\t\tbreak;\n-\t\tif (simplify_away(sb.buf, sb.len, pathspec))\n-\t\t\tbreak;\n-\t\tif (treat_one_path(dir, NULL, istate, &sb, baselen, pathspec,\n-\t\t\t\t DT_DIR, NULL) == path_none)\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_add(&sb, path, prevlen);\n+\t\tmemcpy(de->d_name, path+prevlen, baselen-prevlen);\n+\t\tde->d_name[baselen-prevlen] = '\\0';\n+\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n+\t\t\t\t pathspec);\n+\t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n-\t\tif (len <= baselen) {\n-\t\t\trc = 1;\n+\t\tif (len <= baselen)\n \t\t\tbreak; /* finished checking */\n-\t\t}\n \t}\n+\tadd_path_to_appropriate_result_list(dir, NULL, &cdir, istate,\n+\t\t\t\t\t &sb, baselen, pathspec,\n+\t\t\t\t\t state);\n+\n+\tfree(de);\n \tstrbuf_release(&sb);\n-\treturn rc;\n+\treturn state == path_recurse;\n }\n \n static const char *get_ident_string(void)\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex d6e161ddd8..098fddc75b 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -74,7 +74,7 @@ test_expect_success 'git ls-files -o --directory untracked_dir does not recurse'\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n+test_expect_success 'git ls-files -o --directory untracked_dir/ does not recurse' '\n \techo untracked_dir/ >expect &&\n \tgit ls-files -o --directory untracked_dir/ >actual &&\n \ttest_cmp expect actual\n@@ -86,7 +86,7 @@ test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n+test_expect_success 'git ls-files -o untracked_repo/ does not recurse' '\n \techo untracked_repo/ >expect &&\n \tgit ls-files -o untracked_repo/ >actual &&\n \ttest_cmp expect actual\n@@ -133,7 +133,7 @@ test_expect_success 'git ls-files -o .git shows nothing' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_failure 'git ls-files -o .git/ shows nothing' '\n+test_expect_success 'git ls-files -o .git/ shows nothing' '\n \tgit ls-files -o .git/ >actual &&\n \ttest_must_be_empty actual\n '\n-- \ngitgitgadget\n\n"},{"id":"388466","messageId":"31079dc1cf10d031010846f1d1a0b30a05341943.1576697387.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","subject":"[PATCH v4 8/8] dir: consolidate similar code in treat_directory()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-18T19:29:46Z","receivedAt":"2019-12-18T19:30:00Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nBoth the DIR_SKIP_NESTED_GIT and DIR_NO_GITLINKS cases were checking for\nwhether a path was actually a nonbare repository. That code could be\nshared, with just the result of how to act differing between the two\ncases.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 18 +++++++-----------\n 1 file changed, 7 insertions(+), 11 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex f500fd9279..6912f7eb5e 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1461,6 +1461,8 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int exclude,\n \tconst struct pathspec *pathspec)\n {\n+\tint nested_repo = 0;\n+\n \t/* The \"len-1\" is to strip the final '/' */\n \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n \tcase index_directory:\n@@ -1470,15 +1472,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_none;\n \n \tcase index_nonexistent:\n-\t\tif (dir->flags & DIR_SKIP_NESTED_GIT) {\n-\t\t\tint nested_repo;\n+\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n+\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n \t\t\tstruct strbuf sb = STRBUF_INIT;\n \t\t\tstrbuf_addstr(&sb, dirname);\n \t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n \t\t\tstrbuf_release(&sb);\n-\t\t\tif (nested_repo)\n-\t\t\t\treturn path_none;\n \t\t}\n+\t\tif (nested_repo)\n+\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n+\t\t\t\t(exclude ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n \t\t\tbreak;\n@@ -1506,13 +1509,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t\t\treturn path_none;\n \t\t}\n-\t\tif (!(dir->flags & DIR_NO_GITLINKS)) {\n-\t\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\t\tstrbuf_addstr(&sb, dirname);\n-\t\t\tif (is_nonbare_repository_dir(&sb))\n-\t\t\t\treturn exclude ? path_excluded : path_untracked;\n-\t\t\tstrbuf_release(&sb);\n-\t\t}\n \t\treturn path_recurse;\n \t}\n \n-- \ngitgitgadget\n"},{"id":"388467","messageId":"542c6e57927b968d58e4309e3a09f85914af0408.1576697387.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","subject":"[PATCH v4 7/8] dir: synchronize treat_leading_path() and read_directory_recursive()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-18T19:29:45Z","receivedAt":"2019-12-18T19:30:01Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nOur optimization to avoid calling into read_directory_recursive() when\nall pathspecs have a common leading directory mean that we need to match\nthe logic that read_directory_recursive() would use if we had just\ncalled it from the root. Since it does more than call treat_path() we\nneed to copy that same logic.\n\nAlternatively, we could try to change treat_path to return path_recurse\nfor an untracked directory under the given special circumstances that\nthis logic checks for, but a simple switch results in many test failures\nsuch as 'git clean -d' not wiping out untracked but empty directories.\nTo work around that, we'd need the caller of treat_path to check for\npath_recurse and sometimes special case it into path_untracked. In\nother words, we'd still have extra logic in both places.\n\nNeeding to duplicate logic like this means it is guaranteed someone will\neventually need to make further changes and forget to update both\nlocations. It is tempting to just nuke the leading_directory special\ncasing to avoid such bugs and simplify the code, but unpack_trees'\nverify_clean_subdirectory() also calls read_directory() and does so with\na non-empty leading path, so I'm hesitant to try to restructure further.\nAdd obnoxious warnings to treat_leading_path() and\nread_directory_recursive() to try to warn people of such problems.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 30 +++++++++++++++++++\n ...common-prefixes-and-directory-traversal.sh | 2 +-\n t/t7061-wtstatus-ignore.sh | 2 +-\n 3 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 1de5d7ad33..f500fd9279 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1990,6 +1990,15 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tstruct untracked_cache_dir *untracked, int check_only,\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING WARNING WARNING:\n+\t *\n+\t * Any updates to the traversal logic here may need corresponding\n+\t * updates in treat_leading_path(). See the commit message for the\n+\t * commit adding this warning as well as the commit preceding it\n+\t * for details.\n+\t */\n+\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -2101,6 +2110,15 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING WARNING WARNING:\n+\t *\n+\t * Any updates to the traversal logic here may need corresponding\n+\t * updates in treat_leading_path(). See the commit message for the\n+\t * commit adding this warning as well as the commit preceding it\n+\t * for details.\n+\t */\n+\n \tstruct strbuf sb = STRBUF_INIT;\n \tint prevlen, baselen;\n \tconst char *cp;\n@@ -2156,6 +2174,18 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tde->d_name[baselen-prevlen] = '\\0';\n \t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n \t\t\t\t pathspec);\n+\t\tif (state == path_untracked &&\n+\t\t get_dtype(cdir.de, istate, sb.buf, sb.len) == DT_DIR &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO ||\n+\t\t do_match_pathspec(istate, pathspec, sb.buf, sb.len,\n+\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)) {\n+\t\t\tadd_path_to_appropriate_result_list(dir, NULL, &cdir,\n+\t\t\t\t\t\t\t istate,\n+\t\t\t\t\t\t\t &sb, baselen,\n+\t\t\t\t\t\t\t pathspec, state);\n+\t\t\tstate = path_recurse;\n+\t\t}\n+\n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n \t\tif (len <= baselen)\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex 098fddc75b..3da5b2b6e7 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -195,7 +195,7 @@ test_expect_success 'git ls-files -o consistent between one or two dirs' '\n \n # ls-files doesn't have a way to request showing both untracked and ignored\n # files at the same time, so use `git status --ignored`\n-test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n+test_expect_success 'git status --ignored shows same files under dir with or without pathspec' '\n \tcat <<-EOF >expect &&\n \t?? an_untracked_dir/\n \t!! an_untracked_dir/ignored\ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex 84366050da..e4cf5484f9 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -47,7 +47,7 @@ cat >expected <<\\EOF\n !! untracked/ignored\n EOF\n \n-test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n+test_expect_success 'status of untracked directory with --ignored works with or without prefix' '\n \tgit status --porcelain --ignored >tmp &&\n \tgrep untracked/ tmp >actual &&\n \trm tmp &&\n-- \ngitgitgadget\n\n"},{"id":"388475","messageId":"xmqq1rt1e30q.fsf@gitster-ct.c.googlers.com","threadId":"52419","inReplyTo":"1f3978aa461929923eeb5037e69be6569f0ba331.1576697386.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 6/8] dir: fix checks on common prefix directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-12-18T21:29:25Z","receivedAt":"2019-12-18T21:29:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Elijah Newren via GitGitGadget\" writes:\n\n> ...\n> Fix most these problems by making treat_leading_path() not only loop\n> over each leading path component, but calling treat_path() directly on\n> each. To do so, we have to create a synthetic dir_entry, but that only\n> takes a few lines. Then, pay attention to the path_treatment result we\n> get from treat_path() and don't treat path_excluded, path_untracked, and\n> path_recurse all the same as path_recurse.\n>\n> This leaves one remaining problem, the new inconsistency from commit\n> df5bcdf83ae. That will be addressed in a subsequent commit.\n>\n> Signed-off-by: Elijah Newren \n> ---\n> dir.c | 57 +++++++++++++++----\n> ...common-prefixes-and-directory-traversal.sh | 6 +-\n> 2 files changed, 49 insertions(+), 14 deletions(-)\n>\n> diff --git a/dir.c b/dir.c\n> index 645b44ea64..1de5d7ad33 100644\n> --- a/dir.c\n> +++ b/dir.c\n> @@ -2102,37 +2102,72 @@ static int treat_leading_path(struct dir_struct *dir,\n> \t\t\t const struct pathspec *pathspec)\n> {\n> \tstruct strbuf sb = STRBUF_INIT;\n> -\tint baselen, rc = 0;\n> +\tint prevlen, baselen;\n> \tconst char *cp;\n> +\tstruct cached_dir cdir;\n> +\tstruct dirent *de;\n> +\tenum path_treatment state = path_none;\n> +\n> +\t/*\n> +\t * For each directory component of path, we are going to check whether\n> +\t * that path is relevant given the pathspec. For example, if path is\n> +\t * foo/bar/baz/\n> +\t * then we will ask treat_path() whether we should go into foo, then\n> +\t * whether we should go into bar, then whether baz is relevant.\n> +\t * Checking each is important because e.g. if path is\n> +\t * .git/info/\n> +\t * then we need to check .git to know we shouldn't traverse it.\n> +\t * If the return from treat_path() is:\n> +\t * * path_none, for any path, we return false.\n> +\t * * path_recurse, for all path components, we return true\n> +\t * * for some intermediate component, we make sure\n> +\t * to add that path to the relevant list but return false\n> +\t * signifying that we shouldn't recurse into it.\n> +\t */\n> \n> \twhile (len && path[len - 1] == '/')\n> \t\tlen--;\n> \tif (!len)\n> \t\treturn 1;\n> +\n> +\tde = xcalloc(1, sizeof(struct dirent)+len+1);\n\nThat \"+len+1\" may deserve a comment? If we wanted to shoot for the\nminimum memory consumption (and we do not), we would probably\nallocate\n\n\t(sizeof(struct dirent) - sizeof(de->d_name)) +\n\t\tmax(sizeof(de->d_name), len + 1)\n\nbytes, but unconditionally adding len+1 is simpler and easier to\nunderstand. Either way, we *are* relying on the assumption that\neither:\n\n (1) the \"struct dirent\" would have d_name[] array at the end of the\n struct, and by over-allocating, we can safely fit and carry a\n name that is much longer than sizeof(.d_name[]); OR\n\n (2) the \"struct dirent\" has d_name[] that is large enough to hold len+1\n bytes, if the assumption (1) does not hold.\n\nis true.\n\n> +\tmemset(&cdir, 0, sizeof(cdir));\n> +\tcdir.de = de;\n> +#if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)\n> +\tde->d_type = DT_DIR;\n> +#endif\n> \tbaselen = 0;\n> +\tprevlen = 0;\n> \twhile (1) {\n> -\t\tcp = path + baselen + !!baselen;\n> +\t\tprevlen = baselen + !!baselen;\n> +\t\tcp = path + prevlen;\n> \t\tcp = memchr(cp, '/', path + len - cp);\n> \t\tif (!cp)\n> \t\t\tbaselen = len;\n> \t\telse\n> \t\t\tbaselen = cp - path;\n> -\t\tstrbuf_setlen(&sb, 0);\n> +\t\tstrbuf_reset(&sb);\n> \t\tstrbuf_add(&sb, path, baselen);\n> \t\tif (!is_directory(sb.buf))\n> \t\t\tbreak;\n\n\n\n> -\t\tif (simplify_away(sb.buf, sb.len, pathspec))\n> -\t\t\tbreak;\n> -\t\tif (treat_one_path(dir, NULL, istate, &sb, baselen, pathspec,\n> -\t\t\t\t DT_DIR, NULL) == path_none)\n> +\t\tstrbuf_reset(&sb);\n> +\t\tstrbuf_add(&sb, path, prevlen);\n> +\t\tmemcpy(de->d_name, path+prevlen, baselen-prevlen);\n> +\t\tde->d_name[baselen-prevlen] = '\\0';\n> +\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n> +\t\t\t\t pathspec);\n\nSo this is the crux fo the fix---instead of doing a (poor) imitation\nof what treat_path() does by calling simplify_away() and\ntreat_one_path() ourselves, we make a call to the real thing.\n\nLooking good. Thanks.\n"},{"id":"388566","messageId":"CABPp-BHhAPjL9daPdp7UmVT+vLrcpo6SX0rKt5s1bH4Vz31wxw@mail.gmail.com","threadId":"52419","inReplyTo":"xmqq1rt1e30q.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-12-19T20:23:29Z","receivedAt":"2019-12-19T20:23:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"On Wed, Dec 18, 2019 at 1:29 PM Junio C Hamano wrote:\n>\n> \"Elijah Newren via GitGitGadget\" writes:\n>\n...\n> > while (len && path[len - 1] == '/')\n> > len--;\n> > if (!len)\n> > return 1;\n> > +\n> > + de = xcalloc(1, sizeof(struct dirent)+len+1);\n>\n> That \"+len+1\" may deserve a comment?\n\nGood point, I'll add one and send a re-roll.\n\n> If we wanted to shoot for the\n> minimum memory consumption (and we do not), we would probably\n> allocate\n>\n> (sizeof(struct dirent) - sizeof(de->d_name)) +\n> max(sizeof(de->d_name), len + 1)\n>\n> bytes, but unconditionally adding len+1 is simpler and easier to\n> understand. Either way, we *are* relying on the assumption that\n> either:\n>\n> (1) the \"struct dirent\" would have d_name[] array at the end of the\n> struct, and by over-allocating, we can safely fit and carry a\n> name that is much longer than sizeof(.d_name[]); OR\n>\n> (2) the \"struct dirent\" has d_name[] that is large enough to hold len+1\n> bytes, if the assumption (1) does not hold.\n>\n> is true.\n>\n> > + memset(&cdir, 0, sizeof(cdir));\n> > + cdir.de = de;\n> > +#if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)\n> > + de->d_type = DT_DIR;\n> > +#endif\n> > baselen = 0;\n> > + prevlen = 0;\n> > while (1) {\n> > - cp = path + baselen + !!baselen;\n> > + prevlen = baselen + !!baselen;\n> > + cp = path + prevlen;\n> > cp = memchr(cp, '/', path + len - cp);\n> > if (!cp)\n> > baselen = len;\n> > else\n> > baselen = cp - path;\n> > - strbuf_setlen(&sb, 0);\n> > + strbuf_reset(&sb);\n> > strbuf_add(&sb, path, baselen);\n> > if (!is_directory(sb.buf))\n> > break;\n>\n>\n>\n> > - if (simplify_away(sb.buf, sb.len, pathspec))\n> > - break;\n> > - if (treat_one_path(dir, NULL, istate, &sb, baselen, pathspec,\n> > - DT_DIR, NULL) == path_none)\n> > + strbuf_reset(&sb);\n> > + strbuf_add(&sb, path, prevlen);\n> > + memcpy(de->d_name, path+prevlen, baselen-prevlen);\n> > + de->d_name[baselen-prevlen] = '\\0';\n> > + state = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n> > + pathspec);\n>\n> So this is the crux fo the fix---instead of doing a (poor) imitation\n> of what treat_path() does by calling simplify_away() and\n> treat_one_path() ourselves, we make a call to the real thing.\n>\n> Looking good. Thanks.\n"},{"id":"388568","messageId":"pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v4.git.git.1576697386.gitgitgadget@gmail.com","subject":"[PATCH v5 0/8] Directory traversal bugs","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-19T21:28:18Z","receivedAt":"2019-12-19T21:28:31Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"This series documents multiple fill_directory() bugs, and fixes the one that\nis new to 2.24.0 coming from en/clean-nested-with-ignored-topic, the rest\nhaving been around in versions of git going back up to a decade. \n\nChanges since v4:\n\n * added a comment with the rationale for allocating an extra len+1 bytes\n for the dirent.\n\nElijah Newren (8):\n t3011: demonstrate directory traversal failures\n Revert \"dir.c: make 'git-status --ignored' work within leading\n directories\"\n dir: remove stray quote character in comment\n dir: exit before wildcard fall-through if there is no wildcard\n dir: break part of read_directory_recursive() out for reuse\n dir: fix checks on common prefix directory\n dir: synchronize treat_leading_path() and read_directory_recursive()\n dir: consolidate similar code in treat_directory()\n\n dir.c | 187 ++++++++++++----\n ...common-prefixes-and-directory-traversal.sh | 209 ++++++++++++++++++\n t/t7061-wtstatus-ignore.sh | 9 +-\n 3 files changed, 354 insertions(+), 51 deletions(-)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\n\nbase-commit: da72936f544fec5a335e66432610e4cef4430991\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-676%2Fnewren%2Fls-files-bug-v5\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-676/newren/ls-files-bug-v5\nPull-Request: https://github.com/git/git/pull/676\n\nRange-diff vs v4:\n\n 1: 6d659b2302 = 1: 6d659b2302 t3011: demonstrate directory traversal failures\n 2: 79f2b56174 = 2: 79f2b56174 Revert \"dir.c: make 'git-status --ignored' work within leading directories\"\n 3: d6f858cab1 = 3: d6f858cab1 dir: remove stray quote character in comment\n 4: 8d2d98eec3 = 4: 8d2d98eec3 dir: exit before wildcard fall-through if there is no wildcard\n 5: d2f5623bd7 = 5: d2f5623bd7 dir: break part of read_directory_recursive() out for reuse\n 6: 1f3978aa46 ! 6: 97e145489d dir: fix checks on common prefix directory\n @@ -93,6 +93,16 @@\n \tif (!len)\n \t\treturn 1;\n +\n ++\t/*\n ++\t * We need a manufactured dirent with sufficient space to store a\n ++\t * leading directory component of path in its d_name. Here, we\n ++\t * assume that the dirent's d_name is either declared as\n ++\t * char d_name[BIG_ENOUGH]\n ++\t * or that it is declared at the end of the struct as\n ++\t * char d_name[]\n ++\t * For either case, padding with len+1 bytes at the end will ensure\n ++\t * sufficient storage space.\n ++\t */\n +\tde = xcalloc(1, sizeof(struct dirent)+len+1);\n +\tmemset(&cdir, 0, sizeof(cdir));\n +\tcdir.de = de;\n 7: 542c6e5792 = 7: 5275e6d7f0 dir: synchronize treat_leading_path() and read_directory_recursive()\n 8: 31079dc1cf = 8: e4768931d0 dir: consolidate similar code in treat_directory()\n\n-- \ngitgitgadget\n"},{"id":"388569","messageId":"6d659b230239b530c90fbcc15c9a869f8d5dd5c5.1576790906.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com","subject":"[PATCH v5 1/8] t3011: demonstrate directory traversal failures","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-19T21:28:19Z","receivedAt":"2019-12-19T21:28:33Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nAdd several tests demonstrating directory traversal failures of various\nsorts in dir.c (and one similar looking test that turns out to be a\ngit_fnmatch bug). A lot of these tests look like near duplicates of\neach other, but an optimization path in dir.c to pre-descend into a\ncommon prefix and the specialized treatment of trailing slashes in dir.c\nmean the tiny differences are sometimes important and potentially cause\ndifferent codepaths to be explored.\n\nOf the 7 failing tests, 2 are new to git-2.24.0 (tweaked by side effects\nof the en/clean-nested-with-ignored-topic); the other 5 also failed\nunder git-2.23.0 and earlier.\n\nSigned-off-by: Elijah Newren \n---\n ...common-prefixes-and-directory-traversal.sh | 209 ++++++++++++++++++\n 1 file changed, 209 insertions(+)\n create mode 100755 t/t3011-common-prefixes-and-directory-traversal.sh\n\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nnew file mode 100755\nindex 0000000000..54f80c62b8\n--- /dev/null\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -0,0 +1,209 @@\n+#!/bin/sh\n+\n+test_description='directory traversal handling, especially with common prefixes'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'setup' '\n+\ttest_commit hello &&\n+\n+\t>empty &&\n+\tmkdir untracked_dir &&\n+\t>untracked_dir/empty &&\n+\tgit init untracked_repo &&\n+\t>untracked_repo/empty &&\n+\n+\tcat <<-EOF >.gitignore &&\n+\tignored\n+\tan_ignored_dir/\n+\tEOF\n+\tmkdir an_ignored_dir &&\n+\tmkdir an_untracked_dir &&\n+\t>an_ignored_dir/ignored &&\n+\t>an_ignored_dir/untracked &&\n+\t>an_untracked_dir/ignored &&\n+\t>an_untracked_dir/untracked\n+'\n+\n+test_expect_success 'git ls-files -o shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_ignored_dir/ignored\n+\tan_ignored_dir/untracked\n+\tan_untracked_dir/ignored\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --exclude-standard shows the right entries' '\n+\tcat <<-EOF >expect &&\n+\t.gitignore\n+\tactual\n+\tan_untracked_dir/untracked\n+\tempty\n+\texpect\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --exclude-standard >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ recurses' '\n+\techo untracked_dir/empty >expect &&\n+\tgit ls-files -o untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n+\techo untracked_dir/ >expect &&\n+\tgit ls-files -o --directory untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n+\techo untracked_repo/ >expect &&\n+\tgit ls-files -o untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses into untracked_dir only' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory untracked_dir untracked_repo >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o --directory untracked_dir/ untracked_repo/ does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory untracked_dir/ untracked_repo/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o .git shows nothing' '\n+\tgit ls-files -o .git >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_failure 'git ls-files -o .git/ shows nothing' '\n+\tgit ls-files -o .git/ >actual &&\n+\ttest_must_be_empty actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o untracked_* recurses appropriately' '\n+\tmkdir \"untracked_*\" &&\n+\t>\"untracked_*/empty\" &&\n+\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/empty\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# It turns out fill_directory returns the right paths, but ls-files' post-call\n+# filtering in show_dir_entry() via calling dir_path_match() which ends up\n+# in git_fnmatch() has logic for PATHSPEC_ONESTAR that assumes the pathspec\n+# must match the full path; it doesn't check it for matching a leading\n+# directory.\n+test_expect_failure FUNNYNAMES 'git ls-files -o untracked_*/ recurses appropriately' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/empty\n+\tuntracked_dir/empty\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_* does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory \"untracked_*\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success FUNNYNAMES 'git ls-files -o --directory untracked_*/ does not recurse' '\n+\tcat <<-EOF >expect &&\n+\tuntracked_*/\n+\tuntracked_dir/\n+\tuntracked_repo/\n+\tEOF\n+\tgit ls-files -o --directory \"untracked_*/\" >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_expect_success 'git ls-files -o consistent between one or two dirs' '\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ an_untracked_dir/ >tmp &&\n+\t! grep ^an_ignored_dir/ tmp >expect &&\n+\tgit ls-files -o --exclude-standard an_ignored_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+# ls-files doesn't have a way to request showing both untracked and ignored\n+# files at the same time, so use `git status --ignored`\n+test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n+\tcat <<-EOF >expect &&\n+\t?? an_untracked_dir/\n+\t!! an_untracked_dir/ignored\n+\tEOF\n+\tgit status --porcelain --ignored >output &&\n+\tgrep an_untracked_dir output >expect &&\n+\tgit status --porcelain --ignored an_untracked_dir/ >actual &&\n+\ttest_cmp expect actual\n+'\n+\n+test_done\n-- \ngitgitgadget\n\n"},{"id":"388570","messageId":"d2f5623bd71352f3adbf12f211dbae5820b33666.1576790906.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com","subject":"[PATCH v5 5/8] dir: break part of read_directory_recursive() out for reuse","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-19T21:28:23Z","receivedAt":"2019-12-19T21:28:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCreate an add_path_to_appropriate_result_list() function from the code\nat the end of read_directory_recursive() so we can use it elsewhere.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 60 ++++++++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 37 insertions(+), 23 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 517a569e10..645b44ea64 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1932,6 +1932,40 @@ static void close_cached_dir(struct cached_dir *cdir)\n \t}\n }\n \n+static void add_path_to_appropriate_result_list(struct dir_struct *dir,\n+\tstruct untracked_cache_dir *untracked,\n+\tstruct cached_dir *cdir,\n+\tstruct index_state *istate,\n+\tstruct strbuf *path,\n+\tint baselen,\n+\tconst struct pathspec *pathspec,\n+\tenum path_treatment state)\n+{\n+\t/* add the path to the appropriate result list */\n+\tswitch (state) {\n+\tcase path_excluded:\n+\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\tdir_add_name(dir, istate, path->buf, path->len);\n+\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n+\t\t\t((dir->flags & DIR_COLLECT_IGNORED) &&\n+\t\t\texclude_matches_pathspec(path->buf, path->len,\n+\t\t\t\t\t\t pathspec)))\n+\t\t\tdir_add_ignored(dir, istate, path->buf, path->len);\n+\t\tbreak;\n+\n+\tcase path_untracked:\n+\t\tif (dir->flags & DIR_SHOW_IGNORED)\n+\t\t\tbreak;\n+\t\tdir_add_name(dir, istate, path->buf, path->len);\n+\t\tif (cdir->fdir)\n+\t\t\tadd_untracked(untracked, path->buf + baselen);\n+\t\tbreak;\n+\n+\tdefault:\n+\t\tbreak;\n+\t}\n+}\n+\n /*\n * Read a directory tree. We currently ignore anything but\n * directories, regular files and symlinks. That's because git\n@@ -2035,29 +2069,9 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tcontinue;\n \t\t}\n \n-\t\t/* add the path to the appropriate result list */\n-\t\tswitch (state) {\n-\t\tcase path_excluded:\n-\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n-\t\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n-\t\t\telse if ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n-\t\t\t\t((dir->flags & DIR_COLLECT_IGNORED) &&\n-\t\t\t\texclude_matches_pathspec(path.buf, path.len,\n-\t\t\t\t\t\t\t pathspec)))\n-\t\t\t\tdir_add_ignored(dir, istate, path.buf, path.len);\n-\t\t\tbreak;\n-\n-\t\tcase path_untracked:\n-\t\t\tif (dir->flags & DIR_SHOW_IGNORED)\n-\t\t\t\tbreak;\n-\t\t\tdir_add_name(dir, istate, path.buf, path.len);\n-\t\t\tif (cdir.fdir)\n-\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n-\t\t\tbreak;\n-\n-\t\tdefault:\n-\t\t\tbreak;\n-\t\t}\n+\t\tadd_path_to_appropriate_result_list(dir, untracked, &cdir,\n+\t\t\t\t\t\t istate, &path, baselen,\n+\t\t\t\t\t\t pathspec, state);\n \t}\n \tclose_cached_dir(&cdir);\n out:\n-- \ngitgitgadget\n\n"},{"id":"388571","messageId":"8d2d98eec3c620c55100a322087ef83e4f51c8cf.1576790906.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com","subject":"[PATCH v5 4/8] dir: exit before wildcard fall-through if there is no wildcard","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-19T21:28:22Z","receivedAt":"2019-12-19T21:28:36Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nThe DO_MATCH_LEADING_PATHSPEC had a fall-through case for if there was a\nwildcard, noting that we don't yet have enough information to determine\nif a further paths under the current directory might match due to the\npresence of wildcards. But if we have no wildcards in our pathspec,\nthen we shouldn't get to that fall-through case.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 7 +++++++\n t/t3011-common-prefixes-and-directory-traversal.sh | 4 ++--\n 2 files changed, 9 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 5dacacd469..517a569e10 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -379,6 +379,13 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t\t item->nowildcard_len - prefix))\n \t\t\treturn 0;\n \n+\t\t/*\n+\t\t * name has no wildcard, and it didn't match as a leading\n+\t\t * pathspec so return.\n+\t\t */\n+\t\tif (item->nowildcard_len == item->len)\n+\t\t\treturn 0;\n+\n \t\t/*\n \t\t * Here is where we would perform a wildmatch to check if\n \t\t * \"name\" can be matched as a directory (or a prefix) against\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex 54f80c62b8..d6e161ddd8 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -92,7 +92,7 @@ test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n+test_expect_success 'git ls-files -o untracked_dir untracked_repo recurses into untracked_dir only' '\n \tcat <<-EOF >expect &&\n \tuntracked_dir/empty\n \tuntracked_repo/\n@@ -110,7 +110,7 @@ test_expect_success 'git ls-files -o untracked_dir/ untracked_repo/ recurses int\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n+test_expect_success 'git ls-files -o --directory untracked_dir untracked_repo does not recurse' '\n \tcat <<-EOF >expect &&\n \tuntracked_dir/\n \tuntracked_repo/\n-- \ngitgitgadget\n\n"},{"id":"388572","messageId":"d6f858cab122869425f5801d98713bcbb6d00334.1576790906.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com","subject":"[PATCH v5 3/8] dir: remove stray quote character in comment","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-19T21:28:21Z","receivedAt":"2019-12-19T21:28:37Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nSigned-off-by: Elijah Newren \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex 0dd5266629..5dacacd469 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -373,7 +373,7 @@ static int match_pathspec_item(const struct index_state *istate,\n \t\t !ps_strncmp(item, match, name, namelen))\n \t\t\treturn MATCHED_RECURSIVELY_LEADING_PATHSPEC;\n \n-\t\t/* name\" doesn't match up to the first wild character */\n+\t\t/* name doesn't match up to the first wild character */\n \t\tif (item->nowildcard_len < item->len &&\n \t\t ps_strncmp(item, match, name,\n \t\t\t item->nowildcard_len - prefix))\n-- \ngitgitgadget\n\n"},{"id":"388573","messageId":"79f2b561742372234a52dc7194dd72026c6e96ec.1576790906.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com","subject":"[PATCH v5 2/8] Revert \"dir.c: make 'git-status --ignored' work within leading directories\"","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-19T21:28:20Z","receivedAt":"2019-12-19T21:28:38Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nCommit be8a84c52669 (\"dir.c: make 'git-status --ignored' work within\nleading directories\", 2013-04-15) noted that\n git status --ignored \nwould not list ignored files and directories within if\n was untracked, and modified the behavior to make it show\nthem. However, it did so via a hack that broke consistency; it would\nshow paths under differently than a simple\n git status --ignored | grep \nwould show them. A correct fix is slightly more involved, and\ncomplicated slightly by this hack, so we revert this commit (but keep\ncorrected versions of the testcases) and will later fix the original\nbug with a subsequent patch.\n\nSome history may be helpful:\n\nA very, very similar case to the commit we are reverting was raised in\ncommit 48ffef966c76 (\"ls-files: fix overeager pathspec optimization\",\n2010-01-08); but it actually went in somewhat the opposite direction. In\nthat commit, it mentioned how\n git ls-files -o --exclude-standard t/\nused to show untracked files under t/ even when t/ was ignored, and then\nchanged the behavior to stop showing untracked files under an ignored\ndirectory. More importantly, this commit considered keeping this\nbehavior but noted that it would be inconsistent with the behavior when\nmultiple pathspecs were specified and thus rejected it.\n\nThe reason for this whole inconsistency when one pathspec is specified\nversus zero or two is because common prefixes of pathspecs are sent\nthrough a different set of checks (in treat_leading_path()) than normal\nfile/directory traversal (those go through read_directory_recursive()\nand treat_path()). As such, for consistency, one needs to check that\nboth codepaths produce the same result.\n\nRevert commit be8a84c526691667fc04a8241d93a3de1de298ab, except instead\nof removing the testcase it added, modify it to check for correct and\nconsistent behavior. A subsequent patch in this series will fix the\ntestcase.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 3 ---\n t/t7061-wtstatus-ignore.sh | 9 +++++++--\n 2 files changed, 7 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 61f559f980..0dd5266629 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2083,14 +2083,12 @@ static int treat_leading_path(struct dir_struct *dir,\n \tstruct strbuf sb = STRBUF_INIT;\n \tint baselen, rc = 0;\n \tconst char *cp;\n-\tint old_flags = dir->flags;\n \n \twhile (len && path[len - 1] == '/')\n \t\tlen--;\n \tif (!len)\n \t\treturn 1;\n \tbaselen = 0;\n-\tdir->flags &= ~DIR_SHOW_OTHER_DIRECTORIES;\n \twhile (1) {\n \t\tcp = path + baselen + !!baselen;\n \t\tcp = memchr(cp, '/', path + len - cp);\n@@ -2113,7 +2111,6 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t}\n \t}\n \tstrbuf_release(&sb);\n-\tdir->flags = old_flags;\n \treturn rc;\n }\n \ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex 0c394cf995..84366050da 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -43,11 +43,16 @@ test_expect_success 'status untracked directory with --ignored -u' '\n \ttest_cmp expected actual\n '\n cat >expected <<\\EOF\n-?? untracked/uncommitted\n+?? untracked/\n !! untracked/ignored\n EOF\n \n-test_expect_success 'status prefixed untracked directory with --ignored' '\n+test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n+\tgit status --porcelain --ignored >tmp &&\n+\tgrep untracked/ tmp >actual &&\n+\trm tmp &&\n+\ttest_cmp expected actual &&\n+\n \tgit status --porcelain --ignored untracked/ >actual &&\n \ttest_cmp expected actual\n '\n-- \ngitgitgadget\n\n"},{"id":"388574","messageId":"97e145489d275da2d39179f8d8d928a47936377b.1576790906.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com","subject":"[PATCH v5 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-19T21:28:24Z","receivedAt":"2019-12-19T21:28:41Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nMany years ago, the directory traversing logic had an optimization that\nwould always recurse into any directory that was a common prefix of all\nthe pathspecs without walking the leading directories to get down to\nthe desired directory. Thus,\n git ls-files -o .git/ # case A\nwould notice that .git/ was a common prefix of all pathspecs (since\nit is the only pathspec listed), and then traverse into it and start\nshowing unknown files under that directory. Unfortunately, .git/ is not\na directory we should be traversing into, which made this optimization\nproblematic. This also affected cases like\n git ls-files -o --exclude-standard t/ # case B\nwhere t/ was in the .gitignore file and thus isn't interesting and\nshouldn't be recursed into. It also affected cases like\n git ls-files -o --directory untracked_dir/ # case C\nwhere untracked_dir/ is indeed untracked and thus interesting, but the\n--directory flag means we only want to show the directory itself, not\nrecurse into it and start listing untracked files below it.\n\nThe case B class of bugs were noted and fixed in commits 16e2cfa90993\n(\"read_directory(): further split treat_path()\", 2010-01-08) and\n48ffef966c76 (\"ls-files: fix overeager pathspec optimization\",\n2010-01-08), with the idea being that we first wanted to check whether\nthe common prefix was interesting. The former patch noted that\ntreat_path() couldn't be used when checking the common prefix because\ntreat_path() requires a dir_entry() and we haven't read any directories\nat the point we are checking the common prefix. So, that patch split\ntreat_one_path() out of treat_path(). The latter patch then created a\nnew treat_leading_path() which duplicated by hand the bits of\ntreat_path() that couldn't be broken out and then called\ntreat_one_path() for the remainder. There were three problems with this\napproach:\n\n * The duplicated logic in treat_leading_path() accidentally missed the\n check for special paths (such as is_dot_or_dotdot and matching\n \".git\"), causing case A types of bugs to continue to be an issue.\n * The treat_leading_path() logic assumed we should traverse into\n anything where path_treatment was not path_none, i.e. it perpetuated\n class C types of bugs.\n * It meant we had split logic that needed to kept in sync, running the\n risk that people introduced new inconsistencies (such as in commit\n be8a84c52669, which we reverted earlier in this series, or in commit\n df5bcdf83ae which we'll fix in a subsequent commit)\n\nFix most these problems by making treat_leading_path() not only loop\nover each leading path component, but calling treat_path() directly on\neach. To do so, we have to create a synthetic dir_entry, but that only\ntakes a few lines. Then, pay attention to the path_treatment result we\nget from treat_path() and don't treat path_excluded, path_untracked, and\npath_recurse all the same as path_recurse.\n\nThis leaves one remaining problem, the new inconsistency from commit\ndf5bcdf83ae. That will be addressed in a subsequent commit.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 67 ++++++++++++++++---\n ...common-prefixes-and-directory-traversal.sh | 6 +-\n 2 files changed, 59 insertions(+), 14 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 645b44ea64..a42cc2aa8c 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2102,37 +2102,82 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const struct pathspec *pathspec)\n {\n \tstruct strbuf sb = STRBUF_INIT;\n-\tint baselen, rc = 0;\n+\tint prevlen, baselen;\n \tconst char *cp;\n+\tstruct cached_dir cdir;\n+\tstruct dirent *de;\n+\tenum path_treatment state = path_none;\n+\n+\t/*\n+\t * For each directory component of path, we are going to check whether\n+\t * that path is relevant given the pathspec. For example, if path is\n+\t * foo/bar/baz/\n+\t * then we will ask treat_path() whether we should go into foo, then\n+\t * whether we should go into bar, then whether baz is relevant.\n+\t * Checking each is important because e.g. if path is\n+\t * .git/info/\n+\t * then we need to check .git to know we shouldn't traverse it.\n+\t * If the return from treat_path() is:\n+\t * * path_none, for any path, we return false.\n+\t * * path_recurse, for all path components, we return true\n+\t * * for some intermediate component, we make sure\n+\t * to add that path to the relevant list but return false\n+\t * signifying that we shouldn't recurse into it.\n+\t */\n \n \twhile (len && path[len - 1] == '/')\n \t\tlen--;\n \tif (!len)\n \t\treturn 1;\n+\n+\t/*\n+\t * We need a manufactured dirent with sufficient space to store a\n+\t * leading directory component of path in its d_name. Here, we\n+\t * assume that the dirent's d_name is either declared as\n+\t * char d_name[BIG_ENOUGH]\n+\t * or that it is declared at the end of the struct as\n+\t * char d_name[]\n+\t * For either case, padding with len+1 bytes at the end will ensure\n+\t * sufficient storage space.\n+\t */\n+\tde = xcalloc(1, sizeof(struct dirent)+len+1);\n+\tmemset(&cdir, 0, sizeof(cdir));\n+\tcdir.de = de;\n+#if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)\n+\tde->d_type = DT_DIR;\n+#endif\n \tbaselen = 0;\n+\tprevlen = 0;\n \twhile (1) {\n-\t\tcp = path + baselen + !!baselen;\n+\t\tprevlen = baselen + !!baselen;\n+\t\tcp = path + prevlen;\n \t\tcp = memchr(cp, '/', path + len - cp);\n \t\tif (!cp)\n \t\t\tbaselen = len;\n \t\telse\n \t\t\tbaselen = cp - path;\n-\t\tstrbuf_setlen(&sb, 0);\n+\t\tstrbuf_reset(&sb);\n \t\tstrbuf_add(&sb, path, baselen);\n \t\tif (!is_directory(sb.buf))\n \t\t\tbreak;\n-\t\tif (simplify_away(sb.buf, sb.len, pathspec))\n-\t\t\tbreak;\n-\t\tif (treat_one_path(dir, NULL, istate, &sb, baselen, pathspec,\n-\t\t\t\t DT_DIR, NULL) == path_none)\n+\t\tstrbuf_reset(&sb);\n+\t\tstrbuf_add(&sb, path, prevlen);\n+\t\tmemcpy(de->d_name, path+prevlen, baselen-prevlen);\n+\t\tde->d_name[baselen-prevlen] = '\\0';\n+\t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n+\t\t\t\t pathspec);\n+\t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n-\t\tif (len <= baselen) {\n-\t\t\trc = 1;\n+\t\tif (len <= baselen)\n \t\t\tbreak; /* finished checking */\n-\t\t}\n \t}\n+\tadd_path_to_appropriate_result_list(dir, NULL, &cdir, istate,\n+\t\t\t\t\t &sb, baselen, pathspec,\n+\t\t\t\t\t state);\n+\n+\tfree(de);\n \tstrbuf_release(&sb);\n-\treturn rc;\n+\treturn state == path_recurse;\n }\n \n static const char *get_ident_string(void)\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex d6e161ddd8..098fddc75b 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -74,7 +74,7 @@ test_expect_success 'git ls-files -o --directory untracked_dir does not recurse'\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o --directory untracked_dir/ does not recurse' '\n+test_expect_success 'git ls-files -o --directory untracked_dir/ does not recurse' '\n \techo untracked_dir/ >expect &&\n \tgit ls-files -o --directory untracked_dir/ >actual &&\n \ttest_cmp expect actual\n@@ -86,7 +86,7 @@ test_expect_success 'git ls-files -o untracked_repo does not recurse' '\n \ttest_cmp expect actual\n '\n \n-test_expect_failure 'git ls-files -o untracked_repo/ does not recurse' '\n+test_expect_success 'git ls-files -o untracked_repo/ does not recurse' '\n \techo untracked_repo/ >expect &&\n \tgit ls-files -o untracked_repo/ >actual &&\n \ttest_cmp expect actual\n@@ -133,7 +133,7 @@ test_expect_success 'git ls-files -o .git shows nothing' '\n \ttest_must_be_empty actual\n '\n \n-test_expect_failure 'git ls-files -o .git/ shows nothing' '\n+test_expect_success 'git ls-files -o .git/ shows nothing' '\n \tgit ls-files -o .git/ >actual &&\n \ttest_must_be_empty actual\n '\n-- \ngitgitgadget\n\n"},{"id":"388575","messageId":"e4768931d0e9bca4fd8fcb7f3048287f4fa6d0a5.1576790906.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com","subject":"[PATCH v5 8/8] dir: consolidate similar code in treat_directory()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-19T21:28:26Z","receivedAt":"2019-12-19T21:28:42Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nBoth the DIR_SKIP_NESTED_GIT and DIR_NO_GITLINKS cases were checking for\nwhether a path was actually a nonbare repository. That code could be\nshared, with just the result of how to act differing between the two\ncases.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 18 +++++++-----------\n 1 file changed, 7 insertions(+), 11 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 357f9593c4..e1b74f6478 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1461,6 +1461,8 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \tconst char *dirname, int len, int baselen, int exclude,\n \tconst struct pathspec *pathspec)\n {\n+\tint nested_repo = 0;\n+\n \t/* The \"len-1\" is to strip the final '/' */\n \tswitch (directory_exists_in_index(istate, dirname, len-1)) {\n \tcase index_directory:\n@@ -1470,15 +1472,16 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \t\treturn path_none;\n \n \tcase index_nonexistent:\n-\t\tif (dir->flags & DIR_SKIP_NESTED_GIT) {\n-\t\t\tint nested_repo;\n+\t\tif ((dir->flags & DIR_SKIP_NESTED_GIT) ||\n+\t\t !(dir->flags & DIR_NO_GITLINKS)) {\n \t\t\tstruct strbuf sb = STRBUF_INIT;\n \t\t\tstrbuf_addstr(&sb, dirname);\n \t\t\tnested_repo = is_nonbare_repository_dir(&sb);\n \t\t\tstrbuf_release(&sb);\n-\t\t\tif (nested_repo)\n-\t\t\t\treturn path_none;\n \t\t}\n+\t\tif (nested_repo)\n+\t\t\treturn ((dir->flags & DIR_SKIP_NESTED_GIT) ? path_none :\n+\t\t\t\t(exclude ? path_excluded : path_untracked));\n \n \t\tif (dir->flags & DIR_SHOW_OTHER_DIRECTORIES)\n \t\t\tbreak;\n@@ -1506,13 +1509,6 @@ static enum path_treatment treat_directory(struct dir_struct *dir,\n \n \t\t\treturn path_none;\n \t\t}\n-\t\tif (!(dir->flags & DIR_NO_GITLINKS)) {\n-\t\t\tstruct strbuf sb = STRBUF_INIT;\n-\t\t\tstrbuf_addstr(&sb, dirname);\n-\t\t\tif (is_nonbare_repository_dir(&sb))\n-\t\t\t\treturn exclude ? path_excluded : path_untracked;\n-\t\t\tstrbuf_release(&sb);\n-\t\t}\n \t\treturn path_recurse;\n \t}\n \n-- \ngitgitgadget\n"},{"id":"388576","messageId":"5275e6d7f0cc70468ecd76a88e0afd03339583a0.1576790906.git.gitgitgadget@gmail.com","threadId":"52419","inReplyTo":"pull.676.v5.git.git.1576790906.gitgitgadget@gmail.com","subject":"[PATCH v5 7/8] dir: synchronize treat_leading_path() and read_directory_recursive()","fromName":"Elijah Newren via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2019-12-19T21:28:25Z","receivedAt":"2019-12-19T21:28:43Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"From: Elijah Newren \n\nOur optimization to avoid calling into read_directory_recursive() when\nall pathspecs have a common leading directory mean that we need to match\nthe logic that read_directory_recursive() would use if we had just\ncalled it from the root. Since it does more than call treat_path() we\nneed to copy that same logic.\n\nAlternatively, we could try to change treat_path to return path_recurse\nfor an untracked directory under the given special circumstances that\nthis logic checks for, but a simple switch results in many test failures\nsuch as 'git clean -d' not wiping out untracked but empty directories.\nTo work around that, we'd need the caller of treat_path to check for\npath_recurse and sometimes special case it into path_untracked. In\nother words, we'd still have extra logic in both places.\n\nNeeding to duplicate logic like this means it is guaranteed someone will\neventually need to make further changes and forget to update both\nlocations. It is tempting to just nuke the leading_directory special\ncasing to avoid such bugs and simplify the code, but unpack_trees'\nverify_clean_subdirectory() also calls read_directory() and does so with\na non-empty leading path, so I'm hesitant to try to restructure further.\nAdd obnoxious warnings to treat_leading_path() and\nread_directory_recursive() to try to warn people of such problems.\n\nSigned-off-by: Elijah Newren \n---\n dir.c | 30 +++++++++++++++++++\n ...common-prefixes-and-directory-traversal.sh | 2 +-\n t/t7061-wtstatus-ignore.sh | 2 +-\n 3 files changed, 32 insertions(+), 2 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex a42cc2aa8c..357f9593c4 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1990,6 +1990,15 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tstruct untracked_cache_dir *untracked, int check_only,\n \tint stop_at_first_file, const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING WARNING WARNING:\n+\t *\n+\t * Any updates to the traversal logic here may need corresponding\n+\t * updates in treat_leading_path(). See the commit message for the\n+\t * commit adding this warning as well as the commit preceding it\n+\t * for details.\n+\t */\n+\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n@@ -2101,6 +2110,15 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\t\t const char *path, int len,\n \t\t\t const struct pathspec *pathspec)\n {\n+\t/*\n+\t * WARNING WARNING WARNING:\n+\t *\n+\t * Any updates to the traversal logic here may need corresponding\n+\t * updates in treat_leading_path(). See the commit message for the\n+\t * commit adding this warning as well as the commit preceding it\n+\t * for details.\n+\t */\n+\n \tstruct strbuf sb = STRBUF_INIT;\n \tint prevlen, baselen;\n \tconst char *cp;\n@@ -2166,6 +2184,18 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tde->d_name[baselen-prevlen] = '\\0';\n \t\tstate = treat_path(dir, NULL, &cdir, istate, &sb, prevlen,\n \t\t\t\t pathspec);\n+\t\tif (state == path_untracked &&\n+\t\t get_dtype(cdir.de, istate, sb.buf, sb.len) == DT_DIR &&\n+\t\t (dir->flags & DIR_SHOW_IGNORED_TOO ||\n+\t\t do_match_pathspec(istate, pathspec, sb.buf, sb.len,\n+\t\t\t\t baselen, NULL, DO_MATCH_LEADING_PATHSPEC) == MATCHED_RECURSIVELY_LEADING_PATHSPEC)) {\n+\t\t\tadd_path_to_appropriate_result_list(dir, NULL, &cdir,\n+\t\t\t\t\t\t\t istate,\n+\t\t\t\t\t\t\t &sb, baselen,\n+\t\t\t\t\t\t\t pathspec, state);\n+\t\t\tstate = path_recurse;\n+\t\t}\n+\n \t\tif (state != path_recurse)\n \t\t\tbreak; /* do not recurse into it */\n \t\tif (len <= baselen)\ndiff --git a/t/t3011-common-prefixes-and-directory-traversal.sh b/t/t3011-common-prefixes-and-directory-traversal.sh\nindex 098fddc75b..3da5b2b6e7 100755\n--- a/t/t3011-common-prefixes-and-directory-traversal.sh\n+++ b/t/t3011-common-prefixes-and-directory-traversal.sh\n@@ -195,7 +195,7 @@ test_expect_success 'git ls-files -o consistent between one or two dirs' '\n \n # ls-files doesn't have a way to request showing both untracked and ignored\n # files at the same time, so use `git status --ignored`\n-test_expect_failure 'git status --ignored shows same files under dir with or without pathspec' '\n+test_expect_success 'git status --ignored shows same files under dir with or without pathspec' '\n \tcat <<-EOF >expect &&\n \t?? an_untracked_dir/\n \t!! an_untracked_dir/ignored\ndiff --git a/t/t7061-wtstatus-ignore.sh b/t/t7061-wtstatus-ignore.sh\nindex 84366050da..e4cf5484f9 100755\n--- a/t/t7061-wtstatus-ignore.sh\n+++ b/t/t7061-wtstatus-ignore.sh\n@@ -47,7 +47,7 @@ cat >expected <<\\EOF\n !! untracked/ignored\n EOF\n \n-test_expect_failure 'status of untracked directory with --ignored works with or without prefix' '\n+test_expect_success 'status of untracked directory with --ignored works with or without prefix' '\n \tgit status --porcelain --ignored >tmp &&\n \tgrep untracked/ tmp >actual &&\n \trm tmp &&\n-- \ngitgitgadget\n\n"},{"id":"388597","messageId":"20191219222403.GA705525@coredump.intra.peff.net","threadId":"52419","inReplyTo":"CABPp-BHhAPjL9daPdp7UmVT+vLrcpo6SX0rKt5s1bH4Vz31wxw@mail.gmail.com","subject":"Re: [PATCH v4 6/8] dir: fix checks on common prefix directory","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-12-19T22:24:03Z","receivedAt":"2019-12-19T22:24:06Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Thu, Dec 19, 2019 at 12:23:29PM -0800, Elijah Newren wrote:\n\n> > > while (len && path[len - 1] == '/')\n> > > len--;\n> > > if (!len)\n> > > return 1;\n> > > +\n> > > + de = xcalloc(1, sizeof(struct dirent)+len+1);\n> >\n> > That \"+len+1\" may deserve a comment?\n> \n> Good point, I'll add one and send a re-roll.\n\nPlease use st_add3() while you are at it.\n\nI'd also usually suggest FLEX_ALLOC_MEM() for even more simplicity, but\nit looks like filling the string is handled separately (and done many\ntimes).\n\nI have to wonder, though, if it wouldn't be simpler to move away from\n\"struct dirent\" here (and it looks like Junio suggested the same earlier\nin the thread). I don't know this code very well, but it looks\nlike it could easily get by passing around a name pointer and a dtype\nthrough the cached_dir. The patch below seems like it's not too bad a\ncleanup, but possibly the names could be better.\n\n---\n dir.c | 48 ++++++++++++++++++-----------------\n 1 file changed, 25 insertions(+), 23 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex 43e2f47f66..e1cba688f3 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -41,7 +41,8 @@ struct cached_dir {\n \tint nr_files;\n \tint nr_dirs;\n \n-\tstruct dirent *de;\n+\tconst char *d_name;\n+\tint d_type;\n \tconst char *file;\n \tstruct untracked_cache_dir *ucd;\n };\n@@ -50,8 +51,8 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tstruct index_state *istate, const char *path, int len,\n \tstruct untracked_cache_dir *untracked,\n \tint check_only, int stop_at_first_file, const struct pathspec *pathspec);\n-static int get_dtype(struct dirent *de, struct index_state *istate,\n-\t\t const char *path, int len);\n+static int resolve_dtype(int dtype, struct index_state *istate,\n+\t\t\t const char *path, int len);\n \n int count_slashes(const char *s)\n {\n@@ -1050,8 +1051,7 @@ static struct path_pattern *last_matching_pattern_from_list(const char *pathname\n \t\tint prefix = pattern->nowildcardlen;\n \n \t\tif (pattern->flags & PATTERN_FLAG_MUSTBEDIR) {\n-\t\t\tif (*dtype == DT_UNKNOWN)\n-\t\t\t\t*dtype = get_dtype(NULL, istate, pathname, pathlen);\n+\t\t\t*dtype = resolve_dtype(*dtype, istate, pathname, pathlen);\n \t\t\tif (*dtype != DT_DIR)\n \t\t\t\tcontinue;\n \t\t}\n@@ -1639,10 +1639,9 @@ static int get_index_dtype(struct index_state *istate,\n \treturn DT_UNKNOWN;\n }\n \n-static int get_dtype(struct dirent *de, struct index_state *istate,\n-\t\t const char *path, int len)\n+static int resolve_dtype(int dtype, struct index_state *istate,\n+\t\t\t const char *path, int len)\n {\n-\tint dtype = de ? DTYPE(de) : DT_UNKNOWN;\n \tstruct stat st;\n \n \tif (dtype != DT_UNKNOWN)\n@@ -1667,14 +1666,13 @@ static enum path_treatment treat_one_path(struct dir_struct *dir,\n \t\t\t\t\t struct strbuf *path,\n \t\t\t\t\t int baselen,\n \t\t\t\t\t const struct pathspec *pathspec,\n-\t\t\t\t\t int dtype, struct dirent *de)\n+\t\t\t\t\t int dtype)\n {\n \tint exclude;\n \tint has_path_in_index = !!index_file_exists(istate, path->buf, path->len, ignore_case);\n \tenum path_treatment path_treatment;\n \n-\tif (dtype == DT_UNKNOWN)\n-\t\tdtype = get_dtype(de, istate, path->buf, path->len);\n+\tdtype = resolve_dtype(dtype, istate, path->buf, path->len);\n \n \t/* Always exclude indexed files */\n \tif (dtype != DT_DIR && has_path_in_index)\n@@ -1782,21 +1780,18 @@ static enum path_treatment treat_path(struct dir_struct *dir,\n \t\t\t\t int baselen,\n \t\t\t\t const struct pathspec *pathspec)\n {\n-\tint dtype;\n-\tstruct dirent *de = cdir->de;\n-\n-\tif (!de)\n+\tif (!cdir->d_name)\n \t\treturn treat_path_fast(dir, untracked, cdir, istate, path,\n \t\t\t\t baselen, pathspec);\n-\tif (is_dot_or_dotdot(de->d_name) || !fspathcmp(de->d_name, \".git\"))\n+\tif (is_dot_or_dotdot(cdir->d_name) || !fspathcmp(cdir->d_name, \".git\"))\n \t\treturn path_none;\n \tstrbuf_setlen(path, baselen);\n-\tstrbuf_addstr(path, de->d_name);\n+\tstrbuf_addstr(path, cdir->d_name);\n \tif (simplify_away(path->buf, path->len, pathspec))\n \t\treturn path_none;\n \n-\tdtype = DTYPE(de);\n-\treturn treat_one_path(dir, untracked, istate, path, baselen, pathspec, dtype, de);\n+\treturn treat_one_path(dir, untracked, istate, path, baselen, pathspec,\n+\t\t\t cdir->d_type);\n }\n \n static void add_untracked(struct untracked_cache_dir *dir, const char *name)\n@@ -1884,10 +1879,17 @@ static int open_cached_dir(struct cached_dir *cdir,\n \n static int read_cached_dir(struct cached_dir *cdir)\n {\n+\tstruct dirent *de;\n+\n \tif (cdir->fdir) {\n-\t\tcdir->de = readdir(cdir->fdir);\n-\t\tif (!cdir->de)\n+\t\tde = readdir(cdir->fdir);\n+\t\tif (!de) {\n+\t\t\tcdir->d_name = NULL;\n+\t\t\tcdir->d_type = DT_UNKNOWN;\n \t\t\treturn -1;\n+\t\t}\n+\t\tcdir->d_name = de->d_name;\n+\t\tcdir->d_type = DTYPE(de);\n \t\treturn 0;\n \t}\n \twhile (cdir->nr_dirs < cdir->untracked->dirs_nr) {\n@@ -1970,7 +1972,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t/* recurse into subdir if instructed by treat_path */\n \t\tif ((state == path_recurse) ||\n \t\t\t((state == path_untracked) &&\n-\t\t\t (get_dtype(cdir.de, istate, path.buf, path.len) == DT_DIR) &&\n+\t\t\t (resolve_dtype(cdir.d_type, istate, path.buf, path.len) == DT_DIR) &&\n \t\t\t ((dir->flags & DIR_SHOW_IGNORED_TOO) ||\n \t\t\t (pathspec &&\n \t\t\t do_match_pathspec(istate, pathspec, path.buf, path.len,\n@@ -2103,7 +2105,7 @@ static int treat_leading_path(struct dir_struct *dir,\n \t\tif (simplify_away(sb.buf, sb.len, pathspec))\n \t\t\tbreak;\n \t\tif (treat_one_path(dir, NULL, istate, &sb, baselen, pathspec,\n-\t\t\t\t DT_DIR, NULL) == path_none)\n+\t\t\t\t DT_DIR) == path_none)\n \t\t\tbreak; /* do not recurse into it */\n \t\tif (len <= baselen) {\n \t\t\trc = 1;\n"},{"id":"388616","messageId":"CABPp-BEe7=LfR5KJriu1Lq-v5SVET-zm3GK3ZPjOYuLHrxqimg@mail.gmail.com","threadId":"52419","inReplyTo":"20191219222403.GA705525@coredump.intra.peff.net","subject":"Re: [PATCH v4 6/8] dir: fix checks on common prefix directory","fromName":"Elijah Newren","fromEmail":"newren@gmail.com","sentAt":"2019-12-20T17:00:40Z","receivedAt":"2019-12-20T17:00:55Z","isPatch":true,"sender":{"key":"newren@gmail.com","avatar":"https://avatars.githubusercontent.com/u/5455730?v=4"},"body":"Hi Peff,\n\nOn Thu, Dec 19, 2019 at 2:24 PM Jeff King wrote:\n>\n> On Thu, Dec 19, 2019 at 12:23:29PM -0800, Elijah Newren wrote:\n>\n> > > > while (len && path[len - 1] == '/')\n> > > > len--;\n> > > > if (!len)\n> > > > return 1;\n> > > > +\n> > > > + de = xcalloc(1, sizeof(struct dirent)+len+1);\n> > >\n> > > That \"+len+1\" may deserve a comment?\n> >\n> > Good point, I'll add one and send a re-roll.\n>\n> Please use st_add3() while you are at it.\n\nI would, but Junio already took the patches and applied them to next\nalready. (I am curious, though, why we're worried about overflow in a\ncontext like this?)\n\n> I'd also usually suggest FLEX_ALLOC_MEM() for even more simplicity, but\n> it looks like filling the string is handled separately (and done many\n> times).\n\nYes, the string is handled separately; I don't manufacture a dirent\nper leading directory component of the common prefix, but just\nallocate one and re-use it.\n\n> I have to wonder, though, if it wouldn't be simpler to move away from\n> \"struct dirent\" here (and it looks like Junio suggested the same earlier\n> in the thread). I don't know this code very well, but it looks\n> like it could easily get by passing around a name pointer and a dtype\n> through the cached_dir. The patch below seems like it's not too bad a\n> cleanup, but possibly the names could be better.\n\nThis was mentioned twice upthread, first by me then by Junio (and I'll\ninclude my final response too):\n\n>>> I need to manufacture a dirent myself; short of that, the most\n>>> likely alternative is to drop patches 2 & 5-8 of this series and\n>>> throw my hands in the air and give up.\n>>> ...\n>>> It'd be an awful lot of work to rip [dirent] out...unless someone\n>>> else has some bright ideas about something clever we can do, then I\n>>> think this problem blows up in complexity to a level where I don't\n>>> think it's worth addressing.\n>>> ...\n>>> Any bright ideas about what to do here?\n>>\n>> Restructuring the code so that we do not use \"struct dirent\" in the\n>> first place, even in the original code that used only those obtained\n>> from readdir(), perhaps?\n>\n> Okay, I'll submit a new series dropping most the patches.\n\nIt's possible I vastly overestimated how much work ripping out the\ndirent would be; I mean I've mis-estimated absolutely everything in\ndir.c and assumed each \"little\" thing would all be a small amount of\nwork, so maybe I'm just swinging the pendulum too far the other way.\nBut, although I think this alternative would be the cleanest, I saw a\ncouple things that looked like this was going to turn into a huge can\nof worms when I started to peek at what it all touched. I'd be happy\nfor someone to take this route, but it won't be me (see also\nhttps://lore.kernel.org/git/CABPp-BEkX9cH1=r3dJ4WLzcJKVcF-KpGUkshL34MMp3Xhhhpuw@mail.gmail.com/).\n\nElijah\n"},{"id":"388636","messageId":"xmqqlfr6anbi.fsf@gitster-ct.c.googlers.com","threadId":"52419","inReplyTo":"20191219222403.GA705525@coredump.intra.peff.net","subject":"Re: [PATCH v4 6/8] dir: fix checks on common prefix directory","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2019-12-20T18:01:21Z","receivedAt":"2019-12-20T18:01:26Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> Please use st_add3() while you are at it.\n>\n> I'd also usually suggest FLEX_ALLOC_MEM() for even more simplicity, but\n> it looks like filling the string is handled separately (and done many\n> times).\n>\n> I have to wonder, though, if it wouldn't be simpler to move away from\n> \"struct dirent\" here (and it looks like Junio suggested the same earlier\n> in the thread). I don't know this code very well, but it looks\n> like it could easily get by passing around a name pointer and a dtype\n> through the cached_dir. The patch below seems like it's not too bad a\n> cleanup, but possibly the names could be better.\n\nIt does look like a good clean-up.\n\nIn the meantime, here is to apologize for merging the patch a bit\ntoo early to 'next'.\n\n-- >8 --\nFrom: Junio C Hamano \nDate: Fri, 20 Dec 2019 09:55:53 -0800\nSubject: [PATCH] dir.c: use st_add3() for allocation size\n\nWhen preparing a manufactured dirent instance, we add a length of\npath to the size of struct to decide how many bytes to allocate.\nMake sure this addition does not wrap-around to cause us\nunderallocate.\n\nSuggested-by: Jeff King \nSigned-off-by: Junio C Hamano \n---\n dir.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/dir.c b/dir.c\nindex e1b74f6478..113170aeb9 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2154,7 +2154,7 @@ static int treat_leading_path(struct dir_struct *dir,\n \t * For either case, padding with len+1 bytes at the end will ensure\n \t * sufficient storage space.\n \t */\n-\tde = xcalloc(1, sizeof(struct dirent)+len+1);\n+\tde = xcalloc(1, st_add3(sizeof(struct dirent), len, 1));\n \tmemset(&cdir, 0, sizeof(cdir));\n \tcdir.de = de;\n #if defined(DT_UNKNOWN) && !defined(NO_D_TYPE_IN_DIRENT)\n-- \n2.24.1-769-g187e15c71d\n\n\n\n"},{"id":"388675","messageId":"20191220211400.GB2752079@coredump.intra.peff.net","threadId":"52419","inReplyTo":"CABPp-BEe7=LfR5KJriu1Lq-v5SVET-zm3GK3ZPjOYuLHrxqimg@mail.gmail.com","subject":"Re: [PATCH v4 6/8] dir: fix checks on common prefix directory","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-12-20T21:14:00Z","receivedAt":"2019-12-20T21:14:07Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 20, 2019 at 09:00:40AM -0800, Elijah Newren wrote:\n\n> > > > > + de = xcalloc(1, sizeof(struct dirent)+len+1);\n> > > >\n> > > > That \"+len+1\" may deserve a comment?\n> > >\n> > > Good point, I'll add one and send a re-roll.\n> >\n> > Please use st_add3() while you are at it.\n> \n> I would, but Junio already took the patches and applied them to next\n> already. (I am curious, though, why we're worried about overflow in a\n> context like this?)\n\nIf len is large enough to cause integer overflow when computing the\ntotal size, then we'd allocate a too-small buffer (and then later\noverflow the buffer when writing into it).\n\nI'm not sure how possible that is here. On 32-bit systems, overflowing\nsize_t only needs 4GB. you're not likely to have a 4GB path on a\nfilesystem, but malicious folks could shove them into a tree. I'm not\nsure if this code could be triggered for anything that doesn't actually\nexist on the filesystem, though.\n\nYou're also not likely to actually manage to store a 4GB string in\n\"path\" on a 32-bit system in the first place. But \"len\" is actually an\n\"int\". On a 64-bit system it would be easy to do, though, and int is\nstill 32 bits there. But because the result of sizeof() is a size_t, I\nthink the int will be promoted as well during the addition (and assuming\nit's not negative, will be too small to overflow). (Also, the \"len\"\nparameter probably should to be a size_t in the first place, but that's\nnot new).\n\nSo I don't think it's exploitable, but as you can see there's a bit of\npondering to see that it's so. When I audit I usually look for something\nlike /x[mc]alloc.*[+*] / to find potential problem spots. Even if we're\npretty sure a particular site isn't vulnerable, marking it with st_add()\nerrs on the safe side, and makes those audits easier.\n\n> It's possible I vastly overestimated how much work ripping out the\n> dirent would be; I mean I've mis-estimated absolutely everything in\n> dir.c and assumed each \"little\" thing would all be a small amount of\n> work, so maybe I'm just swinging the pendulum too far the other way.\n> But, although I think this alternative would be the cleanest, I saw a\n> couple things that looked like this was going to turn into a huge can\n> of worms when I started to peek at what it all touched. I'd be happy\n> for someone to take this route, but it won't be me (see also\n> https://lore.kernel.org/git/CABPp-BEkX9cH1=r3dJ4WLzcJKVcF-KpGUkshL34MMp3Xhhhpuw@mail.gmail.com/).\n\nOK. I certainly don't insist on this direction. I just saw the\nportability issues and wondered how bad it would be to do so. Hence the\npatch I sent, which I _think_ is correct, but I really don't know the\ndir.c code very well. And I'm sure it will not surprise you that I have\ngenerally been confused and/or frightened by it when I do look at it. :)\n\n-Peff\n"},{"id":"388676","messageId":"20191220211525.GC2752079@coredump.intra.peff.net","threadId":"52419","inReplyTo":"xmqqlfr6anbi.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 6/8] dir: fix checks on common prefix directory","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2019-12-20T21:15:25Z","receivedAt":"2019-12-20T21:15:28Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Dec 20, 2019 at 10:01:21AM -0800, Junio C Hamano wrote:\n\n> In the meantime, here is to apologize for merging the patch a bit\n> too early to 'next'.\n> \n> -- >8 --\n> From: Junio C Hamano \n> Date: Fri, 20 Dec 2019 09:55:53 -0800\n> Subject: [PATCH] dir.c: use st_add3() for allocation size\n\nThanks, I think this is an easy improvement worth doing (I laid out more\nin my response to Elijah, but: I don't think this is exploitable, but\nI'd rather err on the side of caution and ease of auditing).\n\n-Peff\n"},{"id":"388754","messageId":"nycvar.QRO.7.76.6.1912212305260.46@tvgsbejvaqbjf.bet","threadId":"52419","inReplyTo":"xmqq1rt2ized.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v3 0/3] Directory traversal bugs","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-12-21T22:05:54Z","receivedAt":"2019-12-21T22:06:24Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Tue, 17 Dec 2019, Junio C Hamano wrote:\n\n> Johannes Schindelin writes:\n>\n> > As I said elsewhere, if Git for Windows' FSCache hack is the only thing\n> > that is broken by this patch series, in light of the bugs that it _does_\n> > fix I would rather adjust the FSCache patches to accommodate v2.\n>\n> With \"FSCache hack\", do you refer to the \"d_name is a pointer to\n> elsewhere\" thing?\n\nYes.\n\n> If so, I too very much appreciate the direction you are suggesting.\n> Seeing that these three patches essentially are the same as three (1/8,\n> 3/8 and 4/8) from the v2, I'd keep all the 8 patches from v2 in my tree\n> for now.\n>\n> Thanks, both.\n\nThank you,\nDscho\n"}]}