{"thread":{"id":"66377","subject":"[PATCH 0/2] ls-files: reuse the untracked cache","startedAt":"2026-09-23T17:10:27Z","lastAt":"2026-09-23T22:27:23Z","messageCount":8,"participants":["Tamir Duberstein"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"553075","messageId":"20260923-ls-files-untracked-cache-v1-0-08db4cc1efdb@gmail.com","threadId":"66377","inReplyTo":null,"subject":"[PATCH 0/2] ls-files: reuse the untracked cache","fromName":"Tamir Duberstein","fromEmail":"tamird@gmail.com","sentAt":"2026-09-23T17:10:07Z","receivedAt":"2026-09-23T17:10:27Z","isPatch":true,"body":"Repeated queries such as\n\n  git ls-files --cached --others --exclude-standard -z -- \"**/pyproject.toml\"\n\nwalk the working tree even when status has already populated an untracked\ncache. This series lets ls-files reuse those directory listings. It\nexpands partial scans and collapsed untracked directories as needed,\nwithout writing the converted cache back to the index.\n\nThe first patch fixes inconsistent ignore-file hashes that invalidate an\nunchanged cache. The second enables reuse for ls-files and filters complete\nlistings after the walk. Fixed-prefix queries retain directory pruning;\nattribute and exclude pathspecs retain their existing traversal because\nmatching a directory can differ from matching its children.\n\nOn macOS, a synthetic tree with 100,000 tracked files in 5,000 directories\nand a few untracked files averaged 361 ms before and 50 ms after in five\nhyperfine runs after one warmup. Directory opens fell from 5,058 to seven.\nWith half the files untracked, five alternating runs averaged 659 ms\nbefore and 339 ms after; that case still expands 2,533 partially cached\ndirectories. A cache populated with status.showUntrackedFiles=all avoids\ndirectory opens in both cases. These measurements disable fsmonitor and\nuse a populated cache with a warm filesystem.\n\nUncached controls were noisy; the apparent slowdown in the first batch\ndid not recur. Literal-query results varied in both directions, with a\nseparate five-run hyperfine check measuring 17 ms before and 20 ms after.\n\nPrepared with Codex, including code review and benchmark work by separate\nCodex agents.\n\nSigned-off-by: Tamir Duberstein <tamird@gmail.com>\n---\nTamir Duberstein (2):\n      dir: hash ignore files before adding parser LF\n      ls-files: reuse cached untracked listings\n\n builtin/ls-files.c                | 11 +++++--\n dir.c                             | 64 +++++++++++++++++++++++++++++++++---\n dir.h                             |  2 ++\n t/perf/p3010-ls-files.sh          | 15 +++++++++\n t/t7063-status-untracked-cache.sh | 68 ++++++++++++++++++++++++++++++++++++---\n 5 files changed, 149 insertions(+), 11 deletions(-)\n\n\n---\nbase-commit: 3bc0341126508f78f5869cbfc0005e987efdf0c7\nchange-id: 20260923-ls-files-untracked-cache-3559bed01a3e\n\n"},{"id":"553076","messageId":"20260923-ls-files-untracked-cache-v1-1-08db4cc1efdb@gmail.com","threadId":"66377","inReplyTo":"20260923-ls-files-untracked-cache-v1-0-08db4cc1efdb@gmail.com","subject":"[PATCH 1/2] dir: hash ignore files before adding parser LF","fromName":"Tamir Duberstein","fromEmail":"tamird@gmail.com","sentAt":"2026-09-23T17:10:08Z","receivedAt":"2026-09-23T17:10:29Z","isPatch":true,"body":"add_patterns() appends a newline for the pattern parser before computing\nan ignore file's object ID. Its fallback hash therefore includes a byte\nthat is absent from the file. The fast path instead copies the original\nblob ID from an up-to-date index entry.\n\nSwitching between those paths changes the recorded ignore identity even\nwhen the file has not changed, invalidating the untracked cache below it.\nCompute the hash before appending the parser newline so both paths agree.\nUpdate the expected identities of the untracked ignore files accordingly.\n\nSigned-off-by: Tamir Duberstein <tamird@gmail.com>\n---\n dir.c                             | 2 +-\n t/t7063-status-untracked-cache.sh | 8 ++++----\n 2 files changed, 5 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex d896e7be4b..037780e556 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1195,7 +1195,6 @@ static int add_patterns(const char *fname, const char *base, int baselen,\n \t\t\tclose(fd);\n \t\t\treturn -1;\n \t\t}\n-\t\tbuf[size++] = '\\n';\n \t\tclose(fd);\n \t\tif (oid_stat) {\n \t\t\tint pos;\n@@ -1215,6 +1214,7 @@ static int add_patterns(const char *fname, const char *base, int baselen,\n \t\t\tfill_stat_data(&oid_stat->stat, &st);\n \t\t\toid_stat->valid = 1;\n \t\t}\n+\t\tbuf[size++] = '\\n';\n \t}\n \n \tif (size > PATTERN_MAX_FILE_SIZE) {\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 8929ef481f..10d4850077 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -98,11 +98,11 @@ test_expect_success 'setup' '\n \t: >.git/info/exclude &&\n \tgit update-index --untracked-cache &&\n \ttest_oid_cache <<-EOF\n-\troot sha1:e6fcc8f2ee31bae321d66afd183fcb7237afae6e\n-\troot sha256:b90c672088c015b9c83876e919da311bad4cd39639fb139f988af6a11493b974\n+\troot sha1:8510665149157c2bc901848c3e0b746954e9cbd9\n+\troot sha256:09ef24b38105f396a61ad78d73ba6a18ee3cbd89ce4524b4e13b6c1af191e2d8\n \n-\texclude sha1:13263c0978fb9fad16b2d580fb800b6d811c3ff0\n-\texclude sha256:fe4aaa1bbbbce4cb8f73426748a14c5ad6026b26f90505a0bf2494b165a5b76c\n+\texclude sha1:2bdf67abb163a4ffb2d7f3f0880c9fe5068ce782\n+\texclude sha256:b83643f4390b339c1b3ff2f5132c99bd4a77687dd321d3f386c25953aa6f1ce4\n \n \tdone sha1:1946f0437f90c5005533cbe1736a6451ca301714\n \tdone sha256:7f079501d79f665b3acc50f5e0e9e94509084d5032ac20113a37dd5029b757cc\n\n-- \n2.56.0.rc0.807.ga0c0929ce1.frankengit\n\n"},{"id":"553077","messageId":"20260923-ls-files-untracked-cache-v1-2-08db4cc1efdb@gmail.com","threadId":"66377","inReplyTo":"20260923-ls-files-untracked-cache-v1-0-08db4cc1efdb@gmail.com","subject":"[PATCH 2/2] ls-files: reuse cached untracked listings","fromName":"Tamir Duberstein","fromEmail":"tamird@gmail.com","sentAt":"2026-09-23T17:10:09Z","receivedAt":"2026-09-23T17:10:31Z","isPatch":true,"body":"ls-files --cached --others --exclude-standard with a wildcard pathspec\nwalks the working tree on every invocation, even when status has already\ncached its untracked directories.\n\nAttach the existing untracked cache before loading standard excludes.\nFor whole-tree wildcard queries without a fixed prefix, read complete\ncached listings and filter their results afterward. Preserve traversal\nfiltering for attribute and exclude pathspecs, which can reject a parent\ndirectory independently of its children.\n\nA normal status cache collapses untracked directories and may stop after\nfinding their first file. Rescan those entries while retaining complete\nlistings. Opt in from ls-files, which does not write the index, so the\nsaved status cache and its configured mode remain intact.\n\nOn a synthetic tree with 100,000 tracked files in 5,000 directories and\na few untracked files, five hyperfine runs after one warmup of the\npyproject.toml query average 361 ms before and 50 ms after with a normal\nstatus cache. Directory opens fall from 5,058 to seven. A tree with 50,000\nuntracked files averages 659 ms before and 339 ms after over five\nalternating runs; that case still expands the collapsed untracked\ndirectories. Without an existing cache, traversal is unchanged.\n\nSigned-off-by: Tamir Duberstein <tamird@gmail.com>\n---\n builtin/ls-files.c                | 11 +++++--\n dir.c                             | 62 +++++++++++++++++++++++++++++++++++++--\n dir.h                             |  2 ++\n t/perf/p3010-ls-files.sh          | 15 ++++++++++\n t/t7063-status-untracked-cache.sh | 60 +++++++++++++++++++++++++++++++++++++\n 5 files changed, 144 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex b044520f9e..da1804acf3 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -681,11 +681,16 @@ int cmd_ls_files(int argc,\n \tif (repo_read_index(repo) < 0)\n \t\tdie(\"index file corrupt\");\n \n+\tdir.untracked = repo->index->untracked;\n+\tdir.untracked_cache_expand = 1;\n \targc = parse_options(argc, argv, prefix, builtin_ls_files_options,\n \t\t\tls_files_usage, 0);\n-\tpl = add_pattern_list(&dir, EXC_CMDL, \"--exclude option\");\n-\tfor (i = 0; i < exclude_list.nr; i++) {\n-\t\tadd_pattern(exclude_list.items[i].string, \"\", 0, pl, --exclude_args);\n+\tif (!show_others || dir.flags || !dir.exclude_per_dir || with_tree)\n+\t\tdir.untracked = NULL;\n+\tif (exclude_list.nr) {\n+\t\tpl = add_pattern_list(&dir, EXC_CMDL, \"--exclude option\");\n+\t\tfor (i = 0; i < exclude_list.nr; i++)\n+\t\t\tadd_pattern(exclude_list.items[i].string, \"\", 0, pl, --exclude_args);\n \t}\n \n \tif (format && (show_stage || show_others || show_killed ||\ndiff --git a/dir.c b/dir.c\nindex 037780e556..eccd24e318 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -2974,6 +2974,27 @@ void remove_untracked_cache(struct index_state *istate)\n \t}\n }\n \n+static void expand_untracked_cache(struct untracked_cache *uc,\n+\t\t\t\t   struct untracked_cache_dir *dir)\n+{\n+\tif (!dir)\n+\t\treturn;\n+\t/* Re-read partial scans and listings containing collapsed directories. */\n+\tif (dir->check_only) {\n+\t\tinvalidate_directory(uc, dir);\n+\t\tdir->check_only = 0;\n+\t} else {\n+\t\tfor (size_t i = 0; i < dir->untracked_nr; i++) {\n+\t\t\tif (ends_with(dir->untracked[i], \"/\")) {\n+\t\t\t\tinvalidate_directory(uc, dir);\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tfor (size_t i = 0; i < dir->dirs_nr; i++)\n+\t\texpand_untracked_cache(uc, dir->dirs[i]);\n+}\n+\n static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *dir,\n \t\t\t\t\t\t      int base_len,\n \t\t\t\t\t\t      const struct pathspec *pathspec,\n@@ -3033,6 +3054,12 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \t\treturn NULL;\n \t}\n \n+\t/* Keep complete directory listings when expanding a normal-mode cache. */\n+\tif (dir->untracked_cache_expand && !dir->flags && dir->untracked->dir_flags == (DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES)) {\n+\t\texpand_untracked_cache(dir->untracked, dir->untracked->root);\n+\t\tdir->untracked->dir_flags = 0;\n+\t}\n+\n \t/*\n \t * If the untracked structure we received does not have the same flags\n \t * as requested in this run, we're going to need to either discard the\n@@ -3141,6 +3168,16 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t   const char *path, int len, const struct pathspec *pathspec)\n {\n \tstruct untracked_cache_dir *untracked;\n+\tconst struct pathspec *walk_pathspec = pathspec;\n+\t/* Attribute and exclude pathspecs can prune a directory by its own name. */\n+\tint filter = dir->untracked_cache_expand && dir->untracked &&\n+\t\t     !len && !dir->flags && pathspec && pathspec->nr &&\n+\t\t     !(pathspec->magic & (PATHSPEC_ATTR | PATHSPEC_EXCLUDE));\n+\n+\t/* Keep the usual pruning for queries with a fixed prefix. */\n+\tfor (int i = 0; filter && i < pathspec->nr; i++)\n+\t\tif (pathspec->items[i].nowildcard_len)\n+\t\t\tfilter = 0;\n \n \ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n \tdir->internal.visited_paths = 0;\n@@ -3151,15 +3188,34 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\treturn dir->nr;\n \t}\n \n-\tuntracked = validate_untracked_cache(dir, len, pathspec, istate);\n+\tuntracked = validate_untracked_cache(dir, len,\n+\t\t\t\t\t     filter ? NULL : pathspec, istate);\n \tif (!untracked)\n \t\t/*\n \t\t * make sure untracked cache code path is disabled,\n \t\t * e.g. prep_exclude()\n \t\t */\n \t\tdir->untracked = NULL;\n-\tif (!len || treat_leading_path(dir, istate, path, len, pathspec))\n-\t\tread_directory_recursive(dir, istate, path, len, untracked, 0, 0, pathspec);\n+\telse if (filter)\n+\t\twalk_pathspec = NULL;\n+\tif (!len || treat_leading_path(dir, istate, path, len, walk_pathspec))\n+\t\tread_directory_recursive(dir, istate, path, len, untracked, 0, 0,\n+\t\t\t\t\t walk_pathspec);\n+\tif (filter && untracked) {\n+\t\tint dst = 0;\n+\n+\t\t/* Cache complete listings; apply the query only to the results. */\n+\t\tfor (int i = 0; i < dir->nr; i++) {\n+\t\t\tstruct dir_entry *ent = dir->entries[i];\n+\n+\t\t\tif (match_pathspec(istate, pathspec, ent->name, ent->len,\n+\t\t\t\t\t   0, NULL, 0))\n+\t\t\t\tdir->entries[dst++] = dir->entries[i];\n+\t\t\telse\n+\t\t\t\tfree(dir->entries[i]);\n+\t\t}\n+\t\tdir->nr = dst;\n+\t}\n \tQSORT(dir->entries, dir->nr, cmp_dir_entry);\n \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n \ndiff --git a/dir.h b/dir.h\nindex 83e0f648a8..3403816271 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -296,6 +296,8 @@ struct dir_struct {\n \n \t/* Enable/update untracked file cache if set */\n \tstruct untracked_cache *untracked;\n+\t/* Allow expanding a normal-mode cache without saving it to the index. */\n+\tunsigned int untracked_cache_expand:1;\n \n \t/**\n \t * Deprecated: ls-files is the only allowed caller; all other callers\ndiff --git a/t/perf/p3010-ls-files.sh b/t/perf/p3010-ls-files.sh\nindex ae14449432..59eaea000b 100755\n--- a/t/perf/p3010-ls-files.sh\n+++ b/t/perf/p3010-ls-files.sh\n@@ -28,4 +28,19 @@ test_perf 'ls-files --modified with pathspec' '\n \t\t-- \"$pathspec\" >/dev/null\n '\n \n+test_perf 'ls-files --others with pathspec and no untracked cache' '\n+\tgit -c core.fsmonitor=false -c core.untrackedCache=false \\\n+\t\tls-files --cached --others --exclude-standard -- \"$pathspec\" >/dev/null\n+'\n+\n+test_expect_success 'populate the untracked cache' '\n+\tgit config core.untrackedCache true &&\n+\tgit -c core.fsmonitor=false status --porcelain >/dev/null\n+'\n+\n+test_perf 'ls-files --others with pathspec and untracked cache' '\n+\tgit -c core.fsmonitor=false ls-files --cached --others \\\n+\t\t--exclude-standard -- \"$pathspec\" >/dev/null\n+'\n+\n test_done\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 10d4850077..ccfc354dad 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -991,4 +991,64 @@ test_expect_success 'empty repo (no index) and core.untrackedCache' '\n \tgit -C emptyrepo -c core.untrackedCache=true write-tree\n '\n \n+test_expect_success 'ls-files expands cached directories and filters wildcard results' '\n+\ttest_create_repo ls-files-cache &&\n+\t(\n+\t\tcd ls-files-cache &&\n+\t\tgit config core.untrackedCache true &&\n+\t\tmkdir tracked untracked empty ignored-only &&\n+\t\ttouch tracked/pyproject.toml untracked/pyproject.toml &&\n+\t\ttouch untracked/other ignored-only/file &&\n+\t\techo ignored-only/file >.gitignore &&\n+\t\techo \"*.toml selected\" >.gitattributes &&\n+\t\tgit add .gitignore .gitattributes tracked &&\n+\t\tgit init nested.git &&\n+\t\ttest-tool chmtime =-300 . tracked untracked empty ignored-only &&\n+\t\tfor mode in normal all\n+\t\tdo\n+\t\t\t# Normal mode rescans the root and three partial directories.\n+\t\t\tcase \"$mode\" in\n+\t\t\tnormal) opened=4 ;;\n+\t\t\tall) opened=0 ;;\n+\t\t\tesac &&\n+\t\t\tgit config status.showUntrackedFiles \"$mode\" &&\n+\t\t\tgit status --porcelain >/dev/null &&\n+\t\t\tcp .git/index ../saved-index &&\n+\t\t\t: >\"$TRASH_DIRECTORY/ls-files.trace\" &&\n+\t\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/ls-files.trace\" \\\n+\t\t\t\tgit ls-files --cached --others --exclude-standard -z \\\n+\t\t\t\t-- \"**/pyproject.toml\" >../actual &&\n+\t\t\tprintf \"%s\\0\" untracked/pyproject.toml tracked/pyproject.toml \\\n+\t\t\t\t>../expect &&\n+\t\t\ttest_cmp ../expect ../actual &&\n+\t\t\ttest_cmp_bin ../saved-index .git/index &&\n+\t\t\ttest_grep \"read_directo.*opendir:$opened\\$\" \\\n+\t\t\t\t\"$TRASH_DIRECTORY/ls-files.trace\" &&\n+\t\t\ttest_grep \"read_directo.*gitignore-invalidation:0\\$\" \\\n+\t\t\t\t\"$TRASH_DIRECTORY/ls-files.trace\" || return 1\n+\t\t\tfor pathspec in \"*.git/\" \":(glob)**/*.toml\" \\\n+\t\t\t\t\":(exclude)untracked/\" \":(attr:selected)**/*.toml\" \\\n+\t\t\t\t\"tracked/pyproject.toml\"\n+\t\t\tdo\n+\t\t\t\tGIT_DISABLE_UNTRACKED_CACHE=1 git ls-files --others \\\n+\t\t\t\t\t--exclude-standard -- \"$pathspec\" >../expect &&\n+\t\t\t\tgit ls-files --others --exclude-standard \\\n+\t\t\t\t\t-- \"$pathspec\" >../actual &&\n+\t\t\t\ttest_cmp ../expect ../actual || return 1\n+\t\t\tdone || return 1\n+\t\tdone\n+\t)\n+'\n+\n+test_expect_success 'ls-files revalidates cached files and ignore rules' '\n+\t(\n+\t\tcd ls-files-cache &&\n+\t\techo untracked/ >>.gitignore &&\n+\t\ttouch tracked/new.toml empty/new.toml &&\n+\t\tgit ls-files --others --exclude-standard -- \"**/*.toml\" >../actual &&\n+\t\tprintf \"%s\\n\" empty/new.toml tracked/new.toml >../expect &&\n+\t\ttest_cmp ../expect ../actual\n+\t)\n+'\n+\n test_done\n\n-- \n2.56.0.rc0.807.ga0c0929ce1.frankengit\n\n"},{"id":"553095","messageId":"CAJ-ks9kW4nb8PFMWj+TDx8GtOwt_fTOcmoGvgqpqw04sNeYkqA@mail.gmail.com","threadId":"66377","inReplyTo":"20260923-ls-files-untracked-cache-v1-0-08db4cc1efdb@gmail.com","subject":"Re: [PATCH 0/2] ls-files: reuse the untracked cache","fromName":"Tamir Duberstein","fromEmail":"tamird@gmail.com","sentAt":"2026-09-23T18:38:05Z","receivedAt":"2026-09-23T18:38:44Z","isPatch":true,"body":"On Wed, Sep 23, 2026 at 1:10 PM Tamir Duberstein <tamird@gmail.com> wrote:\n>\n> Uncached controls were noisy; the apparent slowdown in the first batch\n> did not recur. Literal-query results varied in both directions, with a\n> separate five-run hyperfine check measuring 17 ms before and 20 ms after.\n>\n> Prepared with Codex, including code review and benchmark work by separate\n> Codex agents.\n\nJeez, sorry about this slop. This was operator error on my part; I did\nnot intend for these to go out in this form.\n\nThe first commit is in reasonable shape but I'm still iterating on the second.\n"},{"id":"553129","messageId":"20260923-ls-files-untracked-cache-v2-0-d7ee33476eb8@gmail.com","threadId":"66377","inReplyTo":"20260923-ls-files-untracked-cache-v1-0-08db4cc1efdb@gmail.com","subject":"[PATCH v2 0/3] ls-files: reuse and update the untracked cache","fromName":"Tamir Duberstein","fromEmail":"tamird@gmail.com","sentAt":"2026-09-23T22:26:57Z","receivedAt":"2026-09-23T22:27:15Z","isPatch":true,"body":"Repeated queries such as\n\n  git ls-files --cached --others --exclude-standard -z -- \"**/pyproject.toml\"\n\nwalk the working tree even when status has populated an untracked cache.\nThis series lets ls-files reuse those listings and save the work for later\ncommands through Git's existing optional index writes.\n\nThe first patch fixes inconsistent ignore-file hashes that invalidate an\nunchanged cache. The second lets 'git status -unormal' and\n'git status -uall' reuse the same cache. 'git status -unormal' still stops\nscanning an untracked directory after finding an untracked file; a later\ncommand requesting all untracked files completes the listing as needed.\nThe third lets ls-files use the cache and write pending untracked cache\nand fsmonitor updates back to the index. --no-optional-locks suppresses\nthose writes.\n\nFor wildcard pathspecs without a fixed prefix, apply the pathspec after\nreading complete directory listings. A first query can therefore scan\ndirectories that its pathspec would otherwise skip. Fixed-prefix,\nattribute and exclude pathspecs retain their existing traversal.\n\nA new dir_flags value makes older versions of Git rebuild the cache before\nusing it to list untracked files. Changes invalidate cached entries in\nparent directories as well, so updates can reopen more directories than\na cache populated with --untracked-files=all before this series.\n\nOn macOS, with 100,000 files in 5,000 leaf directories, half tracked.\nEach group of 100 leaf directories has 50 tracked and 50 untracked.\nEach binary populated its own cache using the indicated status mode.\n\nhyperfine --warmup 3 --runs 7 (mean ± standard deviation, milliseconds):\n\n  cache   fsmonitor  command          upstream        v1             v2\n  normal  off        query            241.3 ± 14.8    132.7 ± 20.0   33.3 ± 2.4\n  normal  off        query + status   292.7 ± 11.2    187.0 ± 10.8   92.6 ± 3.3\n  all     off        query            231.3 ± 14.6    29.3 ± 1.3     34.7 ± 2.0\n  normal  on         query            223.1 ± 17.8    111.0 ± 13.4   33.8 ± 1.5\n  all     on         query            216.6 ± 12.0    28.0 ± 5.1     34.6 ± 0.8\n\n\"query\" is the ls-files command above; \"status\" is\n\"git status -unormal --porcelain\". These are repeated queries with warm\nfilesystem and untracked caches.\n\nAssisted-by: LLM\nSigned-off-by: Tamir Duberstein <tamird@gmail.com>\n---\nChanges in v2:\n- Share the cache between --untracked-files=normal and --untracked-files=all.\n- Write ls-files untracked cache and fsmonitor updates back to the index.\n- Test alternating modes, cache invalidation and optional index writes.\n- Replace the v1 measurements with results for the revised implementation.\n- Link to v1: https://patch.msgid.link/20260923-ls-files-untracked-cache-v1-0-08db4cc1efdb@gmail.com\n\n---\nTamir Duberstein (3):\n      dir: hash ignore files before appending newline\n      dir: share untracked caches across output modes\n      ls-files: use and update the untracked cache\n\n Documentation/git-ls-files.adoc    |   4 +\n Documentation/gitformat-index.adoc |  13 +-\n builtin/ls-files.c                 |  40 +++++-\n dir.c                              | 251 ++++++++++++++++++----------------\n dir.h                              |  16 +--\n t/perf/p3010-ls-files.sh           |  15 ++\n t/t3001-ls-files-others-exclude.sh |  20 +++\n t/t7063-status-untracked-cache.sh  | 272 +++++++++++++++++++++++++++++--------\n t/t7519-status-fsmonitor.sh        |  44 ++++++\n 9 files changed, 481 insertions(+), 194 deletions(-)\n\nRange-diff versus v1:\n\n1:  896f1aea52 ! 1:  bf5e5eea24 dir: hash ignore files before adding parser LF\n    @@ Metadata\n     Author: Tamir Duberstein <tamird@gmail.com>\n     \n      ## Commit message ##\n    -    dir: hash ignore files before adding parser LF\n    +    dir: hash ignore files before appending newline\n     \n         add_patterns() appends a newline for the pattern parser before computing\n    -    an ignore file's object ID. Its fallback hash therefore includes a byte\n    -    that is absent from the file. The fast path instead copies the original\n    -    blob ID from an up-to-date index entry.\n    +    an ignore file's object ID. Hashing the buffer therefore includes a byte\n    +    that is absent from the file. When the file has an up-to-date index entry\n    +    and needs no content conversion, the function instead uses that entry's\n    +    object ID.\n     \n    -    Switching between those paths changes the recorded ignore identity even\n    -    when the file has not changed, invalidating the untracked cache below it.\n    -    Compute the hash before appending the parser newline so both paths agree.\n    -    Update the expected identities of the untracked ignore files accordingly.\n    +    Switching between these paths changes the cached object ID even when the\n    +    file has not changed, invalidating the untracked cache below it. Compute\n    +    the hash before appending the newline so both paths agree. Update the\n    +    expected object IDs of the untracked ignore files accordingly.\n     \n    +    Assisted-by: LLM\n         Signed-off-by: Tamir Duberstein <tamird@gmail.com>\n     \n      ## dir.c ##\n    @@ dir.c: static int add_patterns(const char *fname, const char *base, int baselen,\n      \t\t\tfill_stat_data(&oid_stat->stat, &st);\n      \t\t\toid_stat->valid = 1;\n      \t\t}\n    ++\t\t/*\n    ++\t\t * The extra newline is only for parsing. Like do_read_blob(),\n    ++\t\t * keep it out of the file's object ID.\n    ++\t\t */\n     +\t\tbuf[size++] = '\\n';\n      \t}\n      \n2:  fc65e91309 < -:  ---------- ls-files: reuse cached untracked listings\n-:  ---------- > 2:  ae86ca623b dir: share untracked caches across output modes\n-:  ---------- > 3:  126241f862 ls-files: use and update the untracked cache\n\n---\nbase-commit: 3bc0341126508f78f5869cbfc0005e987efdf0c7\nchange-id: 20260923-ls-files-untracked-cache-3559bed01a3e\n\n"},{"id":"553130","messageId":"20260923-ls-files-untracked-cache-v2-1-d7ee33476eb8@gmail.com","threadId":"66377","inReplyTo":"20260923-ls-files-untracked-cache-v2-0-d7ee33476eb8@gmail.com","subject":"[PATCH v2 1/3] dir: hash ignore files before appending newline","fromName":"Tamir Duberstein","fromEmail":"tamird@gmail.com","sentAt":"2026-09-23T22:26:58Z","receivedAt":"2026-09-23T22:27:18Z","isPatch":true,"body":"add_patterns() appends a newline for the pattern parser before computing\nan ignore file's object ID. Hashing the buffer therefore includes a byte\nthat is absent from the file. When the file has an up-to-date index entry\nand needs no content conversion, the function instead uses that entry's\nobject ID.\n\nSwitching between these paths changes the cached object ID even when the\nfile has not changed, invalidating the untracked cache below it. Compute\nthe hash before appending the newline so both paths agree. Update the\nexpected object IDs of the untracked ignore files accordingly.\n\nAssisted-by: LLM\nSigned-off-by: Tamir Duberstein <tamird@gmail.com>\n---\n dir.c                             | 6 +++++-\n t/t7063-status-untracked-cache.sh | 8 ++++----\n 2 files changed, 9 insertions(+), 5 deletions(-)\n\ndiff --git a/dir.c b/dir.c\nindex d896e7be4b..5484eb1562 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1195,7 +1195,6 @@ static int add_patterns(const char *fname, const char *base, int baselen,\n \t\t\tclose(fd);\n \t\t\treturn -1;\n \t\t}\n-\t\tbuf[size++] = '\\n';\n \t\tclose(fd);\n \t\tif (oid_stat) {\n \t\t\tint pos;\n@@ -1215,6 +1214,11 @@ static int add_patterns(const char *fname, const char *base, int baselen,\n \t\t\tfill_stat_data(&oid_stat->stat, &st);\n \t\t\toid_stat->valid = 1;\n \t\t}\n+\t\t/*\n+\t\t * The extra newline is only for parsing. Like do_read_blob(),\n+\t\t * keep it out of the file's object ID.\n+\t\t */\n+\t\tbuf[size++] = '\\n';\n \t}\n \n \tif (size > PATTERN_MAX_FILE_SIZE) {\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 8929ef481f..10d4850077 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -98,11 +98,11 @@ test_expect_success 'setup' '\n \t: >.git/info/exclude &&\n \tgit update-index --untracked-cache &&\n \ttest_oid_cache <<-EOF\n-\troot sha1:e6fcc8f2ee31bae321d66afd183fcb7237afae6e\n-\troot sha256:b90c672088c015b9c83876e919da311bad4cd39639fb139f988af6a11493b974\n+\troot sha1:8510665149157c2bc901848c3e0b746954e9cbd9\n+\troot sha256:09ef24b38105f396a61ad78d73ba6a18ee3cbd89ce4524b4e13b6c1af191e2d8\n \n-\texclude sha1:13263c0978fb9fad16b2d580fb800b6d811c3ff0\n-\texclude sha256:fe4aaa1bbbbce4cb8f73426748a14c5ad6026b26f90505a0bf2494b165a5b76c\n+\texclude sha1:2bdf67abb163a4ffb2d7f3f0880c9fe5068ce782\n+\texclude sha256:b83643f4390b339c1b3ff2f5132c99bd4a77687dd321d3f386c25953aa6f1ce4\n \n \tdone sha1:1946f0437f90c5005533cbe1736a6451ca301714\n \tdone sha256:7f079501d79f665b3acc50f5e0e9e94509084d5032ac20113a37dd5029b757cc\n\n-- \n2.56.0.rc0.807.ga0c0929ce1.frankengit\n\n"},{"id":"553131","messageId":"20260923-ls-files-untracked-cache-v2-2-d7ee33476eb8@gmail.com","threadId":"66377","inReplyTo":"20260923-ls-files-untracked-cache-v2-0-d7ee33476eb8@gmail.com","subject":"[PATCH v2 2/3] dir: share untracked caches across output modes","fromName":"Tamir Duberstein","fromEmail":"tamird@gmail.com","sentAt":"2026-09-23T22:26:59Z","receivedAt":"2026-09-23T22:27:20Z","isPatch":true,"body":"The untracked cache stores different directory listings for\n--untracked-files=normal and --untracked-files=all. Switching modes\nbypasses the cache or rebuilds it, even when complete listings could be\nused for either mode.\n\nUse each directory's check_only bit to record an incomplete scan. Rescan\nit when all untracked files are requested, and retain complete listings\nwhen a later 'git status -unormal' stops early. If a partial listing loses\nits cached untracked entry, rescan the directory to find any remaining\nuntracked files.\n\nCall treat_directory() for cached directories as well. For wildcard\npathspecs without a fixed prefix, cache complete listings and apply the\npathspec afterward; retain traversal pruning for fixed-prefix, attribute\nand exclude pathspecs. This follows the cache reuse proposed in [1] while\npreserving the early exit for large untracked trees [2].\n\nUse a new value for dir_flags so older versions of Git rebuild the cache\nbefore using it to list untracked files. Retain DIR_SHOW_OTHER_DIRECTORIES\nso commands such as 'git add' in those versions still invalidate parent\ndirectories when updating the index. Reuse caches written for either\n'normal' or 'all' mode.\n\n[1]: https://lore.kernel.org/git/xmqqtucmag00.fsf@gitster.g/\n[2]: https://lore.kernel.org/git/CAPMMpoixi3x1PHrSHJPV1GRBzMpuOQ4meMr-fipXuDvz-96MEA@mail.gmail.com/\n\nAssisted-by: LLM\nSigned-off-by: Tamir Duberstein <tamird@gmail.com>\n---\n Documentation/gitformat-index.adoc |  13 ++-\n dir.c                              | 217 ++++++++++++++++++++-----------------\n dir.h                              |  16 +--\n t/t7063-status-untracked-cache.sh  | 167 ++++++++++++++++++----------\n 4 files changed, 245 insertions(+), 168 deletions(-)\n\ndiff --git a/Documentation/gitformat-index.adoc b/Documentation/gitformat-index.adoc\nindex f6a427cb49..d7596eb430 100644\n--- a/Documentation/gitformat-index.adoc\n+++ b/Documentation/gitformat-index.adoc\n@@ -296,7 +296,13 @@ Git index format\n \n   - Stat data of core.excludesFile\n \n-  - 32-bit dir_flags (see struct dir_struct)\n+  - 32-bit dir_flags. The values 0 and 6 store the flags from\n+    `struct dir_struct` for `--untracked-files=all` and\n+    `--untracked-files=normal`, respectively. The value 0x80000006 allows\n+    the same cache to be used for both modes. Older versions of Git\n+    rebuild this cache before using it to list untracked files. Its\n+    `DIR_SHOW_OTHER_DIRECTORIES` bit ensures that they still invalidate\n+    parent directories when updating the index.\n \n   - Hash of $GIT_COMMON_DIR/info/exclude. A null hash means the file\n     does not exist.\n@@ -327,8 +333,9 @@ The remaining data of each directory block is grouped by type:\n   - An ewah bitmap, the n-th bit marks whether the n-th directory has\n     valid untracked cache entries.\n \n-  - An ewah bitmap, the n-th bit records \"check-only\" bit of\n-    read_directory_recursive() for the n-th directory.\n+  - An ewah bitmap, the n-th bit records the \"check-only\" bit for the\n+    n-th directory. With dir_flags 0x80000006, a set bit marks a directory\n+    listing that may be incomplete.\n \n   - An ewah bitmap, the n-th bit indicates whether hash and stat data\n     is valid for the n-th directory and exists in the next data.\ndiff --git a/dir.c b/dir.c\nindex 5484eb1562..425fe2af67 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -43,6 +43,15 @@\n   */\n #define PATTERN_MAX_FILE_SIZE (100 * 1024 * 1024)\n \n+/*\n+ * Use the same cache for --untracked-files=normal and --untracked-files=all.\n+ * The high bit makes older versions of Git rebuild the cache before using\n+ * it to list untracked files. Keep DIR_SHOW_OTHER_DIRECTORIES so their\n+ * index updates still invalidate parent directories.\n+ */\n+#define UNTRACKED_CACHE_LAZY ((1U << 31) | DIR_SHOW_OTHER_DIRECTORIES | \\\n+\t\t\t      DIR_HIDE_EMPTY_DIRECTORIES)\n+\n /*\n  * Tells read_directory_recursive how a file or directory should be treated.\n  * Values are ordered by significance, e.g. if a directory contains both\n@@ -1057,7 +1066,7 @@ static void trim_trailing_spaces(char *buf)\n /*\n  * Given a subdirectory name and \"dir\" of the current directory,\n  * search the subdir in \"dir\" and return it, or create a new one if it\n- * does not exist in \"dir\".\n+ * does not exist in \"dir\". If \"uc\" is NULL, do not create a new entry.\n  *\n  * If \"name\" has the trailing slash, it'll be excluded in the search.\n  */\n@@ -1088,6 +1097,8 @@ static struct untracked_cache_dir *lookup_untracked(struct untracked_cache *uc,\n \t\tfirst = next+1;\n \t}\n \n+\tif (!uc)\n+\t\treturn NULL;\n \tuc->dir_created++;\n \tFLEX_ALLOC_MEM(d, name, name, len);\n \n@@ -2410,26 +2421,19 @@ static enum path_treatment treat_path_fast(struct dir_struct *dir,\n \tstrbuf_setlen(path, baselen);\n \tif (!cdir->ucd) {\n \t\tstrbuf_addstr(path, cdir->file);\n-\t\treturn path_untracked;\n+\t\tif (!ends_with(cdir->file, \"/\"))\n+\t\t\treturn path_untracked;\n+\t} else {\n+\t\tstrbuf_addstr(path, cdir->ucd->name);\n+\t\t/* treat_directory() expects a trailing slash. */\n+\t\tstrbuf_complete(path, '/');\n \t}\n-\tstrbuf_addstr(path, cdir->ucd->name);\n-\t/* treat_one_path() does this before it calls treat_directory() */\n-\tstrbuf_complete(path, '/');\n-\tif (cdir->ucd->check_only)\n-\t\t/*\n-\t\t * check_only is set as a result of treat_directory() getting\n-\t\t * to its bottom. Verify again the same set of directories\n-\t\t * with check_only set.\n-\t\t */\n-\t\treturn read_directory_recursive(dir, istate, path->buf, path->len,\n-\t\t\t\t\t\tcdir->ucd, 1, 0, pathspec);\n \t/*\n-\t * We get path_recurse in the first run when\n-\t * directory_exists_in_index() returns index_nonexistent. We\n-\t * are sure that new changes in the index does not impact the\n-\t * outcome. Return now.\n+\t * The output mode may have changed since this directory was cached,\n+\t * and a nested repository may have been created or removed.\n \t */\n-\treturn path_recurse;\n+\treturn treat_directory(dir, istate, cdir->untracked, path->buf,\n+\t\t\t       path->len, baselen, 0, pathspec);\n }\n \n static enum path_treatment treat_path(struct dir_struct *dir,\n@@ -2556,7 +2560,8 @@ static int valid_cached_dir(struct dir_struct *dir,\n \t\t}\n \t}\n \n-\tif (untracked->check_only != !!check_only)\n+\t/* A complete listing can also answer a check_only request. */\n+\tif (untracked->check_only && !check_only)\n \t\treturn 0;\n \n \t/*\n@@ -2618,7 +2623,12 @@ static int read_cached_dir(struct cached_dir *cdir)\n \t\tcdir->d_type = DTYPE(de);\n \t\treturn 0;\n \t}\n-\twhile (cdir->nr_dirs < cdir->untracked->dirs_nr) {\n+\t/*\n+\t * If a cached entry is no longer a nested repository, recursing into\n+\t * it can add it to dirs while we iterate over untracked. Do not visit\n+\t * it twice.\n+\t */\n+\twhile (!cdir->nr_files && cdir->nr_dirs < cdir->untracked->dirs_nr) {\n \t\tstruct untracked_cache_dir *d = cdir->untracked->dirs[cdir->nr_dirs];\n \t\tif (!d->recurse) {\n \t\t\tcdir->nr_dirs++;\n@@ -2629,9 +2639,17 @@ static int read_cached_dir(struct cached_dir *cdir)\n \t\treturn 0;\n \t}\n \tcdir->ucd = NULL;\n-\tif (cdir->nr_files < cdir->untracked->untracked_nr) {\n+\twhile (cdir->nr_files < cdir->untracked->untracked_nr) {\n \t\tstruct untracked_cache_dir *d = cdir->untracked;\n \t\tcdir->file = d->untracked[cdir->nr_files++];\n+\t\t/* A directory may occur in both dirs and untracked. Return it once. */\n+\t\tif (ends_with(cdir->file, \"/\")) {\n+\t\t\tstruct untracked_cache_dir *child =\n+\t\t\t\tlookup_untracked(NULL, d, cdir->file,\n+\t\t\t\t\t\t strlen(cdir->file));\n+\t\t\tif (child && child->recurse)\n+\t\t\t\tcontinue;\n+\t\t}\n \t\treturn 0;\n \t}\n \treturn -1;\n@@ -2641,10 +2659,7 @@ static void close_cached_dir(struct cached_dir *cdir)\n {\n \tif (cdir->fdir)\n \t\tclosedir(cdir->fdir);\n-\t/*\n-\t * We have gone through this directory and found no untracked\n-\t * entries. Mark it valid.\n-\t */\n+\t/* The listing is valid even if check_only marks it as incomplete. */\n \tif (cdir->untracked) {\n \t\tcdir->untracked->valid = 1;\n \t\tcdir->untracked->recurse = 1;\n@@ -2717,6 +2732,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \tstruct cached_dir cdir;\n \tenum path_treatment state, subdir_state, dir_state = path_none;\n \tstruct strbuf path = STRBUF_INIT;\n+\tint incomplete = 0;\n \n \tstrbuf_add(&path, base, baselen);\n \n@@ -2724,9 +2740,6 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\tgoto out;\n \tdir->internal.visited_directories++;\n \n-\tif (untracked)\n-\t\tuntracked->check_only = !!check_only;\n-\n \twhile (!read_cached_dir(&cdir)) {\n \t\t/* check how the file or directory should be treated */\n \t\tstate = treat_path(dir, untracked, &cdir, istate, &path,\n@@ -2776,6 +2789,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t */\n \t\t\t\tif (dir_state >= path_excluded) {\n \t\t\t\t\tdir_state = path_excluded;\n+\t\t\t\t\tincomplete = 1;\n \t\t\t\t\tbreak;\n \t\t\t\t}\n \t\t\t}\n@@ -2784,6 +2798,7 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\tif (dir_state == path_untracked) {\n \t\t\t\tif (cdir.fdir)\n \t\t\t\t\tadd_untracked(untracked, path.buf + baselen);\n+\t\t\t\tincomplete = 1;\n \t\t\t\tbreak;\n \t\t\t}\n \t\t\t/* skip the add_path_to_appropriate_result_list() */\n@@ -2794,7 +2809,27 @@ static enum path_treatment read_directory_recursive(struct dir_struct *dir,\n \t\t\t\t\t\t    istate, &path, baselen,\n \t\t\t\t\t\t    pathspec, state);\n \t}\n-\tclose_cached_dir(&cdir);\n+\t/*\n+\t * Only a filesystem scan replaces the saved completeness. Stopping\n+\t * early while reading a complete cache must not make it partial.\n+\t */\n+\tif (cdir.fdir && untracked)\n+\t\tuntracked->check_only = incomplete;\n+\tif (!cdir.fdir && untracked->check_only &&\n+\t    dir_state != path_untracked) {\n+\t\t/*\n+\t\t * Removing the last untracked file in a cached child need not\n+\t\t * change this directory's mtime. Other children may still have\n+\t\t * untracked files, so rescan the directory before returning.\n+\t\t */\n+\t\tclose_cached_dir(&cdir);\n+\t\tinvalidate_directory(dir->untracked, untracked);\n+\t\tdir_state = read_directory_recursive(dir, istate, base, baselen,\n+\t\t\t\t\t\t     untracked, check_only,\n+\t\t\t\t\t\t     stop_at_first_file, pathspec);\n+\t} else {\n+\t\tclose_cached_dir(&cdir);\n+\t}\n  out:\n \tstrbuf_release(&path);\n \n@@ -2925,33 +2960,12 @@ static void set_untracked_ident(struct untracked_cache *uc)\n \tstrbuf_addch(&uc->ident, 0);\n }\n \n-static unsigned new_untracked_cache_flags(struct index_state *istate)\n-{\n-\tstruct repository *repo = istate->repo;\n-\tconst char *val;\n-\n-\t/*\n-\t * This logic is coordinated with the setting of these flags in\n-\t * wt-status.c#wt_status_collect_untracked(), and the evaluation\n-\t * of the config setting in commit.c#git_status_config()\n-\t */\n-\tif (!repo_config_get_string_tmp(repo, \"status.showuntrackedfiles\", &val) &&\n-\t    !strcmp(val, \"all\"))\n-\t\treturn 0;\n-\n-\t/*\n-\t * The default, if \"all\" is not set, is \"normal\" - leading us here.\n-\t * If the value is \"none\" then it really doesn't matter.\n-\t */\n-\treturn DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES;\n-}\n-\n-static void new_untracked_cache(struct index_state *istate, int flags)\n+static void new_untracked_cache(struct index_state *istate)\n {\n \tstruct untracked_cache *uc = xcalloc(1, sizeof(*uc));\n \tstrbuf_init(&uc->ident, 100);\n \tuc->exclude_per_dir = \".gitignore\";\n-\tuc->dir_flags = flags >= 0 ? flags : new_untracked_cache_flags(istate);\n+\tuc->dir_flags = UNTRACKED_CACHE_LAZY;\n \tset_untracked_ident(uc);\n \tistate->untracked = uc;\n \tistate->cache_changed |= UNTRACKED_CHANGED;\n@@ -2960,11 +2974,11 @@ static void new_untracked_cache(struct index_state *istate, int flags)\n void add_untracked_cache(struct index_state *istate)\n {\n \tif (!istate->untracked) {\n-\t\tnew_untracked_cache(istate, -1);\n+\t\tnew_untracked_cache(istate);\n \t} else {\n \t\tif (!ident_in_untracked(istate->untracked)) {\n \t\t\tfree_untracked_cache(istate->untracked);\n-\t\t\tnew_untracked_cache(istate, -1);\n+\t\t\tnew_untracked_cache(istate);\n \t\t}\n \t}\n }\n@@ -3004,10 +3018,9 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \t\treturn NULL;\n \n \t/*\n-\t * Optimize for the main use case only: whole-tree git\n-\t * status. More work involved in treat_leading_path() if we\n-\t * use cache on just a subset of the worktree. pathspec\n-\t * support could make the matter even worse.\n+\t * The cache needs a whole-tree scan without pathspec pruning.\n+\t * read_directory() handles eligible pathspecs by filtering the results\n+\t * after the scan and passing NULL here.\n \t */\n \tif (base_len || (pathspec && pathspec->nr))\n \t\treturn NULL;\n@@ -3037,47 +3050,22 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \t\treturn NULL;\n \t}\n \n-\t/*\n-\t * If the untracked structure we received does not have the same flags\n-\t * as requested in this run, we're going to need to either discard the\n-\t * existing structure (and potentially later recreate), or bypass the\n-\t * untracked cache mechanism for this run.\n-\t */\n-\tif (dir->flags != dir->untracked->dir_flags) {\n-\t\t/*\n-\t\t * If the untracked structure we received does not have the same flags\n-\t\t * as configured, then we need to reset / create a new \"untracked\"\n-\t\t * structure to match the new config.\n-\t\t *\n-\t\t * Keeping the saved and used untracked cache consistent with the\n-\t\t * configuration provides an opportunity for frequent users of\n-\t\t * \"git status -uall\" to leverage the untracked cache by aligning their\n-\t\t * configuration - setting \"status.showuntrackedfiles\" to \"all\" or\n-\t\t * \"normal\" as appropriate.\n-\t\t *\n-\t\t * Previously using -uall (or setting \"status.showuntrackedfiles\" to\n-\t\t * \"all\") was incompatible with untracked cache and *consistently*\n-\t\t * caused surprisingly bad performance (with fscache and fsmonitor\n-\t\t * enabled) on Windows.\n-\t\t *\n-\t\t * IMPROVEMENT OPPORTUNITY: If we reworked the untracked cache storage\n-\t\t * to not be as bound up with the desired output in a given run,\n-\t\t * and instead iterated through and stored enough information to\n-\t\t * correctly serve both \"modes\", then users could get peak performance\n-\t\t * with or without '-uall' regardless of their\n-\t\t * \"status.showuntrackedfiles\" config.\n-\t\t */\n-\t\tif (dir->untracked->dir_flags != new_untracked_cache_flags(istate)) {\n+\t/* Only --untracked-files=normal and --untracked-files=all are supported. */\n+\tif (dir->flags &&\n+\t    dir->flags != (DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES))\n+\t\treturn NULL;\n+\n+\tif (dir->untracked->dir_flags != UNTRACKED_CACHE_LAZY) {\n+\t\t/* Reuse caches written for either mode by older versions of Git. */\n+\t\tif (dir->untracked->dir_flags &&\n+\t\t    dir->untracked->dir_flags !=\n+\t\t\t    (DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES)) {\n \t\t\tfree_untracked_cache(istate->untracked);\n-\t\t\tnew_untracked_cache(istate, dir->flags);\n+\t\t\tnew_untracked_cache(istate);\n \t\t\tdir->untracked = istate->untracked;\n-\t\t}\n-\t\telse {\n-\t\t\t/*\n-\t\t\t * Current untracked cache data is consistent with config, but not\n-\t\t\t * usable in this request/run; just bypass untracked cache.\n-\t\t\t */\n-\t\t\treturn NULL;\n+\t\t} else {\n+\t\t\tdir->untracked->dir_flags = UNTRACKED_CACHE_LAZY;\n+\t\t\tistate->cache_changed |= UNTRACKED_CHANGED;\n \t\t}\n \t}\n \n@@ -3145,6 +3133,16 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\t   const char *path, int len, const struct pathspec *pathspec)\n {\n \tstruct untracked_cache_dir *untracked;\n+\tconst struct pathspec *walk_pathspec = pathspec;\n+\t/* Attribute and exclude pathspecs can prune a directory by its own name. */\n+\tint filter = dir->untracked &&\n+\t\t     !len && !dir->flags && pathspec && pathspec->nr &&\n+\t\t     !(pathspec->magic & (PATHSPEC_ATTR | PATHSPEC_EXCLUDE));\n+\n+\t/* Keep the usual pruning for pathspecs with a fixed prefix. */\n+\tfor (int i = 0; filter && i < pathspec->nr; i++)\n+\t\tif (pathspec->items[i].nowildcard_len)\n+\t\t\tfilter = 0;\n \n \ttrace2_region_enter(\"dir\", \"read_directory\", istate->repo);\n \tdir->internal.visited_paths = 0;\n@@ -3155,15 +3153,34 @@ int read_directory(struct dir_struct *dir, struct index_state *istate,\n \t\treturn dir->nr;\n \t}\n \n-\tuntracked = validate_untracked_cache(dir, len, pathspec, istate);\n+\tuntracked = validate_untracked_cache(dir, len,\n+\t\t\t\t\t     filter ? NULL : pathspec, istate);\n \tif (!untracked)\n \t\t/*\n \t\t * make sure untracked cache code path is disabled,\n \t\t * e.g. prep_exclude()\n \t\t */\n \t\tdir->untracked = NULL;\n-\tif (!len || treat_leading_path(dir, istate, path, len, pathspec))\n-\t\tread_directory_recursive(dir, istate, path, len, untracked, 0, 0, pathspec);\n+\telse if (filter)\n+\t\twalk_pathspec = NULL;\n+\tif (!len || treat_leading_path(dir, istate, path, len, walk_pathspec))\n+\t\tread_directory_recursive(dir, istate, path, len, untracked, 0, 0,\n+\t\t\t\t\t walk_pathspec);\n+\tif (filter && untracked) {\n+\t\tint dst = 0;\n+\n+\t\t/* Keep complete listings in the cache for later pathspecs. */\n+\t\tfor (int i = 0; i < dir->nr; i++) {\n+\t\t\tstruct dir_entry *ent = dir->entries[i];\n+\n+\t\t\tif (match_pathspec(istate, pathspec, ent->name, ent->len,\n+\t\t\t\t\t   0, NULL, 0))\n+\t\t\t\tdir->entries[dst++] = dir->entries[i];\n+\t\t\telse\n+\t\t\t\tfree(dir->entries[i]);\n+\t\t}\n+\t\tdir->nr = dst;\n+\t}\n \tQSORT(dir->entries, dir->nr, cmp_dir_entry);\n \tQSORT(dir->ignored, dir->ignored_nr, cmp_dir_entry);\n \ndiff --git a/dir.h b/dir.h\nindex 83e0f648a8..5cdad2bc45 100644\n--- a/dir.h\n+++ b/dir.h\n@@ -152,13 +152,10 @@ struct oid_stat {\n  *\n  *   - The list of files and directories of the directory in question\n  *   - The $GIT_DIR/index\n- *   - dir_struct flags\n  *   - The content of $GIT_COMMON_DIR/info/exclude\n  *   - The content of core.excludesfile\n  *   - The content (or the lack) of .gitignore of all parent directories\n  *     from $GIT_WORK_TREE\n- *   - The check_only flag in read_directory_recursive (for\n- *     DIR_HIDE_EMPTY_DIRECTORIES)\n  *\n  *  The first input can be checked using directory mtime. In many\n  *  filesystems, directory mtime (stat_data field) is updated when its\n@@ -168,9 +165,10 @@ struct oid_stat {\n  *  Whenever a file (or a submodule) is added or removed from a\n  *  directory, we invalidate that directory.\n  *\n- *  The remaining inputs are easy, their SHA-1 could be used to verify\n- *  their contents (exclude_sha1[], info_exclude_sha1[] and\n- *  excludes_file_sha1[])\n+ *  Normal and all mode share these listings; other directory flags bypass\n+ *  the cache. Partial scans are marked check_only and completed when needed.\n+ *\n+ *  Ignore files are validated by their object IDs.\n  */\n struct untracked_cache_dir {\n \tstruct untracked_cache_dir **dirs;\n@@ -178,6 +176,7 @@ struct untracked_cache_dir {\n \tstruct stat_data stat_data;\n \tunsigned int untracked_alloc, dirs_nr, dirs_alloc;\n \tunsigned int untracked_nr;\n+\t/* The directory scan stopped early; the listing may be incomplete. */\n \tunsigned int check_only : 1;\n \t/* all data except 'dirs' in this struct are good */\n \tunsigned int valid : 1;\n@@ -193,10 +192,7 @@ struct untracked_cache {\n \tconst char *exclude_per_dir;\n \tchar *exclude_per_dir_to_free;\n \tstruct strbuf ident;\n-\t/*\n-\t * dir_struct#flags must match dir_flags or the untracked\n-\t * cache is ignored.\n-\t */\n+\t/* UNTRACKED_CACHE_LAZY, or dir_struct.flags from older versions of Git. */\n \tunsigned dir_flags;\n \tstruct untracked_cache_dir *root;\n \t/* Statistics */\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 10d4850077..2efbb93b71 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -115,7 +115,7 @@ test_expect_success 'untracked cache is empty' '\n info/exclude $ZERO_OID\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n EOF\n \ttest_cmp ../expect-empty ../actual\n '\n@@ -133,7 +133,7 @@ cat >../dump.expect <<EOF &&\n info/exclude $EMPTY_BLOB\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $ZERO_OID recurse valid\n dthree/\n dtwo/\n@@ -200,11 +200,7 @@ A  two\n ?? three\n EOF\n \n-# Bypassing the untracked cache here is not desirable from an\n-# end-user perspective, but is expected in the current design.\n-# The untracked cache data stored for a -unormal run cannot be\n-# correctly used in a -uall run - it would yield incorrect output.\n-test_expect_success 'untracked cache is bypassed with -uall' '\n+test_expect_success 'untracked cache completes partial directory listings with -uall' '\n \t: >../trace.output &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n \tgit status -uall --porcelain >../actual &&\n@@ -214,16 +210,21 @@ test_expect_success 'untracked cache is bypassed with -uall' '\n \tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <<EOF &&\n  ....path:\n+ ....node-creation:0\n+ ....gitignore-invalidation:0\n+ ....directory-invalidation:2\n+ ....opendir:2\n EOF\n \ttest_cmp ../trace.expect ../trace.relevant\n '\n \n-test_expect_success 'untracked cache remains after bypass' '\n+test_expect_success 'untracked cache retains completed listings' '\n+\tsed \"s/check_only //\" ../dump.expect >../dump_uall.expect &&\n \ttest-tool dump-untracked-cache >../actual &&\n-\ttest_cmp ../dump.expect ../actual\n+\ttest_cmp ../dump_uall.expect ../actual\n '\n \n-test_expect_success 'if -uall is configured, untracked cache gets populated by default' '\n+test_expect_success 'if -uall is configured, untracked cache is reused by default' '\n \ttest_config status.showuntrackedfiles all &&\n \t: >../trace.output &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n@@ -234,28 +235,14 @@ test_expect_success 'if -uall is configured, untracked cache gets populated by d\n \tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <<EOF &&\n  ....path:\n- ....node-creation:3\n- ....gitignore-invalidation:1\n+ ....node-creation:0\n+ ....gitignore-invalidation:0\n  ....directory-invalidation:0\n- ....opendir:4\n+ ....opendir:0\n EOF\n \ttest_cmp ../trace.expect ../trace.relevant\n '\n \n-cat >../dump_uall.expect <<EOF &&\n-info/exclude $EMPTY_BLOB\n-core.excludesfile $ZERO_OID\n-exclude_per_dir .gitignore\n-flags 00000000\n-/ $ZERO_OID recurse valid\n-three\n-/done/ $ZERO_OID recurse valid\n-/dthree/ $ZERO_OID recurse valid\n-three\n-/dtwo/ $ZERO_OID recurse valid\n-two\n-EOF\n-\n test_expect_success 'if -uall was configured, untracked cache is populated' '\n \ttest-tool dump-untracked-cache >../actual &&\n \ttest_cmp ../dump_uall.expect ../actual\n@@ -280,12 +267,7 @@ EOF\n \ttest_cmp ../trace.expect ../trace.relevant\n '\n \n-# Bypassing the untracked cache here is not desirable from an\n-# end-user perspective, but is expected in the current design.\n-# The untracked cache data stored for a -all run cannot be\n-# correctly used in a -unormal run - it would yield incorrect\n-# output.\n-test_expect_success 'if -uall is configured, untracked cache is bypassed with -unormal' '\n+test_expect_success 'if -uall is configured, untracked cache is reused with -unormal' '\n \ttest_config status.showuntrackedfiles all &&\n \t: >../trace.output &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.output\" \\\n@@ -296,12 +278,18 @@ test_expect_success 'if -uall is configured, untracked cache is bypassed with -u\n \tget_relevant_traces ../trace.output ../trace.relevant &&\n \tcat >../trace.expect <<EOF &&\n  ....path:\n+ ....node-creation:0\n+ ....gitignore-invalidation:0\n+ ....directory-invalidation:0\n+ ....opendir:0\n EOF\n \ttest_cmp ../trace.expect ../trace.relevant\n '\n \n-test_expect_success 'repopulate untracked cache for -unormal' '\n-\tgit status --porcelain\n+test_expect_success 'normal status preserves completed untracked cache' '\n+\tgit status --porcelain &&\n+\ttest-tool dump-untracked-cache >../actual &&\n+\ttest_cmp ../dump_uall.expect ../actual\n '\n \n test_expect_success 'modify in root directory, one dir invalidation' '\n@@ -340,16 +328,16 @@ test_expect_success 'verify untracked cache dump' '\n info/exclude $EMPTY_BLOB\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $ZERO_OID recurse valid\n dthree/\n dtwo/\n four\n three\n /done/ $ZERO_OID recurse valid\n-/dthree/ $ZERO_OID recurse check_only valid\n+/dthree/ $ZERO_OID recurse valid\n three\n-/dtwo/ $ZERO_OID recurse check_only valid\n+/dtwo/ $ZERO_OID recurse valid\n two\n EOF\n \ttest_cmp ../expect ../actual\n@@ -390,7 +378,7 @@ test_expect_success 'verify untracked cache dump' '\n info/exclude $EMPTY_BLOB\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $(test_oid root) recurse valid\n .gitignore\n dthree/\n@@ -437,12 +425,12 @@ test_expect_success 'verify untracked cache dump' '\n info/exclude $(test_oid exclude)\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $(test_oid root) recurse valid\n .gitignore\n dtwo/\n /done/ $ZERO_OID recurse valid\n-/dthree/ $ZERO_OID recurse check_only valid\n+/dthree/ $ZERO_OID recurse valid\n /dtwo/ $ZERO_OID recurse check_only valid\n two\n EOF\n@@ -456,10 +444,10 @@ test_expect_success 'move two from tracked to untracked' '\n info/exclude $(test_oid exclude)\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $(test_oid root) recurse\n /done/ $ZERO_OID recurse valid\n-/dthree/ $ZERO_OID recurse check_only valid\n+/dthree/ $ZERO_OID recurse valid\n /dtwo/ $ZERO_OID recurse check_only valid\n two\n EOF\n@@ -497,13 +485,13 @@ test_expect_success 'verify untracked cache dump' '\n info/exclude $(test_oid exclude)\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $(test_oid root) recurse valid\n .gitignore\n dtwo/\n two\n /done/ $ZERO_OID recurse valid\n-/dthree/ $ZERO_OID recurse check_only valid\n+/dthree/ $ZERO_OID recurse valid\n /dtwo/ $ZERO_OID recurse check_only valid\n two\n EOF\n@@ -517,10 +505,10 @@ test_expect_success 'move two from untracked to tracked' '\n info/exclude $(test_oid exclude)\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $(test_oid root) recurse\n /done/ $ZERO_OID recurse valid\n-/dthree/ $ZERO_OID recurse check_only valid\n+/dthree/ $ZERO_OID recurse valid\n /dtwo/ $ZERO_OID recurse check_only valid\n two\n EOF\n@@ -558,12 +546,12 @@ test_expect_success 'verify untracked cache dump' '\n info/exclude $(test_oid exclude)\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $(test_oid root) recurse valid\n .gitignore\n dtwo/\n /done/ $ZERO_OID recurse valid\n-/dthree/ $ZERO_OID recurse check_only valid\n+/dthree/ $ZERO_OID recurse valid\n /dtwo/ $ZERO_OID recurse check_only valid\n two\n EOF\n@@ -606,12 +594,12 @@ test_expect_success 'untracked cache correct after commit' '\n info/exclude $(test_oid exclude)\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $(test_oid root) recurse valid\n .gitignore\n dtwo/\n /done/ $ZERO_OID recurse valid\n-/dthree/ $ZERO_OID recurse check_only valid\n+/dthree/ $ZERO_OID recurse valid\n /dtwo/ $ZERO_OID recurse check_only valid\n two\n EOF\n@@ -669,13 +657,13 @@ test_expect_success 'untracked cache correct after status' '\n info/exclude $(test_oid exclude)\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $(test_oid root) recurse valid\n .gitignore\n dtwo/\n /done/ $(test_oid done) recurse valid\n five\n-/dthree/ $ZERO_OID recurse check_only valid\n+/dthree/ $ZERO_OID recurse valid\n /dtwo/ $ZERO_OID recurse check_only valid\n two\n EOF\n@@ -744,7 +732,7 @@ test_expect_success 'verify untracked cache dump (sparse/subdirs)' '\n info/exclude $(test_oid exclude)\n core.excludesfile $ZERO_OID\n exclude_per_dir .gitignore\n-flags 00000006\n+flags 80000006\n / $(test_oid root) recurse valid\n .gitignore\n dtwo/\n@@ -755,7 +743,7 @@ sub/\n sub/\n /done/sub/sub/ $ZERO_OID recurse check_only valid\n file\n-/dthree/ $ZERO_OID recurse check_only valid\n+/dthree/ $ZERO_OID recurse valid\n /dtwo/ $ZERO_OID recurse check_only valid\n two\n EOF\n@@ -991,4 +979,73 @@ test_expect_success 'empty repo (no index) and core.untrackedCache' '\n \tgit -C emptyrepo -c core.untrackedCache=true write-tree\n '\n \n+test_expect_success 'rescan a partial listing after removing the cached untracked file' '\n+\ttest_create_repo partial-cache &&\n+\t(\n+\t\tcd partial-cache &&\n+\t\ttouch tracked &&\n+\t\tgit add tracked &&\n+\t\tgit commit -m initial &&\n+\t\tgit config core.untrackedCache true &&\n+\t\tmkdir -p d/a d/z &&\n+\t\ttouch d/a/file d/z/file &&\n+\t\ttest-tool chmtime =-300 . d d/a d/z &&\n+\t\tgit status --porcelain >../actual &&\n+\t\ttest-tool dump-untracked-cache >../dump &&\n+\t\tcached_dir=$(sed -n \"s|^/d/\\([^/]*\\)/ .*check_only valid$|\\1|p\" ../dump) &&\n+\t\ttest -n \"$cached_dir\" &&\n+\t\trm \"d/$cached_dir/file\" &&\n+\t\tgit status --porcelain >../actual &&\n+\t\techo \"?? d/\" >../expect &&\n+\t\ttest_cmp ../expect ../actual\n+\t)\n+'\n+\n+test_expect_success 'untracked cache handles nested repository changes' '\n+\ttest_create_repo nested-cache &&\n+\t(\n+\t\tcd nested-cache &&\n+\t\ttouch tracked &&\n+\t\tgit add tracked &&\n+\t\tgit commit -m initial &&\n+\t\tgit config core.untrackedCache true &&\n+\t\tgit init nested &&\n+\t\ttouch nested/file &&\n+\t\ttest-tool chmtime =-300 . nested &&\n+\t\tgit status -uall --porcelain >../actual &&\n+\t\techo \"?? nested/\" >../expect &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\trm -rf nested/.git &&\n+\t\tgit status -uall --porcelain >../actual &&\n+\t\techo \"?? nested/file\" >../expect &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\tgit init nested &&\n+\t\tgit status -uall --porcelain >../actual &&\n+\t\techo \"?? nested/\" >../expect &&\n+\t\ttest_cmp ../expect ../actual\n+\t)\n+'\n+\n+test_expect_success 'filtered status retains complete untracked listings' '\n+\ttest_create_repo filtered-cache &&\n+\t(\n+\t\tcd filtered-cache &&\n+\t\ttouch tracked &&\n+\t\tgit add tracked &&\n+\t\tgit commit -m initial &&\n+\t\tgit config core.untrackedCache true &&\n+\t\tmkdir d &&\n+\t\ttouch d/match.toml d/other &&\n+\t\ttest-tool chmtime =-300 . d &&\n+\t\tgit status -uall --porcelain -- \"**/*.toml\" >../actual &&\n+\t\techo \"?? d/match.toml\" >../expect &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/filtered.trace\" \\\n+\t\t\tgit status -uall --porcelain >../actual &&\n+\t\tprintf \"%s\\n\" \"?? d/match.toml\" \"?? d/other\" >../expect &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\ttest_grep \"read_directo.*opendir:0\\$\" \"$TRASH_DIRECTORY/filtered.trace\"\n+\t)\n+'\n+\n test_done\n\n-- \n2.56.0.rc0.807.ga0c0929ce1.frankengit\n\n"},{"id":"553132","messageId":"20260923-ls-files-untracked-cache-v2-3-d7ee33476eb8@gmail.com","threadId":"66377","inReplyTo":"20260923-ls-files-untracked-cache-v2-0-d7ee33476eb8@gmail.com","subject":"[PATCH v2 3/3] ls-files: use and update the untracked cache","fromName":"Tamir Duberstein","fromEmail":"tamird@gmail.com","sentAt":"2026-09-23T22:27:00Z","receivedAt":"2026-09-23T22:27:23Z","isPatch":true,"body":"Repeated 'git ls-files --cached --others --exclude-standard' calls scan\nthe working tree even when 'git status' has populated an untracked cache.\nReuse those listings and write pending untracked cache and fsmonitor\nupdates back to the index when optional locking is enabled, so later\ncommands can reuse the work.\n\nParse options before reading the index so eligible calls can take the\noptional lock first. With index.skipHash, the index checksum cannot be\nused to detect concurrent index changes. Record the object IDs of the\nstandard exclude files before assigning dir->untracked, so the exclude\noptions can be processed in the same order before reading the index.\n\nA pathspec prefix can prune the in-memory index, and --with-tree can add\nentries to it. Do not write the index in those cases. Respect\n--no-optional-locks and continue without writing when the index lock is\nbusy.\n\nAssisted-by: LLM\nSigned-off-by: Tamir Duberstein <tamird@gmail.com>\n---\n Documentation/git-ls-files.adoc    |  4 ++\n builtin/ls-files.c                 | 40 +++++++++++++---\n dir.c                              | 28 +++++------\n t/perf/p3010-ls-files.sh           | 15 ++++++\n t/t3001-ls-files-others-exclude.sh | 20 ++++++++\n t/t7063-status-untracked-cache.sh  | 97 ++++++++++++++++++++++++++++++++++++++\n t/t7519-status-fsmonitor.sh        | 44 +++++++++++++++++\n 7 files changed, 227 insertions(+), 21 deletions(-)\n\ndiff --git a/Documentation/git-ls-files.adoc b/Documentation/git-ls-files.adoc\nindex 2b175388e1..11323b4ba9 100644\n--- a/Documentation/git-ls-files.adoc\n+++ b/Documentation/git-ls-files.adoc\n@@ -127,6 +127,10 @@ OPTIONS\n --exclude-standard::\n \tAdd the standard Git exclusions: .git/info/exclude, .gitignore\n \tin each directory, and the user's global exclusion file.\n++\n+With `--others`, an enabled untracked cache can be updated in the index\n+after a complete traversal. Use `--no-optional-locks` (see linkgit:git[1])\n+to disable this optional index update.\n \n --error-unmatch::\n \tIf any <file> does not appear in the index, treat this as an\ndiff --git a/builtin/ls-files.c b/builtin/ls-files.c\nindex b044520f9e..e74ffa344d 100644\n--- a/builtin/ls-files.c\n+++ b/builtin/ls-files.c\n@@ -15,6 +15,7 @@\n #include \"quote.h\"\n #include \"dir.h\"\n #include \"gettext.h\"\n+#include \"lockfile.h\"\n #include \"object-name.h\"\n #include \"strbuf.h\"\n #include \"parse-options.h\"\n@@ -597,6 +598,8 @@ int cmd_ls_files(int argc,\n \tstruct dir_struct dir = DIR_INIT;\n \tstruct pattern_list *pl;\n \tstruct string_list exclude_list = STRING_LIST_INIT_NODUP;\n+\tstruct lock_file index_lock = LOCK_INIT;\n+\tint index_fd = -1;\n \tstruct option builtin_ls_files_options[] = {\n \t\t/* Think twice before adding \"--nul\" synonym to this */\n \t\tOPT_SET_INT('z', NULL, &line_terminator,\n@@ -678,14 +681,13 @@ int cmd_ls_files(int argc,\n \t\tprefix_len = strlen(prefix);\n \trepo_config(repo, git_default_config, NULL);\n \n-\tif (repo_read_index(repo) < 0)\n-\t\tdie(\"index file corrupt\");\n-\n \targc = parse_options(argc, argv, prefix, builtin_ls_files_options,\n \t\t\tls_files_usage, 0);\n-\tpl = add_pattern_list(&dir, EXC_CMDL, \"--exclude option\");\n-\tfor (i = 0; i < exclude_list.nr; i++) {\n-\t\tadd_pattern(exclude_list.items[i].string, \"\", 0, pl, --exclude_args);\n+\t/* Even an empty command-line exclude list would disable the cache. */\n+\tif (exclude_list.nr) {\n+\t\tpl = add_pattern_list(&dir, EXC_CMDL, \"--exclude option\");\n+\t\tfor (i = 0; i < exclude_list.nr; i++)\n+\t\t\tadd_pattern(exclude_list.items[i].string, \"\", 0, pl, --exclude_args);\n \t}\n \n \tif (format && (show_stage || show_others || show_killed ||\n@@ -747,6 +749,25 @@ int cmd_ls_files(int argc,\n \t\tmax_prefix = common_prefix(&pathspec);\n \tmax_prefix_len = get_common_prefix_len(max_prefix);\n \n+\t/*\n+\t * Do not save an index pruned by a pathspec or changed by --with-tree.\n+\t * With index.skipHash, the checksum cannot detect concurrent index\n+\t * changes. Lock the index before reading it.\n+\t */\n+\tif (show_others && !dir.flags && dir.exclude_per_dir &&\n+\t    !max_prefix && !with_tree && use_optional_locks() &&\n+\t    !dir.internal.unmanaged_exclude_files && !exclude_list.nr &&\n+\t    repo->settings.core_untracked_cache != UNTRACKED_CACHE_REMOVE)\n+\t\tindex_fd = repo_hold_locked_index(repo, &index_lock, 0);\n+\tif (repo_read_index(repo) < 0)\n+\t\tdie(\"index file corrupt\");\n+\tif (show_others && !dir.flags && dir.exclude_per_dir && !with_tree)\n+\t\tdir.untracked = repo->index->untracked;\n+\tif (index_fd >= 0 && !dir.untracked) {\n+\t\trollback_lock_file(&index_lock);\n+\t\tindex_fd = -1;\n+\t}\n+\n \tprune_index(repo->index, max_prefix, max_prefix_len);\n \n \t/* Treat unmatching pathspec elements as errors */\n@@ -784,6 +805,13 @@ int cmd_ls_files(int argc,\n \t\tret = 1;\n \t}\n \n+\tif (index_fd >= 0) {\n+\t\tif (!ret && dir.untracked)\n+\t\t\trepo_update_index_if_able(repo, &index_lock);\n+\t\telse\n+\t\t\trollback_lock_file(&index_lock);\n+\t}\n+\n \tstring_list_clear(&exclude_list, 0);\n \tdir_clear(&dir);\n \tfree(max_prefix);\ndiff --git a/dir.c b/dir.c\nindex 425fe2af67..949f06c8f7 100644\n--- a/dir.c\n+++ b/dir.c\n@@ -1209,7 +1209,8 @@ static int add_patterns(const char *fname, const char *base, int baselen,\n \t\tclose(fd);\n \t\tif (oid_stat) {\n \t\t\tint pos;\n-\t\t\tif (oid_stat->valid &&\n+\t\t\t/* Racy stat checks need the index timestamp. */\n+\t\t\tif (istate && oid_stat->valid &&\n \t\t\t    !match_stat_data_racy(istate, &oid_stat->stat, &st))\n \t\t\t\t; /* no content change, oid_stat->oid still good */\n \t\t\telse if (istate &&\n@@ -1321,18 +1322,15 @@ struct pattern_list *add_pattern_list(struct dir_struct *dir,\n }\n \n /*\n- * Used to set up core.excludesfile and .git/info/exclude lists.\n+ * Only the standard exclude files have object IDs saved in the untracked\n+ * cache. Other files have no oid_stat and must disable use of the cache.\n  */\n static void add_patterns_from_file_1(struct dir_struct *dir, const char *fname,\n \t\t\t\t     struct oid_stat *oid_stat)\n {\n \tstruct pattern_list *pl;\n-\t/*\n-\t * catch setup_standard_excludes() that's called before\n-\t * dir->untracked is assigned. That function behaves\n-\t * differently when dir->untracked is non-NULL.\n-\t */\n-\tif (!dir->untracked)\n+\n+\tif (!oid_stat)\n \t\tdir->internal.unmanaged_exclude_files++;\n \tpl = add_pattern_list(dir, EXC_FILE, fname);\n \tif (add_patterns(fname, \"\", 0, pl, NULL, 0, oid_stat) < 0)\n@@ -1341,7 +1339,6 @@ static void add_patterns_from_file_1(struct dir_struct *dir, const char *fname,\n \n void add_patterns_from_file(struct dir_struct *dir, const char *fname)\n {\n-\tdir->internal.unmanaged_exclude_files++; /* see validate_untracked_cache() */\n \tadd_patterns_from_file_1(dir, fname, NULL);\n }\n \n@@ -3009,10 +3006,7 @@ static struct untracked_cache_dir *validate_untracked_cache(struct dir_struct *d\n \n \t/*\n \t * We only support $GIT_COMMON_DIR/info/exclude and core.excludesfile\n-\t * as the global ignore rule files. Any other additions\n-\t * (e.g. from command line) invalidate the cache. This\n-\t * condition also catches running setup_standard_excludes()\n-\t * before setting dir->untracked!\n+\t * as the global ignore rule files. Other exclude files bypass the cache.\n \t */\n \tif (dir->internal.unmanaged_exclude_files)\n \t\treturn NULL;\n@@ -3511,17 +3505,21 @@ void setup_standard_excludes(struct dir_struct *dir)\n \n \tdir->exclude_per_dir = \".gitignore\";\n \n+\t/*\n+\t * Option parsing may precede reading the index. Record the object IDs\n+\t * even before the untracked cache is available for validation.\n+\t */\n \t/* core.excludesfile defaulting to $XDG_CONFIG_HOME/git/ignore */\n \tif (excludes_file && !access_or_warn(excludes_file, R_OK, 0))\n \t\tadd_patterns_from_file_1(dir, excludes_file,\n-\t\t\t\t\t dir->untracked ? &dir->internal.ss_excludes_file : NULL);\n+\t\t\t\t\t &dir->internal.ss_excludes_file);\n \n \t/* per repository user preference */\n \tif (startup_info->have_repository) {\n \t\tconst char *path = git_path_info_exclude();\n \t\tif (!access_or_warn(path, R_OK, 0))\n \t\t\tadd_patterns_from_file_1(dir, path,\n-\t\t\t\t\t\t dir->untracked ? &dir->internal.ss_info_exclude : NULL);\n+\t\t\t\t\t\t &dir->internal.ss_info_exclude);\n \t}\n }\n \ndiff --git a/t/perf/p3010-ls-files.sh b/t/perf/p3010-ls-files.sh\nindex ae14449432..bdf398906f 100755\n--- a/t/perf/p3010-ls-files.sh\n+++ b/t/perf/p3010-ls-files.sh\n@@ -28,4 +28,19 @@ test_perf 'ls-files --modified with pathspec' '\n \t\t-- \"$pathspec\" >/dev/null\n '\n \n+test_perf 'ls-files --others with pathspec and no untracked cache' '\n+\tgit -c core.fsmonitor=false -c core.untrackedCache=false \\\n+\t\tls-files --cached --others --exclude-standard -- \"$pathspec\" >/dev/null\n+'\n+\n+test_expect_success 'populate the untracked cache with ls-files' '\n+\tgit config core.untrackedCache true &&\n+\tgit -c core.fsmonitor=false ls-files --others --exclude-standard >/dev/null\n+'\n+\n+test_perf 'ls-files --others with pathspec and untracked cache' '\n+\tgit -c core.fsmonitor=false ls-files --cached --others \\\n+\t\t--exclude-standard -- \"$pathspec\" >/dev/null\n+'\n+\n test_done\ndiff --git a/t/t3001-ls-files-others-exclude.sh b/t/t3001-ls-files-others-exclude.sh\nindex 29a0a25b30..5ca80932ac 100755\n--- a/t/t3001-ls-files-others-exclude.sh\n+++ b/t/t3001-ls-files-others-exclude.sh\n@@ -306,4 +306,24 @@ test_expect_success 'ls-files with \"**\" patterns and no slashes' '\n \ttest_must_be_empty actual\n '\n \n+test_expect_success 'ls-files preserves repeated and ordered exclude options' '\n+\ttest_create_repo exclude-options &&\n+\t(\n+\t\tcd exclude-options &&\n+\t\ttouch keep info-hidden global-hidden &&\n+\t\techo info-hidden >.git/info/exclude &&\n+\t\techo global-hidden >.git/global-ignore &&\n+\t\tgit config core.excludesFile .git/global-ignore &&\n+\t\techo \"!info-hidden\" >.git/custom-ignore &&\n+\t\techo keep >../expect &&\n+\t\tgit ls-files --others --exclude-from=.git/custom-ignore \\\n+\t\t\t--exclude-standard --exclude-standard >../actual &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\tgit ls-files --others --exclude-standard \\\n+\t\t\t--exclude-from=.git/custom-ignore >../actual &&\n+\t\tprintf \"%s\\n\" info-hidden keep >../expect &&\n+\t\ttest_cmp ../expect ../actual\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t7063-status-untracked-cache.sh b/t/t7063-status-untracked-cache.sh\nindex 2efbb93b71..70eed91dd7 100755\n--- a/t/t7063-status-untracked-cache.sh\n+++ b/t/t7063-status-untracked-cache.sh\n@@ -1016,6 +1016,9 @@ test_expect_success 'untracked cache handles nested repository changes' '\n \t\techo \"?? nested/\" >../expect &&\n \t\ttest_cmp ../expect ../actual &&\n \t\trm -rf nested/.git &&\n+\t\tgit ls-files --others --exclude-standard >../actual &&\n+\t\techo nested/file >../expect &&\n+\t\ttest_cmp ../expect ../actual &&\n \t\tgit status -uall --porcelain >../actual &&\n \t\techo \"?? nested/file\" >../expect &&\n \t\ttest_cmp ../expect ../actual &&\n@@ -1048,4 +1051,98 @@ test_expect_success 'filtered status retains complete untracked listings' '\n \t)\n '\n \n+test_expect_success 'ls-files expands cached directories and filters wildcard results' '\n+\ttest_create_repo ls-files-cache &&\n+\t(\n+\t\tcd ls-files-cache &&\n+\t\tgit config core.untrackedCache true &&\n+\t\tmkdir tracked untracked empty ignored-only &&\n+\t\ttouch tracked/pyproject.toml untracked/pyproject.toml &&\n+\t\ttouch untracked/other ignored-only/file &&\n+\t\techo ignored-only/file >.gitignore &&\n+\t\techo \"*.toml selected\" >.gitattributes &&\n+\t\tgit add .gitignore .gitattributes tracked &&\n+\t\tgit init nested.git &&\n+\t\ttest-tool chmtime =-300 . tracked untracked empty ignored-only &&\n+\t\tgit status -unormal --porcelain >/dev/null &&\n+\t\t# Complete the partial listing, then reuse it without opening directories.\n+\t\tfor opened in 1 0\n+\t\tdo\n+\t\t\t: >\"$TRASH_DIRECTORY/ls-files.trace\" &&\n+\t\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/ls-files.trace\" \\\n+\t\t\t\tgit ls-files --cached --others --exclude-standard -z \\\n+\t\t\t\t-- \"**/pyproject.toml\" >../actual &&\n+\t\t\tprintf \"%s\\0\" untracked/pyproject.toml tracked/pyproject.toml \\\n+\t\t\t\t>../expect &&\n+\t\t\ttest_cmp ../expect ../actual &&\n+\t\t\ttest_grep \"read_directo.*opendir:$opened\\$\" \\\n+\t\t\t\t\"$TRASH_DIRECTORY/ls-files.trace\" &&\n+\t\t\ttest_grep \"read_directo.*gitignore-invalidation:0\\$\" \\\n+\t\t\t\t\"$TRASH_DIRECTORY/ls-files.trace\" || return 1\n+\t\tdone &&\n+\t\tfor pathspec in \"*.git/\" \":(glob)**/*.toml\" \\\n+\t\t\t\":(exclude)untracked/\" \":(attr:selected)**/*.toml\"\n+\t\tdo\n+\t\t\tGIT_DISABLE_UNTRACKED_CACHE=1 git ls-files --others \\\n+\t\t\t\t--exclude-standard -- \"$pathspec\" >../expect &&\n+\t\t\tgit ls-files --others --exclude-standard \\\n+\t\t\t\t-- \"$pathspec\" >../actual &&\n+\t\t\ttest_cmp ../expect ../actual || return 1\n+\t\tdone\n+\t)\n+'\n+\n+test_expect_success 'ls-files cache is reused after status -unormal' '\n+\ttest_create_repo persistent-cache &&\n+\t(\n+\t\tcd persistent-cache &&\n+\t\tmkdir tracked untracked &&\n+\t\ttouch tracked/a tracked/b untracked/a untracked/b &&\n+\t\tgit add tracked &&\n+\t\tgit commit -m initial &&\n+\t\tgit config core.untrackedCache true &&\n+\t\ttest-tool chmtime =-300 . tracked untracked &&\n+\t\tgit ls-files --others --exclude-standard >../actual &&\n+\t\tprintf \"%s\\n\" untracked/a untracked/b >../expect &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\tgit status --porcelain >../actual &&\n+\t\techo \"?? untracked/\" >../status-expect &&\n+\t\ttest_cmp ../status-expect ../actual &&\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/persistent.trace\" \\\n+\t\t\tgit ls-files --others --exclude-standard >../actual &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\ttest_grep \"read_directo.*opendir:0\\$\" \"$TRASH_DIRECTORY/persistent.trace\"\n+\t)\n+'\n+\n+test_expect_success 'ls-files respects optional locks and a busy index lock' '\n+\ttest_when_finished \"rm -f persistent-cache/.git/index.lock\" &&\n+\t(\n+\t\tcd persistent-cache &&\n+\t\ttouch untracked/new &&\n+\t\tcp .git/index ../saved-index &&\n+\t\tgit --no-optional-locks ls-files --others --exclude-standard >../actual &&\n+\t\ttest_cmp_bin ../saved-index .git/index &&\n+\t\tprintf \"%s\\n\" untracked/a untracked/b untracked/new >../expect &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\ttouch .git/index.lock &&\n+\t\tgit ls-files --others --exclude-standard >../actual &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\ttest_cmp_bin ../saved-index .git/index\n+\t)\n+'\n+\n+test_expect_success 'ls-files does not write the index with a pathspec prefix or --with-tree' '\n+\t(\n+\t\tcd persistent-cache &&\n+\t\tcp .git/index ../saved-index &&\n+\t\tgit ls-files --cached --others --exclude-standard -- tracked/a >../actual &&\n+\t\ttest_cmp_bin ../saved-index .git/index &&\n+\t\tgit rm --cached tracked/b &&\n+\t\tcp .git/index ../saved-index &&\n+\t\tgit ls-files --cached --others --exclude-standard --with-tree=HEAD >../actual &&\n+\t\ttest_cmp_bin ../saved-index .git/index\n+\t)\n+'\n+\n test_done\ndiff --git a/t/t7519-status-fsmonitor.sh b/t/t7519-status-fsmonitor.sh\nindex 93973ed25a..8feca101ab 100755\n--- a/t/t7519-status-fsmonitor.sh\n+++ b/t/t7519-status-fsmonitor.sh\n@@ -477,4 +477,48 @@ test_expect_success 'status succeeds with sparse index' '\n \t)\n '\n \n+test_expect_success UNTRACKED_CACHE 'ls-files saves the fsmonitor token with index.skipHash' '\n+\ttest_create_repo ls-files-fsmonitor &&\n+\t(\n+\t\t# index.skipHash gives shared indexes a null OID, so they cannot\n+\t\t# be reloaded.\n+\t\tsane_unset GIT_TEST_SPLIT_INDEX &&\n+\t\tcd ls-files-fsmonitor &&\n+\t\ttouch tracked &&\n+\t\tgit add tracked &&\n+\t\tgit commit -m initial &&\n+\t\tgit config core.untrackedCache true &&\n+\t\tgit config index.skipHash true &&\n+\t\tgit config core.fsmonitor .git/hooks/fsmonitor-test &&\n+\t\ttest_hook --setup fsmonitor-test <<-\\EOF &&\n+\t\tprintf \"initial-token\\0/\\0\"\n+\t\tEOF\n+\t\tmkdir untracked &&\n+\t\ttouch untracked/first &&\n+\t\tgit status --porcelain >/dev/null &&\n+\t\ttest_hook --clobber fsmonitor-test <<-\\EOF &&\n+\t\ttest -f .git/index.lock || exit 1\n+\t\tprintf \"%s\\n\" \"$2\" >.git/query-token\n+\t\tprintf \"next-token\\0\"\n+\t\tif test \"$2\" != next-token\n+\t\tthen\n+\t\t\tprintf \"untracked/second\\0\"\n+\t\tfi\n+\t\tEOF\n+\t\ttouch untracked/second &&\n+\t\tgit ls-files --others --exclude-standard >../actual &&\n+\t\tprintf \"%s\\n\" untracked/first untracked/second >../expect &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\techo initial-token >../token-expect &&\n+\t\ttest_cmp ../token-expect .git/query-token &&\n+\t\tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/ls-files-fsmonitor.trace\" \\\n+\t\t\tgit ls-files --others --exclude-standard >../actual &&\n+\t\ttest_cmp ../expect ../actual &&\n+\t\techo next-token >../token-expect &&\n+\t\ttest_cmp ../token-expect .git/query-token &&\n+\t\ttest_grep \"read_directo.*opendir:0\\$\" \\\n+\t\t\t\"$TRASH_DIRECTORY/ls-files-fsmonitor.trace\"\n+\t)\n+'\n+\n test_done\n\n-- \n2.56.0.rc0.807.ga0c0929ce1.frankengit\n\n"}]}