git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v3 0/3] ls-files: reuse and update the untracked cache

From
Tamir Duberstein <tamird@gmail.com>
Date
Oct 23, 2026, 20:03 UTC
Message-ID
<20261023-ls-files-untracked-cache-v3-0-292260e95f53@gmail.com>
In-Reply-To
<20260923-ls-files-untracked-cache-v2-0-d7ee33476eb8@gmail.com>
Repeated queries such as
  git ls-files --cached --others --exclude-standard -z -- "**/pyproject.toml"

walk the working tree even when status has populated an untracked cache. This series lets ls-files reuse those listings and save the work for later commands through Git's existing optional index writes.

The first patch fixes inconsistent ignore-file hashes that invalidate an unchanged cache. The second lets 'git status -unormal' and 'git status -uall' reuse the same cache. 'git status -unormal' still stops scanning an untracked directory after finding an untracked file; a later command requesting all untracked files completes the listing as needed. The third lets ls-files use the cache and write pending untracked cache and fsmonitor updates back to the index. --no-optional-locks suppresses those writes.

For wildcard pathspecs without a fixed prefix, apply the pathspec after reading complete directory listings. A first query can therefore scan directories that its pathspec would otherwise skip. Fixed-prefix, attribute and exclude pathspecs retain their existing traversal.

A new dir_flags value makes older versions of Git rebuild the cache before using it to list untracked files. Changes invalidate cached entries in parent directories as well, so updates can reopen more directories than a cache populated with --untracked-files=all before this series.

On macOS, with 100,000 files in 5,000 leaf directories, half tracked. Each group of 100 leaf directories has 50 tracked and 50 untracked. Each binary populated its own cache using the indicated status mode.

hyperfine --warmup 3 --runs 7 (mean ± standard deviation, milliseconds):
  cache   fsmonitor  command          upstream        v1             v2
  normal  off        query            241.3 ± 14.8    132.7 ± 20.0   33.3 ± 2.4
  normal  off        query + status   292.7 ± 11.2    187.0 ± 10.8   92.6 ± 3.3
  all     off        query            231.3 ± 14.6    29.3 ± 1.3     34.7 ± 2.0
  normal  on         query            223.1 ± 17.8    111.0 ± 13.4   33.8 ± 1.5
  all     on         query            216.6 ± 12.0    28.0 ± 5.1     34.6 ± 0.8

"query" is the ls-files command above; "status" is "git status -unormal --porcelain". These are repeated queries with warm filesystem and untracked caches.

Assisted-by: LLM
Signed-off-by: Tamir Duberstein <tamird@gmail.com>
---
Changes in v3:
- Release the index lock after reading the directory, before printing any
  paths. A slow reader of ls-files output can no longer block index writers.
- Skip the optional write with --debug so the listing shows the stat data
  read from the index.
- Link to v2: https://patch.msgid.link/20260923-ls-files-untracked-cache-v2-0-d7ee33476eb8@gmail.com
Changes in v2:
- Share the cache between --untracked-files=normal and --untracked-files=all.
- Write ls-files untracked cache and fsmonitor updates back to the index.
- Test alternating modes, cache invalidation and optional index writes.
- Replace the v1 measurements with results for the revised implementation.
- Link to v1: https://patch.msgid.link/20260923-ls-files-untracked-cache-v1-0-08db4cc1efdb@gmail.com
---
Tamir Duberstein (3):
      dir: hash ignore files before appending newline
      dir: share untracked caches across output modes
      ls-files: use and update the untracked cache
 Documentation/git-ls-files.adoc    |   4 +
 Documentation/gitformat-index.adoc |  13 +-
 builtin/ls-files.c                 |  59 +++++---
 dir.c                              | 251 +++++++++++++++++---------------
 dir.h                              |  16 +--
 t/perf/p3010-ls-files.sh           |  15 ++
 t/t3001-ls-files-others-exclude.sh |  20 +++
 t/t7063-status-untracked-cache.sh  | 288 +++++++++++++++++++++++++++++--------
 t/t7519-status-fsmonitor.sh        |  44 ++++++
 9 files changed, 506 insertions(+), 204 deletions(-)

--- Range-diff versus v2:

1:  d45e8c7ea9 = 1:  fb0b53b2cc dir: hash ignore files before appending newline
2:  b2ea9e163c = 2:  ca78501354 dir: share untracked caches across output modes
3:  a84c08dabe ! 3:  4622b804a6 ls-files: use and update the untracked cache
    @@ Commit message
         options can be processed in the same order before reading the index.
     
         A pathspec prefix can prune the in-memory index, and --with-tree can add
    -    entries to it. Do not write the index in those cases. Respect
    -    --no-optional-locks and continue without writing when the index lock is
    -    busy.
    +    entries to it. Do not write the index in those cases or with --debug,
    +    which prints stat data that an index write can change. Respect
    +    --no-optional-locks and continue without writing when the lock is busy.
    +
    +    Finish the optional update after the directory walk and before printing
    +    paths, so a slow reader cannot block another index writer.
     
         Assisted-by: LLM
         Signed-off-by: Tamir Duberstein <tamird@gmail.com>
    @@ builtin/ls-files.c
      #include "object-name.h"
      #include "strbuf.h"
      #include "parse-options.h"
    +@@ builtin/ls-files.c: static void show_files(struct repository *repo, struct dir_struct *dir)
    + 	int i;
    + 	struct strbuf fullname = STRBUF_INIT;
    + 
    +-	/* For cached/deleted files we don't need to even do the readdir */
    +-	if (show_others || show_killed) {
    +-		if (!show_others)
    +-			dir->flags |= DIR_COLLECT_KILLED_ONLY;
    +-		fill_directory(dir, repo->index, &pathspec);
    +-		if (show_others)
    +-			show_other_files(repo->index, dir);
    +-		if (show_killed)
    +-			show_killed_files(repo->index, dir);
    +-	}
    ++	if (show_others)
    ++		show_other_files(repo->index, dir);
    ++	if (show_killed)
    ++		show_killed_files(repo->index, dir);
    + 
    + 	if (!(show_cached || show_stage || show_deleted || show_modified))
    + 		return;
     @@ builtin/ls-files.c: int cmd_ls_files(int argc,
      	struct dir_struct dir = DIR_INIT;
      	struct pattern_list *pl;
    @@ builtin/ls-files.c: int cmd_ls_files(int argc,
      
     +	/*
     +	 * Do not save an index pruned by a pathspec or changed by --with-tree.
    ++	 * --debug shows stat data that writing the index can change.
     +	 * With index.skipHash, the checksum cannot detect concurrent index
     +	 * changes. Lock the index before reading it.
     +	 */
     +	if (show_others && !dir.flags && dir.exclude_per_dir &&
    -+	    !max_prefix && !with_tree && use_optional_locks() &&
    ++	    !max_prefix && !with_tree && !debug_mode && use_optional_locks() &&
     +	    !dir.internal.unmanaged_exclude_files && !exclude_list.nr &&
     +	    repo->settings.core_untracked_cache != UNTRACKED_CACHE_REMOVE)
     +		index_fd = repo_hold_locked_index(repo, &index_lock, 0);
    @@ builtin/ls-files.c: int cmd_ls_files(int argc,
      
      	/* Treat unmatching pathspec elements as errors */
     @@ builtin/ls-files.c: int cmd_ls_files(int argc,
    - 		ret = 1;
    + 		overlay_tree_on_index(repo->index, with_tree, max_prefix);
      	}
      
    -+	if (index_fd >= 0) {
    -+		if (!ret && dir.untracked)
    -+			repo_update_index_if_able(repo, &index_lock);
    -+		else
    -+			rollback_lock_file(&index_lock);
    ++	if (show_others || show_killed) {
    ++		if (!show_others)
    ++			dir.flags |= DIR_COLLECT_KILLED_ONLY;
    ++		fill_directory(&dir, repo->index, &pathspec);
     +	}
     +
    - 	string_list_clear(&exclude_list, 0);
    - 	dir_clear(&dir);
    - 	free(max_prefix);
    ++	/* Don't hold the index lock while writing paths to stdout. */
    ++	if (index_fd >= 0 && dir.untracked)
    ++		repo_update_index_if_able(repo, &index_lock);
    ++	rollback_lock_file(&index_lock);
    ++
    + 	show_files(repo, &dir);
    + 
    + 	if (show_resolve_undo)
     
      ## dir.c ##
     @@ dir.c: static int add_patterns(const char *fname, const char *base, int baselen,
    @@ t/t7063-status-untracked-cache.sh: test_expect_success 'filtered status retains
     +		test_cmp_bin ../saved-index .git/index
     +	)
     +'
    ++
    ++test_expect_success 'ls-files --debug shows the stat data read from the index' '
    ++	(
    ++		cd persistent-cache &&
    ++		git config core.trustctime false &&
    ++		printf one >tracked/racy &&
    ++		test-tool chmtime =1000000000 tracked/racy &&
    ++		git add tracked/racy &&
    ++		printf two >tracked/racy &&
    ++		test-tool chmtime =1000000000 tracked/racy .git/index &&
    ++
    ++		git --no-optional-locks ls-files -co --exclude-standard --debug >../expect &&
    ++		git ls-files -co --exclude-standard --debug >../actual &&
    ++		test_cmp ../expect ../actual
    ++	)
    ++'
     +
      test_done
     

--- base-commit: 3bc0341126508f78f5869cbfc0005e987efdf0c7 change-id: 20260923-ls-files-untracked-cache-3559bed01a3e

Previous: Tamir DubersteinNext: Tamir Duberstein
Message 10 of 13 in “ls-files: reuse the untracked cache”
  1. 0/2 ls-files: reuse the untracked cacheTamir Duberstein, Sep 23, 2026
  2. 1/2 dir: hash ignore files before adding parser LFTamir Duberstein, Sep 23, 2026
  3. 2/2 ls-files: reuse cached untracked listingsTamir Duberstein, Sep 23, 2026
  4. Tamir DubersteinSep 23, 2026
  5. 0/3 ls-files: reuse and update the untracked cacheTamir Duberstein, Sep 23, 2026
  6. 1/3 dir: hash ignore files before appending newlineTamir Duberstein, Sep 23, 2026
  7. 2/3 dir: share untracked caches across output modesTamir Duberstein, Sep 23, 2026
  8. 3/3 ls-files: use and update the untracked cacheTamir Duberstein, Sep 23, 2026
  9. Tamir DubersteinOct 23, 2026
  10. 0/3 ls-files: reuse and update the untracked cacheTamir Duberstein, Oct 23, 2026
  11. 1/3 dir: hash ignore files before appending newlineTamir Duberstein, Oct 23, 2026
  12. 2/3 dir: share untracked caches across output modesTamir Duberstein, Oct 23, 2026
  13. 3/3 ls-files: use and update the untracked cacheTamir Duberstein, Oct 23, 2026

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.