git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH 18/20] update-index: manually enable or disable untracked cache

From
Nguyễn Thái Ngọc Duy <pclouds@gmail.com>
Date
May 7, 2014, 14:51 UTC
Message-ID
<1399474320-6840-19-git-send-email-pclouds@gmail.com>
In-Reply-To
<1399474320-6840-1-git-send-email-pclouds@gmail.com>

Some numbers below. In short, the saving on wt_status_collect_untracked() is about 80%. There are some overhead on read/write_cache, but it seems lower than 50ms, while the .._untracked() saving is in the 500ms range (except linux-2.6, about 150ms). "git status" time saving ranges from 33% to 42%.

On gentoo-x86.git (100k files, 23k dirs, quite balance tree, 8.5MB index v4, cache-tree fully populated), before turning untracked cache on (the most important line is wt_status_collect:625)

   184.650 gitmodules_config:201 if (read_cache() < 0) die("index
     0.004 cmd_status:1299 read_cache_preload(&s.pathspec)
   226.231 cmd_status:1300 refresh_index(&the_index, REFRESH_QUIET
     3.096 cmd_status:1304 update_index_if_able(&the_index, &index_lock)
     6.788  wt_status_collect:619 wt_status_collect_changes_worktree(s)
     6.780  wt_status_collect:624 wt_status_collect_changes_index(s)
   772.866  wt_status_collect:625 wt_status_collect_untracked(s)
   786.686 cmd_status:1308 wt_status_collect(&s)

real 0m1.211s user 0m0.566s sys 0m0.638s

and after (saving 42% total time):
   220.888 gitmodules_config:201 if (read_cache() < 0) die("index
    36.368 cmd_status:1299 read_cache_preload(&s.pathspec)
   223.936 cmd_status:1300 refresh_index(&the_index, REFRESH_QUIET
     2.935 cmd_status:1304 update_index_if_able(&the_index, &index_lock)
     7.031  wt_status_collect:619 wt_status_collect_changes_worktree(s)
     7.156  wt_status_collect:624 wt_status_collect_changes_index(s)
   148.022  wt_status_collect:625 wt_status_collect_untracked(s)
   162.443 cmd_status:1308 wt_status_collect(&s)
    36.943 cmd_status:1348 if (the_index.untracked) { struct

real 0m0.696s user 0m0.406s sys 0m0.288s

On webkit.git (182k files, 6k dirs, 14M index v4), before:
   283.182 gitmodules_config:201 if (read_cache() < 0) die("index
     0.004 cmd_status:1299 read_cache_preload(&s.pathspec)
   515.836 cmd_status:1300 refresh_index(&the_index, REFRESH_QUIET
     5.428 cmd_status:1304 update_index_if_able(&the_index, &index_lock)
    11.162  wt_status_collect:619 wt_status_collect_changes_worktree(s)
    11.068  wt_status_collect:624 wt_status_collect_changes_index(s)
   887.247  wt_status_collect:625 wt_status_collect_untracked(s)
   909.722 cmd_status:1308 wt_status_collect(&s)

real 0m1.729s user 0m0.785s sys 0m0.941s

and after (saving 38% total time):
   290.994 gitmodules_config:201 if (read_cache() < 0) die("index
    10.132 cmd_status:1299 read_cache_preload(&s.pathspec)
   516.656 cmd_status:1300 refresh_index(&the_index, REFRESH_QUIET
     5.159 cmd_status:1304 update_index_if_able(&the_index, &index_lock)
    12.605  wt_status_collect:619 wt_status_collect_changes_worktree(s)
    11.262  wt_status_collect:624 wt_status_collect_changes_index(s)
   186.032  wt_status_collect:625 wt_status_collect_untracked(s)
   210.134 cmd_status:1308 wt_status_collect(&s)
    12.332 cmd_status:1348 if (the_index.untracked) { struct index_state

real 0m1.058s user 0m0.525s sys 0m0.532s

And linux-2.6 (45k files, 3k dirs, 3.2MB index v4), before:
    68.668 gitmodules_config:201 if (read_cache() < 0) die("index
     0.004 cmd_status:1299 read_cache_preload(&s.pathspec)
   114.270 cmd_status:1300 refresh_index(&the_index, REFRESH_QUIET
     1.180 cmd_status:1304 update_index_if_able(&the_index, &index_lock)
     4.027  wt_status_collect:619 wt_status_collect_changes_worktree(s)
     4.265  wt_status_collect:624 wt_status_collect_changes_index(s)
   191.285  wt_status_collect:625 wt_status_collect_untracked(s)
   199.825 cmd_status:1308 wt_status_collect(&s)

real 0m0.392s user 0m0.177s sys 0m0.215s

and after (saving 33%):
    71.756 gitmodules_config:201 if (read_cache() < 0) die("index
     5.201 cmd_status:1299 read_cache_preload(&s.pathspec)
   111.064 cmd_status:1300 refresh_index(&the_index, REFRESH_QUIET
     1.171 cmd_status:1304 update_index_if_able(&the_index, &index_lock)
     3.054  wt_status_collect:619 wt_status_collect_changes_worktree(s)
     4.945  wt_status_collect:624 wt_status_collect_changes_index(s)
    27.203  wt_status_collect:625 wt_status_collect_untracked(s)
    35.475 cmd_status:1308 wt_status_collect(&s)
    25.759 cmd_status:1348 if (the_index.untracked) { struct index_state
real    0m0.259s
user    0m0.106s
sys     0m0.132s
---
 builtin/update-index.c | 17 +++++++++++++++++
 1 file changed, 17 insertions(+)
diff --git a/builtin/update-index.c b/builtin/update-index.c
index ba54e19..003e28e 100644
--- a/builtin/update-index.c
+++ b/builtin/update-index.c
@@ -734,6 +734,7 @@ static int reupdate_callback(struct parse_opt_ctx_t *ctx,
 int cmd_update_index(int argc, const char **argv, const char *prefix)
 {
 	int newfd, entries, has_errors = 0, line_termination = '\n';
+	int untracked_cache = -1;
 	int read_from_stdin = 0;
 	int prefix_length = prefix ? strlen(prefix) : 0;
 	int preferred_index_format = 0;
@@ -822,6 +823,8 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 			resolve_undo_clear_callback},
 		OPT_INTEGER(0, "index-version", &preferred_index_format,
 			N_("write index in this format")),
+		OPT_BOOL(0, "untracked-cache", &untracked_cache,
+			N_("enable/disable untracked cache")),
 		OPT_END()
 	};
 
@@ -915,6 +918,20 @@ int cmd_update_index(int argc, const char **argv, const char *prefix)
 		strbuf_release(&buf);
 	}
 
+	if (untracked_cache > 0) {
+		struct untracked_cache *uc;
+
+		uc = xcalloc(1, sizeof(*uc));
+		uc->exclude_per_dir = ".gitignore";
+		/* should be the same flags used by git-status */
+		uc->dir_flags = DIR_SHOW_OTHER_DIRECTORIES | DIR_HIDE_EMPTY_DIRECTORIES;
+		the_index.untracked = uc;
+		active_cache_changed = 1;
+	} else if (!untracked_cache) {
+		the_index.untracked = NULL;
+		active_cache_changed = 1;
+	}
+
 	if (active_cache_changed) {
 		if (newfd < 0) {
 			if (refresh_args.flags & REFRESH_QUIET)

-- 
1.9.1.346.ga2b5940
Previous: Nguyễn Thái Ngọc DuyNext: Nguyễn Thái Ngọc Duy
Message 19 of 21 in “Untracked cache to speed up "git status"”
  1. 00/20 Untracked cache to speed up "git status"Nguyễn Thái Ngọc Duy, May 7, 2014
  2. 01/20 dir.c: coding style fixNguyễn Thái Ngọc Duy, May 7, 2014
  3. 02/20 dir.h: move struct exclude declaration to top levelNguyễn Thái Ngọc Duy, May 7, 2014
  4. 03/20 prep_exclude: remove the artificial PATH_MAX limitNguyễn Thái Ngọc Duy, May 7, 2014
  5. 04/20 dir.c: optionally compute sha-1 of a .gitignore fileNguyễn Thái Ngọc Duy, May 7, 2014
  6. 05/20 untracked cache: record .gitignore information and dir hierarchyNguyễn Thái Ngọc Duy, May 7, 2014
  7. 06/20 untracked cache: initial untracked cache validationNguyễn Thái Ngọc Duy, May 7, 2014
  8. 07/20 untracked cache: invalidate dirs recursively if .gitignore changesNguyễn Thái Ngọc Duy, May 7, 2014
  9. 08/20 untracked cache: record/validate dir mtime and reuse cached outputNguyễn Thái Ngọc Duy, May 7, 2014
  10. 09/20 untracked cache: mark what dirs should be recursed/savedNguyễn Thái Ngọc Duy, May 7, 2014
  11. 10/20 untracked cache: don't open non-existent .gitignoreNguyễn Thái Ngọc Duy, May 7, 2014
  12. 11/20 untracked cache: save to an index extensionNguyễn Thái Ngọc Duy, May 7, 2014
  13. 12/20 untracked cache: load from UNTR index extensionNguyễn Thái Ngọc Duy, May 7, 2014
  14. 13/20 untracked cache: invalidate at index addition or removalNguyễn Thái Ngọc Duy, May 7, 2014
  15. 14/20 untracked cache: print untracked statistics with $GIT_TRACE_UNTRACKEDNguyễn Thái Ngọc Duy, May 7, 2014
  16. 15/20 read-cache.c: split racy stat test to a separate functionNguyễn Thái Ngọc Duy, May 7, 2014
  17. 16/20 untracked cache: avoid racy timestampsNguyễn Thái Ngọc Duy, May 7, 2014
  18. 17/20 status: support untracked cacheNguyễn Thái Ngọc Duy, May 7, 2014
  19. 18/20 update-index: manually enable or disable untracked cacheNguyễn Thái Ngọc Duy, May 7, 2014
  20. 19/20 update-index: test the system before enabling untracked cacheNguyễn Thái Ngọc Duy, May 7, 2014
  21. 20/20 t7063: tests for untracked cacheNguyễn Thái Ngọc Duy, May 7, 2014

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.