git/list[1] front-page[2] threads[3] people[4] search[5] about
 

[PATCH v2 4/5] sparse-index: count lstat() calls

From
Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com>
Date
Jun 26, 2024, 14:29 UTC
Message-ID
<9549f5b8062909253fef1fc9a62dae9114c43481.1719412192.git.gitgitgadget@gmail.com>
In-Reply-To
<pull.1754.v2.git.1719412192.gitgitgadget@gmail.com>
From: Derrick Stolee <stolee@gmail.com>

The clear_skip_worktree.. methods already report some statistics about how many cache entries are checked against path_found() due to having the skip-worktree bit set. However, due to path_found() performing some caching, this isn't the only information that would be helpful to report.

Add a new lstat_count member to the path_found_data struct to count the number of times path_found() calls lstat(). This will be helpful to help explain performance problems in this method as well as to demonstrate future changes to the caching algorithm in a more concrete way than end-to-end timings.

Signed-off-by: Derrick Stolee <stolee@gmail.com>
---
 sparse-index.c | 7 +++++++
 1 file changed, 7 insertions(+)
diff --git a/sparse-index.c b/sparse-index.c
index fec4f393360..8577fa726b8 100644
--- a/sparse-index.c
+++ b/sparse-index.c
@@ -442,6 +442,7 @@ void ensure_correct_sparsity(struct index_state *istate)
 struct path_found_data {
 	struct strbuf dir;
 	int dir_found;
+	size_t lstat_count;
 };
 
 #define PATH_FOUND_DATA_INIT { \
@@ -469,6 +470,7 @@ static int path_found(const char *path, struct path_found_data *data)
 	/*
 	 * If path itself exists, return 1.
 	 */
+	data->lstat_count++;
 	if (!lstat(path, &st))
 		return 1;
 
@@ -493,6 +495,7 @@ static int path_found(const char *path, struct path_found_data *data)
 	strbuf_reset(&data->dir);
 	strbuf_add(&data->dir, path, newdir - path + 1);
 
+	data->lstat_count++;
 	data->dir_found = !lstat(data->dir.buf, &st);
 
 	return 0;
@@ -524,6 +527,8 @@ static int clear_skip_worktree_from_present_files_sparse(struct index_state *ist
 
 	trace2_data_intmax("index", istate->repo,
 			   "sparse_path_count", path_count);
+	trace2_data_intmax("index", istate->repo,
+			   "sparse_lstat_count", data.lstat_count);
 	trace2_region_leave("index", "clear_skip_worktree_from_present_files_sparse",
 			    istate->repo);
 	clear_path_found_data(&data);
@@ -553,6 +558,8 @@ static void clear_skip_worktree_from_present_files_full(struct index_state *ista
 
 	trace2_data_intmax("index", istate->repo,
 			   "full_path_count", path_count);
+	trace2_data_intmax("index", istate->repo,
+			   "full_lstat_count", data.lstat_count);
 	trace2_region_leave("index", "clear_skip_worktree_from_present_files_full",
 			    istate->repo);
 	clear_path_found_data(&data);
-- 
gitgitgadget
Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee via GitGitGadget
Message 30 of 44 in “sparse-index: improve clear_skip_worktree_from_present_files()”
  1. 0/5 sparse-index: improve clear_skip_worktree_from_present_files()Derrick Stolee via GitGitGadget, Jun 20, 2024
  2. 1/5 sparse-index: refactor skip worktree retry logicDerrick Stolee via GitGitGadget, Jun 20, 2024
  3. Elijah NewrenJun 24, 2024
  4. Derrick StoleeJun 26, 2024
  5. 2/5 sparse-index: refactor path_found()Derrick Stolee via GitGitGadget, Jun 20, 2024
  6. Elijah NewrenJun 24, 2024
  7. Derrick StoleeJun 26, 2024
  8. 3/5 sparse-index: use strbuf in path_found()Derrick Stolee via GitGitGadget, Jun 20, 2024
  9. Elijah NewrenJun 24, 2024
  10. 4/5 sparse-index: count lstat() callsDerrick Stolee via GitGitGadget, Jun 20, 2024
  11. Elijah NewrenJun 24, 2024
  12. 5/5 sparse-index: improve lstat caching of sparse pathsDerrick Stolee via GitGitGadget, Jun 20, 2024
  13. Elijah NewrenJun 24, 2024
  14. Junio C HamanoJun 25, 2024
  15. Derrick StoleeJun 26, 2024
  16. Elijah NewrenJun 28, 2024
  17. Junio C HamanoJun 20, 2024
  18. Derrick StoleeJun 20, 2024
  19. Junio C HamanoJun 20, 2024
  20. 0/5 sparse-index: improve clear_skip_worktree_from_present_files()Derrick Stolee via GitGitGadget, Jun 26, 2024
  21. 1/5 sparse-checkout: refactor skip worktree retry logicDerrick Stolee via GitGitGadget, Jun 26, 2024
  22. Junio C HamanoJun 27, 2024
  23. Elijah NewrenJun 28, 2024
  24. Derrick StoleeJun 28, 2024
  25. Junio C HamanoJun 28, 2024
  26. Elijah NewrenJun 28, 2024
  27. Derrick StoleeJun 28, 2024
  28. 2/5 sparse-index: refactor path_found()Derrick Stolee via GitGitGadget, Jun 26, 2024
  29. 3/5 sparse-index: use strbuf in path_found()Derrick Stolee via GitGitGadget, Jun 26, 2024
  30. 4/5 sparse-index: count lstat() callsDerrick Stolee via GitGitGadget, Jun 26, 2024
  31. 5/5 sparse-index: improve lstat caching of sparse pathsDerrick Stolee via GitGitGadget, Jun 26, 2024
  32. Junio C HamanoJun 27, 2024
  33. Derrick StoleeJun 28, 2024
  34. Junio C HamanoJun 27, 2024
  35. Elijah NewrenJun 28, 2024
  36. Derrick StoleeJun 28, 2024
  37. 0/5 sparse-index: improve clear_skip_worktree_from_present_files()Derrick Stolee via GitGitGadget, Jun 28, 2024
  38. 1/5 sparse-checkout: refactor skip worktree retry logicDerrick Stolee via GitGitGadget, Jun 28, 2024
  39. 2/5 sparse-index: refactor path_found()Derrick Stolee via GitGitGadget, Jun 28, 2024
  40. 3/5 sparse-index: use strbuf in path_found()Derrick Stolee via GitGitGadget, Jun 28, 2024
  41. 4/5 sparse-index: count lstat() callsDerrick Stolee via GitGitGadget, Jun 28, 2024
  42. 5/5 sparse-index: improve lstat caching of sparse pathsDerrick Stolee via GitGitGadget, Jun 28, 2024
  43. Elijah NewrenJun 28, 2024
  44. Junio C HamanoJun 28, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.