git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH 4/5] sparse-index: count lstat() calls

From
Elijah Newren <newren@gmail.com>
Date
Jun 24, 2024, 22:13 UTC
Message-ID
<CABPp-BH6apqur+zzE12yqL1ZHNP30_2MFgEEeGxYa2K3f7hNRw@mail.gmail.com>
In-Reply-To
<88a3145e585169fde8cd7d43a435daa07eb82667.1718899877.git.gitgitgadget@gmail.com>

On Thu, Jun 20, 2024 at 10:12 AM Derrick Stolee via GitGitGadget <gitgitgadget@gmail.com> wrote:

Show 68 quoted lines
>
> From: Derrick Stolee <stolee@gmail.com>
>
> The clear_skip_worktree.. methods already report some statistics about
> how many cache entries are checked against path_found() due to having
> the skip-worktree bit set. However, due to path_found() performing some
> caching, this isn't the only information that would be helpful to
> report.
>
> Add a new lstat_count member to the path_found_data struct to count the
> number of times path_found() calls lstat(). This will be helpful to help
> explain performance problems in this method as well as to demonstrate
> future changes to the caching algorithm in a more concrete way than
> end-to-end timings.
>
> Signed-off-by: Derrick Stolee <stolee@gmail.com>
> ---
>  sparse-index.c | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/sparse-index.c b/sparse-index.c
> index fec4f393360..8577fa726b8 100644
> --- a/sparse-index.c
> +++ b/sparse-index.c
> @@ -442,6 +442,7 @@ void ensure_correct_sparsity(struct index_state *istate)
>  struct path_found_data {
>         struct strbuf dir;
>         int dir_found;
> +       size_t lstat_count;
>  };
>
>  #define PATH_FOUND_DATA_INIT { \
> @@ -469,6 +470,7 @@ static int path_found(const char *path, struct path_found_data *data)
>         /*
>          * If path itself exists, return 1.
>          */
> +       data->lstat_count++;
>         if (!lstat(path, &st))
>                 return 1;
>
> @@ -493,6 +495,7 @@ static int path_found(const char *path, struct path_found_data *data)
>         strbuf_reset(&data->dir);
>         strbuf_add(&data->dir, path, newdir - path + 1);
>
> +       data->lstat_count++;
>         data->dir_found = !lstat(data->dir.buf, &st);
>
>         return 0;
> @@ -524,6 +527,8 @@ static int clear_skip_worktree_from_present_files_sparse(struct index_state *ist
>
>         trace2_data_intmax("index", istate->repo,
>                            "sparse_path_count", path_count);
> +       trace2_data_intmax("index", istate->repo,
> +                          "sparse_lstat_count", data.lstat_count);
>         trace2_region_leave("index", "clear_skip_worktree_from_present_files_sparse",
>                             istate->repo);
>         clear_path_found_data(&data);
> @@ -553,6 +558,8 @@ static void clear_skip_worktree_from_present_files_full(struct index_state *ista
>
>         trace2_data_intmax("index", istate->repo,
>                            "full_path_count", path_count);
> +       trace2_data_intmax("index", istate->repo,
> +                          "full_lstat_count", data.lstat_count);
>         trace2_region_leave("index", "clear_skip_worktree_from_present_files_full",
>                             istate->repo);
>         clear_path_found_data(&data);
> --
> gitgitgadget

Makes sense. I'm getting really curious to see how this all fits into the big improvements you made; I guess I'll see that in the next patch...

Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee via GitGitGadget
Message 11 of 44 in “sparse-index: improve clear_skip_worktree_from_present_files()”
  1. 0/5 sparse-index: improve clear_skip_worktree_from_present_files()Derrick Stolee via GitGitGadget, Jun 20, 2024
  2. 1/5 sparse-index: refactor skip worktree retry logicDerrick Stolee via GitGitGadget, Jun 20, 2024
  3. Elijah NewrenJun 24, 2024
  4. Derrick StoleeJun 26, 2024
  5. 2/5 sparse-index: refactor path_found()Derrick Stolee via GitGitGadget, Jun 20, 2024
  6. Elijah NewrenJun 24, 2024
  7. Derrick StoleeJun 26, 2024
  8. 3/5 sparse-index: use strbuf in path_found()Derrick Stolee via GitGitGadget, Jun 20, 2024
  9. Elijah NewrenJun 24, 2024
  10. 4/5 sparse-index: count lstat() callsDerrick Stolee via GitGitGadget, Jun 20, 2024
  11. Elijah NewrenJun 24, 2024
  12. 5/5 sparse-index: improve lstat caching of sparse pathsDerrick Stolee via GitGitGadget, Jun 20, 2024
  13. Elijah NewrenJun 24, 2024
  14. Junio C HamanoJun 25, 2024
  15. Derrick StoleeJun 26, 2024
  16. Elijah NewrenJun 28, 2024
  17. Junio C HamanoJun 20, 2024
  18. Derrick StoleeJun 20, 2024
  19. Junio C HamanoJun 20, 2024
  20. 0/5 sparse-index: improve clear_skip_worktree_from_present_files()Derrick Stolee via GitGitGadget, Jun 26, 2024
  21. 1/5 sparse-checkout: refactor skip worktree retry logicDerrick Stolee via GitGitGadget, Jun 26, 2024
  22. Junio C HamanoJun 27, 2024
  23. Elijah NewrenJun 28, 2024
  24. Derrick StoleeJun 28, 2024
  25. Junio C HamanoJun 28, 2024
  26. Elijah NewrenJun 28, 2024
  27. Derrick StoleeJun 28, 2024
  28. 2/5 sparse-index: refactor path_found()Derrick Stolee via GitGitGadget, Jun 26, 2024
  29. 3/5 sparse-index: use strbuf in path_found()Derrick Stolee via GitGitGadget, Jun 26, 2024
  30. 4/5 sparse-index: count lstat() callsDerrick Stolee via GitGitGadget, Jun 26, 2024
  31. 5/5 sparse-index: improve lstat caching of sparse pathsDerrick Stolee via GitGitGadget, Jun 26, 2024
  32. Junio C HamanoJun 27, 2024
  33. Derrick StoleeJun 28, 2024
  34. Junio C HamanoJun 27, 2024
  35. Elijah NewrenJun 28, 2024
  36. Derrick StoleeJun 28, 2024
  37. 0/5 sparse-index: improve clear_skip_worktree_from_present_files()Derrick Stolee via GitGitGadget, Jun 28, 2024
  38. 1/5 sparse-checkout: refactor skip worktree retry logicDerrick Stolee via GitGitGadget, Jun 28, 2024
  39. 2/5 sparse-index: refactor path_found()Derrick Stolee via GitGitGadget, Jun 28, 2024
  40. 3/5 sparse-index: use strbuf in path_found()Derrick Stolee via GitGitGadget, Jun 28, 2024
  41. 4/5 sparse-index: count lstat() callsDerrick Stolee via GitGitGadget, Jun 28, 2024
  42. 5/5 sparse-index: improve lstat caching of sparse pathsDerrick Stolee via GitGitGadget, Jun 28, 2024
  43. Elijah NewrenJun 28, 2024
  44. Junio C HamanoJun 28, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.