git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v2 5/5] sparse-index: improve lstat caching of sparse paths

From
Junio C Hamano <gitster@pobox.com>
Date
Jun 27, 2024, 21:14 UTC
Message-ID
<xmqqy16qrsup.fsf@gitster.g>
In-Reply-To
<0cb344ac14fa942f781221663e2324fd2225fb5f.1719412192.git.gitgitgadget@gmail.com>
"Derrick Stolee via GitGitGadget" <gitgitgadget@gmail.com> writes:
Show 28 quoted lines
>  struct path_found_data {
> +	/**
> +	 * The path stored in 'dir', if non-empty, corresponds to the most-
> +	 * recent path that we checked where:
> +	 *
> +	 *   1. The path should be a directory, according to the index.
> +	 *   2. The path does not exist.
> +	 *   3. The parent path _does_ exist. (This may be the root of the
> +	 *      working directory.)
> +	 */
>  	struct strbuf dir;
> -	int dir_found;
>  	size_t lstat_count;
>  };
>  
>  #define PATH_FOUND_DATA_INIT { \
> -	.dir = STRBUF_INIT, \
> -	.dir_found = 1 \
> +	.dir = STRBUF_INIT \
>  }
>  
>  static void clear_path_found_data(struct path_found_data *data)
> @@ -455,49 +462,108 @@ static void clear_path_found_data(struct path_found_data *data)
>  	strbuf_release(&data->dir);
>  }
>  
> +/**
> + * Return the length of the largest common substring that ends in a
"largest" here is understandable (it means longest).
> + * slash ('/') to indicate the largest common parent directory. Returns

here I find it a bit confusing. It is the deepest common parent directory between the two (or "the common parent directory with the longest path"), but my initial reaction was "largest common parent directory? wouldn't the root level by definition the 'largest' (having the largest number of paths underneath) directory that is common?)".

Show 19 quoted lines
> + * zero if no common directory exists.
> + */
> +static size_t max_common_dir_prefix(const char *path1, const char *path2)
> +{
> +	size_t common_prefix = 0;
> +	for (size_t i = 0; path1[i] && path2[i]; i++) {
> +		if (path1[i] != path2[i])
> +			break;
> +
> +		/*
> +		 * If they agree at a directory separator, then add one
> +		 * to make sure it is included in the common prefix string.
> +		 */
> +		if (path1[i] == '/')
> +			common_prefix = i + 1;
> +	}
> +
> +	return common_prefix;
> +}

Looking good. I assume that these two paths are relative to the top-level of the working tree (in other words, they do not begin with a slash)?

Show 16 quoted lines
>  static int path_found(const char *path, struct path_found_data *data)
>  {
> ...
> +	 * At this point, we know that 'path' doesn't exist, and we know that
> +	 * the parent directory of 'data->dir' does exist. Let's set 'data->dir'
> +	 * to be the top-most non-existing directory of 'path'. If the first
> +	 * parent of 'path' exists, then we will act as though 'path'
> +	 * corresponds to a directory (by adding a slash).
>  	 */
> -	newdir = strrchr(path, '/');
> -	if (!newdir)
> -		return 0; /* Didn't find a parent dir; just return 0 now. */
> +	common_prefix = max_common_dir_prefix(path, data->dir.buf);
> ...
> +	strbuf_setlen(&data->dir, common_prefix);
> +	while (1) {

Oooh, nice. So you learned /a/b/c/d did not exist, so check /a/b/c, and then /a/b/ and stop, because you know /a does exist already. With luck, our next query is for /a/b/c/e or for /a/b/f, and knowing that /a/b/ does not exist would allow us to say "no, they do not exist" without having to lstat(). OK.

Previous: Derrick Stolee via GitGitGadgetNext: Derrick Stolee
Message 32 of 44 in “sparse-index: improve clear_skip_worktree_from_present_files()”
  1. 0/5 sparse-index: improve clear_skip_worktree_from_present_files()Derrick Stolee via GitGitGadget, Jun 20, 2024
  2. 1/5 sparse-index: refactor skip worktree retry logicDerrick Stolee via GitGitGadget, Jun 20, 2024
  3. Elijah NewrenJun 24, 2024
  4. Derrick StoleeJun 26, 2024
  5. 2/5 sparse-index: refactor path_found()Derrick Stolee via GitGitGadget, Jun 20, 2024
  6. Elijah NewrenJun 24, 2024
  7. Derrick StoleeJun 26, 2024
  8. 3/5 sparse-index: use strbuf in path_found()Derrick Stolee via GitGitGadget, Jun 20, 2024
  9. Elijah NewrenJun 24, 2024
  10. 4/5 sparse-index: count lstat() callsDerrick Stolee via GitGitGadget, Jun 20, 2024
  11. Elijah NewrenJun 24, 2024
  12. 5/5 sparse-index: improve lstat caching of sparse pathsDerrick Stolee via GitGitGadget, Jun 20, 2024
  13. Elijah NewrenJun 24, 2024
  14. Junio C HamanoJun 25, 2024
  15. Derrick StoleeJun 26, 2024
  16. Elijah NewrenJun 28, 2024
  17. Junio C HamanoJun 20, 2024
  18. Derrick StoleeJun 20, 2024
  19. Junio C HamanoJun 20, 2024
  20. 0/5 sparse-index: improve clear_skip_worktree_from_present_files()Derrick Stolee via GitGitGadget, Jun 26, 2024
  21. 1/5 sparse-checkout: refactor skip worktree retry logicDerrick Stolee via GitGitGadget, Jun 26, 2024
  22. Junio C HamanoJun 27, 2024
  23. Elijah NewrenJun 28, 2024
  24. Derrick StoleeJun 28, 2024
  25. Junio C HamanoJun 28, 2024
  26. Elijah NewrenJun 28, 2024
  27. Derrick StoleeJun 28, 2024
  28. 2/5 sparse-index: refactor path_found()Derrick Stolee via GitGitGadget, Jun 26, 2024
  29. 3/5 sparse-index: use strbuf in path_found()Derrick Stolee via GitGitGadget, Jun 26, 2024
  30. 4/5 sparse-index: count lstat() callsDerrick Stolee via GitGitGadget, Jun 26, 2024
  31. 5/5 sparse-index: improve lstat caching of sparse pathsDerrick Stolee via GitGitGadget, Jun 26, 2024
  32. Junio C HamanoJun 27, 2024
  33. Derrick StoleeJun 28, 2024
  34. Junio C HamanoJun 27, 2024
  35. Elijah NewrenJun 28, 2024
  36. Derrick StoleeJun 28, 2024
  37. 0/5 sparse-index: improve clear_skip_worktree_from_present_files()Derrick Stolee via GitGitGadget, Jun 28, 2024
  38. 1/5 sparse-checkout: refactor skip worktree retry logicDerrick Stolee via GitGitGadget, Jun 28, 2024
  39. 2/5 sparse-index: refactor path_found()Derrick Stolee via GitGitGadget, Jun 28, 2024
  40. 3/5 sparse-index: use strbuf in path_found()Derrick Stolee via GitGitGadget, Jun 28, 2024
  41. 4/5 sparse-index: count lstat() callsDerrick Stolee via GitGitGadget, Jun 28, 2024
  42. 5/5 sparse-index: improve lstat caching of sparse pathsDerrick Stolee via GitGitGadget, Jun 28, 2024
  43. Elijah NewrenJun 28, 2024
  44. Junio C HamanoJun 28, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.