git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v5 0/4] bloom: enable bloom filter optimization for multiple pathspec elements in revision traversal

From
Derrick Stolee <stolee@gmail.com>
Date
Jul 10, 2025, 13:49 UTC
Message-ID
<afb68948-218b-4b56-9faa-29578ef9c73c@gmail.com>
In-Reply-To
<20250710084829.2171855-1-502024330056@smail.nju.edu.cn>
On 7/10/2025 4:48 AM, Lidong Yan wrote:
Show 39 quoted lines
> The difference from v4 is:
>   - for the bloom_key_* functions, we now pass struct bloom_key *
>     as the first parameter.
>   - bloom_keyvec_fill_key() and bloom_keyvec_new() have been merged
>     into a single function, so that each key is filled during the
>     creation of the bloom_keyvec.
> 
> Below is a comparison of the time taken to run git log on Git and
> LLVM repositories before and after applying this patch.
> 
> Setup commit-graph:
>   $ cd ~/src/git && git commit-graph write --split --reachable --changed-paths
>   $ cd ~/src/llvm && git commit-graph write --split --reachable --changed-paths
> 
> Run git log on Git repository
>   $ cd ~/src/git
>   $ hash -p /usr/bin/git git # my system git binary is 2.43.0
>   $ time git log -100 -- commit.c commit-graph.c >/dev/null
>   real	0m0.055s
>   user	0m0.040s
>   sys	  0m0.015s
>   $ hash -p ~/bin/git/bin/git git
>   $ time git log -100 -- commit.c commit-graph.c >/dev/null
>   real	0m0.039s
>   user	0m0.020s
>   sys	  0m0.020s
> 
> Run git log in LLVM repository
>   $ cd ~/src/llvm
>   $ hash -p /usr/bin/git git # my system git binary is 2.43.0
>   $ time git log -100 -- llvm/lib/Support/CommandLine.cpp llvm/lib/Support/CommandLine.h >/dev/null
>   real	0m2.365s
>   user	0m2.252s
>   sys	  0m0.113s
>   $ hash -p ~/bin/git/bin/git git
>   $ time git log -100 -- llvm/lib/Support/CommandLine.cpp llvm/lib/Support/CommandLine.h >/dev/null
>   real	0m0.240s
>   user	0m0.158s
>   sys	  0m0.064s

Thanks for these stats, though I'd recommend trying hyperfine [1] which with this setup would let you run these experiments as the following:

$ $ hyperfine --warmup=3 \
> -n 'old' '~/_git/git-sparse-checkout-clean/git log -100 -- commit.c commit-graph.c' \
> -n 'new' '~/_git/git/git log -100 -- commit.c commit-graph.c'
Benchmark 1: old
  Time (mean ± σ):      73.1 ms ±   2.9 ms    [User: 48.8 ms, System: 23.9 ms]
  Range (min … max):    69.9 ms …  84.5 ms    42 runs
 
Benchmark 2: new
  Time (mean ± σ):      55.1 ms ±   2.9 ms    [User: 30.5 ms, System: 24.4 ms]
  Range (min … max):    51.1 ms …  61.2 ms    52 runs
 
Summary
  'new' ran
    1.33 ± 0.09 times faster than 'old'
And for LLVM:
$ hyperfine --warmup=3 \
> -n 'old' '~/_git/git-sparse-checkout-clean/git log -100 -- llvm/lib/Support/CommandLine.cpp llvm/lib/Support/CommandLine.h' \
> -n 'new' '~/_git/git/git log -100 -- llvm/lib/Support/CommandLine.cpp llvm/lib/Support/CommandLine.h'
Benchmark 1: old
  Time (mean ± σ):      1.974 s ±  0.006 s    [User: 1.877 s, System: 0.097 s]
  Range (min … max):    1.960 s …  1.983 s    10 runs
 
Benchmark 2: new
  Time (mean ± σ):     262.9 ms ±   2.4 ms    [User: 214.2 ms, System: 48.4 ms]
  Range (min … max):   257.7 ms … 266.2 ms    11 runs
 
Summary
  'new' ran
    7.51 ± 0.07 times faster than 'old'
[1] https://github.com/sharkdp/hyperfine

Finally, putting these performance numbers in the commit message will make the results permanently findable in the repo history.

> 3:  c042907b92 ! 3:  60a3b16bbb bloom: replace struct bloom_key * with struct bloom_keyvec
Show 43 quoted lines
>     -+struct bloom_keyvec *bloom_keyvec_new(size_t count)
>     ++struct bloom_keyvec *bloom_keyvec_new(const char *path, size_t len,
>     ++				      const struct bloom_filter_settings *settings)
>      +{
>      +	struct bloom_keyvec *vec;
>     -+	size_t sz = sizeof(struct bloom_keyvec);
>     -+	sz += count * sizeof(struct bloom_key);
>     ++	const char *p;
>     ++	size_t sz;
>     ++	size_t nr = 1;
>     ++
>     ++	p = path;
>     ++	while (*p) {
>     ++		/*
>     ++		 * At this point, the path is normalized to use Unix-style
>     ++		 * path separators. This is required due to how the
>     ++		 * changed-path Bloom filters store the paths.
>     ++		 */
>     ++		if (*p == '/')
>     ++			nr++;
>     ++		p++;
>     ++	}
>     ++
>     ++	sz = sizeof(struct bloom_keyvec);
>     ++	sz += nr * sizeof(struct bloom_key);
>      +	vec = (struct bloom_keyvec *)xcalloc(1, sz);
>     -+	vec->count = count;
>     ++	if (!vec)
>     ++		return NULL;
>     ++	vec->count = nr;
>     ++
>     ++	bloom_key_fill(&vec->key[0], path, len, settings);
>     ++	nr = 1;
>     ++	p = path + len - 1;
>     ++	while (p > path) {
>     ++		if (*p == '/') {
>     ++			bloom_key_fill(&vec->key[nr++], path, p - path, settings);
>     ++		}
>     ++		p--;
>     ++	}
>     ++	assert(nr == vec->count);
>      +	return vec;
>      +}

These additions to bloom_keyvec_new() certainly help simplify some of the code movement I was talking about, but there is more that can be done to simplify patch 4. I'll send a couple example patches in reply to patch 4 with what I mean.

Overall, the patch series is correct. My complaints are stylistic more than anything.

Thanks, -Stolee

Previous: Lidong YanNext: Lidong Yan
Message 47 of 72 in “bloom: use bloom filter given multiple pathspec”
  1. 0/2 bloom: use bloom filter given multiple pathspecLidong Yan, Jun 25, 2025
  2. 1/2 bloom: replace struct bloom_key * with struct bloom_keyvecLidong Yan, Jun 25, 2025
  3. Junio C HamanoJun 25, 2025
  4. Lidong YanJun 26, 2025
  5. 2/2 bloom: enable multiple pathspec bloom keysLidong Yan, Jun 25, 2025
  6. Junio C HamanoJun 27, 2025
  7. Lidong YanJun 27, 2025
  8. Junio C HamanoJun 27, 2025
  9. Lidong YanJul 1, 2025
  10. Junio C HamanoJul 1, 2025
  11. Lidong YanJul 2, 2025
  12. Junio C HamanoJul 2, 2025
  13. Lidong YanJul 3, 2025
  14. Lidong YanJul 4, 2025
  15. SZEDER GáborJul 1, 2025
  16. Lidong YanJul 1, 2025
  17. Junio C HamanoJul 1, 2025
  18. Junio C HamanoJun 27, 2025
  19. Lidong YanJun 28, 2025
  20. Junio C HamanoJun 25, 2025
  21. Lidong YanJun 26, 2025
  22. Junio C HamanoJun 26, 2025
  23. 0/2 bloom: enable bloom filter optimization for multiple pathspec elements in revision traversalLidong Yan, Jun 27, 2025
  24. 0/2 bloom: enable bloom filter optimization for multiple pathspec elements in revision traversalLidong Yan, Jun 28, 2025
  25. 0/4 bloom: enable bloom filter optimization for multiple pathspec elements in revision traversalLidong Yan, Jul 4, 2025
  26. 1/4 bloom: add test helper to return murmur3 hashLidong Yan, Jul 4, 2025
  27. 2/4 bloom: rename function operates on bloom_keyLidong Yan, Jul 4, 2025
  28. 3/4 bloom: replace struct bloom_key * with struct bloom_keyvecLidong Yan, Jul 4, 2025
  29. Derrick StoleeJul 7, 2025
  30. Lidong YanJul 7, 2025
  31. 4/4 bloom: optimize multiple pathspec items in revision traversalLidong Yan, Jul 4, 2025
  32. Derrick StoleeJul 7, 2025
  33. Lidong YanJul 7, 2025
  34. Junio C HamanoJul 7, 2025
  35. 0/4 bloom: enable bloom filter optimization for multiple pathspec elements in revision traversalLidong Yan, Jul 10, 2025
  36. 1/4 bloom: add test helper to return murmur3 hashLidong Yan, Jul 10, 2025
  37. 2/4 bloom: rename function operates on bloom_keyLidong Yan, Jul 10, 2025
  38. 3/4 bloom: replace struct bloom_key * with struct bloom_keyvecLidong Yan, Jul 10, 2025
  39. Junio C HamanoJul 10, 2025
  40. Lidong YanJul 11, 2025
  41. Junio C HamanoJul 11, 2025
  42. 4/4 bloom: optimize multiple pathspec items in revision traversalLidong Yan, Jul 10, 2025
  43. 5/4 revision: make helper for pathspec to bloom keyDerrick Stolee, Jul 10, 2025
  44. Lidong YanJul 10, 2025
  45. 4/4 bloom: optimize multiple pathspec items in revisionDerrick Stolee, Jul 10, 2025
  46. Lidong YanJul 10, 2025
  47. Derrick StoleeJul 10, 2025
  48. 0/5 bloom: enable bloom filter optimization for multiple pathspec elements in revision traversalLidong Yan, Jul 12, 2025
  49. 1/5 bloom: add test helper to return murmur3 hashLidong Yan, Jul 12, 2025
  50. 2/5 bloom: rename function operates on bloom_keyLidong Yan, Jul 12, 2025
  51. 3/5 bloom: replace struct bloom_key * with struct bloom_keyvecLidong Yan, Jul 12, 2025
  52. 4/5 revision: make helper for pathspec to bloom keyvecLidong Yan, Jul 12, 2025
  53. 5/5 To enable optimize multiple pathspec items in revision traversal, return 0 if all pathspec item is literal in forbid_bloom_filters(). Add for loops to initialize and check each pathspec item's bloom_keyvec when optimization is possible.Lidong Yan, Jul 12, 2025
  54. Lidong YanJul 12, 2025
  55. 5/5 bloom: optimize multiple pathspec items in revisionLidong Yan, Jul 12, 2025
  56. Derrick StoleeJul 14, 2025
  57. Junio C HamanoJul 14, 2025
  58. Lidong YanJul 15, 2025
  59. [RESEND][PATCH v6 5/5] bloom: optimize multiple pathspec items in revisionLidong Yan, Jul 15, 2025
  60. Derrick StoleeJul 14, 2025
  61. Junio C HamanoJul 14, 2025
  62. Lidong YanJul 15, 2025
  63. Derrick StoleeJul 15, 2025
  64. Junio C HamanoJul 15, 2025
  65. 1/2 bloom: replace struct bloom_key * with struct bloom_keyvecLidong Yan, Jun 28, 2025
  66. Patrick SteinhardtJul 2, 2025
  67. Lidong YanJul 2, 2025
  68. Junio C HamanoJul 2, 2025
  69. Lidong YanJul 3, 2025
  70. 2/2 bloom: optimize multiple pathspec items in revision traversalLidong Yan, Jun 28, 2025
  71. 1/2 bloom: replace struct bloom_key * with struct bloom_keyvecLidong Yan, Jun 27, 2025
  72. 2/2 bloom: optimize multiple pathspec items in revision traversalLidong Yan, Jun 27, 2025

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.