git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v6 13/16] commit-graph: new Bloom filter version that fixes murmur3

From
SZEDER Gábor <szeder.dev@gmail.com>
Date
Feb 25, 2024, 22:15 UTC
Message-ID
<20240225221524.GA1940392@szeder.dev>
In-Reply-To
<d2f11c082d3bf10d9127c330a7d59b7e47ac4f21.1706741516.git.me@ttaylorr.com>
On Wed, Jan 31, 2024 at 05:52:54PM -0500, Taylor Blau wrote:
Show 12 quoted lines
> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh
> index 93b8d096cf..a7bf3a7dca 100755
> --- a/t/t4216-log-bloom.sh
> +++ b/t/t4216-log-bloom.sh
> @@ -485,6 +485,33 @@ test_expect_success 'merge graph layers with incompatible Bloom settings' '
>  	test_must_be_empty err
>  '
>  
> +test_expect_success 'ensure Bloom filter with incompatible versions are ignored' '
> +	rm "$repo/$graph" &&
> +
> +	git -C $repo log --oneline --no-decorate -- $CENT >expect &&

The $CENT variable is not set at this point in the test script. However, even if it were, the repository used in this test case doesn't contain any file with that name.

Show 21 quoted lines
> +
> +	# Compute v1 Bloom filters for commits at the bottom.
> +	git -C $repo rev-parse HEAD^ >in &&
> +	git -C $repo commit-graph write --stdin-commits --changed-paths \
> +		--split <in &&
> +
> +	# Compute v2 Bloomfilters for the rest of the commits at the top.
> +	git -C $repo rev-parse HEAD >in &&
> +	git -C $repo -c commitGraph.changedPathsVersion=2 commit-graph write \
> +		--stdin-commits --changed-paths --split=no-merge <in &&
> +
> +	test_line_count = 2 $repo/$chain &&
> +
> +	git -C $repo log --oneline --no-decorate -- $CENT >actual 2>err &&
> +	test_cmp expect actual &&
> +
> +	layer="$(head -n 1 $repo/$chain)" &&
> +	cat >expect.err <<-EOF &&
> +	warning: disabling Bloom filters for commit-graph layer $SQ$layer$SQ due to incompatible settings
> +	EOF
> +	test_cmp expect.err err

The variable $repo is used 9 times in this test case. I think it would be simpler and easier follow if it used a ( cd $repo && ... ) subshell, like many of the previous tests in this test script,

Show 115 quoted lines
> +'
> +
>  get_first_changed_path_filter () {
>  	test-tool read-graph bloom-filters >filters.dat &&
>  	head -n 1 filters.dat
> @@ -536,6 +563,120 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '
>  	)
>  '
>  
> +test_expect_success 'version 1 changed-path not used when version 2 requested' '
> +	(
> +		cd highbit1 &&
> +		git config --add commitgraph.changedPathsVersion 2 &&
> +		test_bloom_filters_not_used "-- another$CENT"
> +	)
> +'
> +
> +test_expect_success 'version 1 changed-path used when autodetect requested' '
> +	(
> +		cd highbit1 &&
> +		git config --add commitgraph.changedPathsVersion -1 &&
> +		test_bloom_filters_used "-- another$CENT"
> +	)
> +'
> +
> +test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '
> +	test_commit -C highbit1 c1double "$CENT$CENT" &&
> +	git -C highbit1 commit-graph write --reachable --changed-paths &&
> +	(
> +		cd highbit1 &&
> +		git config --add commitgraph.changedPathsVersion -1 &&
> +		echo "options: bloom(1,10,7) read_generation_data" >expect &&
> +		test-tool read-graph >full &&
> +		grep options full >actual &&
> +		test_cmp expect actual
> +	)
> +'
> +
> +test_expect_success 'set up repo with high bit path, version 2 changed-path' '
> +	git init highbit2 &&
> +	git -C highbit2 config --add commitgraph.changedPathsVersion 2 &&
> +	test_commit -C highbit2 c2 "$CENT" &&
> +	git -C highbit2 commit-graph write --reachable --changed-paths
> +'
> +
> +test_expect_success 'check value of version 2 changed-path' '
> +	(
> +		cd highbit2 &&
> +		echo "c01f" >expect &&
> +		get_first_changed_path_filter >actual &&
> +		test_cmp expect actual
> +	)
> +'
> +
> +test_expect_success 'setup make another commit' '
> +	# "git log" does not use Bloom filters for root commits - see how, in
> +	# revision.c, rev_compare_tree() (the only code path that eventually calls
> +	# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit
> +	# has one parent. Therefore, make another commit so that we perform the tests on
> +	# a non-root commit.
> +	test_commit -C highbit2 anotherc2 "another$CENT"
> +'
> +
> +test_expect_success 'version 2 changed-path used when version 2 requested' '
> +	(
> +		cd highbit2 &&
> +		test_bloom_filters_used "-- another$CENT"
> +	)
> +'
> +
> +test_expect_success 'version 2 changed-path not used when version 1 requested' '
> +	(
> +		cd highbit2 &&
> +		git config --add commitgraph.changedPathsVersion 1 &&
> +		test_bloom_filters_not_used "-- another$CENT"
> +	)
> +'
> +
> +test_expect_success 'version 2 changed-path used when autodetect requested' '
> +	(
> +		cd highbit2 &&
> +		git config --add commitgraph.changedPathsVersion -1 &&
> +		test_bloom_filters_used "-- another$CENT"
> +	)
> +'
> +
> +test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '
> +	test_commit -C highbit2 c2double "$CENT$CENT" &&
> +	git -C highbit2 commit-graph write --reachable --changed-paths &&
> +	(
> +		cd highbit2 &&
> +		git config --add commitgraph.changedPathsVersion -1 &&
> +		echo "options: bloom(2,10,7) read_generation_data" >expect &&
> +		test-tool read-graph >full &&
> +		grep options full >actual &&
> +		test_cmp expect actual
> +	)
> +'
> +
> +test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '
> +	git init doublewrite &&
> +	test_commit -C doublewrite c "$CENT" &&
> +	git -C doublewrite config --add commitgraph.changedPathsVersion 1 &&
> +	git -C doublewrite commit-graph write --reachable --changed-paths &&
> +	for v in -2 3
> +	do
> +		git -C doublewrite config --add commitgraph.changedPathsVersion $v &&
> +		git -C doublewrite commit-graph write --reachable --changed-paths 2>err &&
> +		cat >expect <<-EOF &&
> +		warning: attempting to write a commit-graph, but ${SQ}commitgraph.changedPathsVersion${SQ} ($v) is not supported
> +		EOF
> +		test_cmp expect err || return 1
> +	done &&
> +	git -C doublewrite config --add commitgraph.changedPathsVersion 2 &&
> +	git -C doublewrite commit-graph write --reachable --changed-paths &&

The path 'doublewrite' is used 8 times in this test case before finally cd-ing into that directory in a subshell...

Show 14 quoted lines
> +	(
> +		cd doublewrite &&
> +		echo "c01f" >expect &&
> +		get_first_changed_path_filter >actual &&
> +		test_cmp expect actual
> +	)
> +'
> +
>  corrupt_graph () {
>  	test_when_finished "rm -rf $graph" &&
>  	git commit-graph write --reachable --changed-paths &&
> -- 
> 2.43.0.509.g253f65a7fc
> 
Previous: Taylor BlauNext: Jiang Xin
Message 15 of 36 in “bloom: changed-path Bloom filters v2 (& sundries)”
  1. 00/16 bloom: changed-path Bloom filters v2 (& sundries)Taylor Blau, Jan 31, 2024
  2. 01/16 t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`Taylor Blau, Jan 31, 2024
  3. 02/16 revision.c: consult Bloom filters for root commitsTaylor Blau, Jan 31, 2024
  4. 03/16 commit-graph: ensure Bloom filters are read with consistent settingsTaylor Blau, Jan 31, 2024
  5. 04/16 gitformat-commit-graph: describe version 2 of BDATTaylor Blau, Jan 31, 2024
  6. 05/16 t/helper/test-read-graph.c: extract `dump_graph_info()`Taylor Blau, Jan 31, 2024
  7. 06/16 bloom.h: make `load_bloom_filter_from_graph()` publicTaylor Blau, Jan 31, 2024
  8. 07/16 t/helper/test-read-graph: implement `bloom-filters` modeTaylor Blau, Jan 31, 2024
  9. 08/16 t4216: test changed path filters with high bit pathsTaylor Blau, Jan 31, 2024
  10. 09/16 repo-settings: introduce commitgraph.changedPathsVersionTaylor Blau, Jan 31, 2024
  11. 10/16 bloom: annotate filters with hash versionTaylor Blau, Jan 31, 2024
  12. 11/16 bloom: prepare to discard incompatible Bloom filtersTaylor Blau, Jan 31, 2024
  13. 12/16 commit-graph: unconditionally load Bloom filtersTaylor Blau, Jan 31, 2024
  14. 13/16 commit-graph: new Bloom filter version that fixes murmur3Taylor Blau, Jan 31, 2024
  15. SZEDER GáborFeb 25, 2024
  16. Jiang XinFeb 28, 2024
  17. 14/16 object.h: fix mis-aligned flag bits tableTaylor Blau, Jan 31, 2024
  18. 15/16 commit-graph: reuse existing Bloom filters where possibleTaylor Blau, Jan 31, 2024
  19. 16/16 bloom: introduce `deinit_bloom_filters()`Taylor Blau, Jan 31, 2024
  20. 00/16 bloom: changed-path Bloom filters v2 (& sundries)Taylor Blau, Jun 25, 2024
  21. 01/16 t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`Taylor Blau, Jun 25, 2024
  22. 02/16 revision.c: consult Bloom filters for root commitsTaylor Blau, Jun 25, 2024
  23. 03/16 commit-graph: ensure Bloom filters are read with consistent settingsTaylor Blau, Jun 25, 2024
  24. 04/16 gitformat-commit-graph: describe version 2 of BDATTaylor Blau, Jun 25, 2024
  25. 05/16 t/helper/test-read-graph.c: extract `dump_graph_info()`Taylor Blau, Jun 25, 2024
  26. 06/16 bloom.h: make `load_bloom_filter_from_graph()` publicTaylor Blau, Jun 25, 2024
  27. 07/16 t/helper/test-read-graph: implement `bloom-filters` modeTaylor Blau, Jun 25, 2024
  28. 08/16 t4216: test changed path filters with high bit pathsTaylor Blau, Jun 25, 2024
  29. 09/16 repo-settings: introduce commitgraph.changedPathsVersionTaylor Blau, Jun 25, 2024
  30. 10/16 bloom: annotate filters with hash versionTaylor Blau, Jun 25, 2024
  31. 11/16 bloom: prepare to discard incompatible Bloom filtersTaylor Blau, Jun 25, 2024
  32. 12/16 commit-graph: unconditionally load Bloom filtersTaylor Blau, Jun 25, 2024
  33. 13/16 commit-graph: new Bloom filter version that fixes murmur3Taylor Blau, Jun 25, 2024
  34. 14/16 object.h: fix mis-aligned flag bits tableTaylor Blau, Jun 25, 2024
  35. 15/16 commit-graph: reuse existing Bloom filters where possibleTaylor Blau, Jun 25, 2024
  36. 16/16 bloom: introduce `deinit_bloom_filters()`Taylor Blau, Jun 25, 2024

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.