git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [PATCH v3 13/17] commit-graph.c: unconditionally load Bloom filters

From
Patrick Steinhardt <ps@pks.im>
Date
Oct 17, 2023, 08:45 UTC
Message-ID
<ZS5JqdUmUWZ8OMoo@tanuki>
In-Reply-To
<09d8669c3a074e7a2ace9d650a345244b2362f7e.1696969994.git.me@ttaylorr.com>
On Tue, Oct 10, 2023 at 04:33:59PM -0400, Taylor Blau wrote:
> In 9e4df4da07 (commit-graph: new filter ver. that fixes murmur3,
Nit: It's a bit funny to read this reference to a commit ID when the
commit in question is part of the same series. Isn't it likely to grow
stale?
Show 73 quoted lines
> 2023-08-01), we began ignoring the Bloom data ("BDAT") chunk for
> commit-graphs whose Bloom filters were computed using a hash version
> incompatible with the value of `commitGraph.changedPathVersion`.
> 
> Now that the Bloom API has been hardened to discard these incompatible
> filters (with the exception of low-level APIs), we can safely load these
> Bloom filters unconditionally.
> 
> We no longer want to return early from `graph_read_bloom_data()`, and
> similarly do not want to set the bloom_settings' `hash_version` field as
> a side-effect. The latter is because we want to wait until we know which
> Bloom settings we're using (either the defaults, from the GIT_TEST
> variables, or from the previous commit-graph layer) before deciding what
> hash_version to use.
> 
> If we detect an existing BDAT chunk, we'll infer the rest of the
> settings (e.g., number of hashes, bits per entry, and maximum number of
> changed paths) from the earlier graph layer. The hash_version will be
> inferred from the previous layer as well, unless one has already been
> specified via configuration.
> 
> Once all of that is done, we normalize the value of the hash_version to
> either "1" or "2".
> 
> Signed-off-by: Taylor Blau <me@ttaylorr.com>
> ---
>  commit-graph.c | 19 ++++++++++---------
>  1 file changed, 10 insertions(+), 9 deletions(-)
> 
> diff --git a/commit-graph.c b/commit-graph.c
> index db623afd09..fa3b58e762 100644
> --- a/commit-graph.c
> +++ b/commit-graph.c
> @@ -327,12 +327,6 @@ static int graph_read_bloom_data(const unsigned char *chunk_start,
>  	uint32_t hash_version;
>  	hash_version = get_be32(chunk_start);
>  
> -	if (*c->commit_graph_changed_paths_version == -1) {
> -		*c->commit_graph_changed_paths_version = hash_version;
> -	} else if (hash_version != *c->commit_graph_changed_paths_version) {
> -		return 0;
> -	}
> -
>  	g->chunk_bloom_data = chunk_start;
>  	g->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));
>  	g->bloom_filter_settings->hash_version = hash_version;
> @@ -2473,8 +2467,7 @@ int write_commit_graph(struct object_directory *odb,
>  	ctx->write_generation_data = (get_configured_generation_version(r) == 2);
>  	ctx->num_generation_data_overflows = 0;
>  
> -	bloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2
> -		? 2 : 1;
> +	bloom_settings.hash_version = r->settings.commit_graph_changed_paths_version;
>  	bloom_settings.bits_per_entry = git_env_ulong("GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY",
>  						      bloom_settings.bits_per_entry);
>  	bloom_settings.num_hashes = git_env_ulong("GIT_TEST_BLOOM_SETTINGS_NUM_HASHES",
> @@ -2506,10 +2499,18 @@ int write_commit_graph(struct object_directory *odb,
>  		/* We have changed-paths already. Keep them in the next graph */
>  		if (g && g->bloom_filter_settings) {
>  			ctx->changed_paths = 1;
> -			ctx->bloom_settings = g->bloom_filter_settings;
> +
> +			/* don't propagate the hash_version unless unspecified */
> +			if (bloom_settings.hash_version == -1)
> +				bloom_settings.hash_version = g->bloom_filter_settings->hash_version;
> +			bloom_settings.bits_per_entry = g->bloom_filter_settings->bits_per_entry;
> +			bloom_settings.num_hashes = g->bloom_filter_settings->num_hashes;
> +			bloom_settings.max_changed_paths = g->bloom_filter_settings->max_changed_paths;
>  		}
>  	}
>  
> +	bloom_settings.hash_version = bloom_settings.hash_version == 2 ? 2 : 1;
> +

What if there is a future version of Git that writes Bloom filters with hash version 3? Should we really normalize that to `1`?

Patrick
Show 6 quoted lines
>  	if (ctx->split) {
>  		struct commit_graph *g = ctx->r->objects->commit_graph;
>  
> -- 
> 2.42.0.342.g8bb3a896ee
> 
Previous: Taylor BlauNext: Taylor Blau
Message 64 of 76 in “bloom: changed-path Bloom filters v2”
  1. 00/15 bloom: changed-path Bloom filters v2Taylor Blau, Aug 21, 2023
  2. 01/15 gitformat-commit-graph: describe version 2 of BDATTaylor Blau, Aug 21, 2023
  3. 02/15 t/helper/test-read-graph.c: extract `dump_graph_info()`Taylor Blau, Aug 21, 2023
  4. 03/15 bloom.h: make `load_bloom_filter_from_graph()` publicTaylor Blau, Aug 21, 2023
  5. 04/15 t/helper/test-read-graph: implement `bloom-filters` modeTaylor Blau, Aug 21, 2023
  6. 05/15 t4216: test changed path filters with high bit pathsTaylor Blau, Aug 21, 2023
  7. 06/15 repo-settings: introduce commitgraph.changedPathsVersionTaylor Blau, Aug 21, 2023
  8. 07/15 commit-graph: new filter ver. that fixes murmur3Taylor Blau, Aug 21, 2023
  9. SZEDER GáborAug 26, 2023
  10. Jonathan TanAug 29, 2023
  11. SZEDER GáborAug 30, 2023
  12. Jonathan TanSep 1, 2023
  13. Taylor BlauSep 25, 2023
  14. SZEDER GáborOct 8, 2023
  15. Taylor BlauOct 9, 2023
  16. Taylor BlauOct 9, 2023
  17. Junio C HamanoOct 9, 2023
  18. Taylor BlauOct 10, 2023
  19. 08/15 bloom: annotate filters with hash versionTaylor Blau, Aug 21, 2023
  20. 09/15 bloom: prepare to discard incompatible Bloom filtersTaylor Blau, Aug 21, 2023
  21. 10/15 t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`Taylor Blau, Aug 21, 2023
  22. 11/15 commit-graph.c: unconditionally load Bloom filtersTaylor Blau, Aug 21, 2023
  23. 12/15 commit-graph: drop unnecessary `graph_read_bloom_data_context`Taylor Blau, Aug 21, 2023
  24. 13/15 object.h: fix mis-aligned flag bits tableTaylor Blau, Aug 21, 2023
  25. 14/15 commit-graph: reuse existing Bloom filters where possibleTaylor Blau, Aug 21, 2023
  26. 15/15 bloom: introduce `deinit_bloom_filters()`Taylor Blau, Aug 21, 2023
  27. Jonathan TanAug 24, 2023
  28. Jonathan TanAug 25, 2023
  29. Jonathan TanAug 29, 2023
  30. Junio C HamanoAug 29, 2023
  31. 00/15 bloom: changed-path Bloom filters v2Jonathan Tan, Aug 30, 2023
  32. 13/15 object.h: fix mis-aligned flag bits tableJonathan Tan, Aug 30, 2023
  33. 06/15 repo-settings: introduce commitgraph.changedPathsVersionJonathan Tan, Aug 30, 2023
  34. 04/15 t/helper/test-read-graph: implement `bloom-filters` modeJonathan Tan, Aug 30, 2023
  35. 08/15 bloom: annotate filters with hash versionJonathan Tan, Aug 30, 2023
  36. 01/15 gitformat-commit-graph: describe version 2 of BDATJonathan Tan, Aug 30, 2023
  37. 11/15 commit-graph.c: unconditionally load Bloom filtersJonathan Tan, Aug 30, 2023
  38. 10/15 t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`Jonathan Tan, Aug 30, 2023
  39. 09/15 bloom: prepare to discard incompatible Bloom filtersJonathan Tan, Aug 30, 2023
  40. 12/15 commit-graph: drop unnecessary `graph_read_bloom_data_context`Jonathan Tan, Aug 30, 2023
  41. 07/15 commit-graph: new filter ver. that fixes murmur3Jonathan Tan, Aug 30, 2023
  42. 02/15 t/helper/test-read-graph.c: extract `dump_graph_info()`Jonathan Tan, Aug 30, 2023
  43. 05/15 t4216: test changed path filters with high bit pathsJonathan Tan, Aug 30, 2023
  44. 14/15 commit-graph: reuse existing Bloom filters where possibleJonathan Tan, Aug 30, 2023
  45. 03/15 bloom.h: make `load_bloom_filter_from_graph()` publicJonathan Tan, Aug 30, 2023
  46. 15/15 bloom: introduce `deinit_bloom_filters()`Jonathan Tan, Aug 30, 2023
  47. Junio C HamanoAug 30, 2023
  48. 00/17 bloom: changed-path Bloom filters v2 (& sundries)Taylor Blau, Oct 10, 2023
  49. 01/17 t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`Taylor Blau, Oct 10, 2023
  50. 02/17 revision.c: consult Bloom filters for root commitsTaylor Blau, Oct 10, 2023
  51. 04/17 gitformat-commit-graph: describe version 2 of BDATTaylor Blau, Oct 10, 2023
  52. 05/17 t/helper/test-read-graph.c: extract `dump_graph_info()`Taylor Blau, Oct 10, 2023
  53. Patrick SteinhardtOct 17, 2023
  54. Taylor BlauOct 18, 2023
  55. Junio C HamanoOct 18, 2023
  56. 03/17 commit-graph: ensure Bloom filters are read with consistent settingsTaylor Blau, Oct 10, 2023
  57. Patrick SteinhardtOct 17, 2023
  58. 06/17 bloom.h: make `load_bloom_filter_from_graph()` publicTaylor Blau, Oct 10, 2023
  59. 07/17 t/helper/test-read-graph: implement `bloom-filters` modeTaylor Blau, Oct 10, 2023
  60. 09/17 repo-settings: introduce commitgraph.changedPathsVersionTaylor Blau, Oct 10, 2023
  61. 15/17 object.h: fix mis-aligned flag bits tableTaylor Blau, Oct 10, 2023
  62. 12/17 bloom: prepare to discard incompatible Bloom filtersTaylor Blau, Oct 10, 2023
  63. 13/17 commit-graph.c: unconditionally load Bloom filtersTaylor Blau, Oct 10, 2023
  64. Patrick SteinhardtOct 17, 2023
  65. 11/17 bloom: annotate filters with hash versionTaylor Blau, Oct 10, 2023
  66. 14/17 commit-graph: drop unnecessary `graph_read_bloom_data_context`Taylor Blau, Oct 10, 2023
  67. 10/17 commit-graph: new filter ver. that fixes murmur3Taylor Blau, Oct 10, 2023
  68. Patrick SteinhardtOct 17, 2023
  69. Taylor BlauOct 18, 2023
  70. 08/17 t4216: test changed path filters with high bit pathsTaylor Blau, Oct 10, 2023
  71. Patrick SteinhardtOct 17, 2023
  72. Taylor BlauOct 18, 2023
  73. 16/17 commit-graph: reuse existing Bloom filters where possibleTaylor Blau, Oct 10, 2023
  74. 17/17 bloom: introduce `deinit_bloom_filters()`Taylor Blau, Oct 10, 2023
  75. Patrick SteinhardtOct 17, 2023
  76. Taylor BlauOct 18, 2023

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.