{"thread":{"id":"60142","subject":"[PATCH 00/15] bloom: changed-path Bloom filters v2","startedAt":"2023-08-21T21:44:00Z","lastAt":"2023-10-18T23:56:21Z","messageCount":76,"participants":["Taylor Blau","Jonathan Tan","SZEDER Gábor","Junio C Hamano","Patrick Steinhardt"],"isPatch":true,"patchVersion":1,"patchTotal":15},"messages":[{"id":"480887","messageId":"cover.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":null,"subject":"[PATCH 00/15] bloom: changed-path Bloom filters v2","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:43:56Z","receivedAt":"2023-08-21T21:44:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"This series combines the efforts in [1] and [2] to culminate in v2 of\nthe changed-path Bloom filter format (which uses a bona-fide murmur3\nimplementation) as well as an efficient upgrade path for repositories\nwith few non-ASCII paths.\n\nThe first seven patches are from [1], and are unchanged from that\nversion. Most of the remaining patches are from [2], and they have been\nmodified based on the review therein.\n\nThe final patch is new, and avoids leaking memory when the Bloom\nsubsystem is initialized both in the read- and read/write-cases.\n\nThanks to Jonathan and Peff who have both helped a great deal in putting\nthese patches together. And, as always, thanks in advance for your\nreview!\n\n[1]: https://lore.kernel.org/git/cover.1684790529.git.jonathantanmy@google.com/\n[2]: https://lore.kernel.org/git/cover.1691426160.git.me@ttaylorr.com/\n\nJonathan Tan (4):\n gitformat-commit-graph: describe version 2 of BDAT\n t4216: test changed path filters with high bit paths\n repo-settings: introduce commitgraph.changedPathsVersion\n commit-graph: new filter ver. that fixes murmur3\n\nTaylor Blau (11):\n t/helper/test-read-graph.c: extract `dump_graph_info()`\n bloom.h: make `load_bloom_filter_from_graph()` public\n t/helper/test-read-graph: implement `bloom-filters` mode\n bloom: annotate filters with hash version\n bloom: prepare to discard incompatible Bloom filters\n t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`\n commit-graph.c: unconditionally load Bloom filters\n commit-graph: drop unnecessary `graph_read_bloom_data_context`\n object.h: fix mis-aligned flag bits table\n commit-graph: reuse existing Bloom filters where possible\n bloom: introduce `deinit_bloom_filters()`\n\n Documentation/config/commitgraph.txt | 26 ++-\n Documentation/gitformat-commit-graph.txt | 9 +-\n bloom.c | 208 +++++++++++++++++++++--\n bloom.h | 38 ++++-\n commit-graph.c | 36 +++-\n object.h | 3 +-\n oss-fuzz/fuzz-commit-graph.c | 2 +-\n repo-settings.c | 6 +-\n repository.h | 2 +-\n t/helper/test-bloom.c | 9 +-\n t/helper/test-read-graph.c | 67 ++++++--\n t/t0095-bloom.sh | 8 +\n t/t4216-log-bloom.sh | 184 +++++++++++++++++++-\n 13 files changed, 548 insertions(+), 50 deletions(-)\n\n-- \n2.42.0.4.g52b49bb434\n"},{"id":"480888","messageId":"0241b6e18867dab34c8dd8b49a861884e9818d37.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 01/15] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:43:59Z","receivedAt":"2023-08-21T21:44:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jonathan Tan \n\nThe code change to Git to support version 2 will be done in subsequent\ncommits.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n Documentation/gitformat-commit-graph.txt | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\nindex 31cad585e2..3e906e8030 100644\n--- a/Documentation/gitformat-commit-graph.txt\n+++ b/Documentation/gitformat-commit-graph.txt\n@@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n \n ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n * It starts with header consisting of three unsigned 32-bit integers:\n- - Version of the hash algorithm being used. We currently only support\n-\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n+ - Version of the hash algorithm being used. We currently support\n+\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n \timplemented exactly as described in\n \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n-\tin Probabilistic Verification\"\n+\tin Probabilistic Verification\". Version 1 Bloom filters have a bug that appears\n+\twhen char is signed and the repository has path names that have characters >=\n+\t0x80; Git supports reading and writing them, but this ability will be removed\n+\tin a future version of Git.\n - The number of times a path is hashed and hence the number of bit positions\n \t that cumulatively determine whether a file is present in the commit.\n - The minimum number of bits 'b' per entry in the Bloom filter. If the filter\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480889","messageId":"9c129c733e689f02abafdc97011395653c2708d4.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 02/15] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:02Z","receivedAt":"2023-08-21T21:44:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for the 'read-graph' test helper to perform other tasks besides\ndumping high-level information about the commit-graph by extracting its\nmain routine into a separate function.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 31 ++++++++++++++++++-------------\n 1 file changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 8c7a83f578..3375392f6c 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -5,20 +5,8 @@\n #include \"bloom.h\"\n #include \"setup.h\"\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_info(struct commit_graph *graph)\n {\n-\tstruct commit_graph *graph = NULL;\n-\tstruct object_directory *odb;\n-\n-\tsetup_git_directory();\n-\todb = the_repository->objects->odb;\n-\n-\tprepare_repo_settings(the_repository);\n-\n-\tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n@@ -57,6 +45,23 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tif (graph->topo_levels)\n \t\tprintf(\" topo_levels\");\n \tprintf(\"\\n\");\n+}\n+\n+int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tstruct object_directory *odb;\n+\n+\tsetup_git_directory();\n+\todb = the_repository->objects->odb;\n+\n+\tprepare_repo_settings(the_repository);\n+\n+\tgraph = read_commit_graph_one(the_repository, odb);\n+\tif (!graph)\n+\t\treturn 1;\n+\n+\tdump_graph_info(graph);\n \n \tUNLEAK(graph);\n \n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480890","messageId":"766feac8cb8e796eaf99ecd4b92ff66293356814.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 03/15] bloom.h: make `load_bloom_filter_from_graph()` public","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:04Z","receivedAt":"2023-08-21T21:44:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for a future commit to use the load_bloom_filter_from_graph()\nfunction directly to load specific Bloom filters out of the commit-graph\nfor manual inspection (to be used during tests).\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n bloom.c | 6 +++---\n bloom.h | 5 +++++\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex aef6b5fea2..3e78cfe79d 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -29,9 +29,9 @@ static inline unsigned char get_bitmask(uint32_t pos)\n \treturn ((unsigned char)1) << (pos & (BITS_PER_WORD - 1));\n }\n \n-static int load_bloom_filter_from_graph(struct commit_graph *g,\n-\t\t\t\t\tstruct bloom_filter *filter,\n-\t\t\t\t\tuint32_t graph_pos)\n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos)\n {\n \tuint32_t lex_pos, start_index, end_index;\n \ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..1e4f612d2c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -3,6 +3,7 @@\n \n struct commit;\n struct repository;\n+struct commit_graph;\n \n struct bloom_filter_settings {\n \t/*\n@@ -68,6 +69,10 @@ struct bloom_key {\n \tuint32_t *hashes;\n };\n \n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos);\n+\n /*\n * Calculate the murmur3 32-bit hash value for the given data\n * using the given seed.\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480891","messageId":"9a98ca09dc85d1bad79a79425fe88c83f5756c14.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 04/15] t/helper/test-read-graph: implement `bloom-filters` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:07Z","receivedAt":"2023-08-21T21:44:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement a mode of the \"read-graph\" test helper to dump out the\nhexadecimal contents of the Bloom filter(s) contained in a commit-graph.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 44 +++++++++++++++++++++++++++++++++-----\n 1 file changed, 39 insertions(+), 5 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 3375392f6c..da9ac8584d 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -47,10 +47,32 @@ static void dump_graph_info(struct commit_graph *graph)\n \tprintf(\"\\n\");\n }\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_bloom_filters(struct commit_graph *graph)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < graph->num_commits + graph->num_commits_in_base; i++) {\n+\t\tstruct bloom_filter filter = { 0 };\n+\t\tsize_t j;\n+\n+\t\tif (load_bloom_filter_from_graph(graph, &filter, i) < 0) {\n+\t\t\tfprintf(stderr, \"missing Bloom filter for graph \"\n+\t\t\t\t\"position %\"PRIu32\"\\n\", i);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (j = 0; j < filter.len; j++)\n+\t\t\tprintf(\"%02x\", filter.data[j]);\n+\t\tif (filter.len)\n+\t\t\tprintf(\"\\n\");\n+\t}\n+}\n+\n+int cmd__read_graph(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n \tstruct object_directory *odb;\n+\tint ret = 0;\n \n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n@@ -58,12 +80,24 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tprepare_repo_settings(the_repository);\n \n \tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n+\tif (!graph) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n \n-\tdump_graph_info(graph);\n+\tif (argc <= 1)\n+\t\tdump_graph_info(graph);\n+\telse if (!strcmp(argv[1], \"bloom-filters\"))\n+\t\tdump_graph_bloom_filters(graph);\n+\telse {\n+\t\tfprintf(stderr, \"unknown sub-command: '%s'\\n\", argv[1]);\n+\t\tret = 1;\n+\t}\n \n+done:\n \tUNLEAK(graph);\n \n-\treturn 0;\n+\treturn ret;\n }\n+\n+\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480892","messageId":"de585ce7f3c0bac504b59656abf444ecd360fd97.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 05/15] t4216: test changed path filters with high bit paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:10Z","receivedAt":"2023-08-21T21:44:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jonathan Tan \n\nSubsequent commits will teach Git another version of changed path\nfilter that has different behavior with paths that contain at least\none character with its high bit set, so test the existing behavior as\na baseline.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 43 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..2d4a3fefee 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -404,4 +404,47 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n+get_first_changed_path_filter () {\n+\ttest-tool read-graph bloom-filters >filters.dat &&\n+\thead -n 1 filters.dat\n+}\n+\n+# chosen to be the same under all Unicode normalization forms\n+CENT=$(printf \"\\302\\242\")\n+\n+test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n+\tgit init highbit1 &&\n+\ttest_commit -C highbit1 c1 \"$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'setup check value of version 1 changed-path' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\techo \"52a9\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+# expect will not match actual if char is unsigned by default. Write the test\n+# in this way, so that a user running this test script can still see if the two\n+# files match. (It will appear as an ordinary success if they match, and a skip\n+# if not.)\n+if test_cmp highbit1/expect highbit1/actual\n+then\n+\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n+fi\n+test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n+\t# Only the prereq matters for this test.\n+\ttrue\n+'\n+\n+test_expect_success 'version 1 changed-path used when version 1 requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\"\n+\t)\n+'\n+\n test_done\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480893","messageId":"ef5b54c53404fe5a02424e795d5722f2cf3f8ae8.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 06/15] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:12Z","receivedAt":"2023-08-21T21:44:20Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jonathan Tan \n\nA subsequent commit will introduce another version of the changed-path\nfilter in the commit graph file. In order to control which version to\nwrite (and read), a config variable is needed.\n\nTherefore, introduce this config variable. For forwards compatibility,\nteach Git to not read commit graphs when the config variable\nis set to an unsupported version. Because we teach Git this,\ncommitgraph.readChangedPaths is now redundant, so deprecate it and\ndefine its behavior in terms of the config variable we introduce.\n\nThis commit does not change the behavior of writing (Git writes changed\npath filters when explicitly instructed regardless of any config\nvariable), but a subsequent commit will restrict Git such that it will\nonly write when commitgraph.changedPathsVersion is a recognized value.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 23 ++++++++++++++++++++---\n commit-graph.c | 2 +-\n oss-fuzz/fuzz-commit-graph.c | 2 +-\n repo-settings.c | 6 +++++-\n repository.h | 2 +-\n 5 files changed, 28 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c..2dc9170622 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -9,6 +9,23 @@ commitGraph.maxNewFilters::\n \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n \n commitGraph.readChangedPaths::\n-\tIf true, then git will use the changed-path Bloom filters in the\n-\tcommit-graph file (if it exists, and they are present). Defaults to\n-\ttrue. See linkgit:git-commit-graph[1] for more information.\n+\tDeprecated. Equivalent to commitGraph.changedPathsVersion=-1 if true, and\n+\tcommitGraph.changedPathsVersion=0 if false. (If commitGraph.changedPathVersion\n+\tis also set, commitGraph.changedPathsVersion takes precedence.)\n+\n+commitGraph.changedPathsVersion::\n+\tSpecifies the version of the changed-path Bloom filters that Git will read and\n+\twrite. May be -1, 0 or 1.\n++\n+Defaults to -1.\n++\n+If -1, Git will use the version of the changed-path Bloom filters in the\n+repository, defaulting to 1 if there are none.\n++\n+If 0, Git will not read any Bloom filters, and will write version 1 Bloom\n+filters when instructed to write.\n++\n+If 1, Git will only read version 1 Bloom filters, and will write version 1\n+Bloom filters.\n++\n+See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0aa1640d15..da99f15fdf 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -401,7 +401,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_read_changed_paths) {\n+\tif (s->commit_graph_changed_paths_version) {\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\ndiff --git a/oss-fuzz/fuzz-commit-graph.c b/oss-fuzz/fuzz-commit-graph.c\nindex 2992079dd9..325c0b991a 100644\n--- a/oss-fuzz/fuzz-commit-graph.c\n+++ b/oss-fuzz/fuzz-commit-graph.c\n@@ -19,7 +19,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \t * possible.\n \t */\n \tthe_repository->settings.commit_graph_generation_version = 2;\n-\tthe_repository->settings.commit_graph_read_changed_paths = 1;\n+\tthe_repository->settings.commit_graph_changed_paths_version = 1;\n \tg = parse_commit_graph(&the_repository->settings, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 525f69c0c7..db8fe817f3 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n \tint value;\n \tconst char *strval;\n \tint manyfiles;\n+\tint read_changed_paths;\n \n \tif (!r->gitdir)\n \t\tBUG(\"Cannot add settings for uninitialized repository\");\n@@ -54,7 +55,10 @@ void prepare_repo_settings(struct repository *r)\n \t/* Commit graph config or default, does not cascade (simple) */\n \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n-\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &read_changed_paths, 1);\n+\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n+\t\t &r->settings.commit_graph_changed_paths_version,\n+\t\t read_changed_paths ? -1 : 0);\n \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n \ndiff --git a/repository.h b/repository.h\nindex 5f18486f64..f71154e12c 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -29,7 +29,7 @@ struct repo_settings {\n \n \tint core_commit_graph;\n \tint commit_graph_generation_version;\n-\tint commit_graph_read_changed_paths;\n+\tint commit_graph_changed_paths_version;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \tint command_requires_full_index;\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480894","messageId":"72150bd026befc758b41386cf571c6a33a754d4e.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:15Z","receivedAt":"2023-08-21T21:44:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jonathan Tan \n\nThe murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 (thus,\nthe changed path filters wouldn't be readable by other off-the-shelf\nimplementatios of murmur3) and even with Git binaries that were compiled\nwith different signedness of char. This bug affects both how Git writes\nchanged path filters to disk and how Git interprets changed path filters\non disk.\n\nTherefore, introduce a new version (2) of changed path filters that\ncorrects this problem. The existing version (1) is still supported and\nis still the default, but users should migrate away from it as soon\nas possible.\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter.\n\nSo this patch does not include any mechanism to \"salvage\" changed path\nfilters from repositories. There is also no \"mixed\" mode - for each\ninvocation of Git, reading and writing changed path filters are done\nwith the same version number; this version number may be explicitly\nstated (typically if the user knows which version they need) or\nautomatically determined from the version of the existing changed path\nfilters in the repository.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in t0095 was obtained from another murmur3 implementation\nusing the following Go source code:\n\n package main\n\n import \"fmt\"\n import \"github.com/spaolacci/murmur3\"\n\n func main() {\n fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n }\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 5 +-\n bloom.c | 69 +++++++++++++++++++-\n bloom.h | 8 ++-\n commit-graph.c | 32 ++++++++--\n t/helper/test-bloom.c | 9 ++-\n t/t0095-bloom.sh | 8 +++\n t/t4216-log-bloom.sh | 96 ++++++++++++++++++++++++++++\n 7 files changed, 214 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 2dc9170622..acc74a2f27 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -15,7 +15,7 @@ commitGraph.readChangedPaths::\n \n commitGraph.changedPathsVersion::\n \tSpecifies the version of the changed-path Bloom filters that Git will read and\n-\twrite. May be -1, 0 or 1.\n+\twrite. May be -1, 0, 1, or 2.\n +\n Defaults to -1.\n +\n@@ -28,4 +28,7 @@ filters when instructed to write.\n If 1, Git will only read version 1 Bloom filters, and will write version 1\n Bloom filters.\n +\n+If 2, Git will only read version 2 Bloom filters, and will write version 2\n+Bloom filters.\n++\n See linkgit:git-commit-graph[1] for more information.\ndiff --git a/bloom.c b/bloom.c\nindex 3e78cfe79d..ebef5cfd2f 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -66,7 +66,64 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n * Not considered to be cryptographically secure.\n * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n+{\n+\tconst uint32_t c1 = 0xcc9e2d51;\n+\tconst uint32_t c2 = 0x1b873593;\n+\tconst uint32_t r1 = 15;\n+\tconst uint32_t r2 = 13;\n+\tconst uint32_t m = 5;\n+\tconst uint32_t n = 0xe6546b64;\n+\tint i;\n+\tuint32_t k1 = 0;\n+\tconst char *tail;\n+\n+\tint len4 = len / sizeof(uint32_t);\n+\n+\tuint32_t k;\n+\tfor (i = 0; i < len4; i++) {\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n+\t\tk = byte1 | byte2 | byte3 | byte4;\n+\t\tk *= c1;\n+\t\tk = rotate_left(k, r1);\n+\t\tk *= c2;\n+\n+\t\tseed ^= k;\n+\t\tseed = rotate_left(seed, r2) * m + n;\n+\t}\n+\n+\ttail = (data + len4 * sizeof(uint32_t));\n+\n+\tswitch (len & (sizeof(uint32_t) - 1)) {\n+\tcase 3:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n+\t\t/*-fallthrough*/\n+\tcase 2:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n+\t\t/*-fallthrough*/\n+\tcase 1:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n+\t\tk1 *= c1;\n+\t\tk1 = rotate_left(k1, r1);\n+\t\tk1 *= c2;\n+\t\tseed ^= k1;\n+\t\tbreak;\n+\t}\n+\n+\tseed ^= (uint32_t)len;\n+\tseed ^= (seed >> 16);\n+\tseed *= 0x85ebca6b;\n+\tseed ^= (seed >> 13);\n+\tseed *= 0xc2b2ae35;\n+\tseed ^= (seed >> 16);\n+\n+\treturn seed;\n+}\n+\n+static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n {\n \tconst uint32_t c1 = 0xcc9e2d51;\n \tconst uint32_t c2 = 0x1b873593;\n@@ -131,8 +188,14 @@ void fill_bloom_key(const char *data,\n \tint i;\n \tconst uint32_t seed0 = 0x293ae76f;\n \tconst uint32_t seed1 = 0x7e646e2c;\n-\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n-\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n+\tuint32_t hash0, hash1;\n+\tif (settings->hash_version == 2) {\n+\t\thash0 = murmur3_seeded_v2(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v2(seed1, data, len);\n+\t} else {\n+\t\thash0 = murmur3_seeded_v1(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v1(seed1, data, len);\n+\t}\n \n \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n \tfor (i = 0; i < settings->num_hashes; i++)\ndiff --git a/bloom.h b/bloom.h\nindex 1e4f612d2c..138d57a86b 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -8,9 +8,11 @@ struct commit_graph;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * The newest version is 2, which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -80,7 +82,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n * Not considered to be cryptographically secure.\n * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n \n void fill_bloom_key(const char *data,\n \t\t size_t len,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex da99f15fdf..f7322c4fff 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -304,17 +304,26 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct graph_read_bloom_data_context {\n+\tstruct commit_graph *g;\n+\tint *commit_graph_changed_paths_version;\n+};\n+\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t size_t chunk_size, void *data)\n {\n-\tstruct commit_graph *g = data;\n+\tstruct graph_read_bloom_data_context *c = data;\n+\tstruct commit_graph *g = c->g;\n \tuint32_t hash_version;\n-\tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n+\tif (*c->commit_graph_changed_paths_version == -1) {\n+\t\t*c->commit_graph_changed_paths_version = hash_version;\n+\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n \t\treturn 0;\n+\t}\n \n+\tg->chunk_bloom_data = chunk_start;\n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n \tg->bloom_filter_settings->hash_version = hash_version;\n \tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n@@ -402,10 +411,14 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t}\n \n \tif (s->commit_graph_changed_paths_version) {\n+\t\tstruct graph_read_bloom_data_context context = {\n+\t\t\t.g = graph,\n+\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n+\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t graph_read_bloom_data, graph);\n+\t\t\t graph_read_bloom_data, &context);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -2376,6 +2389,13 @@ int write_commit_graph(struct object_directory *odb,\n \t}\n \tif (!commit_graph_compatible(r))\n \t\treturn 0;\n+\tif (r->settings.commit_graph_changed_paths_version < -1\n+\t || r->settings.commit_graph_changed_paths_version > 2) {\n+\t\twarning(_(\"attempting to write a commit-graph, but \"\n+\t\t\t \"'commitgraph.changedPathsVersion' (%d) is not supported\"),\n+\t\t\tr->settings.commit_graph_changed_paths_version);\n+\t\treturn 0;\n+\t}\n \n \tCALLOC_ARRAY(ctx, 1);\n \tctx->r = r;\n@@ -2388,6 +2408,8 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n+\t\t? 2 : 1;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2417,7 +2439,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex aabe31d724..3cbc0a5b50 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -50,6 +50,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \" test-tool bloom get_murmur3 \\n\"\n+\" test-tool bloom get_murmur3_seven_highbit\\n\"\n \" test-tool bloom generate_filter [...]\\n\"\n \" test-tool bloom get_filter_for_commit \\n\";\n \n@@ -64,7 +65,13 @@ int cmd__bloom(int argc, const char **argv)\n \t\tuint32_t hashed;\n \t\tif (argc < 3)\n \t\t\tusage(bloom_usage);\n-\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n+\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 2d4a3fefee..775e59d864 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -447,4 +447,100 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n \t)\n '\n \n+test_expect_success 'version 1 changed-path not used when version 2 requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 1 changed-path used when autodetect requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n+\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(1,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n+\tgit init highbit2 &&\n+\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n+\ttest_commit -C highbit2 c2 \"$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'check value of version 2 changed-path' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\techo \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path used when version 2 requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path not used when version 1 requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path used when autodetect requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n+\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(2,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n+\tgit init doublewrite &&\n+\ttest_commit -C doublewrite c \"$CENT\" &&\n+\tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n+\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n+\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd doublewrite &&\n+\t\techo \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480895","messageId":"48347796c81f03f676295cf0217715e39192a522.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 08/15] bloom: annotate filters with hash version","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:19Z","receivedAt":"2023-08-21T21:44:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, we will want to load existing Bloom filters out\nof a commit-graph, even when the hash version they were computed with\ndoes not match the value of `commitGraph.changedPathVersion`.\n\nIn order to differentiate between the two, add a \"version\" field to each\nBloom filter.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 11 ++++++++---\n bloom.h | 1 +\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex ebef5cfd2f..9b6a30f6f6 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -55,6 +55,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n \t\t\t\t\tsizeof(unsigned char) * start_index +\n \t\t\t\t\tBLOOMDATA_CHUNK_HEADER_SIZE);\n+\tfilter->version = g->bloom_filter_settings->hash_version;\n \n \treturn 1;\n }\n@@ -240,11 +241,13 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data UNUSED,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-static void init_truncated_large_filter(struct bloom_filter *filter)\n+static void init_truncated_large_filter(struct bloom_filter *filter,\n+\t\t\t\t\tint version)\n {\n \tfilter->data = xmalloc(1);\n \tfilter->data[0] = 0xFF;\n \tfilter->len = 1;\n+\tfilter->version = version;\n }\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n@@ -329,13 +332,15 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t}\n \n \t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths) {\n-\t\t\tinit_truncated_large_filter(filter);\n+\t\t\tinit_truncated_large_filter(filter,\n+\t\t\t\t\t\t settings->hash_version);\n \t\t\tif (computed)\n \t\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n \t\t\tgoto cleanup;\n \t\t}\n \n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tfilter->version = settings->hash_version;\n \t\tif (!filter->len) {\n \t\t\tif (computed)\n \t\t\t\t*computed |= BLOOM_TRUNC_EMPTY;\n@@ -355,7 +360,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n-\t\tinit_truncated_large_filter(filter);\n+\t\tinit_truncated_large_filter(filter, settings->hash_version);\n \n \t\tif (computed)\n \t\t\t*computed |= BLOOM_TRUNC_LARGE;\ndiff --git a/bloom.h b/bloom.h\nindex 138d57a86b..330a140520 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -55,6 +55,7 @@ struct bloom_filter_settings {\n struct bloom_filter {\n \tunsigned char *data;\n \tsize_t len;\n+\tint version;\n };\n \n /*\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480896","messageId":"7c5a0090b52f829cab3ab2e107029a74b7ecfa87.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 09/15] bloom: prepare to discard incompatible Bloom filters","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:25Z","receivedAt":"2023-08-21T21:44:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Callers use the inline `get_bloom_filter()` implementation as a thin\nwrapper around `get_or_compute_bloom_filter()`. The former calls the\nlatter with a value of \"0\" for `compute_if_not_present`, making\n`get_bloom_filter()` the default read-only path for fetching an existing\nBloom filter.\n\nCallers expect the value returned from `get_bloom_filter()` is usable,\nthat is that it's compatible with the configured value corresponding to\n`commitGraph.changedPathsVersion`.\n\nThis is OK, since the commit-graph machinery only initializes its BDAT\nchunk (thereby enabling it to service Bloom filter queries) when the\nBloom filter hash_version is compatible with our settings. So any value\nreturned by `get_bloom_filter()` is trivially useable.\n\nHowever, subsequent commits will load the BDAT chunk even when the Bloom\nfilters are built with incompatible hash versions. Prepare to handle\nthis by teaching `get_bloom_filter()` to discard filters that are\nincompatible with the configured hash version.\n\nCallers who wish to read incompatible filters (e.g., for upgrading\nfilters from v1 to v2) may use the lower level routine,\n`get_or_compute_bloom_filter()`.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 20 +++++++++++++++++++-\n bloom.h | 20 ++++++++++++++++++--\n 2 files changed, 37 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 9b6a30f6f6..739fa093ba 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -250,6 +250,23 @@ static void init_truncated_large_filter(struct bloom_filter *filter,\n \tfilter->version = version;\n }\n \n+struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c)\n+{\n+\tstruct bloom_filter *filter;\n+\tint hash_version;\n+\n+\tfilter = get_or_compute_bloom_filter(r, c, 0, NULL, NULL);\n+\tif (!filter)\n+\t\treturn NULL;\n+\n+\tprepare_repo_settings(r);\n+\thash_version = r->settings.commit_graph_changed_paths_version;\n+\n+\tif (!(hash_version == -1 || hash_version == filter->version))\n+\t\treturn NULL; /* unusable filter */\n+\treturn filter;\n+}\n+\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n@@ -275,7 +292,8 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t filter, graph_pos);\n \t}\n \n-\tif (filter->data && filter->len)\n+\tif ((filter->data && filter->len) &&\n+\t (!settings || settings->hash_version == filter->version))\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\ndiff --git a/bloom.h b/bloom.h\nindex 330a140520..bfe389e29c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -110,8 +110,24 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t enum bloom_filter_computed *computed);\n \n-#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n-\t(r), (c), 0, NULL, NULL)\n+/*\n+ * Find the Bloom filter associated with the given commit \"c\".\n+ *\n+ * If any of the following are true\n+ *\n+ * - the repository does not have a commit-graph, or\n+ * - the repository disables reading from the commit-graph, or\n+ * - the given commit does not have a Bloom filter computed, or\n+ * - there is a Bloom filter for commit \"c\", but it cannot be read\n+ * because the filter uses an incompatible version of murmur3\n+ *\n+ * , then `get_bloom_filter()` will return NULL. Otherwise, the corresponding\n+ * Bloom filter will be returned.\n+ *\n+ * For callers who wish to inspect Bloom filters with incompatible hash\n+ * versions, use get_or_compute_bloom_filter().\n+ */\n+struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c);\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480897","messageId":"d7c2c517de81bec237653a92033bfb45e8f9eded.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 10/15] t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:28Z","receivedAt":"2023-08-21T21:44:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The existing implementation of test_bloom_filters_not_used() asserts\nthat the Bloom filter sub-system has not been initialized at all, by\nchecking for the absence of any data from it from trace2.\n\nIn the following commit, it will become possible to load Bloom filters\nwithout using them (e.g., because `commitGraph.changedPathVersion` is\nincompatible with the hash version with which the commit-graph's Bloom\nfilters were written).\n\nWhen this is the case, it's possible to initialize the Bloom filter\nsub-system, while still not using any Bloom filters. When this is the\ncase, check that the data dump from the Bloom sub-system is all zeros,\nindicating that no filters were used.\n\nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 775e59d864..a77caca789 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -81,7 +81,19 @@ test_bloom_filters_used () {\n test_bloom_filters_not_used () {\n \tlog_args=$1\n \tsetup \"$log_args\" &&\n-\t! grep -q \"statistics:{\\\"filter_not_present\\\":\" \"$TRASH_DIRECTORY/trace.perf\" &&\n+\n+\tif grep -q \"statistics:{\\\"filter_not_present\\\":\" \"$TRASH_DIRECTORY/trace.perf\"\n+\tthen\n+\t\t# if the Bloom filter system is initialized, ensure that no\n+\t\t# filters were used\n+\t\tdata=\"statistics:{\"\n+\t\tdata=\"$data\\\"filter_not_present\\\":0,\"\n+\t\tdata=\"$data\\\"maybe\\\":0,\"\n+\t\tdata=\"$data\\\"definitely_not\\\":0,\"\n+\t\tdata=\"$data\\\"false_positive\\\":0}\"\n+\n+\t\tgrep -q \"$data\" \"$TRASH_DIRECTORY/trace.perf\"\n+\tfi &&\n \ttest_cmp log_wo_bloom log_w_bloom\n }\n \n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480898","messageId":"05516aa898aa5008eac113381b6aa58ef7b90065.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 11/15] commit-graph.c: unconditionally load Bloom filters","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:32Z","receivedAt":"2023-08-21T21:44:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 9e4df4da07 (commit-graph: new filter ver. that fixes murmur3,\n2023-08-01), we began ignoring the Bloom data (\"BDAT\") chunk for\ncommit-graphs whose Bloom filters were computed using a hash version\nincompatible with the value of `commitGraph.changedPathVersion`.\n\nNow that the Bloom API has been hardened to discard these incompatible\nfilters (with the exception of low-level APIs), we can safely load these\nBloom filters unconditionally.\n\nWe no longer want to return early from `graph_read_bloom_data()`, and\nsimilarly do not want to set the bloom_settings' `hash_version` field as\na side-effect. The latter is because we want to wait until we know which\nBloom settings we're using (either the defaults, from the GIT_TEST\nvariables, or from the previous commit-graph layer) before deciding what\nhash_version to use.\n\nIf we detect an existing BDAT chunk, we'll infer the rest of the\nsettings (e.g., number of hashes, bits per entry, and maximum number of\nchanged paths) from the earlier graph layer. The hash_version will be\ninferred from the previous layer as well, unless one has already been\nspecified via configuration.\n\nOnce all of that is done, we normalize the value of the hash_version to\neither \"1\" or \"2\".\n\nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 19 ++++++++++---------\n 1 file changed, 10 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f7322c4fff..665a3edf78 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -317,12 +317,6 @@ static int graph_read_bloom_data(const unsigned char *chunk_start,\n \tuint32_t hash_version;\n \thash_version = get_be32(chunk_start);\n \n-\tif (*c->commit_graph_changed_paths_version == -1) {\n-\t\t*c->commit_graph_changed_paths_version = hash_version;\n-\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n-\t\treturn 0;\n-\t}\n-\n \tg->chunk_bloom_data = chunk_start;\n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n \tg->bloom_filter_settings->hash_version = hash_version;\n@@ -2408,8 +2402,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n-\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n-\t\t? 2 : 1;\n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2441,10 +2434,18 @@ int write_commit_graph(struct object_directory *odb,\n \t\t/* We have changed-paths already. Keep them in the next graph */\n \t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n-\t\t\tctx->bloom_settings = g->bloom_filter_settings;\n+\n+\t\t\t/* don't propagate the hash_version unless unspecified */\n+\t\t\tif (bloom_settings.hash_version == -1)\n+\t\t\t\tbloom_settings.hash_version = g->bloom_filter_settings->hash_version;\n+\t\t\tbloom_settings.bits_per_entry = g->bloom_filter_settings->bits_per_entry;\n+\t\t\tbloom_settings.num_hashes = g->bloom_filter_settings->num_hashes;\n+\t\t\tbloom_settings.max_changed_paths = g->bloom_filter_settings->max_changed_paths;\n \t\t}\n \t}\n \n+\tbloom_settings.hash_version = bloom_settings.hash_version == 2 ? 2 : 1;\n+\n \tif (ctx->split) {\n \t\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n \n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480899","messageId":"7c144a535d30dbccff5e9b32b18571058d16c809.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 12/15] commit-graph: drop unnecessary `graph_read_bloom_data_context`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:36Z","receivedAt":"2023-08-21T21:44:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `graph_read_bloom_data_context` struct was introduced in an earlier\ncommit in order to pass pointers to the commit-graph and changed-path\nBloom filter version when reading the BDAT chunk.\n\nThe previous commit no longer writes through the changed_paths_version\npointer, making the surrounding context structure unnecessary. Drop it\nand pass a pointer to the commit-graph directly when reading the BDAT\nchunk.\n\nNoticed-by: Jonathan Tan \nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 14 ++------------\n 1 file changed, 2 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 665a3edf78..a8e33c0739 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -304,16 +304,10 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n-struct graph_read_bloom_data_context {\n-\tstruct commit_graph *g;\n-\tint *commit_graph_changed_paths_version;\n-};\n-\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t size_t chunk_size, void *data)\n {\n-\tstruct graph_read_bloom_data_context *c = data;\n-\tstruct commit_graph *g = c->g;\n+\tstruct commit_graph *g = data;\n \tuint32_t hash_version;\n \thash_version = get_be32(chunk_start);\n \n@@ -405,14 +399,10 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t}\n \n \tif (s->commit_graph_changed_paths_version) {\n-\t\tstruct graph_read_bloom_data_context context = {\n-\t\t\t.g = graph,\n-\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n-\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t graph_read_bloom_data, &context);\n+\t\t\t graph_read_bloom_data, graph);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480900","messageId":"a86f7259b66dcb5a2bc26e9a59397213d24313e2.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 13/15] object.h: fix mis-aligned flag bits table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:39Z","receivedAt":"2023-08-21T21:44:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Bit position 23 is one column too far to the left.\n\nSigned-off-by: Taylor Blau \n---\n object.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object.h b/object.h\nindex 114d45954d..db25714b4e 100644\n--- a/object.h\n+++ b/object.h\n@@ -62,7 +62,7 @@ void object_array_init(struct object_array *array);\n \n /*\n * object flag allocation:\n- * revision.h: 0---------10 15 23------27\n+ * revision.h: 0---------10 15 23------27\n * fetch-pack.c: 01 67\n * negotiator/default.c: 2--5\n * walker.c: 0-2\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480901","messageId":"4f69a4c41d935ede83bd89871f2f84b334953d8f.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 14/15] commit-graph: reuse existing Bloom filters where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:42Z","receivedAt":"2023-08-21T21:44:53Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 9e4df4da07 (commit-graph: new filter ver. that fixes murmur3,\n2023-08-01), a bug was described where it's possible for Git to produce\nnon-murmur3 hashes when the platform's \"char\" type is signed, and there\nare paths with characters whose highest bit is set (i.e. all characters\n>= 0x80).\n\nThat patch allows the caller to control which version of Bloom filters\nare read and written. However, even on platforms with a signed \"char\"\ntype, it is possible to reuse existing Bloom filters if and only if\nthere are no changed paths in any commit's first parent tree-diff whose\ncharacters have their highest bit set.\n\nWhen this is the case, we can reuse the existing filter without having\nto compute a new one. This is done by marking trees which are known to\nhave (or not have) any such paths. When a commit's root tree is verified\nto not have any such paths, we mark it as such and declare that the\ncommit's Bloom filter is reusable.\n\nNote that this heuristic only goes in one direction. If neither a commit\nnor its first parent have any paths in their trees with non-ASCII\ncharacters, then we know for certain that a path with non-ASCII\ncharacters will not appear in a tree-diff against that commit's first\nparent. The reverse isn't necessarily true: just because the tree-diff\ndoesn't contain any such paths does not imply that no such paths exist\nin either tree.\n\nSo we end up recomputing some Bloom filters that we don't strictly have\nto (i.e. their bits are the same no matter which version of murmur3 we\nuse). But culling these out is impossible, since we'd have to perform\nthe full tree-diff, which is the same effort as computing the Bloom\nfilter from scratch.\n\nBut because we can cache our results in each tree's flag bits, we can\noften avoid recomputing many filters, thereby reducing the time it takes\nto run\n\n $ git commit-graph write --changed-paths --reachable\n\nwhen upgrading from v1 to v2 Bloom filters.\n\nTo benchmark this, let's generate a commit-graph in linux.git with v1\nchanged-paths in generation order[^1]:\n\n $ git clone git@github.com:torvalds/linux.git\n $ cd linux\n $ git commit-graph write --reachable --changed-paths\n $ graph=\".git/objects/info/commit-graph\"\n $ mv $graph{,.bak}\n\nThen let's time how long it takes to go from v1 to v2 filters (with and\nwithout the upgrade path enabled), resetting the state of the\ncommit-graph each time:\n\n $ git config commitGraph.changedPathsVersion 2\n $ hyperfine -p 'cp -f $graph.bak $graph' -L v 0,1 \\\n 'GIT_TEST_UPGRADE_BLOOM_FILTERS={v} git.compile commit-graph write --reachable --changed-paths'\n\nOn linux.git (where there aren't any non-ASCII paths), the timings\nindicate that this patch represents a speed-up over recomputing all\nBloom filters from scratch:\n\n Benchmark 1: GIT_TEST_UPGRADE_BLOOM_FILTERS=0 git.compile commit-graph write --reachable --changed-paths\n Time (mean ± σ): 124.873 s ± 0.316 s [User: 124.081 s, System: 0.643 s]\n Range (min … max): 124.621 s … 125.227 s 3 runs\n\n Benchmark 2: GIT_TEST_UPGRADE_BLOOM_FILTERS=1 git.compile commit-graph write --reachable --changed-paths\n Time (mean ± σ): 79.271 s ± 0.163 s [User: 74.611 s, System: 4.521 s]\n Range (min … max): 79.112 s … 79.437 s 3 runs\n\n Summary\n 'GIT_TEST_UPGRADE_BLOOM_FILTERS=1 git.compile commit-graph write --reachable --changed-paths' ran\n 1.58 ± 0.01 times faster than 'GIT_TEST_UPGRADE_BLOOM_FILTERS=0 git.compile commit-graph write --reachable --changed-paths'\n\nOn git.git, we do have some non-ASCII paths, giving us a more modest\nimprovement from 4.163 seconds to 3.348 seconds, for a 1.24x speed-up.\nOn my machine, the stats for git.git are:\n\n - 8,285 Bloom filters computed from scratch\n - 10 Bloom filters generated as empty\n - 4 Bloom filters generated as truncated due to too many changed paths\n - 65,114 Bloom filters were reused when transitioning from v1 to v2.\n\n[^1]: Note that this is is important, since `--stdin-packs` or\n `--stdin-commits` orders commits in the commit-graph by their pack\n position (with `--stdin-packs`) or in the raw input (with\n `--stdin-commits`).\n\n Since we compute Bloom filters in the same order that commits appear\n in the graph, we must see a commit's (first) parent before we process\n the commit itself. This is only guaranteed to happen when sorting\n commits by their generation number.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 90 ++++++++++++++++++++++++++++++++++++++++++--\n bloom.h | 1 +\n commit-graph.c | 5 +++\n object.h | 1 +\n t/t4216-log-bloom.sh | 35 ++++++++++++++++-\n 5 files changed, 127 insertions(+), 5 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 739fa093ba..24dd874e46 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -7,6 +7,9 @@\n #include \"commit-graph.h\"\n #include \"commit.h\"\n #include \"commit-slab.h\"\n+#include \"tree.h\"\n+#include \"tree-walk.h\"\n+#include \"config.h\"\n \n define_commit_slab(bloom_filter_slab, struct bloom_filter);\n \n@@ -250,6 +253,73 @@ static void init_truncated_large_filter(struct bloom_filter *filter,\n \tfilter->version = version;\n }\n \n+#define VISITED (1u<<21)\n+#define HIGH_BITS (1u<<22)\n+\n+static int has_entries_with_high_bit(struct repository *r, struct tree *t)\n+{\n+\tif (parse_tree(t))\n+\t\treturn 1;\n+\n+\tif (!(t->object.flags & VISITED)) {\n+\t\tstruct tree_desc desc;\n+\t\tstruct name_entry entry;\n+\n+\t\tinit_tree_desc(&desc, t->buffer, t->size);\n+\t\twhile (tree_entry(&desc, &entry)) {\n+\t\t\tsize_t i;\n+\t\t\tfor (i = 0; i < entry.pathlen; i++) {\n+\t\t\t\tif (entry.path[i] & 0x80) {\n+\t\t\t\t\tt->object.flags |= HIGH_BITS;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\t\tstruct tree *sub = lookup_tree(r, &entry.oid);\n+\t\t\t\tif (sub && has_entries_with_high_bit(r, sub)) {\n+\t\t\t\t\tt->object.flags |= HIGH_BITS;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t}\n+\n+done:\n+\t\tt->object.flags |= VISITED;\n+\t}\n+\n+\treturn !!(t->object.flags & HIGH_BITS);\n+}\n+\n+static int commit_tree_has_high_bit_paths(struct repository *r,\n+\t\t\t\t\t struct commit *c)\n+{\n+\tstruct tree *t;\n+\tif (repo_parse_commit(r, c))\n+\t\treturn 1;\n+\tt = repo_get_commit_tree(r, c);\n+\tif (!t)\n+\t\treturn 1;\n+\treturn has_entries_with_high_bit(r, t);\n+}\n+\n+static struct bloom_filter *upgrade_filter(struct repository *r, struct commit *c,\n+\t\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t\t int hash_version)\n+{\n+\tstruct commit_list *p = c->parents;\n+\tif (commit_tree_has_high_bit_paths(r, c))\n+\t\treturn NULL;\n+\n+\tif (p && commit_tree_has_high_bit_paths(r, p->item))\n+\t\treturn NULL;\n+\n+\tfilter->version = hash_version;\n+\n+\treturn filter;\n+}\n+\n struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c)\n {\n \tstruct bloom_filter *filter;\n@@ -292,9 +362,23 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t filter, graph_pos);\n \t}\n \n-\tif ((filter->data && filter->len) &&\n-\t (!settings || settings->hash_version == filter->version))\n-\t\treturn filter;\n+\tif (filter->data && filter->len) {\n+\t\tstruct bloom_filter *upgrade;\n+\t\tif (!settings || settings->hash_version == filter->version)\n+\t\t\treturn filter;\n+\n+\t\t/* version mismatch, see if we can upgrade */\n+\t\tif (compute_if_not_present &&\n+\t\t git_env_bool(\"GIT_TEST_UPGRADE_BLOOM_FILTERS\", 1)) {\n+\t\t\tupgrade = upgrade_filter(r, c, filter,\n+\t\t\t\t\t\t settings->hash_version);\n+\t\t\tif (upgrade) {\n+\t\t\t\tif (computed)\n+\t\t\t\t\t*computed |= BLOOM_UPGRADED;\n+\t\t\t\treturn upgrade;\n+\t\t\t}\n+\t\t}\n+\t}\n \tif (!compute_if_not_present)\n \t\treturn NULL;\n \ndiff --git a/bloom.h b/bloom.h\nindex bfe389e29c..e3a9b68905 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -102,6 +102,7 @@ enum bloom_filter_computed {\n \tBLOOM_COMPUTED = (1 << 1),\n \tBLOOM_TRUNC_LARGE = (1 << 2),\n \tBLOOM_TRUNC_EMPTY = (1 << 3),\n+\tBLOOM_UPGRADED = (1 << 4),\n };\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a8e33c0739..a3473df515 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1045,6 +1045,7 @@ struct write_commit_graph_context {\n \tint count_bloom_filter_not_computed;\n \tint count_bloom_filter_trunc_empty;\n \tint count_bloom_filter_trunc_large;\n+\tint count_bloom_filter_upgraded;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1651,6 +1652,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \t\t\t ctx->count_bloom_filter_trunc_empty);\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-large\",\n \t\t\t ctx->count_bloom_filter_trunc_large);\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-upgraded\",\n+\t\t\t ctx->count_bloom_filter_upgraded);\n }\n \n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n@@ -1692,6 +1695,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\tctx->count_bloom_filter_trunc_empty++;\n \t\t\tif (computed & BLOOM_TRUNC_LARGE)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n+\t\t} else if (computed & BLOOM_UPGRADED) {\n+\t\t\tctx->count_bloom_filter_upgraded++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n \t\t\tctx->count_bloom_filter_not_computed++;\n \t\tctx->total_bloom_filter_data_size += filter\ndiff --git a/object.h b/object.h\nindex db25714b4e..2e5e08725f 100644\n--- a/object.h\n+++ b/object.h\n@@ -75,6 +75,7 @@ void object_array_init(struct object_array *array);\n * commit-reach.c: 16-----19\n * sha1-name.c: 20\n * list-objects-filter.c: 21\n+ * bloom.c: 2122\n * builtin/fsck.c: 0--3\n * builtin/gc.c: 0\n * builtin/index-pack.c: 2021\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex a77caca789..48f8109a66 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -217,6 +217,10 @@ test_filter_trunc_large () {\n \tgrep \"\\\"key\\\":\\\"filter-trunc-large\\\",\\\"value\\\":\\\"$1\\\"\" $2\n }\n \n+test_filter_upgraded () {\n+\tgrep \"\\\"key\\\":\\\"filter-upgraded\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n test_expect_success 'correctly report changes over limit' '\n \tgit init limits &&\n \t(\n@@ -543,10 +547,19 @@ test_expect_success 'when writing another commit graph, preserve existing versio\n test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n \tgit init doublewrite &&\n \ttest_commit -C doublewrite c \"$CENT\" &&\n+\n \tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n-\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 1 trace2.txt &&\n+\ttest_filter_upgraded 0 trace2.txt &&\n+\n \tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n-\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 1 trace2.txt &&\n+\ttest_filter_upgraded 0 trace2.txt &&\n+\n \t(\n \t\tcd doublewrite &&\n \t\techo \"c01f\" >expect &&\n@@ -555,4 +568,22 @@ test_expect_success 'when writing commit graph, do not reuse changed-path of ano\n \t)\n '\n \n+test_expect_success 'when writing commit graph, reuse changed-path of another version where possible' '\n+\tgit init upgrade &&\n+\n+\ttest_commit -C upgrade base no-high-bits &&\n+\n+\tgit -C upgrade config --add commitgraph.changedPathsVersion 1 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C upgrade commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 1 trace2.txt &&\n+\ttest_filter_upgraded 0 trace2.txt &&\n+\n+\tgit -C upgrade config --add commitgraph.changedPathsVersion 2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C upgrade commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 0 trace2.txt &&\n+\ttest_filter_upgraded 1 trace2.txt\n+'\n+\n test_done\n-- \n2.42.0.4.g52b49bb434\n\n"},{"id":"480902","messageId":"eed1e1a32af0ad343023a5143722226d2ef595bb.1692654233.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH 15/15] bloom: introduce `deinit_bloom_filters()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-21T21:44:46Z","receivedAt":"2023-08-21T21:44:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"After we are done using Bloom filters, we do not currently clean up any\nmemory allocated by the commit slab used to store those filters in the\nfirst place.\n\nBesides the bloom_filter structures themselves, there is mostly nothing\nto free() in the first place, since in the read-only path all Bloom\nfilter's `data` members point to a memory mapped region in the\ncommit-graph file itself.\n\nBut when generating Bloom filters from scratch (or initializing\ntruncated filters) we allocate additional memory to store the filter's\ndata.\n\nKeep track of when we need to free() this additional chunk of memory by\nusing an extra pointer `to_free`. Most of the time this will be NULL\n(indicating that we are representing an existing Bloom filter stored in\na memory mapped region). When it is non-NULL, free it before discarding\nthe Bloom filters slab.\n\nSuggested-by: Jonathan Tan \nSigned-off-by: Taylor Blau \n---\n bloom.c | 16 +++++++++++++++-\n bloom.h | 3 +++\n commit-graph.c | 4 ++++\n 3 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 24dd874e46..ff131893cd 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -59,6 +59,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n \t\t\t\t\tsizeof(unsigned char) * start_index +\n \t\t\t\t\tBLOOMDATA_CHUNK_HEADER_SIZE);\n \tfilter->version = g->bloom_filter_settings->hash_version;\n+\tfilter->to_free = NULL;\n \n \treturn 1;\n }\n@@ -231,6 +232,18 @@ void init_bloom_filters(void)\n \tinit_bloom_filter_slab(&bloom_filters);\n }\n \n+static void free_one_bloom_filter(struct bloom_filter *filter)\n+{\n+\tif (!filter)\n+\t\treturn;\n+\tfree(filter->to_free);\n+}\n+\n+void deinit_bloom_filters(void)\n+{\n+\tdeep_clear_bloom_filter_slab(&bloom_filters, free_one_bloom_filter);\n+}\n+\n static int pathmap_cmp(const void *hashmap_cmp_fn_data UNUSED,\n \t\t const struct hashmap_entry *eptr,\n \t\t const struct hashmap_entry *entry_or_key,\n@@ -247,7 +260,7 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data UNUSED,\n static void init_truncated_large_filter(struct bloom_filter *filter,\n \t\t\t\t\tint version)\n {\n-\tfilter->data = xmalloc(1);\n+\tfilter->data = filter->to_free = xmalloc(1);\n \tfilter->data[0] = 0xFF;\n \tfilter->len = 1;\n \tfilter->version = version;\n@@ -449,6 +462,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tfilter->len = 1;\n \t\t}\n \t\tCALLOC_ARRAY(filter->data, filter->len);\n+\t\tfilter->to_free = filter->data;\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n \t\t\tstruct bloom_key key;\ndiff --git a/bloom.h b/bloom.h\nindex e3a9b68905..d20e64bfbb 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -56,6 +56,8 @@ struct bloom_filter {\n \tunsigned char *data;\n \tsize_t len;\n \tint version;\n+\n+\tvoid *to_free;\n };\n \n /*\n@@ -96,6 +98,7 @@ void add_key_to_filter(const struct bloom_key *key,\n \t\t const struct bloom_filter_settings *settings);\n \n void init_bloom_filters(void);\n+void deinit_bloom_filters(void);\n \n enum bloom_filter_computed {\n \tBLOOM_NOT_COMPUTED = (1 << 0),\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a3473df515..585539da2f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -723,6 +723,7 @@ static void close_commit_graph_one(struct commit_graph *g)\n void close_commit_graph(struct raw_object_store *o)\n {\n \tclose_commit_graph_one(o->commit_graph);\n+\tdeinit_bloom_filters();\n \to->commit_graph = NULL;\n }\n \n@@ -2523,6 +2524,9 @@ int write_commit_graph(struct object_directory *odb,\n \n \tres = write_commit_graph_file(ctx);\n \n+\tif (ctx->changed_paths)\n+\t\tdeinit_bloom_filters();\n+\n \tif (ctx->split)\n \t\tmark_commit_graphs(ctx);\n \n-- \n2.42.0.4.g52b49bb434\n"},{"id":"481022","messageId":"20230824222246.2320443-1-jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"Re: [PATCH 00/15] bloom: changed-path Bloom filters v2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-24T22:22:46Z","receivedAt":"2023-08-24T22:24:00Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau writes:\n> Jonathan Tan (4):\n> gitformat-commit-graph: describe version 2 of BDAT\n> t4216: test changed path filters with high bit paths\n> repo-settings: introduce commitgraph.changedPathsVersion\n> commit-graph: new filter ver. that fixes murmur3\n> \n> Taylor Blau (11):\n> t/helper/test-read-graph.c: extract `dump_graph_info()`\n> bloom.h: make `load_bloom_filter_from_graph()` public\n> t/helper/test-read-graph: implement `bloom-filters` mode\n> bloom: annotate filters with hash version\n> bloom: prepare to discard incompatible Bloom filters\n> t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`\n> commit-graph.c: unconditionally load Bloom filters\n> commit-graph: drop unnecessary `graph_read_bloom_data_context`\n> object.h: fix mis-aligned flag bits table\n> commit-graph: reuse existing Bloom filters where possible\n> bloom: introduce `deinit_bloom_filters()`\n\nThanks. I had one small comment (sent as an email reply to one of the\npatches), but everything else looks good.\n \n"},{"id":"481029","messageId":"20230825170656.2598201-1-jonathantanmy@google.com","threadId":"60142","inReplyTo":"20230824222246.2320443-1-jonathantanmy@google.com","subject":"Re: [PATCH 00/15] bloom: changed-path Bloom filters v2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-25T17:06:56Z","receivedAt":"2023-08-25T17:07:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Jonathan Tan writes:\n> Taylor Blau writes:\n> > Jonathan Tan (4):\n> > gitformat-commit-graph: describe version 2 of BDAT\n> > t4216: test changed path filters with high bit paths\n> > repo-settings: introduce commitgraph.changedPathsVersion\n> > commit-graph: new filter ver. that fixes murmur3\n> > \n> > Taylor Blau (11):\n> > t/helper/test-read-graph.c: extract `dump_graph_info()`\n> > bloom.h: make `load_bloom_filter_from_graph()` public\n> > t/helper/test-read-graph: implement `bloom-filters` mode\n> > bloom: annotate filters with hash version\n> > bloom: prepare to discard incompatible Bloom filters\n> > t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`\n> > commit-graph.c: unconditionally load Bloom filters\n> > commit-graph: drop unnecessary `graph_read_bloom_data_context`\n> > object.h: fix mis-aligned flag bits table\n> > commit-graph: reuse existing Bloom filters where possible\n> > bloom: introduce `deinit_bloom_filters()`\n> \n> Thanks. I had one small comment (sent as an email reply to one of the\n> patches), but everything else looks good.\n\nI mistakenly sent my reply to an earlier version [1]. (Taylor has seen\nit, so this note is more for future readers who might be curious about\nwhat that email reply contains.)\n\n[1] https://lore.kernel.org/git/20230824222051.2320003-1-jonathantanmy@google.com/\n"},{"id":"481041","messageId":"20230826150610.GA1928@szeder.dev","threadId":"60142","inReplyTo":"72150bd026befc758b41386cf571c6a33a754d4e.1692654233.git.me@ttaylorr.com","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2023-08-26T15:06:10Z","receivedAt":"2023-08-26T15:07:10Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Aug 21, 2023 at 05:44:15PM -0400, Taylor Blau wrote:\n> From: Jonathan Tan \n> \n> The murmur3 implementation in bloom.c has a bug when converting series\n> of 4 bytes into network-order integers when char is signed (which is\n> controllable by a compiler option, and the default signedness of char is\n> platform-specific). When a string contains characters with the high bit\n> set, this bug causes results that, although internally consistent within\n> Git, does not accord with other implementations of murmur3 (thus,\n> the changed path filters wouldn't be readable by other off-the-shelf\n> implementatios of murmur3) and even with Git binaries that were compiled\n> with different signedness of char. This bug affects both how Git writes\n> changed path filters to disk and how Git interprets changed path filters\n> on disk.\n> \n> Therefore, introduce a new version (2) of changed path filters that\n> corrects this problem. The existing version (1) is still supported and\n> is still the default, but users should migrate away from it as soon\n> as possible.\n> \n> Because this bug only manifests with characters that have the high bit\n> set, it may be possible that some (or all) commits in a given repo would\n> have the same changed path filter both before and after this fix is\n> applied. However, in order to determine whether this is the case, the\n> changed paths would first have to be computed, at which point it is not\n> much more expensive to just compute a new changed path filter.\n> \n> So this patch does not include any mechanism to \"salvage\" changed path\n> filters from repositories. There is also no \"mixed\" mode - for each\n> invocation of Git, reading and writing changed path filters are done\n> with the same version number; this version number may be explicitly\n> stated (typically if the user knows which version they need) or\n> automatically determined from the version of the existing changed path\n> filters in the repository.\n> \n> There is a change in write_commit_graph(). graph_read_bloom_data()\n> makes it possible for chunk_bloom_data to be non-NULL but\n> bloom_filter_settings to be NULL, which causes a segfault later on. I\n> produced such a segfault while developing this patch, but couldn't find\n> a way to reproduce it neither after this complete patch (or before),\n> but in any case it seemed like a good thing to include that might help\n> future patch authors.\n> \n> The value in t0095 was obtained from another murmur3 implementation\n> using the following Go source code:\n> \n> package main\n> \n> import \"fmt\"\n> import \"github.com/spaolacci/murmur3\"\n> \n> func main() {\n> fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n> fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n> }\n> \n> Signed-off-by: Jonathan Tan \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/config/commitgraph.txt | 5 +-\n> bloom.c | 69 +++++++++++++++++++-\n> bloom.h | 8 ++-\n> commit-graph.c | 32 ++++++++--\n> t/helper/test-bloom.c | 9 ++-\n> t/t0095-bloom.sh | 8 +++\n> t/t4216-log-bloom.sh | 96 ++++++++++++++++++++++++++++\n> 7 files changed, 214 insertions(+), 13 deletions(-)\n> \n> diff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\n> index 2dc9170622..acc74a2f27 100644\n> --- a/Documentation/config/commitgraph.txt\n> +++ b/Documentation/config/commitgraph.txt\n> @@ -15,7 +15,7 @@ commitGraph.readChangedPaths::\n> \n> commitGraph.changedPathsVersion::\n> \tSpecifies the version of the changed-path Bloom filters that Git will read and\n> -\twrite. May be -1, 0 or 1.\n> +\twrite. May be -1, 0, 1, or 2.\n> +\n> Defaults to -1.\n> +\n> @@ -28,4 +28,7 @@ filters when instructed to write.\n> If 1, Git will only read version 1 Bloom filters, and will write version 1\n> Bloom filters.\n> +\n> +If 2, Git will only read version 2 Bloom filters, and will write version 2\n> +Bloom filters.\n> ++\n> See linkgit:git-commit-graph[1] for more information.\n> diff --git a/bloom.c b/bloom.c\n> index 3e78cfe79d..ebef5cfd2f 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -66,7 +66,64 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n> * Not considered to be cryptographically secure.\n> * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n> */\n> -uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n> +uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n\nNit: MurmurHash3 implementations in C/C++ (apart from the sample\nimplementation on Wikipedia), and other hash functions taking data\npointer and buffer size parameters in general, have a 'const void\n*data' parameter, not 'const char*'.\n\n> +{\n> +\tconst uint32_t c1 = 0xcc9e2d51;\n> +\tconst uint32_t c2 = 0x1b873593;\n> +\tconst uint32_t r1 = 15;\n> +\tconst uint32_t r2 = 13;\n> +\tconst uint32_t m = 5;\n> +\tconst uint32_t n = 0xe6546b64;\n> +\tint i;\n> +\tuint32_t k1 = 0;\n> +\tconst char *tail;\n> +\n> +\tint len4 = len / sizeof(uint32_t);\n> +\n> +\tuint32_t k;\n> +\tfor (i = 0; i < len4; i++) {\n> +\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n> +\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n> +\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n> +\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n> +\t\tk = byte1 | byte2 | byte3 | byte4;\n> +\t\tk *= c1;\n> +\t\tk = rotate_left(k, r1);\n> +\t\tk *= c2;\n> +\n> +\t\tseed ^= k;\n> +\t\tseed = rotate_left(seed, r2) * m + n;\n> +\t}\n> +\n> +\ttail = (data + len4 * sizeof(uint32_t));\n> +\n> +\tswitch (len & (sizeof(uint32_t) - 1)) {\n> +\tcase 3:\n> +\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n> +\t\t/*-fallthrough*/\n> +\tcase 2:\n> +\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n> +\t\t/*-fallthrough*/\n> +\tcase 1:\n> +\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n> +\t\tk1 *= c1;\n> +\t\tk1 = rotate_left(k1, r1);\n> +\t\tk1 *= c2;\n> +\t\tseed ^= k1;\n> +\t\tbreak;\n> +\t}\n> +\n> +\tseed ^= (uint32_t)len;\n> +\tseed ^= (seed >> 16);\n> +\tseed *= 0x85ebca6b;\n> +\tseed ^= (seed >> 13);\n> +\tseed *= 0xc2b2ae35;\n> +\tseed ^= (seed >> 16);\n> +\n> +\treturn seed;\n> +}\n> +\n> +static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n> {\n> \tconst uint32_t c1 = 0xcc9e2d51;\n> \tconst uint32_t c2 = 0x1b873593;\n> @@ -131,8 +188,14 @@ void fill_bloom_key(const char *data,\n> \tint i;\n> \tconst uint32_t seed0 = 0x293ae76f;\n> \tconst uint32_t seed1 = 0x7e646e2c;\n> -\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n> -\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n> +\tuint32_t hash0, hash1;\n> +\tif (settings->hash_version == 2) {\n> +\t\thash0 = murmur3_seeded_v2(seed0, data, len);\n> +\t\thash1 = murmur3_seeded_v2(seed1, data, len);\n> +\t} else {\n> +\t\thash0 = murmur3_seeded_v1(seed0, data, len);\n> +\t\thash1 = murmur3_seeded_v1(seed1, data, len);\n> +\t}\n> \n> \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n> \tfor (i = 0; i < settings->num_hashes; i++)\n> diff --git a/bloom.h b/bloom.h\n> index 1e4f612d2c..138d57a86b 100644\n> --- a/bloom.h\n> +++ b/bloom.h\n> @@ -8,9 +8,11 @@ struct commit_graph;\n> struct bloom_filter_settings {\n> \t/*\n> \t * The version of the hashing technique being used.\n> -\t * We currently only support version = 1 which is\n> +\t * The newest version is 2, which is\n> \t * the seeded murmur3 hashing technique implemented\n> -\t * in bloom.c.\n> +\t * in bloom.c. Bloom filters of version 1 were created\n> +\t * with prior versions of Git, which had a bug in the\n> +\t * implementation of the hash function.\n> \t */\n> \tuint32_t hash_version;\n> \n> @@ -80,7 +82,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n> * Not considered to be cryptographically secure.\n> * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n> */\n> -uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n> +uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n> \n> void fill_bloom_key(const char *data,\n> \t\t size_t len,\n> diff --git a/commit-graph.c b/commit-graph.c\n> index da99f15fdf..f7322c4fff 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -304,17 +304,26 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n> \treturn 0;\n> }\n> \n> +struct graph_read_bloom_data_context {\n> +\tstruct commit_graph *g;\n> +\tint *commit_graph_changed_paths_version;\n> +};\n> +\n> static int graph_read_bloom_data(const unsigned char *chunk_start,\n> \t\t\t\t size_t chunk_size, void *data)\n> {\n> -\tstruct commit_graph *g = data;\n> +\tstruct graph_read_bloom_data_context *c = data;\n> +\tstruct commit_graph *g = c->g;\n> \tuint32_t hash_version;\n> -\tg->chunk_bloom_data = chunk_start;\n> \thash_version = get_be32(chunk_start);\n> \n> -\tif (hash_version != 1)\n> +\tif (*c->commit_graph_changed_paths_version == -1) {\n> +\t\t*c->commit_graph_changed_paths_version = hash_version;\n> +\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n> \t\treturn 0;\n> +\t}\n> \n> +\tg->chunk_bloom_data = chunk_start;\n> \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n> \tg->bloom_filter_settings->hash_version = hash_version;\n> \tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n> @@ -402,10 +411,14 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n> \t}\n> \n> \tif (s->commit_graph_changed_paths_version) {\n> +\t\tstruct graph_read_bloom_data_context context = {\n> +\t\t\t.g = graph,\n> +\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n> +\t\t};\n> \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n> \t\t\t &graph->chunk_bloom_indexes);\n> \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n> -\t\t\t graph_read_bloom_data, graph);\n> +\t\t\t graph_read_bloom_data, &context);\n> \t}\n> \n> \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n> @@ -2376,6 +2389,13 @@ int write_commit_graph(struct object_directory *odb,\n> \t}\n> \tif (!commit_graph_compatible(r))\n> \t\treturn 0;\n> +\tif (r->settings.commit_graph_changed_paths_version < -1\n> +\t || r->settings.commit_graph_changed_paths_version > 2) {\n> +\t\twarning(_(\"attempting to write a commit-graph, but \"\n> +\t\t\t \"'commitgraph.changedPathsVersion' (%d) is not supported\"),\n> +\t\t\tr->settings.commit_graph_changed_paths_version);\n> +\t\treturn 0;\n> +\t}\n> \n> \tCALLOC_ARRAY(ctx, 1);\n> \tctx->r = r;\n> @@ -2388,6 +2408,8 @@ int write_commit_graph(struct object_directory *odb,\n> \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n> \tctx->num_generation_data_overflows = 0;\n> \n> +\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n> +\t\t? 2 : 1;\n> \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n> \t\t\t\t\t\t bloom_settings.bits_per_entry);\n> \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n> @@ -2417,7 +2439,7 @@ int write_commit_graph(struct object_directory *odb,\n> \t\tg = ctx->r->objects->commit_graph;\n> \n> \t\t/* We have changed-paths already. Keep them in the next graph */\n> -\t\tif (g && g->chunk_bloom_data) {\n> +\t\tif (g && g->bloom_filter_settings) {\n> \t\t\tctx->changed_paths = 1;\n> \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n> \t\t}\n> diff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\n> index aabe31d724..3cbc0a5b50 100644\n> --- a/t/helper/test-bloom.c\n> +++ b/t/helper/test-bloom.c\n> @@ -50,6 +50,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n> \n> static const char *bloom_usage = \"\\n\"\n> \" test-tool bloom get_murmur3 \\n\"\n> +\" test-tool bloom get_murmur3_seven_highbit\\n\"\n> \" test-tool bloom generate_filter [...]\\n\"\n> \" test-tool bloom get_filter_for_commit \\n\";\n> \n> @@ -64,7 +65,13 @@ int cmd__bloom(int argc, const char **argv)\n> \t\tuint32_t hashed;\n> \t\tif (argc < 3)\n> \t\t\tusage(bloom_usage);\n> -\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n> +\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n> +\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n> +\t}\n> +\n> +\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n> +\t\tuint32_t hashed;\n> +\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n> \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n> \t}\n> \n> diff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\n> index b567383eb8..c8d84ab606 100755\n> --- a/t/t0095-bloom.sh\n> +++ b/t/t0095-bloom.sh\n> @@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n> +\tcat >expect <<-\\EOF &&\n> +\tMurmur3 Hash with seed=0:0xa183ccfd\n> +\tEOF\n> +\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_expect_success 'compute bloom key for empty string' '\n> \tcat >expect <<-\\EOF &&\n> \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index 2d4a3fefee..775e59d864 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -447,4 +447,100 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n> \t)\n> '\n> \n> +test_expect_success 'version 1 changed-path not used when version 2 requested' '\n> +\t(\n> +\t\tcd highbit1 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n> +\t\ttest_bloom_filters_not_used \"-- $CENT\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'version 1 changed-path used when autodetect requested' '\n> +\t(\n> +\t\tcd highbit1 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n> +\t\ttest_bloom_filters_used \"-- $CENT\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n> +\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n> +\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n\nNit: Since there is a subshell cd-ing into the 'highbit1' directory\nanyway, it would look clearer to put these two commands into that\nsubshell as well.\nThis applies to some of the later test cases as well.\n\n> +\t(\n> +\t\tcd highbit1 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n> +\t\techo \"options: bloom(1,10,7) read_generation_data\" >expect &&\n> +\t\ttest-tool read-graph >full &&\n> +\t\tgrep options full >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n> +\tgit init highbit2 &&\n> +\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n> +\ttest_commit -C highbit2 c2 \"$CENT\" &&\n> +\tgit -C highbit2 commit-graph write --reachable --changed-paths\n> +'\n> +\n> +test_expect_success 'check value of version 2 changed-path' '\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\techo \"c01f\" >expect &&\n> +\t\tget_first_changed_path_filter >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'version 2 changed-path used when version 2 requested' '\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\ttest_bloom_filters_used \"-- $CENT\"\n\nThis test_bloom_filter_used helper runs two pathspec-limited 'git log'\ninvocations, one with disabled and the other with enabled\ncommit-graph, and thus with disabled and enabled modified path Bloom\nfilters, and compares their output.\n\nOne of the flaws of the current modified path Bloom filters\nimplementation is that it doesn't check Bloom filters for root\ncommits.\n\nIn several of the above test cases test_bloom_filters_used is invoked\nin a repository with only a root commit, so they don't check that\nthe output is the same with and without Bloom filters.\n\n> +\t)\n> +'\n> +\n> +test_expect_success 'version 2 changed-path not used when version 1 requested' '\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n> +\t\ttest_bloom_filters_not_used \"-- $CENT\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'version 2 changed-path used when autodetect requested' '\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n> +\t\ttest_bloom_filters_used \"-- $CENT\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n> +\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n> +\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n> +\t\techo \"options: bloom(2,10,7) read_generation_data\" >expect &&\n> +\t\ttest-tool read-graph >full &&\n> +\t\tgrep options full >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n> +\tgit init doublewrite &&\n> +\ttest_commit -C doublewrite c \"$CENT\" &&\n> +\tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n> +\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n> +\tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n> +\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n> +\t(\n> +\t\tcd doublewrite &&\n> +\t\techo \"c01f\" >expect &&\n> +\t\tget_first_changed_path_filter >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n\nThe string \"split\" occurs twice in this patch series, but only in\npatch hunk contexts, and it doesn't occur at all in the previous long\nthread about the original patch series.\n\nUnfortunately, split commit-graphs weren't really considered in the\ndesign of the modified path Bloom filters feature, and layers with\ndifferent Bloom filter settings weren't considered at all. I've\nreported it back then, but the fixes so far were incomplete, and e.g.\nthe test cases shown in\n\n https://public-inbox.org/git/20201015132147.GB24954@szeder.dev/\n\nstill fail.\n\nSince the interaction of different versions and split commit-graphs\nwas neither mentioned in any of the commit messages nor discussed\nduring the previous rounds, and there isn't any test case excercising\nit, and since the Bloom filter version information is stored in the\nsame 'g->bloom_filter_settings' structure as the number of hashes, I'm\nafraid (though haven't actually checked) that handling commit-graph\nlayers with different Bloom filter versions is prone to the same\nissues as well.\n\n\n"},{"id":"481118","messageId":"20230829163124.363157-1-jonathantanmy@google.com","threadId":"60142","inReplyTo":"20230826150610.GA1928@szeder.dev","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-29T16:31:23Z","receivedAt":"2023-08-29T16:32:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"SZEDER Gábor writes:\n> > @@ -66,7 +66,64 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n> > * Not considered to be cryptographically secure.\n> > * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n> > */\n> > -uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n> > +uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n> \n> Nit: MurmurHash3 implementations in C/C++ (apart from the sample\n> implementation on Wikipedia), and other hash functions taking data\n> pointer and buffer size parameters in general, have a 'const void\n> *data' parameter, not 'const char*'.\n\nI think either works, so I'll stick with what the existing code uses.\n(Well, we probably should have used \"unsigned char\" in the first place.)\n\n> > +test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n> > +\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n> > +\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n> \n> Nit: Since there is a subshell cd-ing into the 'highbit1' directory\n> anyway, it would look clearer to put these two commands into that\n> subshell as well.\n> This applies to some of the later test cases as well.\n\nMakes sense, but this patch series has already been reviewed a few times\nso I don't think it's worth making this change at this point in the\nreview process for a small increase in readability.\n\n> > +test_expect_success 'version 2 changed-path used when version 2 requested' '\n> > +\t(\n> > +\t\tcd highbit2 &&\n> > +\t\ttest_bloom_filters_used \"-- $CENT\"\n> \n> This test_bloom_filter_used helper runs two pathspec-limited 'git log'\n> invocations, one with disabled and the other with enabled\n> commit-graph, and thus with disabled and enabled modified path Bloom\n> filters, and compares their output.\n> \n> One of the flaws of the current modified path Bloom filters\n> implementation is that it doesn't check Bloom filters for root\n> commits.\n> \n> In several of the above test cases test_bloom_filters_used is invoked\n> in a repository with only a root commit, so they don't check that\n> the output is the same with and without Bloom filters.\n\nAh...you are right. Indeed, when I flip one of the tests from\ntest_bloom_filters_used to _not_, the test still passes. I'll change\nthe tests.\n\n> The string \"split\" occurs twice in this patch series, but only in\n> patch hunk contexts, and it doesn't occur at all in the previous long\n> thread about the original patch series.\n> \n> Unfortunately, split commit-graphs weren't really considered in the\n> design of the modified path Bloom filters feature, and layers with\n> different Bloom filter settings weren't considered at all. I've\n> reported it back then, but the fixes so far were incomplete, and e.g.\n> the test cases shown in\n> \n> https://public-inbox.org/git/20201015132147.GB24954@szeder.dev/\n> \n> still fail.\n> \n> Since the interaction of different versions and split commit-graphs\n> was neither mentioned in any of the commit messages nor discussed\n> during the previous rounds, and there isn't any test case excercising\n> it, and since the Bloom filter version information is stored in the\n> same 'g->bloom_filter_settings' structure as the number of hashes, I'm\n> afraid (though haven't actually checked) that handling commit-graph\n> layers with different Bloom filter versions is prone to the same\n> issues as well.\n\nMy original design (up to patch 7 in this patch set) defends against\nthis by taking the very first version detected and rejecting every\nother version, and Taylor's subsequent design reads every version, but\nannotates filters with its version. So I think we're covered.\n"},{"id":"481138","messageId":"20230829221849.562285-1-jonathantanmy@google.com","threadId":"60142","inReplyTo":"20230825170656.2598201-1-jonathantanmy@google.com","subject":"Re: [PATCH 00/15] bloom: changed-path Bloom filters v2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-29T22:18:49Z","receivedAt":"2023-08-29T22:19:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Jonathan Tan writes:\n> I mistakenly sent my reply to an earlier version [1]. (Taylor has seen\n> it, so this note is more for future readers who might be curious about\n> what that email reply contains.)\n> \n> [1] https://lore.kernel.org/git/20230824222051.2320003-1-jonathantanmy@google.com/\n\nI think that this patch set is good for merging. There is a discussion\nbetween Taylor and me at the link above in which Taylor thinks something\nis OK and I think that that thing is OK except in a very specific\nsituation (there exists a commit-graph chain in which the different\nlayers have different Bloom filter versions). (Hopefully that's a good\nsummary of the discussion.) I think that the situation is narrow enough\nand have seen that Taylor (whose capabilities I have a high opinion of)\nhas looked into it. I think the only way for me to be fully convinced\nis to write a test that includes these different layers and see how the\ncode functions.\n\nSo I think that we can merge this patch set. An alternative would be\nto wait for me (or someone else) to write such a test, but if it's just\nme that's worried about this situation, it's probably not worth waiting\njust for this.\n\nThe other outstanding thing is that Szeder Gabor pointed out that Bloom\nfilters are not applied to root commits so some of the tests don't\ntest what you would expect [2]. I've updated the tests and pushed the\nresults to GitHub [3]. I'm OK with using that version, or the current\nversion (in which case I'll resend the updated tests once this version\nis merged).\n\n[2] https://lore.kernel.org/git/20230826150610.GA1928@szeder.dev/\n[3] https://github.com/jonathantanmy/git/tree/changedpath\n"},{"id":"481139","messageId":"xmqqcyz5bfsu.fsf@gitster.g","threadId":"60142","inReplyTo":"20230829221849.562285-1-jonathantanmy@google.com","subject":"Re: [PATCH 00/15] bloom: changed-path Bloom filters v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-29T23:16:17Z","receivedAt":"2023-08-29T23:17:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> So I think that we can merge this patch set. An alternative would be\n> to wait for me (or someone else) to write such a test, but if it's just\n> me that's worried about this situation, it's probably not worth waiting\n> just for this.\n>\n> The other outstanding thing is that Szeder Gabor pointed out that Bloom\n> filters are not applied to root commits so some of the tests don't\n> test what you would expect [2]. I've updated the tests and pushed the\n> results to GitHub [3]. I'm OK with using that version, or the current\n> version (in which case I'll resend the updated tests once this version\n> is merged).\n>\n> [2] https://lore.kernel.org/git/20230826150610.GA1928@szeder.dev/\n> [3] https://github.com/jonathantanmy/git/tree/changedpath\n\nThanks for a concise and well thought out summary.\n\nIf we know that some tests in the current round is not doing the\nright thing, and we already have an updated set of tests to fix\nthem, I doubt that the topic is so urgent that merging a known to be\nincomplete version is preferrable than seeing the hopefully final\nversion on the list that everybody can agree with.\n\nThanks.\n"},{"id":"481170","messageId":"a3b4e7ef599bea153b74ae9b7edcec3641797776.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 13/15] object.h: fix mis-aligned flag bits table","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:53Z","receivedAt":"2023-08-30T18:29:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nBit position 23 is one column too far to the left.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n object.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object.h b/object.h\nindex 114d45954d..db25714b4e 100644\n--- a/object.h\n+++ b/object.h\n@@ -62,7 +62,7 @@ void object_array_init(struct object_array *array);\n \n /*\n * object flag allocation:\n- * revision.h: 0---------10 15 23------27\n+ * revision.h: 0---------10 15 23------27\n * fetch-pack.c: 01 67\n * negotiator/default.c: 2--5\n * walker.c: 0-2\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481171","messageId":"94ad289dbb1cedfed2b8c66a3b805028e8038317.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 06/15] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:46Z","receivedAt":"2023-08-30T18:29:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A subsequent commit will introduce another version of the changed-path\nfilter in the commit graph file. In order to control which version to\nwrite (and read), a config variable is needed.\n\nTherefore, introduce this config variable. For forwards compatibility,\nteach Git to not read commit graphs when the config variable\nis set to an unsupported version. Because we teach Git this,\ncommitgraph.readChangedPaths is now redundant, so deprecate it and\ndefine its behavior in terms of the config variable we introduce.\n\nThis commit does not change the behavior of writing (Git writes changed\npath filters when explicitly instructed regardless of any config\nvariable), but a subsequent commit will restrict Git such that it will\nonly write when commitgraph.changedPathsVersion is a recognized value.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n Documentation/config/commitgraph.txt | 23 ++++++++++++++++++++---\n commit-graph.c | 2 +-\n oss-fuzz/fuzz-commit-graph.c | 2 +-\n repo-settings.c | 6 +++++-\n repository.h | 2 +-\n 5 files changed, 28 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c..2dc9170622 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -9,6 +9,23 @@ commitGraph.maxNewFilters::\n \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n \n commitGraph.readChangedPaths::\n-\tIf true, then git will use the changed-path Bloom filters in the\n-\tcommit-graph file (if it exists, and they are present). Defaults to\n-\ttrue. See linkgit:git-commit-graph[1] for more information.\n+\tDeprecated. Equivalent to commitGraph.changedPathsVersion=-1 if true, and\n+\tcommitGraph.changedPathsVersion=0 if false. (If commitGraph.changedPathVersion\n+\tis also set, commitGraph.changedPathsVersion takes precedence.)\n+\n+commitGraph.changedPathsVersion::\n+\tSpecifies the version of the changed-path Bloom filters that Git will read and\n+\twrite. May be -1, 0 or 1.\n++\n+Defaults to -1.\n++\n+If -1, Git will use the version of the changed-path Bloom filters in the\n+repository, defaulting to 1 if there are none.\n++\n+If 0, Git will not read any Bloom filters, and will write version 1 Bloom\n+filters when instructed to write.\n++\n+If 1, Git will only read version 1 Bloom filters, and will write version 1\n+Bloom filters.\n++\n+See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0aa1640d15..da99f15fdf 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -401,7 +401,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_read_changed_paths) {\n+\tif (s->commit_graph_changed_paths_version) {\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\ndiff --git a/oss-fuzz/fuzz-commit-graph.c b/oss-fuzz/fuzz-commit-graph.c\nindex 2992079dd9..325c0b991a 100644\n--- a/oss-fuzz/fuzz-commit-graph.c\n+++ b/oss-fuzz/fuzz-commit-graph.c\n@@ -19,7 +19,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \t * possible.\n \t */\n \tthe_repository->settings.commit_graph_generation_version = 2;\n-\tthe_repository->settings.commit_graph_read_changed_paths = 1;\n+\tthe_repository->settings.commit_graph_changed_paths_version = 1;\n \tg = parse_commit_graph(&the_repository->settings, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 525f69c0c7..db8fe817f3 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n \tint value;\n \tconst char *strval;\n \tint manyfiles;\n+\tint read_changed_paths;\n \n \tif (!r->gitdir)\n \t\tBUG(\"Cannot add settings for uninitialized repository\");\n@@ -54,7 +55,10 @@ void prepare_repo_settings(struct repository *r)\n \t/* Commit graph config or default, does not cascade (simple) */\n \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n-\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &read_changed_paths, 1);\n+\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n+\t\t &r->settings.commit_graph_changed_paths_version,\n+\t\t read_changed_paths ? -1 : 0);\n \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n \ndiff --git a/repository.h b/repository.h\nindex 5f18486f64..f71154e12c 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -29,7 +29,7 @@ struct repo_settings {\n \n \tint core_commit_graph;\n \tint commit_graph_generation_version;\n-\tint commit_graph_read_changed_paths;\n+\tint commit_graph_changed_paths_version;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \tint command_requires_full_index;\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481172","messageId":"3a25f90c15b65df29c31b205346d303abd54ce96.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 04/15] t/helper/test-read-graph: implement `bloom-filters` mode","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:44Z","receivedAt":"2023-08-30T18:29:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nImplement a mode of the \"read-graph\" test helper to dump out the\nhexadecimal contents of the Bloom filter(s) contained in a commit-graph.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n t/helper/test-read-graph.c | 42 +++++++++++++++++++++++++++++++++-----\n 1 file changed, 37 insertions(+), 5 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 3375392f6c..899b5f41cc 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -47,10 +47,32 @@ static void dump_graph_info(struct commit_graph *graph)\n \tprintf(\"\\n\");\n }\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_bloom_filters(struct commit_graph *graph)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < graph->num_commits + graph->num_commits_in_base; i++) {\n+\t\tstruct bloom_filter filter = { 0 };\n+\t\tsize_t j;\n+\n+\t\tif (load_bloom_filter_from_graph(graph, &filter, i) < 0) {\n+\t\t\tfprintf(stderr, \"missing Bloom filter for graph \"\n+\t\t\t\t\"position %\"PRIu32\"\\n\", i);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (j = 0; j < filter.len; j++)\n+\t\t\tprintf(\"%02x\", filter.data[j]);\n+\t\tif (filter.len)\n+\t\t\tprintf(\"\\n\");\n+\t}\n+}\n+\n+int cmd__read_graph(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n \tstruct object_directory *odb;\n+\tint ret = 0;\n \n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n@@ -58,12 +80,22 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tprepare_repo_settings(the_repository);\n \n \tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n+\tif (!graph) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n \n-\tdump_graph_info(graph);\n+\tif (argc <= 1)\n+\t\tdump_graph_info(graph);\n+\telse if (!strcmp(argv[1], \"bloom-filters\"))\n+\t\tdump_graph_bloom_filters(graph);\n+\telse {\n+\t\tfprintf(stderr, \"unknown sub-command: '%s'\\n\", argv[1]);\n+\t\tret = 1;\n+\t}\n \n+done:\n \tUNLEAK(graph);\n \n-\treturn 0;\n+\treturn ret;\n }\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481173","messageId":"a5bf23a7d303a2d16d3d75205bfc12d0149945ca.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 08/15] bloom: annotate filters with hash version","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:48Z","receivedAt":"2023-08-30T18:29:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nIn subsequent commits, we will want to load existing Bloom filters out\nof a commit-graph, even when the hash version they were computed with\ndoes not match the value of `commitGraph.changedPathVersion`.\n\nIn order to differentiate between the two, add a \"version\" field to each\nBloom filter.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n bloom.c | 11 ++++++++---\n bloom.h | 1 +\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex ebef5cfd2f..9b6a30f6f6 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -55,6 +55,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n \t\t\t\t\tsizeof(unsigned char) * start_index +\n \t\t\t\t\tBLOOMDATA_CHUNK_HEADER_SIZE);\n+\tfilter->version = g->bloom_filter_settings->hash_version;\n \n \treturn 1;\n }\n@@ -240,11 +241,13 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data UNUSED,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-static void init_truncated_large_filter(struct bloom_filter *filter)\n+static void init_truncated_large_filter(struct bloom_filter *filter,\n+\t\t\t\t\tint version)\n {\n \tfilter->data = xmalloc(1);\n \tfilter->data[0] = 0xFF;\n \tfilter->len = 1;\n+\tfilter->version = version;\n }\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n@@ -329,13 +332,15 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t}\n \n \t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths) {\n-\t\t\tinit_truncated_large_filter(filter);\n+\t\t\tinit_truncated_large_filter(filter,\n+\t\t\t\t\t\t settings->hash_version);\n \t\t\tif (computed)\n \t\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n \t\t\tgoto cleanup;\n \t\t}\n \n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tfilter->version = settings->hash_version;\n \t\tif (!filter->len) {\n \t\t\tif (computed)\n \t\t\t\t*computed |= BLOOM_TRUNC_EMPTY;\n@@ -355,7 +360,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n-\t\tinit_truncated_large_filter(filter);\n+\t\tinit_truncated_large_filter(filter, settings->hash_version);\n \n \t\tif (computed)\n \t\t\t*computed |= BLOOM_TRUNC_LARGE;\ndiff --git a/bloom.h b/bloom.h\nindex 138d57a86b..330a140520 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -55,6 +55,7 @@ struct bloom_filter_settings {\n struct bloom_filter {\n \tunsigned char *data;\n \tsize_t len;\n+\tint version;\n };\n \n /*\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481174","messageId":"dcfc98774123862ddb9390690127fa358af20d05.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 01/15] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:41Z","receivedAt":"2023-08-30T18:29:29Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The code change to Git to support version 2 will be done in subsequent\ncommits.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n Documentation/gitformat-commit-graph.txt | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\nindex 31cad585e2..3e906e8030 100644\n--- a/Documentation/gitformat-commit-graph.txt\n+++ b/Documentation/gitformat-commit-graph.txt\n@@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n \n ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n * It starts with header consisting of three unsigned 32-bit integers:\n- - Version of the hash algorithm being used. We currently only support\n-\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n+ - Version of the hash algorithm being used. We currently support\n+\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n \timplemented exactly as described in\n \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n-\tin Probabilistic Verification\"\n+\tin Probabilistic Verification\". Version 1 Bloom filters have a bug that appears\n+\twhen char is signed and the repository has path names that have characters >=\n+\t0x80; Git supports reading and writing them, but this ability will be removed\n+\tin a future version of Git.\n - The number of times a path is hashed and hence the number of bit positions\n \t that cumulatively determine whether a file is present in the commit.\n - The minimum number of bits 'b' per entry in the Bloom filter. If the filter\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481178","messageId":"cover.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH v2 00/15] bloom: changed-path Bloom filters v2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:40Z","receivedAt":"2023-08-30T18:29:29Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Here's an updated patch set containing updates to the tests, making them\nrun on non-root commits to ensure that we exercise the Bloom filters.\nThanks to SZEDER Gábor for spotting this.\n\nJonathan Tan (4):\n gitformat-commit-graph: describe version 2 of BDAT\n t4216: test changed path filters with high bit paths\n repo-settings: introduce commitgraph.changedPathsVersion\n commit-graph: new filter ver. that fixes murmur3\n\nTaylor Blau (11):\n t/helper/test-read-graph.c: extract `dump_graph_info()`\n bloom.h: make `load_bloom_filter_from_graph()` public\n t/helper/test-read-graph: implement `bloom-filters` mode\n bloom: annotate filters with hash version\n bloom: prepare to discard incompatible Bloom filters\n t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`\n commit-graph.c: unconditionally load Bloom filters\n commit-graph: drop unnecessary `graph_read_bloom_data_context`\n object.h: fix mis-aligned flag bits table\n commit-graph: reuse existing Bloom filters where possible\n bloom: introduce `deinit_bloom_filters()`\n\n Documentation/config/commitgraph.txt | 26 ++-\n Documentation/gitformat-commit-graph.txt | 9 +-\n bloom.c | 208 +++++++++++++++++++++--\n bloom.h | 38 ++++-\n commit-graph.c | 36 +++-\n object.h | 3 +-\n oss-fuzz/fuzz-commit-graph.c | 2 +-\n repo-settings.c | 6 +-\n repository.h | 2 +-\n t/helper/test-bloom.c | 9 +-\n t/helper/test-read-graph.c | 65 +++++--\n t/t0095-bloom.sh | 8 +\n t/t4216-log-bloom.sh | 202 +++++++++++++++++++++-\n 13 files changed, 564 insertions(+), 50 deletions(-)\n\nRange-diff against v1:\n 1: dcfc987741 = 1: dcfc987741 gitformat-commit-graph: describe version 2 of BDAT\n 2: 0c56f2a9e9 = 2: 0c56f2a9e9 t/helper/test-read-graph.c: extract `dump_graph_info()`\n 3: 8405b845e5 = 3: 8405b845e5 bloom.h: make `load_bloom_filter_from_graph()` public\n 4: 3a25f90c15 = 4: 3a25f90c15 t/helper/test-read-graph: implement `bloom-filters` mode\n 5: e300d338e1 ! 5: 7858010665 t4216: test changed path filters with high bit paths\n @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm\n +\ttrue\n +'\n +\n ++test_expect_success 'setup make another commit' '\n ++\t# \"git log\" does not use Bloom filters for root commits - see how, in\n ++\t# revision.c, rev_compare_tree() (the only code path that eventually calls\n ++\t# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit\n ++\t# has one parent. Therefore, make another commit so that we perform the tests on\n ++\t# a non-root commit.\n ++\ttest_commit -C highbit1 anotherc1 \"another$CENT\"\n ++'\n ++\n +test_expect_success 'version 1 changed-path used when version 1 requested' '\n +\t(\n +\t\tcd highbit1 &&\n -+\t\ttest_bloom_filters_used \"-- $CENT\"\n ++\t\ttest_bloom_filters_used \"-- another$CENT\"\n +\t)\n +'\n +\n 6: 6bc665e1d3 = 6: 94ad289dbb repo-settings: introduce commitgraph.changedPathsVersion\n 7: 7ef3b2bbbd ! 7: 44d3163125 commit-graph: new filter ver. that fixes murmur3\n @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n +\t(\n +\t\tcd highbit1 &&\n +\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n -+\t\ttest_bloom_filters_not_used \"-- $CENT\"\n ++\t\ttest_bloom_filters_not_used \"-- another$CENT\"\n +\t)\n +'\n +\n @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n +\t(\n +\t\tcd highbit1 &&\n +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n -+\t\ttest_bloom_filters_used \"-- $CENT\"\n ++\t\ttest_bloom_filters_used \"-- another$CENT\"\n +\t)\n +'\n +\n @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n +\t)\n +'\n +\n ++test_expect_success 'setup make another commit' '\n ++\t# \"git log\" does not use Bloom filters for root commits - see how, in\n ++\t# revision.c, rev_compare_tree() (the only code path that eventually calls\n ++\t# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit\n ++\t# has one parent. Therefore, make another commit so that we perform the tests on\n ++\t# a non-root commit.\n ++\ttest_commit -C highbit2 anotherc2 \"another$CENT\"\n ++'\n ++\n +test_expect_success 'version 2 changed-path used when version 2 requested' '\n +\t(\n +\t\tcd highbit2 &&\n -+\t\ttest_bloom_filters_used \"-- $CENT\"\n ++\t\ttest_bloom_filters_used \"-- another$CENT\"\n +\t)\n +'\n +\n @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n +\t(\n +\t\tcd highbit2 &&\n +\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n -+\t\ttest_bloom_filters_not_used \"-- $CENT\"\n ++\t\ttest_bloom_filters_not_used \"-- another$CENT\"\n +\t)\n +'\n +\n @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n +\t(\n +\t\tcd highbit2 &&\n +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n -+\t\ttest_bloom_filters_used \"-- $CENT\"\n ++\t\ttest_bloom_filters_used \"-- another$CENT\"\n +\t)\n +'\n +\n 8: 302caee39d = 8: a5bf23a7d3 bloom: annotate filters with hash version\n 9: d2b0726266 = 9: 3de6cd8460 bloom: prepare to discard incompatible Bloom filters\n10: 9a9992220f = 10: ef04389a0e t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`\n11: 607945ab05 = 11: e0c1c1ccec commit-graph.c: unconditionally load Bloom filters\n12: e397d83895 = 12: 4d57f51854 commit-graph: drop unnecessary `graph_read_bloom_data_context`\n13: bd3ad6b6c0 = 13: a3b4e7ef59 object.h: fix mis-aligned flag bits table\n14: 2996f0fdb6 = 14: 05357f9533 commit-graph: reuse existing Bloom filters where possible\n15: a3b5b22db0 = 15: 58a1d90e6d bloom: introduce `deinit_bloom_filters()`\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481175","messageId":"e0c1c1ccec9aa61fc5613aff0519f2a6dd6493fd.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 11/15] commit-graph.c: unconditionally load Bloom filters","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:51Z","receivedAt":"2023-08-30T18:29:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nIn 9e4df4da07 (commit-graph: new filter ver. that fixes murmur3,\n2023-08-01), we began ignoring the Bloom data (\"BDAT\") chunk for\ncommit-graphs whose Bloom filters were computed using a hash version\nincompatible with the value of `commitGraph.changedPathVersion`.\n\nNow that the Bloom API has been hardened to discard these incompatible\nfilters (with the exception of low-level APIs), we can safely load these\nBloom filters unconditionally.\n\nWe no longer want to return early from `graph_read_bloom_data()`, and\nsimilarly do not want to set the bloom_settings' `hash_version` field as\na side-effect. The latter is because we want to wait until we know which\nBloom settings we're using (either the defaults, from the GIT_TEST\nvariables, or from the previous commit-graph layer) before deciding what\nhash_version to use.\n\nIf we detect an existing BDAT chunk, we'll infer the rest of the\nsettings (e.g., number of hashes, bits per entry, and maximum number of\nchanged paths) from the earlier graph layer. The hash_version will be\ninferred from the previous layer as well, unless one has already been\nspecified via configuration.\n\nOnce all of that is done, we normalize the value of the hash_version to\neither \"1\" or \"2\".\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n commit-graph.c | 19 ++++++++++---------\n 1 file changed, 10 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f7322c4fff..665a3edf78 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -317,12 +317,6 @@ static int graph_read_bloom_data(const unsigned char *chunk_start,\n \tuint32_t hash_version;\n \thash_version = get_be32(chunk_start);\n \n-\tif (*c->commit_graph_changed_paths_version == -1) {\n-\t\t*c->commit_graph_changed_paths_version = hash_version;\n-\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n-\t\treturn 0;\n-\t}\n-\n \tg->chunk_bloom_data = chunk_start;\n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n \tg->bloom_filter_settings->hash_version = hash_version;\n@@ -2408,8 +2402,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n-\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n-\t\t? 2 : 1;\n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2441,10 +2434,18 @@ int write_commit_graph(struct object_directory *odb,\n \t\t/* We have changed-paths already. Keep them in the next graph */\n \t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n-\t\t\tctx->bloom_settings = g->bloom_filter_settings;\n+\n+\t\t\t/* don't propagate the hash_version unless unspecified */\n+\t\t\tif (bloom_settings.hash_version == -1)\n+\t\t\t\tbloom_settings.hash_version = g->bloom_filter_settings->hash_version;\n+\t\t\tbloom_settings.bits_per_entry = g->bloom_filter_settings->bits_per_entry;\n+\t\t\tbloom_settings.num_hashes = g->bloom_filter_settings->num_hashes;\n+\t\t\tbloom_settings.max_changed_paths = g->bloom_filter_settings->max_changed_paths;\n \t\t}\n \t}\n \n+\tbloom_settings.hash_version = bloom_settings.hash_version == 2 ? 2 : 1;\n+\n \tif (ctx->split) {\n \t\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n \n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481176","messageId":"ef04389a0e791610551d420a8aaf503b503f6d0a.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 10/15] t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:50Z","receivedAt":"2023-08-30T18:29:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nThe existing implementation of test_bloom_filters_not_used() asserts\nthat the Bloom filter sub-system has not been initialized at all, by\nchecking for the absence of any data from it from trace2.\n\nIn the following commit, it will become possible to load Bloom filters\nwithout using them (e.g., because `commitGraph.changedPathVersion` is\nincompatible with the hash version with which the commit-graph's Bloom\nfilters were written).\n\nWhen this is the case, it's possible to initialize the Bloom filter\nsub-system, while still not using any Bloom filters. When this is the\ncase, check that the data dump from the Bloom sub-system is all zeros,\nindicating that no filters were used.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n t/t4216-log-bloom.sh | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 1d0e11d7c1..940a71d8b8 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -81,7 +81,19 @@ test_bloom_filters_used () {\n test_bloom_filters_not_used () {\n \tlog_args=$1\n \tsetup \"$log_args\" &&\n-\t! grep -q \"statistics:{\\\"filter_not_present\\\":\" \"$TRASH_DIRECTORY/trace.perf\" &&\n+\n+\tif grep -q \"statistics:{\\\"filter_not_present\\\":\" \"$TRASH_DIRECTORY/trace.perf\"\n+\tthen\n+\t\t# if the Bloom filter system is initialized, ensure that no\n+\t\t# filters were used\n+\t\tdata=\"statistics:{\"\n+\t\tdata=\"$data\\\"filter_not_present\\\":0,\"\n+\t\tdata=\"$data\\\"maybe\\\":0,\"\n+\t\tdata=\"$data\\\"definitely_not\\\":0,\"\n+\t\tdata=\"$data\\\"false_positive\\\":0}\"\n+\n+\t\tgrep -q \"$data\" \"$TRASH_DIRECTORY/trace.perf\"\n+\tfi &&\n \ttest_cmp log_wo_bloom log_w_bloom\n }\n \n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481182","messageId":"3de6cd8460de51c3d866b0df9c219dd985f7433f.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 09/15] bloom: prepare to discard incompatible Bloom filters","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:49Z","receivedAt":"2023-08-30T18:29:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nCallers use the inline `get_bloom_filter()` implementation as a thin\nwrapper around `get_or_compute_bloom_filter()`. The former calls the\nlatter with a value of \"0\" for `compute_if_not_present`, making\n`get_bloom_filter()` the default read-only path for fetching an existing\nBloom filter.\n\nCallers expect the value returned from `get_bloom_filter()` is usable,\nthat is that it's compatible with the configured value corresponding to\n`commitGraph.changedPathsVersion`.\n\nThis is OK, since the commit-graph machinery only initializes its BDAT\nchunk (thereby enabling it to service Bloom filter queries) when the\nBloom filter hash_version is compatible with our settings. So any value\nreturned by `get_bloom_filter()` is trivially useable.\n\nHowever, subsequent commits will load the BDAT chunk even when the Bloom\nfilters are built with incompatible hash versions. Prepare to handle\nthis by teaching `get_bloom_filter()` to discard filters that are\nincompatible with the configured hash version.\n\nCallers who wish to read incompatible filters (e.g., for upgrading\nfilters from v1 to v2) may use the lower level routine,\n`get_or_compute_bloom_filter()`.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n bloom.c | 20 +++++++++++++++++++-\n bloom.h | 20 ++++++++++++++++++--\n 2 files changed, 37 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 9b6a30f6f6..739fa093ba 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -250,6 +250,23 @@ static void init_truncated_large_filter(struct bloom_filter *filter,\n \tfilter->version = version;\n }\n \n+struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c)\n+{\n+\tstruct bloom_filter *filter;\n+\tint hash_version;\n+\n+\tfilter = get_or_compute_bloom_filter(r, c, 0, NULL, NULL);\n+\tif (!filter)\n+\t\treturn NULL;\n+\n+\tprepare_repo_settings(r);\n+\thash_version = r->settings.commit_graph_changed_paths_version;\n+\n+\tif (!(hash_version == -1 || hash_version == filter->version))\n+\t\treturn NULL; /* unusable filter */\n+\treturn filter;\n+}\n+\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n@@ -275,7 +292,8 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t filter, graph_pos);\n \t}\n \n-\tif (filter->data && filter->len)\n+\tif ((filter->data && filter->len) &&\n+\t (!settings || settings->hash_version == filter->version))\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\ndiff --git a/bloom.h b/bloom.h\nindex 330a140520..bfe389e29c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -110,8 +110,24 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t enum bloom_filter_computed *computed);\n \n-#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n-\t(r), (c), 0, NULL, NULL)\n+/*\n+ * Find the Bloom filter associated with the given commit \"c\".\n+ *\n+ * If any of the following are true\n+ *\n+ * - the repository does not have a commit-graph, or\n+ * - the repository disables reading from the commit-graph, or\n+ * - the given commit does not have a Bloom filter computed, or\n+ * - there is a Bloom filter for commit \"c\", but it cannot be read\n+ * because the filter uses an incompatible version of murmur3\n+ *\n+ * , then `get_bloom_filter()` will return NULL. Otherwise, the corresponding\n+ * Bloom filter will be returned.\n+ *\n+ * For callers who wish to inspect Bloom filters with incompatible hash\n+ * versions, use get_or_compute_bloom_filter().\n+ */\n+struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c);\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481183","messageId":"4d57f5185400fd44956b596fe8c32cbd1ba185de.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 12/15] commit-graph: drop unnecessary `graph_read_bloom_data_context`","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:52Z","receivedAt":"2023-08-30T18:29:37Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nThe `graph_read_bloom_data_context` struct was introduced in an earlier\ncommit in order to pass pointers to the commit-graph and changed-path\nBloom filter version when reading the BDAT chunk.\n\nThe previous commit no longer writes through the changed_paths_version\npointer, making the surrounding context structure unnecessary. Drop it\nand pass a pointer to the commit-graph directly when reading the BDAT\nchunk.\n\nNoticed-by: Jonathan Tan \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n commit-graph.c | 14 ++------------\n 1 file changed, 2 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 665a3edf78..a8e33c0739 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -304,16 +304,10 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n-struct graph_read_bloom_data_context {\n-\tstruct commit_graph *g;\n-\tint *commit_graph_changed_paths_version;\n-};\n-\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t size_t chunk_size, void *data)\n {\n-\tstruct graph_read_bloom_data_context *c = data;\n-\tstruct commit_graph *g = c->g;\n+\tstruct commit_graph *g = data;\n \tuint32_t hash_version;\n \thash_version = get_be32(chunk_start);\n \n@@ -405,14 +399,10 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t}\n \n \tif (s->commit_graph_changed_paths_version) {\n-\t\tstruct graph_read_bloom_data_context context = {\n-\t\t\t.g = graph,\n-\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n-\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t graph_read_bloom_data, &context);\n+\t\t\t graph_read_bloom_data, graph);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481185","messageId":"44d3163125026843d75ec609401d96acc3387f28.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:47Z","receivedAt":"2023-08-30T18:29:40Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 (thus,\nthe changed path filters wouldn't be readable by other off-the-shelf\nimplementatios of murmur3) and even with Git binaries that were compiled\nwith different signedness of char. This bug affects both how Git writes\nchanged path filters to disk and how Git interprets changed path filters\non disk.\n\nTherefore, introduce a new version (2) of changed path filters that\ncorrects this problem. The existing version (1) is still supported and\nis still the default, but users should migrate away from it as soon\nas possible.\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter.\n\nSo this patch does not include any mechanism to \"salvage\" changed path\nfilters from repositories. There is also no \"mixed\" mode - for each\ninvocation of Git, reading and writing changed path filters are done\nwith the same version number; this version number may be explicitly\nstated (typically if the user knows which version they need) or\nautomatically determined from the version of the existing changed path\nfilters in the repository.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in t0095 was obtained from another murmur3 implementation\nusing the following Go source code:\n\n package main\n\n import \"fmt\"\n import \"github.com/spaolacci/murmur3\"\n\n func main() {\n fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n }\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n Documentation/config/commitgraph.txt | 5 +-\n bloom.c | 69 +++++++++++++++++-\n bloom.h | 8 +-\n commit-graph.c | 32 ++++++--\n t/helper/test-bloom.c | 9 ++-\n t/t0095-bloom.sh | 8 ++\n t/t4216-log-bloom.sh | 105 +++++++++++++++++++++++++++\n 7 files changed, 223 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 2dc9170622..acc74a2f27 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -15,7 +15,7 @@ commitGraph.readChangedPaths::\n \n commitGraph.changedPathsVersion::\n \tSpecifies the version of the changed-path Bloom filters that Git will read and\n-\twrite. May be -1, 0 or 1.\n+\twrite. May be -1, 0, 1, or 2.\n +\n Defaults to -1.\n +\n@@ -28,4 +28,7 @@ filters when instructed to write.\n If 1, Git will only read version 1 Bloom filters, and will write version 1\n Bloom filters.\n +\n+If 2, Git will only read version 2 Bloom filters, and will write version 2\n+Bloom filters.\n++\n See linkgit:git-commit-graph[1] for more information.\ndiff --git a/bloom.c b/bloom.c\nindex 3e78cfe79d..ebef5cfd2f 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -66,7 +66,64 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n * Not considered to be cryptographically secure.\n * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n+{\n+\tconst uint32_t c1 = 0xcc9e2d51;\n+\tconst uint32_t c2 = 0x1b873593;\n+\tconst uint32_t r1 = 15;\n+\tconst uint32_t r2 = 13;\n+\tconst uint32_t m = 5;\n+\tconst uint32_t n = 0xe6546b64;\n+\tint i;\n+\tuint32_t k1 = 0;\n+\tconst char *tail;\n+\n+\tint len4 = len / sizeof(uint32_t);\n+\n+\tuint32_t k;\n+\tfor (i = 0; i < len4; i++) {\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n+\t\tk = byte1 | byte2 | byte3 | byte4;\n+\t\tk *= c1;\n+\t\tk = rotate_left(k, r1);\n+\t\tk *= c2;\n+\n+\t\tseed ^= k;\n+\t\tseed = rotate_left(seed, r2) * m + n;\n+\t}\n+\n+\ttail = (data + len4 * sizeof(uint32_t));\n+\n+\tswitch (len & (sizeof(uint32_t) - 1)) {\n+\tcase 3:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n+\t\t/*-fallthrough*/\n+\tcase 2:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n+\t\t/*-fallthrough*/\n+\tcase 1:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n+\t\tk1 *= c1;\n+\t\tk1 = rotate_left(k1, r1);\n+\t\tk1 *= c2;\n+\t\tseed ^= k1;\n+\t\tbreak;\n+\t}\n+\n+\tseed ^= (uint32_t)len;\n+\tseed ^= (seed >> 16);\n+\tseed *= 0x85ebca6b;\n+\tseed ^= (seed >> 13);\n+\tseed *= 0xc2b2ae35;\n+\tseed ^= (seed >> 16);\n+\n+\treturn seed;\n+}\n+\n+static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n {\n \tconst uint32_t c1 = 0xcc9e2d51;\n \tconst uint32_t c2 = 0x1b873593;\n@@ -131,8 +188,14 @@ void fill_bloom_key(const char *data,\n \tint i;\n \tconst uint32_t seed0 = 0x293ae76f;\n \tconst uint32_t seed1 = 0x7e646e2c;\n-\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n-\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n+\tuint32_t hash0, hash1;\n+\tif (settings->hash_version == 2) {\n+\t\thash0 = murmur3_seeded_v2(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v2(seed1, data, len);\n+\t} else {\n+\t\thash0 = murmur3_seeded_v1(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v1(seed1, data, len);\n+\t}\n \n \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n \tfor (i = 0; i < settings->num_hashes; i++)\ndiff --git a/bloom.h b/bloom.h\nindex 1e4f612d2c..138d57a86b 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -8,9 +8,11 @@ struct commit_graph;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * The newest version is 2, which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -80,7 +82,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n * Not considered to be cryptographically secure.\n * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n \n void fill_bloom_key(const char *data,\n \t\t size_t len,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex da99f15fdf..f7322c4fff 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -304,17 +304,26 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct graph_read_bloom_data_context {\n+\tstruct commit_graph *g;\n+\tint *commit_graph_changed_paths_version;\n+};\n+\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t size_t chunk_size, void *data)\n {\n-\tstruct commit_graph *g = data;\n+\tstruct graph_read_bloom_data_context *c = data;\n+\tstruct commit_graph *g = c->g;\n \tuint32_t hash_version;\n-\tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n+\tif (*c->commit_graph_changed_paths_version == -1) {\n+\t\t*c->commit_graph_changed_paths_version = hash_version;\n+\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n \t\treturn 0;\n+\t}\n \n+\tg->chunk_bloom_data = chunk_start;\n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n \tg->bloom_filter_settings->hash_version = hash_version;\n \tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n@@ -402,10 +411,14 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t}\n \n \tif (s->commit_graph_changed_paths_version) {\n+\t\tstruct graph_read_bloom_data_context context = {\n+\t\t\t.g = graph,\n+\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n+\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t graph_read_bloom_data, graph);\n+\t\t\t graph_read_bloom_data, &context);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -2376,6 +2389,13 @@ int write_commit_graph(struct object_directory *odb,\n \t}\n \tif (!commit_graph_compatible(r))\n \t\treturn 0;\n+\tif (r->settings.commit_graph_changed_paths_version < -1\n+\t || r->settings.commit_graph_changed_paths_version > 2) {\n+\t\twarning(_(\"attempting to write a commit-graph, but \"\n+\t\t\t \"'commitgraph.changedPathsVersion' (%d) is not supported\"),\n+\t\t\tr->settings.commit_graph_changed_paths_version);\n+\t\treturn 0;\n+\t}\n \n \tCALLOC_ARRAY(ctx, 1);\n \tctx->r = r;\n@@ -2388,6 +2408,8 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n+\t\t? 2 : 1;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2417,7 +2439,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex aabe31d724..3cbc0a5b50 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -50,6 +50,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \" test-tool bloom get_murmur3 \\n\"\n+\" test-tool bloom get_murmur3_seven_highbit\\n\"\n \" test-tool bloom generate_filter [...]\\n\"\n \" test-tool bloom get_filter_for_commit \\n\";\n \n@@ -64,7 +65,13 @@ int cmd__bloom(int argc, const char **argv)\n \t\tuint32_t hashed;\n \t\tif (argc < 3)\n \t\t\tusage(bloom_usage);\n-\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n+\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 0aa9719934..1d0e11d7c1 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -456,4 +456,109 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n \t)\n '\n \n+test_expect_success 'version 1 changed-path not used when version 2 requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n+\t\ttest_bloom_filters_not_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 1 changed-path used when autodetect requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n+\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(1,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n+\tgit init highbit2 &&\n+\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n+\ttest_commit -C highbit2 c2 \"$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'check value of version 2 changed-path' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\techo \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'setup make another commit' '\n+\t# \"git log\" does not use Bloom filters for root commits - see how, in\n+\t# revision.c, rev_compare_tree() (the only code path that eventually calls\n+\t# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit\n+\t# has one parent. Therefore, make another commit so that we perform the tests on\n+\t# a non-root commit.\n+\ttest_commit -C highbit2 anotherc2 \"another$CENT\"\n+'\n+\n+test_expect_success 'version 2 changed-path used when version 2 requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\ttest_bloom_filters_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path not used when version 1 requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n+\t\ttest_bloom_filters_not_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path used when autodetect requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n+\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(2,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n+\tgit init doublewrite &&\n+\ttest_commit -C doublewrite c \"$CENT\" &&\n+\tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n+\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n+\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd doublewrite &&\n+\t\techo \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481184","messageId":"0c56f2a9e970b0ba2063c87254cd8e5d24f506de.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 02/15] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:42Z","receivedAt":"2023-08-30T18:29:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nPrepare for the 'read-graph' test helper to perform other tasks besides\ndumping high-level information about the commit-graph by extracting its\nmain routine into a separate function.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n t/helper/test-read-graph.c | 31 ++++++++++++++++++-------------\n 1 file changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 8c7a83f578..3375392f6c 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -5,20 +5,8 @@\n #include \"bloom.h\"\n #include \"setup.h\"\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_info(struct commit_graph *graph)\n {\n-\tstruct commit_graph *graph = NULL;\n-\tstruct object_directory *odb;\n-\n-\tsetup_git_directory();\n-\todb = the_repository->objects->odb;\n-\n-\tprepare_repo_settings(the_repository);\n-\n-\tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n@@ -57,6 +45,23 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tif (graph->topo_levels)\n \t\tprintf(\" topo_levels\");\n \tprintf(\"\\n\");\n+}\n+\n+int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tstruct object_directory *odb;\n+\n+\tsetup_git_directory();\n+\todb = the_repository->objects->odb;\n+\n+\tprepare_repo_settings(the_repository);\n+\n+\tgraph = read_commit_graph_one(the_repository, odb);\n+\tif (!graph)\n+\t\treturn 1;\n+\n+\tdump_graph_info(graph);\n \n \tUNLEAK(graph);\n \n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481187","messageId":"785801066563fdc9ec74635886aaea36347e45dd.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 05/15] t4216: test changed path filters with high bit paths","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:45Z","receivedAt":"2023-08-30T18:29:49Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Subsequent commits will teach Git another version of changed path\nfilter that has different behavior with paths that contain at least\none character with its high bit set, so test the existing behavior as\na baseline.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n t/t4216-log-bloom.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..0aa9719934 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -404,4 +404,56 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n+get_first_changed_path_filter () {\n+\ttest-tool read-graph bloom-filters >filters.dat &&\n+\thead -n 1 filters.dat\n+}\n+\n+# chosen to be the same under all Unicode normalization forms\n+CENT=$(printf \"\\302\\242\")\n+\n+test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n+\tgit init highbit1 &&\n+\ttest_commit -C highbit1 c1 \"$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'setup check value of version 1 changed-path' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\techo \"52a9\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+# expect will not match actual if char is unsigned by default. Write the test\n+# in this way, so that a user running this test script can still see if the two\n+# files match. (It will appear as an ordinary success if they match, and a skip\n+# if not.)\n+if test_cmp highbit1/expect highbit1/actual\n+then\n+\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n+fi\n+test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n+\t# Only the prereq matters for this test.\n+\ttrue\n+'\n+\n+test_expect_success 'setup make another commit' '\n+\t# \"git log\" does not use Bloom filters for root commits - see how, in\n+\t# revision.c, rev_compare_tree() (the only code path that eventually calls\n+\t# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit\n+\t# has one parent. Therefore, make another commit so that we perform the tests on\n+\t# a non-root commit.\n+\ttest_commit -C highbit1 anotherc1 \"another$CENT\"\n+'\n+\n+test_expect_success 'version 1 changed-path used when version 1 requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\ttest_bloom_filters_used \"-- another$CENT\"\n+\t)\n+'\n+\n test_done\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481189","messageId":"05357f9533d6ca6dd51cce3cd399fbbcedfcf93a.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 14/15] commit-graph: reuse existing Bloom filters where possible","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:54Z","receivedAt":"2023-08-30T18:29:54Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nIn 9e4df4da07 (commit-graph: new filter ver. that fixes murmur3,\n2023-08-01), a bug was described where it's possible for Git to produce\nnon-murmur3 hashes when the platform's \"char\" type is signed, and there\nare paths with characters whose highest bit is set (i.e. all characters\n>= 0x80).\n\nThat patch allows the caller to control which version of Bloom filters\nare read and written. However, even on platforms with a signed \"char\"\ntype, it is possible to reuse existing Bloom filters if and only if\nthere are no changed paths in any commit's first parent tree-diff whose\ncharacters have their highest bit set.\n\nWhen this is the case, we can reuse the existing filter without having\nto compute a new one. This is done by marking trees which are known to\nhave (or not have) any such paths. When a commit's root tree is verified\nto not have any such paths, we mark it as such and declare that the\ncommit's Bloom filter is reusable.\n\nNote that this heuristic only goes in one direction. If neither a commit\nnor its first parent have any paths in their trees with non-ASCII\ncharacters, then we know for certain that a path with non-ASCII\ncharacters will not appear in a tree-diff against that commit's first\nparent. The reverse isn't necessarily true: just because the tree-diff\ndoesn't contain any such paths does not imply that no such paths exist\nin either tree.\n\nSo we end up recomputing some Bloom filters that we don't strictly have\nto (i.e. their bits are the same no matter which version of murmur3 we\nuse). But culling these out is impossible, since we'd have to perform\nthe full tree-diff, which is the same effort as computing the Bloom\nfilter from scratch.\n\nBut because we can cache our results in each tree's flag bits, we can\noften avoid recomputing many filters, thereby reducing the time it takes\nto run\n\n $ git commit-graph write --changed-paths --reachable\n\nwhen upgrading from v1 to v2 Bloom filters.\n\nTo benchmark this, let's generate a commit-graph in linux.git with v1\nchanged-paths in generation order[^1]:\n\n $ git clone git@github.com:torvalds/linux.git\n $ cd linux\n $ git commit-graph write --reachable --changed-paths\n $ graph=\".git/objects/info/commit-graph\"\n $ mv $graph{,.bak}\n\nThen let's time how long it takes to go from v1 to v2 filters (with and\nwithout the upgrade path enabled), resetting the state of the\ncommit-graph each time:\n\n $ git config commitGraph.changedPathsVersion 2\n $ hyperfine -p 'cp -f $graph.bak $graph' -L v 0,1 \\\n 'GIT_TEST_UPGRADE_BLOOM_FILTERS={v} git.compile commit-graph write --reachable --changed-paths'\n\nOn linux.git (where there aren't any non-ASCII paths), the timings\nindicate that this patch represents a speed-up over recomputing all\nBloom filters from scratch:\n\n Benchmark 1: GIT_TEST_UPGRADE_BLOOM_FILTERS=0 git.compile commit-graph write --reachable --changed-paths\n Time (mean ± σ): 124.873 s ± 0.316 s [User: 124.081 s, System: 0.643 s]\n Range (min … max): 124.621 s … 125.227 s 3 runs\n\n Benchmark 2: GIT_TEST_UPGRADE_BLOOM_FILTERS=1 git.compile commit-graph write --reachable --changed-paths\n Time (mean ± σ): 79.271 s ± 0.163 s [User: 74.611 s, System: 4.521 s]\n Range (min … max): 79.112 s … 79.437 s 3 runs\n\n Summary\n 'GIT_TEST_UPGRADE_BLOOM_FILTERS=1 git.compile commit-graph write --reachable --changed-paths' ran\n 1.58 ± 0.01 times faster than 'GIT_TEST_UPGRADE_BLOOM_FILTERS=0 git.compile commit-graph write --reachable --changed-paths'\n\nOn git.git, we do have some non-ASCII paths, giving us a more modest\nimprovement from 4.163 seconds to 3.348 seconds, for a 1.24x speed-up.\nOn my machine, the stats for git.git are:\n\n - 8,285 Bloom filters computed from scratch\n - 10 Bloom filters generated as empty\n - 4 Bloom filters generated as truncated due to too many changed paths\n - 65,114 Bloom filters were reused when transitioning from v1 to v2.\n\n[^1]: Note that this is is important, since `--stdin-packs` or\n `--stdin-commits` orders commits in the commit-graph by their pack\n position (with `--stdin-packs`) or in the raw input (with\n `--stdin-commits`).\n\n Since we compute Bloom filters in the same order that commits appear\n in the graph, we must see a commit's (first) parent before we process\n the commit itself. This is only guaranteed to happen when sorting\n commits by their generation number.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n bloom.c | 90 ++++++++++++++++++++++++++++++++++++++++++--\n bloom.h | 1 +\n commit-graph.c | 5 +++\n object.h | 1 +\n t/t4216-log-bloom.sh | 35 ++++++++++++++++-\n 5 files changed, 127 insertions(+), 5 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 739fa093ba..24dd874e46 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -7,6 +7,9 @@\n #include \"commit-graph.h\"\n #include \"commit.h\"\n #include \"commit-slab.h\"\n+#include \"tree.h\"\n+#include \"tree-walk.h\"\n+#include \"config.h\"\n \n define_commit_slab(bloom_filter_slab, struct bloom_filter);\n \n@@ -250,6 +253,73 @@ static void init_truncated_large_filter(struct bloom_filter *filter,\n \tfilter->version = version;\n }\n \n+#define VISITED (1u<<21)\n+#define HIGH_BITS (1u<<22)\n+\n+static int has_entries_with_high_bit(struct repository *r, struct tree *t)\n+{\n+\tif (parse_tree(t))\n+\t\treturn 1;\n+\n+\tif (!(t->object.flags & VISITED)) {\n+\t\tstruct tree_desc desc;\n+\t\tstruct name_entry entry;\n+\n+\t\tinit_tree_desc(&desc, t->buffer, t->size);\n+\t\twhile (tree_entry(&desc, &entry)) {\n+\t\t\tsize_t i;\n+\t\t\tfor (i = 0; i < entry.pathlen; i++) {\n+\t\t\t\tif (entry.path[i] & 0x80) {\n+\t\t\t\t\tt->object.flags |= HIGH_BITS;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\t\tstruct tree *sub = lookup_tree(r, &entry.oid);\n+\t\t\t\tif (sub && has_entries_with_high_bit(r, sub)) {\n+\t\t\t\t\tt->object.flags |= HIGH_BITS;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t}\n+\n+done:\n+\t\tt->object.flags |= VISITED;\n+\t}\n+\n+\treturn !!(t->object.flags & HIGH_BITS);\n+}\n+\n+static int commit_tree_has_high_bit_paths(struct repository *r,\n+\t\t\t\t\t struct commit *c)\n+{\n+\tstruct tree *t;\n+\tif (repo_parse_commit(r, c))\n+\t\treturn 1;\n+\tt = repo_get_commit_tree(r, c);\n+\tif (!t)\n+\t\treturn 1;\n+\treturn has_entries_with_high_bit(r, t);\n+}\n+\n+static struct bloom_filter *upgrade_filter(struct repository *r, struct commit *c,\n+\t\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t\t int hash_version)\n+{\n+\tstruct commit_list *p = c->parents;\n+\tif (commit_tree_has_high_bit_paths(r, c))\n+\t\treturn NULL;\n+\n+\tif (p && commit_tree_has_high_bit_paths(r, p->item))\n+\t\treturn NULL;\n+\n+\tfilter->version = hash_version;\n+\n+\treturn filter;\n+}\n+\n struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c)\n {\n \tstruct bloom_filter *filter;\n@@ -292,9 +362,23 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t filter, graph_pos);\n \t}\n \n-\tif ((filter->data && filter->len) &&\n-\t (!settings || settings->hash_version == filter->version))\n-\t\treturn filter;\n+\tif (filter->data && filter->len) {\n+\t\tstruct bloom_filter *upgrade;\n+\t\tif (!settings || settings->hash_version == filter->version)\n+\t\t\treturn filter;\n+\n+\t\t/* version mismatch, see if we can upgrade */\n+\t\tif (compute_if_not_present &&\n+\t\t git_env_bool(\"GIT_TEST_UPGRADE_BLOOM_FILTERS\", 1)) {\n+\t\t\tupgrade = upgrade_filter(r, c, filter,\n+\t\t\t\t\t\t settings->hash_version);\n+\t\t\tif (upgrade) {\n+\t\t\t\tif (computed)\n+\t\t\t\t\t*computed |= BLOOM_UPGRADED;\n+\t\t\t\treturn upgrade;\n+\t\t\t}\n+\t\t}\n+\t}\n \tif (!compute_if_not_present)\n \t\treturn NULL;\n \ndiff --git a/bloom.h b/bloom.h\nindex bfe389e29c..e3a9b68905 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -102,6 +102,7 @@ enum bloom_filter_computed {\n \tBLOOM_COMPUTED = (1 << 1),\n \tBLOOM_TRUNC_LARGE = (1 << 2),\n \tBLOOM_TRUNC_EMPTY = (1 << 3),\n+\tBLOOM_UPGRADED = (1 << 4),\n };\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a8e33c0739..a3473df515 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1045,6 +1045,7 @@ struct write_commit_graph_context {\n \tint count_bloom_filter_not_computed;\n \tint count_bloom_filter_trunc_empty;\n \tint count_bloom_filter_trunc_large;\n+\tint count_bloom_filter_upgraded;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1651,6 +1652,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \t\t\t ctx->count_bloom_filter_trunc_empty);\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-large\",\n \t\t\t ctx->count_bloom_filter_trunc_large);\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-upgraded\",\n+\t\t\t ctx->count_bloom_filter_upgraded);\n }\n \n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n@@ -1692,6 +1695,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\tctx->count_bloom_filter_trunc_empty++;\n \t\t\tif (computed & BLOOM_TRUNC_LARGE)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n+\t\t} else if (computed & BLOOM_UPGRADED) {\n+\t\t\tctx->count_bloom_filter_upgraded++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n \t\t\tctx->count_bloom_filter_not_computed++;\n \t\tctx->total_bloom_filter_data_size += filter\ndiff --git a/object.h b/object.h\nindex db25714b4e..2e5e08725f 100644\n--- a/object.h\n+++ b/object.h\n@@ -75,6 +75,7 @@ void object_array_init(struct object_array *array);\n * commit-reach.c: 16-----19\n * sha1-name.c: 20\n * list-objects-filter.c: 21\n+ * bloom.c: 2122\n * builtin/fsck.c: 0--3\n * builtin/gc.c: 0\n * builtin/index-pack.c: 2021\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 940a71d8b8..502115abc3 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -217,6 +217,10 @@ test_filter_trunc_large () {\n \tgrep \"\\\"key\\\":\\\"filter-trunc-large\\\",\\\"value\\\":\\\"$1\\\"\" $2\n }\n \n+test_filter_upgraded () {\n+\tgrep \"\\\"key\\\":\\\"filter-upgraded\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n test_expect_success 'correctly report changes over limit' '\n \tgit init limits &&\n \t(\n@@ -561,10 +565,19 @@ test_expect_success 'when writing another commit graph, preserve existing versio\n test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n \tgit init doublewrite &&\n \ttest_commit -C doublewrite c \"$CENT\" &&\n+\n \tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n-\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 1 trace2.txt &&\n+\ttest_filter_upgraded 0 trace2.txt &&\n+\n \tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n-\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 1 trace2.txt &&\n+\ttest_filter_upgraded 0 trace2.txt &&\n+\n \t(\n \t\tcd doublewrite &&\n \t\techo \"c01f\" >expect &&\n@@ -573,4 +586,22 @@ test_expect_success 'when writing commit graph, do not reuse changed-path of ano\n \t)\n '\n \n+test_expect_success 'when writing commit graph, reuse changed-path of another version where possible' '\n+\tgit init upgrade &&\n+\n+\ttest_commit -C upgrade base no-high-bits &&\n+\n+\tgit -C upgrade config --add commitgraph.changedPathsVersion 1 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C upgrade commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 1 trace2.txt &&\n+\ttest_filter_upgraded 0 trace2.txt &&\n+\n+\tgit -C upgrade config --add commitgraph.changedPathsVersion 2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C upgrade commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 0 trace2.txt &&\n+\ttest_filter_upgraded 1 trace2.txt\n+'\n+\n test_done\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481190","messageId":"8405b845e5a8f56a03c98708185c3402e172d311.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 03/15] bloom.h: make `load_bloom_filter_from_graph()` public","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:43Z","receivedAt":"2023-08-30T18:29:56Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nPrepare for a future commit to use the load_bloom_filter_from_graph()\nfunction directly to load specific Bloom filters out of the commit-graph\nfor manual inspection (to be used during tests).\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n bloom.c | 6 +++---\n bloom.h | 5 +++++\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex aef6b5fea2..3e78cfe79d 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -29,9 +29,9 @@ static inline unsigned char get_bitmask(uint32_t pos)\n \treturn ((unsigned char)1) << (pos & (BITS_PER_WORD - 1));\n }\n \n-static int load_bloom_filter_from_graph(struct commit_graph *g,\n-\t\t\t\t\tstruct bloom_filter *filter,\n-\t\t\t\t\tuint32_t graph_pos)\n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos)\n {\n \tuint32_t lex_pos, start_index, end_index;\n \ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..1e4f612d2c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -3,6 +3,7 @@\n \n struct commit;\n struct repository;\n+struct commit_graph;\n \n struct bloom_filter_settings {\n \t/*\n@@ -68,6 +69,10 @@ struct bloom_key {\n \tuint32_t *hashes;\n };\n \n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos);\n+\n /*\n * Calculate the murmur3 32-bit hash value for the given data\n * using the given seed.\n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481191","messageId":"58a1d90e6d83506e151bba51fdddcd5cf17b9f29.1693413637.git.jonathantanmy@google.com","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"[PATCH v2 15/15] bloom: introduce `deinit_bloom_filters()`","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-30T16:43:55Z","receivedAt":"2023-08-30T18:29:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau \n\nAfter we are done using Bloom filters, we do not currently clean up any\nmemory allocated by the commit slab used to store those filters in the\nfirst place.\n\nBesides the bloom_filter structures themselves, there is mostly nothing\nto free() in the first place, since in the read-only path all Bloom\nfilter's `data` members point to a memory mapped region in the\ncommit-graph file itself.\n\nBut when generating Bloom filters from scratch (or initializing\ntruncated filters) we allocate additional memory to store the filter's\ndata.\n\nKeep track of when we need to free() this additional chunk of memory by\nusing an extra pointer `to_free`. Most of the time this will be NULL\n(indicating that we are representing an existing Bloom filter stored in\na memory mapped region). When it is non-NULL, free it before discarding\nthe Bloom filters slab.\n\nSuggested-by: Jonathan Tan \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n bloom.c | 16 +++++++++++++++-\n bloom.h | 3 +++\n commit-graph.c | 4 ++++\n 3 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 24dd874e46..ff131893cd 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -59,6 +59,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n \t\t\t\t\tsizeof(unsigned char) * start_index +\n \t\t\t\t\tBLOOMDATA_CHUNK_HEADER_SIZE);\n \tfilter->version = g->bloom_filter_settings->hash_version;\n+\tfilter->to_free = NULL;\n \n \treturn 1;\n }\n@@ -231,6 +232,18 @@ void init_bloom_filters(void)\n \tinit_bloom_filter_slab(&bloom_filters);\n }\n \n+static void free_one_bloom_filter(struct bloom_filter *filter)\n+{\n+\tif (!filter)\n+\t\treturn;\n+\tfree(filter->to_free);\n+}\n+\n+void deinit_bloom_filters(void)\n+{\n+\tdeep_clear_bloom_filter_slab(&bloom_filters, free_one_bloom_filter);\n+}\n+\n static int pathmap_cmp(const void *hashmap_cmp_fn_data UNUSED,\n \t\t const struct hashmap_entry *eptr,\n \t\t const struct hashmap_entry *entry_or_key,\n@@ -247,7 +260,7 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data UNUSED,\n static void init_truncated_large_filter(struct bloom_filter *filter,\n \t\t\t\t\tint version)\n {\n-\tfilter->data = xmalloc(1);\n+\tfilter->data = filter->to_free = xmalloc(1);\n \tfilter->data[0] = 0xFF;\n \tfilter->len = 1;\n \tfilter->version = version;\n@@ -449,6 +462,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tfilter->len = 1;\n \t\t}\n \t\tCALLOC_ARRAY(filter->data, filter->len);\n+\t\tfilter->to_free = filter->data;\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n \t\t\tstruct bloom_key key;\ndiff --git a/bloom.h b/bloom.h\nindex e3a9b68905..d20e64bfbb 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -56,6 +56,8 @@ struct bloom_filter {\n \tunsigned char *data;\n \tsize_t len;\n \tint version;\n+\n+\tvoid *to_free;\n };\n \n /*\n@@ -96,6 +98,7 @@ void add_key_to_filter(const struct bloom_key *key,\n \t\t const struct bloom_filter_settings *settings);\n \n void init_bloom_filters(void);\n+void deinit_bloom_filters(void);\n \n enum bloom_filter_computed {\n \tBLOOM_NOT_COMPUTED = (1 << 0),\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a3473df515..585539da2f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -723,6 +723,7 @@ static void close_commit_graph_one(struct commit_graph *g)\n void close_commit_graph(struct raw_object_store *o)\n {\n \tclose_commit_graph_one(o->commit_graph);\n+\tdeinit_bloom_filters();\n \to->commit_graph = NULL;\n }\n \n@@ -2523,6 +2524,9 @@ int write_commit_graph(struct object_directory *odb,\n \n \tres = write_commit_graph_file(ctx);\n \n+\tif (ctx->changed_paths)\n+\t\tdeinit_bloom_filters();\n+\n \tif (ctx->split)\n \t\tmark_commit_graphs(ctx);\n \n-- \n2.42.0.rc2.253.gd59a3bf2b4-goog\n\n"},{"id":"481193","messageId":"xmqqo7io8gmo.fsf@gitster.g","threadId":"60142","inReplyTo":"cover.1693413637.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 00/15] bloom: changed-path Bloom filters v2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-30T19:38:55Z","receivedAt":"2023-08-30T20:08:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Here's an updated patch set containing updates to the tests, making them\n> run on non-root commits to ensure that we exercise the Bloom filters.\n\nThanks for a quick turnaround.\n\n> Thanks to SZEDER Gábor for spotting this.\n\nWith this everybody involved are happy, I presume. Thank you,\neverybody, who's worked on polishing and assembling these patches.\n"},{"id":"481194","messageId":"20230830200218.GA5147@szeder.dev","threadId":"60142","inReplyTo":"20230829163124.363157-1-jonathantanmy@google.com","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2023-08-30T20:02:18Z","receivedAt":"2023-08-30T20:18:29Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 29, 2023 at 09:31:23AM -0700, Jonathan Tan wrote:\n> SZEDER Gábor writes:\n \n> > > +test_expect_success 'version 2 changed-path used when version 2 requested' '\n> > > +\t(\n> > > +\t\tcd highbit2 &&\n> > > +\t\ttest_bloom_filters_used \"-- $CENT\"\n> > \n> > This test_bloom_filter_used helper runs two pathspec-limited 'git log'\n> > invocations, one with disabled and the other with enabled\n> > commit-graph, and thus with disabled and enabled modified path Bloom\n> > filters, and compares their output.\n> > \n> > One of the flaws of the current modified path Bloom filters\n> > implementation is that it doesn't check Bloom filters for root\n> > commits.\n> > \n> > In several of the above test cases test_bloom_filters_used is invoked\n> > in a repository with only a root commit, so they don't check that\n> > the output is the same with and without Bloom filters.\n> \n> Ah...you are right. Indeed, when I flip one of the tests from\n> test_bloom_filters_used to _not_, the test still passes. I'll change\n> the tests.\n\nI'd prefer to leave the test cases unchanged, and make the revision\nwalking machinery look at Bloom filters even for root commits, because\nthis is an annoying and recurring testing issue. I remember it\nannoyed me back then, when I wanted to reproduce a couple of bugs that\nI knew were there, but my initial test cases didn't fail because the\nBloom filter of the root commit was ignored; Derrick overlooked this\nin b16a8277644, you overlooked it now, and none of the reviewers then\nand now caught it, either.\n\n> > The string \"split\" occurs twice in this patch series, but only in\n> > patch hunk contexts, and it doesn't occur at all in the previous long\n> > thread about the original patch series.\n> > \n> > Unfortunately, split commit-graphs weren't really considered in the\n> > design of the modified path Bloom filters feature, and layers with\n> > different Bloom filter settings weren't considered at all. I've\n> > reported it back then, but the fixes so far were incomplete, and e.g.\n> > the test cases shown in\n> > \n> > https://public-inbox.org/git/20201015132147.GB24954@szeder.dev/\n> > \n> > still fail.\n> > \n> > Since the interaction of different versions and split commit-graphs\n> > was neither mentioned in any of the commit messages nor discussed\n> > during the previous rounds, and there isn't any test case excercising\n> > it, and since the Bloom filter version information is stored in the\n> > same 'g->bloom_filter_settings' structure as the number of hashes, I'm\n> > afraid (though haven't actually checked) that handling commit-graph\n> > layers with different Bloom filter versions is prone to the same\n> > issues as well.\n> \n> My original design (up to patch 7 in this patch set) defends against\n> this by taking the very first version detected and rejecting every\n> other version, and Taylor's subsequent design reads every version, but\n> annotates filters with its version. So I think we're covered.\n\nIn the meantime I adapted the test cases from the above linked message\nto write commit-graph layers with different Bloom filter versions, and\nit does fail, because commits from the bottom commit-graph layer are\nomitted from the revision walk's output. And the test case doesn't\neven need a middle layer without modified path Bloom filters to \"hide\"\nthe different version in the bottom layer. Merging the layers seems\nto work, though.\n\nBesides fixing this issue, I think that the interaction of different\nBloom filter versions and split commit-graphs needs to be thoroughly\ncovered with test cases and discussed in the commit messages before\nthis series could be considered good for merging.\n\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 48f8109a66..55f67e5110 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -586,4 +586,40 @@ test_expect_success 'when writing commit graph, reuse changed-path of another ve\n \ttest_filter_upgraded 1 trace2.txt\n '\n \n+test_expect_success 'split commit graph vs changed paths breakage - setup' '\n+\tgit init split-breakage &&\n+\t(\n+\t\tcd split-breakage &&\n+\t\tgit commit --allow-empty -m \"Bloom filters are written but still ignored for root commits :(\" &&\n+\t\tfor i in 1 2 3\n+\t\tdo\n+\t\t\techo $i >$CENT &&\n+\t\t\tgit add $CENT &&\n+\t\t\tgit commit -m \"$i\" || return 1\n+\t\tdone &&\n+\t\tgit log --oneline -- $CENT >expect\n+\t)\n+'\n+\n+test_expect_failure 'split commit graph vs changed paths breakage - split' '\n+\t(\n+\t\tcd split-breakage &&\n+\n+\t\t# Compute v1 Bloom filters for the commits at the bottom.\n+\t\tgit rev-parse HEAD^ | git commit-graph write --stdin-commits --changed-paths --split &&\n+\t\t# Compute v2 Bloom filters for the rest of the commits at the top.\n+\t\tgit rev-parse HEAD | git -c commitgraph.changedPathsVersion=2 commit-graph write --stdin-commits --changed-paths --split=no-merge &&\n+\n+\t\t# Just to make sure that there are as many graph layers as I\n+\t\t# think there should be.\n+\t\ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain &&\n+\n+\t\t# This checks Bloom filters using version information in the\n+\t\t# top layer, thus misses commits modifying the file in the\n+\t\t# bottom commit-graph layer.\n+\t\tgit log --oneline -- $CENT >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n\nIt fails with:\n\n + cd split-breakage\n + git rev-parse HEAD^\n + git commit-graph write --stdin-commits --changed-paths --split\n + git rev-parse HEAD\n + git -c commitgraph.changedPathsVersion=2 commit-graph write --stdin-commits --changed-paths --split=no-merge\n + test_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n + git log --oneline -- ¢\n + test_cmp expect actual\n --- expect 2023-08-30 19:07:59.882066827 +0000\n +++ actual 2023-08-30 19:07:59.894067148 +0000\n @@ -1,3 +1 @@\n 1db2248 3\n -cfcc97f 2\n -bd9c2c8 1\n error: last command exited with $?=1\n\n"},{"id":"481315","messageId":"20230901205616.3572722-1-jonathantanmy@google.com","threadId":"60142","inReplyTo":"20230830200218.GA5147@szeder.dev","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-09-01T20:56:15Z","receivedAt":"2023-09-01T20:56:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"SZEDER Gábor writes:\n> I'd prefer to leave the test cases unchanged, and make the revision\n> walking machinery look at Bloom filters even for root commits, because\n> this is an annoying and recurring testing issue. I remember it\n> annoyed me back then, when I wanted to reproduce a couple of bugs that\n> I knew were there, but my initial test cases didn't fail because the\n> Bloom filter of the root commit was ignored; Derrick overlooked this\n> in b16a8277644, you overlooked it now, and none of the reviewers then\n> and now caught it, either.\n\nI agree that making the revwalk look at Bloom filters of root commits\nis an urgent matter for the reasons you describe (it's something\neasily missed that slows down future development and/or makes future\ndevelopment error-prone), but so is moving to a correct murmur3\nimplementation, I think, and one shouldn't stop the other. There could\nbe an argument that because the revwalk doesn't look at root commit\nBloom filters, any development on a new Bloom filter hash version is\nsuspect, but I don't think that has to be completely true.\n\n> > My original design (up to patch 7 in this patch set) defends against\n> > this by taking the very first version detected and rejecting every\n> > other version, and Taylor's subsequent design reads every version, but\n> > annotates filters with its version. So I think we're covered.\n> \n> In the meantime I adapted the test cases from the above linked message\n> to write commit-graph layers with different Bloom filter versions, and\n> it does fail, because commits from the bottom commit-graph layer are\n> omitted from the revision walk's output. And the test case doesn't\n> even need a middle layer without modified path Bloom filters to \"hide\"\n> the different version in the bottom layer. Merging the layers seems\n> to work, though.\n\nFor what it's worth, your test case below (with test_expect_success\ninstead of test_expect_failure) passes with my original design. With the\nfull patch set, it does fail, but for what it's worth, I did spot this,\nproviding an incomplete solution [1] and then a complete one [2]. Your\ntest case passes if I also include the following:\n\n diff --git a/bloom.c b/bloom.c\n index ff131893cd..1bafd62a4e 100644\n --- a/bloom.c\n +++ b/bloom.c\n @@ -344,6 +344,10 @@ struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c)\n \n prepare_repo_settings(r);\n hash_version = r->settings.commit_graph_changed_paths_version;\n + if (hash_version == -1) {\n + struct bloom_filter_settings *s = get_bloom_filter_settings(r);\n + hash_version = (s && s->hash_version == 2) ? 2 : 1;\n + }\n \n if (!(hash_version == -1 || hash_version == filter->version))\n return NULL; /* unusable filter */\n\n[1] https://lore.kernel.org/git/20230824222051.2320003-1-jonathantanmy@google.com/\n[2] https://lore.kernel.org/git/20230829220432.558674-1-jonathantanmy@google.com/\n\n> Besides fixing this issue, I think that the interaction of different\n> Bloom filter versions and split commit-graphs needs to be thoroughly\n> covered with test cases and discussed in the commit messages before\n> this series could be considered good for merging.\n\nRegarding commit messages, I can see that in \"commit-graph: new filter\nver. that fixes murmur3\", I could add \"(the first version read if there\nare multiple versions of changed path filters in the repository)\" after\n\"automatically determined from the version of the existing changed path\nfilters in the repository\". Taylor's patches already inherently cover\nmultiple versions, I think, since Bloom filters are annotated with their\nversions, individually.\n\nRegarding tests, yes, we could add the one you provided.\n\nIf you (or anyone else) can spot anything else that should be added,\nplease let us know.\n"},{"id":"482312","messageId":"ZRIRtlbsYadg7EUx@nand.local","threadId":"60142","inReplyTo":"20230901205616.3572722-1-jonathantanmy@google.com","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-09-25T23:03:18Z","receivedAt":"2023-09-25T23:03:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 01, 2023 at 01:56:15PM -0700, Jonathan Tan wrote:\n> > > My original design (up to patch 7 in this patch set) defends against\n> > > this by taking the very first version detected and rejecting every\n> > > other version, and Taylor's subsequent design reads every version, but\n> > > annotates filters with its version. So I think we're covered.\n> >\n> > In the meantime I adapted the test cases from the above linked message\n> > to write commit-graph layers with different Bloom filter versions, and\n> > it does fail, because commits from the bottom commit-graph layer are\n> > omitted from the revision walk's output. And the test case doesn't\n> > even need a middle layer without modified path Bloom filters to \"hide\"\n> > the different version in the bottom layer. Merging the layers seems\n> > to work, though.\n>\n> For what it's worth, your test case below (with test_expect_success\n> instead of test_expect_failure) passes with my original design. With the\n> full patch set, it does fail, but for what it's worth, I did spot this,\n> providing an incomplete solution [1] and then a complete one [2]. Your\n> test case passes if I also include the following:\n>\n> diff --git a/bloom.c b/bloom.c\n> index ff131893cd..1bafd62a4e 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -344,6 +344,10 @@ struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c)\n>\n> prepare_repo_settings(r);\n> hash_version = r->settings.commit_graph_changed_paths_version;\n> + if (hash_version == -1) {\n> + struct bloom_filter_settings *s = get_bloom_filter_settings(r);\n> + hash_version = (s && s->hash_version == 2) ? 2 : 1;\n> + }\n>\n> if (!(hash_version == -1 || hash_version == filter->version))\n> return NULL; /* unusable filter */\n>\n> [1] https://lore.kernel.org/git/20230824222051.2320003-1-jonathantanmy@google.com/\n> [2] https://lore.kernel.org/git/20230829220432.558674-1-jonathantanmy@google.com/\n\nHmm. I am confused -- are you saying that this series breaks existing\nfunctionality, or merely does not patch an existing breakage? I *think*\nthat it's the latter, since this test case fails identically on master,\nbut I am not sure.\n\nIf my understanding is correct, I think that patching this would involve\nannotating each Bloom filter with a pointer to the bloom_filter_settings\nit was written with, and then using those settings when querying it.\n\nThanks,\nTaylor\n"},{"id":"482821","messageId":"20231008143523.GA18858@szeder.dev","threadId":"60142","inReplyTo":"ZRIRtlbsYadg7EUx@nand.local","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2023-10-08T14:35:23Z","receivedAt":"2023-10-08T14:35:41Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Mon, Sep 25, 2023 at 07:03:18PM -0400, Taylor Blau wrote:\n> On Fri, Sep 01, 2023 at 01:56:15PM -0700, Jonathan Tan wrote:\n> > > > My original design (up to patch 7 in this patch set) defends against\n> > > > this by taking the very first version detected and rejecting every\n> > > > other version, and Taylor's subsequent design reads every version, but\n> > > > annotates filters with its version. So I think we're covered.\n> > >\n> > > In the meantime I adapted the test cases from the above linked message\n> > > to write commit-graph layers with different Bloom filter versions, and\n> > > it does fail, because commits from the bottom commit-graph layer are\n> > > omitted from the revision walk's output. And the test case doesn't\n> > > even need a middle layer without modified path Bloom filters to \"hide\"\n> > > the different version in the bottom layer. Merging the layers seems\n> > > to work, though.\n> >\n> > For what it's worth, your test case below (with test_expect_success\n> > instead of test_expect_failure) passes with my original design. With the\n> > full patch set, it does fail, but for what it's worth, I did spot this,\n> > providing an incomplete solution [1] and then a complete one [2]. Your\n> > test case passes if I also include the following:\n> >\n> > diff --git a/bloom.c b/bloom.c\n> > index ff131893cd..1bafd62a4e 100644\n> > --- a/bloom.c\n> > +++ b/bloom.c\n> > @@ -344,6 +344,10 @@ struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c)\n> >\n> > prepare_repo_settings(r);\n> > hash_version = r->settings.commit_graph_changed_paths_version;\n> > + if (hash_version == -1) {\n> > + struct bloom_filter_settings *s = get_bloom_filter_settings(r);\n> > + hash_version = (s && s->hash_version == 2) ? 2 : 1;\n> > + }\n> >\n> > if (!(hash_version == -1 || hash_version == filter->version))\n> > return NULL; /* unusable filter */\n> >\n> > [1] https://lore.kernel.org/git/20230824222051.2320003-1-jonathantanmy@google.com/\n> > [2] https://lore.kernel.org/git/20230829220432.558674-1-jonathantanmy@google.com/\n> \n> Hmm. I am confused -- are you saying that this series breaks existing\n> functionality, or merely does not patch an existing breakage? I *think*\n> that it's the latter,\n\nIt's neither: the new functionality added in this series is broken.\n\n> since this test case fails identically on master,\n> but I am not sure.\n\nNot sure what test you are referring to. My test demonstrating the\nbreakage succeeds when adaped to master, because master doesn't\nunderstand the commitgraph.changedPathsVersion=2 setting, and keeps\nwriting v1 Bloom filter chunks instead, so all commit-graphs layers\ncontain the same version.\n\n> If my understanding is correct, I think that patching this would involve\n> annotating each Bloom filter with a pointer to the bloom_filter_settings\n> it was written with, and then using those settings when querying it.\n\nIn general we are better off when we don't write Bloom filter chunks\nwith different settings in the first place.\n\n"},{"id":"482889","messageId":"ZSRD0tK3bk67aDw4@nand.local","threadId":"60142","inReplyTo":"20231008143523.GA18858@szeder.dev","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-09T18:17:54Z","receivedAt":"2023-10-09T18:17:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sun, Oct 08, 2023 at 04:35:23PM +0200, SZEDER Gábor wrote:\n> > Hmm. I am confused -- are you saying that this series breaks existing\n> > functionality, or merely does not patch an existing breakage? I *think*\n> > that it's the latter,\n>\n> It's neither: the new functionality added in this series is broken.\n>\n> > since this test case fails identically on master,\n> > but I am not sure.\n>\n> Not sure what test you are referring to. My test demonstrating the\n> breakage succeeds when adaped to master, because master doesn't\n> understand the commitgraph.changedPathsVersion=2 setting, and keeps\n> writing v1 Bloom filter chunks instead, so all commit-graphs layers\n> contain the same version.\n\nI was referring to the test you sent back in:\n\n https://public-inbox.org/git/20201015132147.GB24954@szeder.dev/\n\nbut I think that I should have been looking at the one you sent more\nrecently in:\n\n https://lore.kernel.org/git/20230830200218.GA5147@szeder.dev/\n\nThanks,\nTaylor\n"},{"id":"482897","messageId":"ZSRVJ3iASRaDGc80@nand.local","threadId":"60142","inReplyTo":"ZSRD0tK3bk67aDw4@nand.local","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-09T19:31:51Z","receivedAt":"2023-10-09T19:32:05Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 09, 2023 at 02:17:54PM -0400, Taylor Blau wrote:\n> On Sun, Oct 08, 2023 at 04:35:23PM +0200, SZEDER Gábor wrote:\n> > > Hmm. I am confused -- are you saying that this series breaks existing\n> > > functionality, or merely does not patch an existing breakage? I *think*\n> > > that it's the latter,\n> >\n> > It's neither: the new functionality added in this series is broken.\n> >\n> > > since this test case fails identically on master,\n> > > but I am not sure.\n> >\n> > Not sure what test you are referring to. My test demonstrating the\n> > breakage succeeds when adaped to master, because master doesn't\n> > understand the commitgraph.changedPathsVersion=2 setting, and keeps\n> > writing v1 Bloom filter chunks instead, so all commit-graphs layers\n> > contain the same version.\n>\n> I was referring to the test you sent back in:\n>\n> https://public-inbox.org/git/20201015132147.GB24954@szeder.dev/\n>\n> but I think that I should have been looking at the one you sent more\n> recently in:\n>\n> https://lore.kernel.org/git/20230830200218.GA5147@szeder.dev/\n\nOK, I think that I now have my head wrapped around what you're saying.\n\nHowever, I am not entirely sure I agree with you that this is a \"new\"\nissue. At least in the sense that Git (on 'master') does not currently\nknow how to deal with Bloom filters that have different settings across\ncommit-graph layers.\n\nIOW, you could produce this problem today using the test you wrote in\n<20201015132147.GB24954@szeder.dev> using different values of the\nGIT_BLOOM_SETTINGS environment variables as a proxy for different values\nof the commitGraph.changedPathsVersion configuration variable introduced\nin this series.\n\nSo I think that this series makes it easier to fall into that trap, but\nthe trap itself is not new. I think a reasonable stopgap (which IIUC you\nhave suggested earlier) is to prevent writing a new commit-graph layer\nwith a different hash version than the previous layer.\n\nHow does that sound?\n\nThanks,\nTaylor\n"},{"id":"482898","messageId":"xmqqr0m3y3o2.fsf@gitster.g","threadId":"60142","inReplyTo":"ZSRVJ3iASRaDGc80@nand.local","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-09T19:52:13Z","receivedAt":"2023-10-09T19:52:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> However, I am not entirely sure I agree with you that this is a \"new\"\n> issue. At least in the sense that Git (on 'master') does not currently\n> know how to deal with Bloom filters that have different settings across\n> commit-graph layers.\n>\n> IOW, you could produce this problem today using the test you wrote in\n> <20201015132147.GB24954@szeder.dev> using different values of the\n> GIT_BLOOM_SETTINGS environment variables as a proxy for different values\n> of the commitGraph.changedPathsVersion configuration variable introduced\n> in this series.\n>\n> So I think that this series makes it easier to fall into that trap, but\n> the trap itself is not new. I think a reasonable stopgap (which IIUC you\n> have suggested earlier) is to prevent writing a new commit-graph layer\n> with a different hash version than the previous layer.\n\nWhat we probably want more urgently than that stopgap is to perhaps\nteach the code pretend as if commit-graph did not exist when we\ndetect multiple layers use different hash versions (or perhaps only\nuse the base layer and ignore the rest as an anti-pessimization), to\nprotect correctness first, no?\n\n\n"},{"id":"483004","messageId":"cover.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1692654233.git.me@ttaylorr.com","subject":"[PATCH v3 00/17] bloom: changed-path Bloom filters v2 (& sundries)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:17Z","receivedAt":"2023-10-10T20:33:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Rebased onto the tip of 'master', which is 3a06386e31 (The fifteenth\nbatch, 2023-10-04), at the time of writing).\n\nThis series is a reroll of the combined efforts of [1] and [2] to\nintroduce the v2 changed-path Bloom filters, which fixes a bug in our\nexisting implementation of murmur3 paths with non-ASCII characters (when\nthe \"char\" type is signed).\n\nIn large part, this is the same as the previous round. But this round\nincludes some extra bits that address issues pointed out by SZEDER\nGábor, which are:\n\n - not reading Bloom filters for root commits\n - corrupting Bloom filter reads by tweaking the filter settings\n between layers.\n\nThese issues were discussed in (among other places) [3], and [4],\nrespectively.\n\nThanks to Jonathan, Peff, and SZEDER who have helped a great deal in\nassembling these patches. As usual, a range-diff is included below.\nThanks in advance for your\nreview!\n\n[1]: https://lore.kernel.org/git/cover.1684790529.git.jonathantanmy@google.com/\n[2]: https://lore.kernel.org/git/cover.1691426160.git.me@ttaylorr.com/\n[3]: https://public-inbox.org/git/20201015132147.GB24954@szeder.dev/\n[4]: https://lore.kernel.org/git/20230830200218.GA5147@szeder.dev/\n\nJonathan Tan (4):\n gitformat-commit-graph: describe version 2 of BDAT\n t4216: test changed path filters with high bit paths\n repo-settings: introduce commitgraph.changedPathsVersion\n commit-graph: new filter ver. that fixes murmur3\n\nTaylor Blau (13):\n t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`\n revision.c: consult Bloom filters for root commits\n commit-graph: ensure Bloom filters are read with consistent settings\n t/helper/test-read-graph.c: extract `dump_graph_info()`\n bloom.h: make `load_bloom_filter_from_graph()` public\n t/helper/test-read-graph: implement `bloom-filters` mode\n bloom: annotate filters with hash version\n bloom: prepare to discard incompatible Bloom filters\n commit-graph.c: unconditionally load Bloom filters\n commit-graph: drop unnecessary `graph_read_bloom_data_context`\n object.h: fix mis-aligned flag bits table\n commit-graph: reuse existing Bloom filters where possible\n bloom: introduce `deinit_bloom_filters()`\n\n Documentation/config/commitgraph.txt | 26 ++-\n Documentation/gitformat-commit-graph.txt | 9 +-\n bloom.c | 208 ++++++++++++++++-\n bloom.h | 38 +++-\n commit-graph.c | 61 ++++-\n object.h | 3 +-\n oss-fuzz/fuzz-commit-graph.c | 2 +-\n repo-settings.c | 6 +-\n repository.h | 2 +-\n revision.c | 26 ++-\n t/helper/test-bloom.c | 9 +-\n t/helper/test-read-graph.c | 67 ++++--\n t/t0095-bloom.sh | 8 +\n t/t4216-log-bloom.sh | 272 ++++++++++++++++++++++-\n 14 files changed, 682 insertions(+), 55 deletions(-)\n\nRange-diff against v2:\n10: 002a06d1e9 ! 1: fe671d616c t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`\n @@ Commit message\n indicating that no filters were used.\n \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## t/t4216-log-bloom.sh ##\n @@ t/t4216-log-bloom.sh: test_bloom_filters_used () {\n -: ---------- > 2: 7d0fa93543 revision.c: consult Bloom filters for root commits\n -: ---------- > 3: 2ecc0a2d58 commit-graph: ensure Bloom filters are read with consistent settings\n 1: 5fa681b58e ! 4: 17703ed89a gitformat-commit-graph: describe version 2 of BDAT\n @@ Commit message\n Signed-off-by: Jonathan Tan \n Signed-off-by: Junio C Hamano \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## Documentation/gitformat-commit-graph.txt ##\n @@ Documentation/gitformat-commit-graph.txt: All multi-byte numbers are in network byte order.\n 2: 623d840575 ! 5: 94552abf45 t/helper/test-read-graph.c: extract `dump_graph_info()`\n @@ Commit message\n Signed-off-by: Jonathan Tan \n Signed-off-by: Junio C Hamano \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## t/helper/test-read-graph.c ##\n @@\n 3: bc9d77ae60 ! 6: 3d81efa27b bloom.h: make `load_bloom_filter_from_graph()` public\n @@ Commit message\n Signed-off-by: Jonathan Tan \n Signed-off-by: Junio C Hamano \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## bloom.c ##\n @@ bloom.c: static inline unsigned char get_bitmask(uint32_t pos)\n 4: ac7008aed3 ! 7: d23cd89037 t/helper/test-read-graph: implement `bloom-filters` mode\n @@ Commit message\n Signed-off-by: Jonathan Tan \n Signed-off-by: Junio C Hamano \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## t/helper/test-read-graph.c ##\n @@ t/helper/test-read-graph.c: static void dump_graph_info(struct commit_graph *graph)\n @@ t/helper/test-read-graph.c: int cmd__read_graph(int argc UNUSED, const char **ar\n -\treturn 0;\n +\treturn ret;\n }\n ++\n ++\n 5: 71755ba856 ! 8: cba766f224 t4216: test changed path filters with high bit paths\n @@ Commit message\n Signed-off-by: Taylor Blau \n \n ## t/t4216-log-bloom.sh ##\n -@@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty commits' '\n - \t)\n +@@ t/t4216-log-bloom.sh: test_expect_success 'merge graph layers with incompatible Bloom settings' '\n + \t! grep \"disabling Bloom filters\" err\n '\n \n +get_first_changed_path_filter () {\n 6: 9768d92c0f ! 9: a08a961f41 repo-settings: introduce commitgraph.changedPathsVersion\n @@ Commit message\n Signed-off-by: Jonathan Tan \n Signed-off-by: Junio C Hamano \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## Documentation/config/commitgraph.txt ##\n @@ Documentation/config/commitgraph.txt: commitGraph.maxNewFilters::\n 7: f911b4bfab = 10: 61d44519a5 commit-graph: new filter ver. that fixes murmur3\n 8: 35009900df ! 11: a8c10f8de8 bloom: annotate filters with hash version\n @@ Commit message\n Bloom filter.\n \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## bloom.c ##\n @@ bloom.c: int load_bloom_filter_from_graph(struct commit_graph *g,\n 9: 138bc16905 ! 12: 2ba10a4b4b bloom: prepare to discard incompatible Bloom filters\n @@ Commit message\n `get_or_compute_bloom_filter()`.\n \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## bloom.c ##\n @@ bloom.c: static void init_truncated_large_filter(struct bloom_filter *filter,\n11: 2437e62813 ! 13: 09d8669c3a commit-graph.c: unconditionally load Bloom filters\n @@ Commit message\n either \"1\" or \"2\".\n \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## commit-graph.c ##\n @@ commit-graph.c: static int graph_read_bloom_data(const unsigned char *chunk_start,\n12: fe8fb2f5fe ! 14: 0d4f9dc4ee commit-graph: drop unnecessary `graph_read_bloom_data_context`\n @@ Commit message\n \n Noticed-by: Jonathan Tan \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## commit-graph.c ##\n @@ commit-graph.c: static int graph_read_oid_lookup(const unsigned char *chunk_start,\n13: 825af91e11 ! 15: 1f7f27bc47 object.h: fix mis-aligned flag bits table\n @@ Commit message\n Bit position 23 is one column too far to the left.\n \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## object.h ##\n @@ object.h: void object_array_init(struct object_array *array);\n14: 593b317192 ! 16: abbef95ae8 commit-graph: reuse existing Bloom filters where possible\n @@ Commit message\n commits by their generation number.\n \n Signed-off-by: Taylor Blau \n - Signed-off-by: Junio C Hamano \n - Signed-off-by: Taylor Blau \n \n ## bloom.c ##\n @@\n15: 8bf2c9cf98 = 17: ca362408d5 bloom: introduce `deinit_bloom_filters()`\n-- \n2.42.0.342.g8bb3a896ee\n"},{"id":"483005","messageId":"fe671d616c6640fc41dcea7a9738b9b640420a2c.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 01/17] t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:20Z","receivedAt":"2023-10-10T20:33:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The existing implementation of test_bloom_filters_not_used() asserts\nthat the Bloom filter sub-system has not been initialized at all, by\nchecking for the absence of any data from it from trace2.\n\nIn the following commit, it will become possible to load Bloom filters\nwithout using them (e.g., because `commitGraph.changedPathVersion` is\nincompatible with the hash version with which the commit-graph's Bloom\nfilters were written).\n\nWhen this is the case, it's possible to initialize the Bloom filter\nsub-system, while still not using any Bloom filters. When this is the\ncase, check that the data dump from the Bloom sub-system is all zeros,\nindicating that no filters were used.\n\nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 14 +++++++++++++-\n 1 file changed, 13 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..487fc3d6b9 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -81,7 +81,19 @@ test_bloom_filters_used () {\n test_bloom_filters_not_used () {\n \tlog_args=$1\n \tsetup \"$log_args\" &&\n-\t! grep -q \"statistics:{\\\"filter_not_present\\\":\" \"$TRASH_DIRECTORY/trace.perf\" &&\n+\n+\tif grep -q \"statistics:{\\\"filter_not_present\\\":\" \"$TRASH_DIRECTORY/trace.perf\"\n+\tthen\n+\t\t# if the Bloom filter system is initialized, ensure that no\n+\t\t# filters were used\n+\t\tdata=\"statistics:{\"\n+\t\tdata=\"$data\\\"filter_not_present\\\":0,\"\n+\t\tdata=\"$data\\\"maybe\\\":0,\"\n+\t\tdata=\"$data\\\"definitely_not\\\":0,\"\n+\t\tdata=\"$data\\\"false_positive\\\":0}\"\n+\n+\t\tgrep -q \"$data\" \"$TRASH_DIRECTORY/trace.perf\"\n+\tfi &&\n \ttest_cmp log_wo_bloom log_w_bloom\n }\n \n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483006","messageId":"7d0fa9354328799606f7e9a85fb5cedd06da466c.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 02/17] revision.c: consult Bloom filters for root commits","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:23Z","receivedAt":"2023-10-10T20:33:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The commit-graph stores changed-path Bloom filters which represent the\nset of paths included in a tree-level diff between a commit's root tree\nand that of its parent.\n\nWhen a commit has no parents, the tree-diff is computed against that\ncommit's root tree and the empty tree. In other words, every path in\nthat commit's tree is stored in the Bloom filter (since they all appear\nin the diff).\n\nConsult these filters during pathspec-limited traversals in the function\n`rev_same_tree_as_empty()`. Doing so yields a performance improvement\nwhere we can avoid enumerating the full set of paths in a parentless\ncommit's root tree when we know that the path(s) of interest were not\nlisted in that commit's changed-path Bloom filter.\n\nSuggested-by: SZEDER Gábor \nOriginal-patch-by: Jonathan Tan \nSigned-off-by: Taylor Blau \n---\n revision.c | 26 ++++++++++++++++++++++----\n t/t4216-log-bloom.sh | 8 ++++++--\n 2 files changed, 28 insertions(+), 6 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex e789834dd1..dae569a547 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -834,17 +834,28 @@ static int rev_compare_tree(struct rev_info *revs,\n \treturn tree_difference;\n }\n \n-static int rev_same_tree_as_empty(struct rev_info *revs, struct commit *commit)\n+static int rev_same_tree_as_empty(struct rev_info *revs, struct commit *commit,\n+\t\t\t\t int nth_parent)\n {\n \tstruct tree *t1 = repo_get_commit_tree(the_repository, commit);\n+\tint bloom_ret = 1;\n \n \tif (!t1)\n \t\treturn 0;\n \n+\tif (nth_parent == 1 && revs->bloom_keys_nr) {\n+\t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, commit);\n+\t\tif (!bloom_ret)\n+\t\t\treturn 1;\n+\t}\n+\n \ttree_difference = REV_TREE_SAME;\n \trevs->pruning.flags.has_changes = 0;\n \tdiff_tree_oid(NULL, &t1->object.oid, \"\", &revs->pruning);\n \n+\tif (bloom_ret == 1 && tree_difference == REV_TREE_SAME)\n+\t\tcount_bloom_filter_false_positive++;\n+\n \treturn tree_difference == REV_TREE_SAME;\n }\n \n@@ -882,7 +893,7 @@ static int compact_treesame(struct rev_info *revs, struct commit *commit, unsign\n \t\tif (nth_parent != 0)\n \t\t\tdie(\"compact_treesame %u\", nth_parent);\n \t\told_same = !!(commit->object.flags & TREESAME);\n-\t\tif (rev_same_tree_as_empty(revs, commit))\n+\t\tif (rev_same_tree_as_empty(revs, commit, nth_parent))\n \t\t\tcommit->object.flags |= TREESAME;\n \t\telse\n \t\t\tcommit->object.flags &= ~TREESAME;\n@@ -978,7 +989,14 @@ static void try_to_simplify_commit(struct rev_info *revs, struct commit *commit)\n \t\treturn;\n \n \tif (!commit->parents) {\n-\t\tif (rev_same_tree_as_empty(revs, commit))\n+\t\t/*\n+\t\t * Pretend as if we are comparing ourselves to the\n+\t\t * (non-existent) first parent of this commit object. Even\n+\t\t * though no such parent exists, its changed-path Bloom filter\n+\t\t * (if one exists) is relative to the empty tree, using Bloom\n+\t\t * filters is allowed here.\n+\t\t */\n+\t\tif (rev_same_tree_as_empty(revs, commit, 1))\n \t\t\tcommit->object.flags |= TREESAME;\n \t\treturn;\n \t}\n@@ -1059,7 +1077,7 @@ static void try_to_simplify_commit(struct rev_info *revs, struct commit *commit)\n \n \t\tcase REV_TREE_NEW:\n \t\t\tif (revs->remove_empty_trees &&\n-\t\t\t rev_same_tree_as_empty(revs, p)) {\n+\t\t\t rev_same_tree_as_empty(revs, p, nth_parent)) {\n \t\t\t\t/* We are adding all the specified\n \t\t\t\t * paths from this parent, so the\n \t\t\t\t * history beyond this parent is not\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 487fc3d6b9..322640feeb 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -87,7 +87,11 @@ test_bloom_filters_not_used () {\n \t\t# if the Bloom filter system is initialized, ensure that no\n \t\t# filters were used\n \t\tdata=\"statistics:{\"\n-\t\tdata=\"$data\\\"filter_not_present\\\":0,\"\n+\t\t# unusable filters (e.g., those computed with a\n+\t\t# different value of commitGraph.changedPathsVersion)\n+\t\t# are counted in the filter_not_present bucket, so any\n+\t\t# value is OK there.\n+\t\tdata=\"$data\\\"filter_not_present\\\":[0-9][0-9]*,\"\n \t\tdata=\"$data\\\"maybe\\\":0,\"\n \t\tdata=\"$data\\\"definitely_not\\\":0,\"\n \t\tdata=\"$data\\\"false_positive\\\":0}\"\n@@ -174,7 +178,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":9\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":10\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483007","messageId":"17703ed89ae9a60a26161682ce1c5a468d4ab3e0.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 04/17] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:30Z","receivedAt":"2023-10-10T20:33:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jonathan Tan \n\nThe code change to Git to support version 2 will be done in subsequent\ncommits.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n Documentation/gitformat-commit-graph.txt | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\nindex 31cad585e2..3e906e8030 100644\n--- a/Documentation/gitformat-commit-graph.txt\n+++ b/Documentation/gitformat-commit-graph.txt\n@@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n \n ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n * It starts with header consisting of three unsigned 32-bit integers:\n- - Version of the hash algorithm being used. We currently only support\n-\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n+ - Version of the hash algorithm being used. We currently support\n+\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n \timplemented exactly as described in\n \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n-\tin Probabilistic Verification\"\n+\tin Probabilistic Verification\". Version 1 Bloom filters have a bug that appears\n+\twhen char is signed and the repository has path names that have characters >=\n+\t0x80; Git supports reading and writing them, but this ability will be removed\n+\tin a future version of Git.\n - The number of times a path is hashed and hence the number of bit positions\n \t that cumulatively determine whether a file is present in the commit.\n - The minimum number of bits 'b' per entry in the Bloom filter. If the filter\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483008","messageId":"94552abf455c6d341a0811333ae4edb4a8cea259.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 05/17] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:33Z","receivedAt":"2023-10-10T20:33:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for the 'read-graph' test helper to perform other tasks besides\ndumping high-level information about the commit-graph by extracting its\nmain routine into a separate function.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 31 ++++++++++++++++++-------------\n 1 file changed, 18 insertions(+), 13 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 8c7a83f578..3375392f6c 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -5,20 +5,8 @@\n #include \"bloom.h\"\n #include \"setup.h\"\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_info(struct commit_graph *graph)\n {\n-\tstruct commit_graph *graph = NULL;\n-\tstruct object_directory *odb;\n-\n-\tsetup_git_directory();\n-\todb = the_repository->objects->odb;\n-\n-\tprepare_repo_settings(the_repository);\n-\n-\tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n@@ -57,6 +45,23 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tif (graph->topo_levels)\n \t\tprintf(\" topo_levels\");\n \tprintf(\"\\n\");\n+}\n+\n+int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tstruct object_directory *odb;\n+\n+\tsetup_git_directory();\n+\todb = the_repository->objects->odb;\n+\n+\tprepare_repo_settings(the_repository);\n+\n+\tgraph = read_commit_graph_one(the_repository, odb);\n+\tif (!graph)\n+\t\treturn 1;\n+\n+\tdump_graph_info(graph);\n \n \tUNLEAK(graph);\n \n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483009","messageId":"2ecc0a2d58432b149d73a3e2abfa948eb1f0aa0b.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 03/17] commit-graph: ensure Bloom filters are read with consistent settings","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:26Z","receivedAt":"2023-10-10T20:33:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The changed-path Bloom filter mechanism is parameterized by a couple of\nvariables, notably the number of bits per hash (typically \"m\" in Bloom\nfilter literature) and the number of hashes themselves (typically \"k\").\n\nIt is critically important that filters are read with the Bloom filter\nsettings that they were written with. Failing to do so would mean that\neach query is liable to compute different fingerprints, meaning that the\nfilter itself could return a false negative. This goes against a basic\nassumption of using Bloom filters (that they may return false positives,\nbut never false negatives) and can lead to incorrect results.\n\nWe have some existing logic to carry forward existing Bloom filter\nsettings from one layer to the next. In `write_commit_graph()`, we have\nsomething like:\n\n if (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\n struct commit_graph *g = ctx->r->objects->commit_graph;\n\n /* We have changed-paths already. Keep them in the next graph */\n if (g && g->chunk_bloom_data) {\n ctx->changed_paths = 1;\n ctx->bloom_settings = g->bloom_filter_settings;\n }\n }\n\n, which drags forward Bloom filter settings across adjacent layers.\n\nThis doesn't quite address all cases, however, since it is possible for\nintermediate layers to contain no Bloom filters at all. For example,\nsuppose we have two layers in a commit-graph chain, say, {G1, G2}. If G1\ncontains Bloom filters, but G2 doesn't, a new G3 (whose base graph is\nG2) may be written with arbitrary Bloom filter settings, because we only\ncheck the immediately adjacent layer's settings for compatibility.\n\nThis behavior has existed since the introduction of changed-path Bloom\nfilters. But in practice, this is not such a big deal, since the only\nway up until this point to modify the Bloom filter settings at write\ntime is with the undocumented environment variables:\n\n - GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\n - GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\n - GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\n\n(it is still possible to tweak MAX_CHANGED_PATHS between layers, but\nthis does not affect reads, so is allowed to differ across multiple\ngraph layers).\n\nBut in future commits, we will introduce another parameter to change the\nhash algorithm used to compute Bloom fingerprints itself. This will be\nexposed via a configuration setting, making this foot-gun easier to use.\n\nTo prevent this potential issue, validate that all layers of a split\ncommit-graph have compatible settings with the newest layer which\ncontains Bloom filters.\n\nReported-by: SZEDER Gábor \nOriginal-test-by: SZEDER Gábor \nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 25 +++++++++++++++++\n t/t4216-log-bloom.sh | 64 ++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 89 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1a56efcf69..ae0902f7f4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -498,6 +498,30 @@ static int validate_mixed_generation_chain(struct commit_graph *g)\n \treturn 0;\n }\n \n+static void validate_mixed_bloom_settings(struct commit_graph *g)\n+{\n+\tstruct bloom_filter_settings *settings = NULL;\n+\tfor (; g; g = g->base_graph) {\n+\t\tif (!g->bloom_filter_settings)\n+\t\t\tcontinue;\n+\t\tif (!settings) {\n+\t\t\tsettings = g->bloom_filter_settings;\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tif (g->bloom_filter_settings->bits_per_entry != settings->bits_per_entry ||\n+\t\t g->bloom_filter_settings->num_hashes != settings->num_hashes) {\n+\t\t\tg->chunk_bloom_indexes = NULL;\n+\t\t\tg->chunk_bloom_data = NULL;\n+\t\t\tFREE_AND_NULL(g->bloom_filter_settings);\n+\n+\t\t\twarning(_(\"disabling Bloom filters for commit-graph \"\n+\t\t\t\t \"layer '%s' due to incompatible settings\"),\n+\t\t\t\toid_to_hex(&g->oid));\n+\t\t}\n+\t}\n+}\n+\n static int add_graph_to_chain(struct commit_graph *g,\n \t\t\t struct commit_graph *chain,\n \t\t\t struct object_id *oids,\n@@ -614,6 +638,7 @@ struct commit_graph *load_commit_graph_chain_fd_st(struct repository *r,\n \t}\n \n \tvalidate_mixed_generation_chain(graph_chain);\n+\tvalidate_mixed_bloom_settings(graph_chain);\n \n \tfree(oids);\n \tfclose(fp);\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 322640feeb..f49a8f2fbf 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -420,4 +420,68 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n+graph=.git/objects/info/commit-graph\n+graphdir=.git/objects/info/commit-graphs\n+chain=$graphdir/commit-graph-chain\n+\n+test_expect_success 'setup for mixed Bloom setting tests' '\n+\trepo=mixed-bloom-settings &&\n+\n+\tgit init $repo &&\n+\tfor i in one two three\n+\tdo\n+\t\ttest_commit -C $repo $i file || return 1\n+\tdone\n+'\n+\n+test_expect_success 'split' '\n+\t# Compute Bloom filters with \"unusual\" settings.\n+\tgit -C $repo rev-parse one >in &&\n+\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=3 git -C $repo commit-graph write \\\n+\t\t--stdin-commits --changed-paths --split in &&\n+\tgit -C $repo commit-graph write --stdin-commits --no-changed-paths \\\n+\t\t--split=no-merge in &&\n+\tgit -C $repo commit-graph write --stdin-commits --changed-paths \\\n+\t\t--split=no-merge expect 2>err &&\n+\tgit -C $repo log --oneline --no-decorate -- file >actual 2>err &&\n+\ttest_cmp expect actual &&\n+\tgrep \"disabling Bloom filters for commit-graph layer .$layer.\" err\n+'\n+\n+test_expect_success 'merge graph layers with incompatible Bloom settings' '\n+\t# Ensure that incompatible Bloom filters are ignored when\n+\t# generating new layers.\n+\tgit -C $repo commit-graph write --reachable --changed-paths 2>err &&\n+\tgrep \"disabling Bloom filters for commit-graph layer .$layer.\" err &&\n+\n+\ttest_path_is_file $repo/$graph &&\n+\ttest_dir_is_empty $repo/$graphdir &&\n+\n+\t# ...and merging existing ones.\n+\tgit -C $repo -c core.commitGraph=false log --oneline --no-decorate -- file \\\n+\t\t>expect 2>err &&\n+\tGIT_TRACE2_PERF=\"$(pwd)/trace.perf\" \\\n+\t\tgit -C $repo log --oneline --no-decorate -- file >actual 2>err &&\n+\n+\ttest_cmp expect actual && cat err &&\n+\tgrep \"statistics:{\\\"filter_not_present\\\":0\" trace.perf &&\n+\t! grep \"disabling Bloom filters\" err\n+'\n+\n test_done\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483010","messageId":"3d81efa27b9b7c5b0cc2c77d080c7b65a4e46faa.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 06/17] bloom.h: make `load_bloom_filter_from_graph()` public","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:36Z","receivedAt":"2023-10-10T20:33:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for a future commit to use the load_bloom_filter_from_graph()\nfunction directly to load specific Bloom filters out of the commit-graph\nfor manual inspection (to be used during tests).\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n bloom.c | 6 +++---\n bloom.h | 5 +++++\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex aef6b5fea2..3e78cfe79d 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -29,9 +29,9 @@ static inline unsigned char get_bitmask(uint32_t pos)\n \treturn ((unsigned char)1) << (pos & (BITS_PER_WORD - 1));\n }\n \n-static int load_bloom_filter_from_graph(struct commit_graph *g,\n-\t\t\t\t\tstruct bloom_filter *filter,\n-\t\t\t\t\tuint32_t graph_pos)\n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos)\n {\n \tuint32_t lex_pos, start_index, end_index;\n \ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..1e4f612d2c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -3,6 +3,7 @@\n \n struct commit;\n struct repository;\n+struct commit_graph;\n \n struct bloom_filter_settings {\n \t/*\n@@ -68,6 +69,10 @@ struct bloom_key {\n \tuint32_t *hashes;\n };\n \n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos);\n+\n /*\n * Calculate the murmur3 32-bit hash value for the given data\n * using the given seed.\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483011","messageId":"d23cd8903722966ec40fca7d76e502bf815e611e.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 07/17] t/helper/test-read-graph: implement `bloom-filters` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:39Z","receivedAt":"2023-10-10T20:34:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement a mode of the \"read-graph\" test helper to dump out the\nhexadecimal contents of the Bloom filter(s) contained in a commit-graph.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 44 +++++++++++++++++++++++++++++++++-----\n 1 file changed, 39 insertions(+), 5 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 3375392f6c..da9ac8584d 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -47,10 +47,32 @@ static void dump_graph_info(struct commit_graph *graph)\n \tprintf(\"\\n\");\n }\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_bloom_filters(struct commit_graph *graph)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < graph->num_commits + graph->num_commits_in_base; i++) {\n+\t\tstruct bloom_filter filter = { 0 };\n+\t\tsize_t j;\n+\n+\t\tif (load_bloom_filter_from_graph(graph, &filter, i) < 0) {\n+\t\t\tfprintf(stderr, \"missing Bloom filter for graph \"\n+\t\t\t\t\"position %\"PRIu32\"\\n\", i);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (j = 0; j < filter.len; j++)\n+\t\t\tprintf(\"%02x\", filter.data[j]);\n+\t\tif (filter.len)\n+\t\t\tprintf(\"\\n\");\n+\t}\n+}\n+\n+int cmd__read_graph(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n \tstruct object_directory *odb;\n+\tint ret = 0;\n \n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n@@ -58,12 +80,24 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tprepare_repo_settings(the_repository);\n \n \tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n+\tif (!graph) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n \n-\tdump_graph_info(graph);\n+\tif (argc <= 1)\n+\t\tdump_graph_info(graph);\n+\telse if (!strcmp(argv[1], \"bloom-filters\"))\n+\t\tdump_graph_bloom_filters(graph);\n+\telse {\n+\t\tfprintf(stderr, \"unknown sub-command: '%s'\\n\", argv[1]);\n+\t\tret = 1;\n+\t}\n \n+done:\n \tUNLEAK(graph);\n \n-\treturn 0;\n+\treturn ret;\n }\n+\n+\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483012","messageId":"a08a961f4170f58c34eec7d89b2e7979724f8a1d.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 09/17] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:46Z","receivedAt":"2023-10-10T20:34:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jonathan Tan \n\nA subsequent commit will introduce another version of the changed-path\nfilter in the commit graph file. In order to control which version to\nwrite (and read), a config variable is needed.\n\nTherefore, introduce this config variable. For forwards compatibility,\nteach Git to not read commit graphs when the config variable\nis set to an unsupported version. Because we teach Git this,\ncommitgraph.readChangedPaths is now redundant, so deprecate it and\ndefine its behavior in terms of the config variable we introduce.\n\nThis commit does not change the behavior of writing (Git writes changed\npath filters when explicitly instructed regardless of any config\nvariable), but a subsequent commit will restrict Git such that it will\nonly write when commitgraph.changedPathsVersion is a recognized value.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 23 ++++++++++++++++++++---\n commit-graph.c | 2 +-\n oss-fuzz/fuzz-commit-graph.c | 2 +-\n repo-settings.c | 6 +++++-\n repository.h | 2 +-\n 5 files changed, 28 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c..2dc9170622 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -9,6 +9,23 @@ commitGraph.maxNewFilters::\n \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n \n commitGraph.readChangedPaths::\n-\tIf true, then git will use the changed-path Bloom filters in the\n-\tcommit-graph file (if it exists, and they are present). Defaults to\n-\ttrue. See linkgit:git-commit-graph[1] for more information.\n+\tDeprecated. Equivalent to commitGraph.changedPathsVersion=-1 if true, and\n+\tcommitGraph.changedPathsVersion=0 if false. (If commitGraph.changedPathVersion\n+\tis also set, commitGraph.changedPathsVersion takes precedence.)\n+\n+commitGraph.changedPathsVersion::\n+\tSpecifies the version of the changed-path Bloom filters that Git will read and\n+\twrite. May be -1, 0 or 1.\n++\n+Defaults to -1.\n++\n+If -1, Git will use the version of the changed-path Bloom filters in the\n+repository, defaulting to 1 if there are none.\n++\n+If 0, Git will not read any Bloom filters, and will write version 1 Bloom\n+filters when instructed to write.\n++\n+If 1, Git will only read version 1 Bloom filters, and will write version 1\n+Bloom filters.\n++\n+See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ae0902f7f4..ea677c87fb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -411,7 +411,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_read_changed_paths) {\n+\tif (s->commit_graph_changed_paths_version) {\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\ndiff --git a/oss-fuzz/fuzz-commit-graph.c b/oss-fuzz/fuzz-commit-graph.c\nindex 2992079dd9..325c0b991a 100644\n--- a/oss-fuzz/fuzz-commit-graph.c\n+++ b/oss-fuzz/fuzz-commit-graph.c\n@@ -19,7 +19,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \t * possible.\n \t */\n \tthe_repository->settings.commit_graph_generation_version = 2;\n-\tthe_repository->settings.commit_graph_read_changed_paths = 1;\n+\tthe_repository->settings.commit_graph_changed_paths_version = 1;\n \tg = parse_commit_graph(&the_repository->settings, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 525f69c0c7..db8fe817f3 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n \tint value;\n \tconst char *strval;\n \tint manyfiles;\n+\tint read_changed_paths;\n \n \tif (!r->gitdir)\n \t\tBUG(\"Cannot add settings for uninitialized repository\");\n@@ -54,7 +55,10 @@ void prepare_repo_settings(struct repository *r)\n \t/* Commit graph config or default, does not cascade (simple) */\n \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n-\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &read_changed_paths, 1);\n+\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n+\t\t &r->settings.commit_graph_changed_paths_version,\n+\t\t read_changed_paths ? -1 : 0);\n \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n \ndiff --git a/repository.h b/repository.h\nindex 5f18486f64..f71154e12c 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -29,7 +29,7 @@ struct repo_settings {\n \n \tint core_commit_graph;\n \tint commit_graph_generation_version;\n-\tint commit_graph_read_changed_paths;\n+\tint commit_graph_changed_paths_version;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \tint command_requires_full_index;\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483013","messageId":"1f7f27bc47eba053731e75fb7a4cb330065f6caf.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 15/17] object.h: fix mis-aligned flag bits table","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:34:05Z","receivedAt":"2023-10-10T20:34:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Bit position 23 is one column too far to the left.\n\nSigned-off-by: Taylor Blau \n---\n object.h | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/object.h b/object.h\nindex 114d45954d..db25714b4e 100644\n--- a/object.h\n+++ b/object.h\n@@ -62,7 +62,7 @@ void object_array_init(struct object_array *array);\n \n /*\n * object flag allocation:\n- * revision.h: 0---------10 15 23------27\n+ * revision.h: 0---------10 15 23------27\n * fetch-pack.c: 01 67\n * negotiator/default.c: 2--5\n * walker.c: 0-2\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483014","messageId":"2ba10a4b4b890d3c75f128a972a93889edc4f60e.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 12/17] bloom: prepare to discard incompatible Bloom filters","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:55Z","receivedAt":"2023-10-10T20:34:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Callers use the inline `get_bloom_filter()` implementation as a thin\nwrapper around `get_or_compute_bloom_filter()`. The former calls the\nlatter with a value of \"0\" for `compute_if_not_present`, making\n`get_bloom_filter()` the default read-only path for fetching an existing\nBloom filter.\n\nCallers expect the value returned from `get_bloom_filter()` is usable,\nthat is that it's compatible with the configured value corresponding to\n`commitGraph.changedPathsVersion`.\n\nThis is OK, since the commit-graph machinery only initializes its BDAT\nchunk (thereby enabling it to service Bloom filter queries) when the\nBloom filter hash_version is compatible with our settings. So any value\nreturned by `get_bloom_filter()` is trivially useable.\n\nHowever, subsequent commits will load the BDAT chunk even when the Bloom\nfilters are built with incompatible hash versions. Prepare to handle\nthis by teaching `get_bloom_filter()` to discard filters that are\nincompatible with the configured hash version.\n\nCallers who wish to read incompatible filters (e.g., for upgrading\nfilters from v1 to v2) may use the lower level routine,\n`get_or_compute_bloom_filter()`.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 20 +++++++++++++++++++-\n bloom.h | 20 ++++++++++++++++++--\n 2 files changed, 37 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 9b6a30f6f6..739fa093ba 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -250,6 +250,23 @@ static void init_truncated_large_filter(struct bloom_filter *filter,\n \tfilter->version = version;\n }\n \n+struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c)\n+{\n+\tstruct bloom_filter *filter;\n+\tint hash_version;\n+\n+\tfilter = get_or_compute_bloom_filter(r, c, 0, NULL, NULL);\n+\tif (!filter)\n+\t\treturn NULL;\n+\n+\tprepare_repo_settings(r);\n+\thash_version = r->settings.commit_graph_changed_paths_version;\n+\n+\tif (!(hash_version == -1 || hash_version == filter->version))\n+\t\treturn NULL; /* unusable filter */\n+\treturn filter;\n+}\n+\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n@@ -275,7 +292,8 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t filter, graph_pos);\n \t}\n \n-\tif (filter->data && filter->len)\n+\tif ((filter->data && filter->len) &&\n+\t (!settings || settings->hash_version == filter->version))\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\ndiff --git a/bloom.h b/bloom.h\nindex 330a140520..bfe389e29c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -110,8 +110,24 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t enum bloom_filter_computed *computed);\n \n-#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n-\t(r), (c), 0, NULL, NULL)\n+/*\n+ * Find the Bloom filter associated with the given commit \"c\".\n+ *\n+ * If any of the following are true\n+ *\n+ * - the repository does not have a commit-graph, or\n+ * - the repository disables reading from the commit-graph, or\n+ * - the given commit does not have a Bloom filter computed, or\n+ * - there is a Bloom filter for commit \"c\", but it cannot be read\n+ * because the filter uses an incompatible version of murmur3\n+ *\n+ * , then `get_bloom_filter()` will return NULL. Otherwise, the corresponding\n+ * Bloom filter will be returned.\n+ *\n+ * For callers who wish to inspect Bloom filters with incompatible hash\n+ * versions, use get_or_compute_bloom_filter().\n+ */\n+struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c);\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483015","messageId":"09d8669c3a074e7a2ace9d650a345244b2362f7e.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 13/17] commit-graph.c: unconditionally load Bloom filters","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:59Z","receivedAt":"2023-10-10T20:34:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 9e4df4da07 (commit-graph: new filter ver. that fixes murmur3,\n2023-08-01), we began ignoring the Bloom data (\"BDAT\") chunk for\ncommit-graphs whose Bloom filters were computed using a hash version\nincompatible with the value of `commitGraph.changedPathVersion`.\n\nNow that the Bloom API has been hardened to discard these incompatible\nfilters (with the exception of low-level APIs), we can safely load these\nBloom filters unconditionally.\n\nWe no longer want to return early from `graph_read_bloom_data()`, and\nsimilarly do not want to set the bloom_settings' `hash_version` field as\na side-effect. The latter is because we want to wait until we know which\nBloom settings we're using (either the defaults, from the GIT_TEST\nvariables, or from the previous commit-graph layer) before deciding what\nhash_version to use.\n\nIf we detect an existing BDAT chunk, we'll infer the rest of the\nsettings (e.g., number of hashes, bits per entry, and maximum number of\nchanged paths) from the earlier graph layer. The hash_version will be\ninferred from the previous layer as well, unless one has already been\nspecified via configuration.\n\nOnce all of that is done, we normalize the value of the hash_version to\neither \"1\" or \"2\".\n\nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 19 ++++++++++---------\n 1 file changed, 10 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex db623afd09..fa3b58e762 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -327,12 +327,6 @@ static int graph_read_bloom_data(const unsigned char *chunk_start,\n \tuint32_t hash_version;\n \thash_version = get_be32(chunk_start);\n \n-\tif (*c->commit_graph_changed_paths_version == -1) {\n-\t\t*c->commit_graph_changed_paths_version = hash_version;\n-\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n-\t\treturn 0;\n-\t}\n-\n \tg->chunk_bloom_data = chunk_start;\n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n \tg->bloom_filter_settings->hash_version = hash_version;\n@@ -2473,8 +2467,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n-\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n-\t\t? 2 : 1;\n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2506,10 +2499,18 @@ int write_commit_graph(struct object_directory *odb,\n \t\t/* We have changed-paths already. Keep them in the next graph */\n \t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n-\t\t\tctx->bloom_settings = g->bloom_filter_settings;\n+\n+\t\t\t/* don't propagate the hash_version unless unspecified */\n+\t\t\tif (bloom_settings.hash_version == -1)\n+\t\t\t\tbloom_settings.hash_version = g->bloom_filter_settings->hash_version;\n+\t\t\tbloom_settings.bits_per_entry = g->bloom_filter_settings->bits_per_entry;\n+\t\t\tbloom_settings.num_hashes = g->bloom_filter_settings->num_hashes;\n+\t\t\tbloom_settings.max_changed_paths = g->bloom_filter_settings->max_changed_paths;\n \t\t}\n \t}\n \n+\tbloom_settings.hash_version = bloom_settings.hash_version == 2 ? 2 : 1;\n+\n \tif (ctx->split) {\n \t\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n \n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483016","messageId":"a8c10f8de83eddb9a0d3e39bd31bf30808cb59ce.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 11/17] bloom: annotate filters with hash version","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:52Z","receivedAt":"2023-10-10T20:34:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In subsequent commits, we will want to load existing Bloom filters out\nof a commit-graph, even when the hash version they were computed with\ndoes not match the value of `commitGraph.changedPathVersion`.\n\nIn order to differentiate between the two, add a \"version\" field to each\nBloom filter.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 11 ++++++++---\n bloom.h | 1 +\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex ebef5cfd2f..9b6a30f6f6 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -55,6 +55,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n \t\t\t\t\tsizeof(unsigned char) * start_index +\n \t\t\t\t\tBLOOMDATA_CHUNK_HEADER_SIZE);\n+\tfilter->version = g->bloom_filter_settings->hash_version;\n \n \treturn 1;\n }\n@@ -240,11 +241,13 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data UNUSED,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-static void init_truncated_large_filter(struct bloom_filter *filter)\n+static void init_truncated_large_filter(struct bloom_filter *filter,\n+\t\t\t\t\tint version)\n {\n \tfilter->data = xmalloc(1);\n \tfilter->data[0] = 0xFF;\n \tfilter->len = 1;\n+\tfilter->version = version;\n }\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n@@ -329,13 +332,15 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t}\n \n \t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths) {\n-\t\t\tinit_truncated_large_filter(filter);\n+\t\t\tinit_truncated_large_filter(filter,\n+\t\t\t\t\t\t settings->hash_version);\n \t\t\tif (computed)\n \t\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n \t\t\tgoto cleanup;\n \t\t}\n \n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tfilter->version = settings->hash_version;\n \t\tif (!filter->len) {\n \t\t\tif (computed)\n \t\t\t\t*computed |= BLOOM_TRUNC_EMPTY;\n@@ -355,7 +360,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n-\t\tinit_truncated_large_filter(filter);\n+\t\tinit_truncated_large_filter(filter, settings->hash_version);\n \n \t\tif (computed)\n \t\t\t*computed |= BLOOM_TRUNC_LARGE;\ndiff --git a/bloom.h b/bloom.h\nindex 138d57a86b..330a140520 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -55,6 +55,7 @@ struct bloom_filter_settings {\n struct bloom_filter {\n \tunsigned char *data;\n \tsize_t len;\n+\tint version;\n };\n \n /*\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483017","messageId":"0d4f9dc4ee58feb81928f92f6f8ac465e49083c0.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 14/17] commit-graph: drop unnecessary `graph_read_bloom_data_context`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:34:02Z","receivedAt":"2023-10-10T20:34:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The `graph_read_bloom_data_context` struct was introduced in an earlier\ncommit in order to pass pointers to the commit-graph and changed-path\nBloom filter version when reading the BDAT chunk.\n\nThe previous commit no longer writes through the changed_paths_version\npointer, making the surrounding context structure unnecessary. Drop it\nand pass a pointer to the commit-graph directly when reading the BDAT\nchunk.\n\nNoticed-by: Jonathan Tan \nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 14 ++------------\n 1 file changed, 2 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex fa3b58e762..e0fc62e110 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -314,16 +314,10 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n-struct graph_read_bloom_data_context {\n-\tstruct commit_graph *g;\n-\tint *commit_graph_changed_paths_version;\n-};\n-\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t size_t chunk_size, void *data)\n {\n-\tstruct graph_read_bloom_data_context *c = data;\n-\tstruct commit_graph *g = c->g;\n+\tstruct commit_graph *g = data;\n \tuint32_t hash_version;\n \thash_version = get_be32(chunk_start);\n \n@@ -415,14 +409,10 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t}\n \n \tif (s->commit_graph_changed_paths_version) {\n-\t\tstruct graph_read_bloom_data_context context = {\n-\t\t\t.g = graph,\n-\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n-\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t graph_read_bloom_data, &context);\n+\t\t\t graph_read_bloom_data, graph);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483018","messageId":"61d44519a5ffaf2c040198cf8d80d05a09de5de5.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 10/17] commit-graph: new filter ver. that fixes murmur3","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:49Z","receivedAt":"2023-10-10T20:34:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jonathan Tan \n\nThe murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 (thus,\nthe changed path filters wouldn't be readable by other off-the-shelf\nimplementatios of murmur3) and even with Git binaries that were compiled\nwith different signedness of char. This bug affects both how Git writes\nchanged path filters to disk and how Git interprets changed path filters\non disk.\n\nTherefore, introduce a new version (2) of changed path filters that\ncorrects this problem. The existing version (1) is still supported and\nis still the default, but users should migrate away from it as soon\nas possible.\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter.\n\nSo this patch does not include any mechanism to \"salvage\" changed path\nfilters from repositories. There is also no \"mixed\" mode - for each\ninvocation of Git, reading and writing changed path filters are done\nwith the same version number; this version number may be explicitly\nstated (typically if the user knows which version they need) or\nautomatically determined from the version of the existing changed path\nfilters in the repository.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in t0095 was obtained from another murmur3 implementation\nusing the following Go source code:\n\n package main\n\n import \"fmt\"\n import \"github.com/spaolacci/murmur3\"\n\n func main() {\n fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n }\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 5 +-\n bloom.c | 69 +++++++++++++++++-\n bloom.h | 8 +-\n commit-graph.c | 32 ++++++--\n t/helper/test-bloom.c | 9 ++-\n t/t0095-bloom.sh | 8 ++\n t/t4216-log-bloom.sh | 105 +++++++++++++++++++++++++++\n 7 files changed, 223 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 2dc9170622..acc74a2f27 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -15,7 +15,7 @@ commitGraph.readChangedPaths::\n \n commitGraph.changedPathsVersion::\n \tSpecifies the version of the changed-path Bloom filters that Git will read and\n-\twrite. May be -1, 0 or 1.\n+\twrite. May be -1, 0, 1, or 2.\n +\n Defaults to -1.\n +\n@@ -28,4 +28,7 @@ filters when instructed to write.\n If 1, Git will only read version 1 Bloom filters, and will write version 1\n Bloom filters.\n +\n+If 2, Git will only read version 2 Bloom filters, and will write version 2\n+Bloom filters.\n++\n See linkgit:git-commit-graph[1] for more information.\ndiff --git a/bloom.c b/bloom.c\nindex 3e78cfe79d..ebef5cfd2f 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -66,7 +66,64 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n * Not considered to be cryptographically secure.\n * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n+{\n+\tconst uint32_t c1 = 0xcc9e2d51;\n+\tconst uint32_t c2 = 0x1b873593;\n+\tconst uint32_t r1 = 15;\n+\tconst uint32_t r2 = 13;\n+\tconst uint32_t m = 5;\n+\tconst uint32_t n = 0xe6546b64;\n+\tint i;\n+\tuint32_t k1 = 0;\n+\tconst char *tail;\n+\n+\tint len4 = len / sizeof(uint32_t);\n+\n+\tuint32_t k;\n+\tfor (i = 0; i < len4; i++) {\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n+\t\tk = byte1 | byte2 | byte3 | byte4;\n+\t\tk *= c1;\n+\t\tk = rotate_left(k, r1);\n+\t\tk *= c2;\n+\n+\t\tseed ^= k;\n+\t\tseed = rotate_left(seed, r2) * m + n;\n+\t}\n+\n+\ttail = (data + len4 * sizeof(uint32_t));\n+\n+\tswitch (len & (sizeof(uint32_t) - 1)) {\n+\tcase 3:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n+\t\t/*-fallthrough*/\n+\tcase 2:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n+\t\t/*-fallthrough*/\n+\tcase 1:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n+\t\tk1 *= c1;\n+\t\tk1 = rotate_left(k1, r1);\n+\t\tk1 *= c2;\n+\t\tseed ^= k1;\n+\t\tbreak;\n+\t}\n+\n+\tseed ^= (uint32_t)len;\n+\tseed ^= (seed >> 16);\n+\tseed *= 0x85ebca6b;\n+\tseed ^= (seed >> 13);\n+\tseed *= 0xc2b2ae35;\n+\tseed ^= (seed >> 16);\n+\n+\treturn seed;\n+}\n+\n+static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n {\n \tconst uint32_t c1 = 0xcc9e2d51;\n \tconst uint32_t c2 = 0x1b873593;\n@@ -131,8 +188,14 @@ void fill_bloom_key(const char *data,\n \tint i;\n \tconst uint32_t seed0 = 0x293ae76f;\n \tconst uint32_t seed1 = 0x7e646e2c;\n-\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n-\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n+\tuint32_t hash0, hash1;\n+\tif (settings->hash_version == 2) {\n+\t\thash0 = murmur3_seeded_v2(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v2(seed1, data, len);\n+\t} else {\n+\t\thash0 = murmur3_seeded_v1(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v1(seed1, data, len);\n+\t}\n \n \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n \tfor (i = 0; i < settings->num_hashes; i++)\ndiff --git a/bloom.h b/bloom.h\nindex 1e4f612d2c..138d57a86b 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -8,9 +8,11 @@ struct commit_graph;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * The newest version is 2, which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -80,7 +82,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n * Not considered to be cryptographically secure.\n * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n \n void fill_bloom_key(const char *data,\n \t\t size_t len,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ea677c87fb..db623afd09 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -314,17 +314,26 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct graph_read_bloom_data_context {\n+\tstruct commit_graph *g;\n+\tint *commit_graph_changed_paths_version;\n+};\n+\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t size_t chunk_size, void *data)\n {\n-\tstruct commit_graph *g = data;\n+\tstruct graph_read_bloom_data_context *c = data;\n+\tstruct commit_graph *g = c->g;\n \tuint32_t hash_version;\n-\tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n+\tif (*c->commit_graph_changed_paths_version == -1) {\n+\t\t*c->commit_graph_changed_paths_version = hash_version;\n+\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n \t\treturn 0;\n+\t}\n \n+\tg->chunk_bloom_data = chunk_start;\n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n \tg->bloom_filter_settings->hash_version = hash_version;\n \tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n@@ -412,10 +421,14 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t}\n \n \tif (s->commit_graph_changed_paths_version) {\n+\t\tstruct graph_read_bloom_data_context context = {\n+\t\t\t.g = graph,\n+\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n+\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t graph_read_bloom_data, graph);\n+\t\t\t graph_read_bloom_data, &context);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -2441,6 +2454,13 @@ int write_commit_graph(struct object_directory *odb,\n \t}\n \tif (!commit_graph_compatible(r))\n \t\treturn 0;\n+\tif (r->settings.commit_graph_changed_paths_version < -1\n+\t || r->settings.commit_graph_changed_paths_version > 2) {\n+\t\twarning(_(\"attempting to write a commit-graph, but \"\n+\t\t\t \"'commitgraph.changedPathsVersion' (%d) is not supported\"),\n+\t\t\tr->settings.commit_graph_changed_paths_version);\n+\t\treturn 0;\n+\t}\n \n \tCALLOC_ARRAY(ctx, 1);\n \tctx->r = r;\n@@ -2453,6 +2473,8 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n+\t\t? 2 : 1;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2482,7 +2504,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex aabe31d724..3cbc0a5b50 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -50,6 +50,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \" test-tool bloom get_murmur3 \\n\"\n+\" test-tool bloom get_murmur3_seven_highbit\\n\"\n \" test-tool bloom generate_filter [...]\\n\"\n \" test-tool bloom get_filter_for_commit \\n\";\n \n@@ -64,7 +65,13 @@ int cmd__bloom(int argc, const char **argv)\n \t\tuint32_t hashed;\n \t\tif (argc < 3)\n \t\t\tusage(bloom_usage);\n-\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n+\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex da67c40134..8f8b5d4966 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -536,4 +536,109 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n \t)\n '\n \n+test_expect_success 'version 1 changed-path not used when version 2 requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n+\t\ttest_bloom_filters_not_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 1 changed-path used when autodetect requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n+\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(1,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n+\tgit init highbit2 &&\n+\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n+\ttest_commit -C highbit2 c2 \"$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'check value of version 2 changed-path' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\techo \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'setup make another commit' '\n+\t# \"git log\" does not use Bloom filters for root commits - see how, in\n+\t# revision.c, rev_compare_tree() (the only code path that eventually calls\n+\t# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit\n+\t# has one parent. Therefore, make another commit so that we perform the tests on\n+\t# a non-root commit.\n+\ttest_commit -C highbit2 anotherc2 \"another$CENT\"\n+'\n+\n+test_expect_success 'version 2 changed-path used when version 2 requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\ttest_bloom_filters_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path not used when version 1 requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n+\t\ttest_bloom_filters_not_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path used when autodetect requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- another$CENT\"\n+\t)\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n+\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(2,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n+\tgit init doublewrite &&\n+\ttest_commit -C doublewrite c \"$CENT\" &&\n+\tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n+\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n+\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd doublewrite &&\n+\t\techo \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483019","messageId":"cba766f224b0d2b4fd952b11bef8068c07dfcf88.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 08/17] t4216: test changed path filters with high bit paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:33:42Z","receivedAt":"2023-10-10T20:34:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Jonathan Tan \n\nSubsequent commits will teach Git another version of changed path\nfilter that has different behavior with paths that contain at least\none character with its high bit set, so test the existing behavior as\na baseline.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 52 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex f49a8f2fbf..da67c40134 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -484,4 +484,56 @@ test_expect_success 'merge graph layers with incompatible Bloom settings' '\n \t! grep \"disabling Bloom filters\" err\n '\n \n+get_first_changed_path_filter () {\n+\ttest-tool read-graph bloom-filters >filters.dat &&\n+\thead -n 1 filters.dat\n+}\n+\n+# chosen to be the same under all Unicode normalization forms\n+CENT=$(printf \"\\302\\242\")\n+\n+test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n+\tgit init highbit1 &&\n+\ttest_commit -C highbit1 c1 \"$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'setup check value of version 1 changed-path' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\techo \"52a9\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+# expect will not match actual if char is unsigned by default. Write the test\n+# in this way, so that a user running this test script can still see if the two\n+# files match. (It will appear as an ordinary success if they match, and a skip\n+# if not.)\n+if test_cmp highbit1/expect highbit1/actual\n+then\n+\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n+fi\n+test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n+\t# Only the prereq matters for this test.\n+\ttrue\n+'\n+\n+test_expect_success 'setup make another commit' '\n+\t# \"git log\" does not use Bloom filters for root commits - see how, in\n+\t# revision.c, rev_compare_tree() (the only code path that eventually calls\n+\t# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit\n+\t# has one parent. Therefore, make another commit so that we perform the tests on\n+\t# a non-root commit.\n+\ttest_commit -C highbit1 anotherc1 \"another$CENT\"\n+'\n+\n+test_expect_success 'version 1 changed-path used when version 1 requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\ttest_bloom_filters_used \"-- another$CENT\"\n+\t)\n+'\n+\n test_done\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483020","messageId":"abbef95ae8051401606fc181e9ded9e9fcf51872.1696969994.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 16/17] commit-graph: reuse existing Bloom filters where possible","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:34:08Z","receivedAt":"2023-10-10T20:34:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In 9e4df4da07 (commit-graph: new filter ver. that fixes murmur3,\n2023-08-01), a bug was described where it's possible for Git to produce\nnon-murmur3 hashes when the platform's \"char\" type is signed, and there\nare paths with characters whose highest bit is set (i.e. all characters\n>= 0x80).\n\nThat patch allows the caller to control which version of Bloom filters\nare read and written. However, even on platforms with a signed \"char\"\ntype, it is possible to reuse existing Bloom filters if and only if\nthere are no changed paths in any commit's first parent tree-diff whose\ncharacters have their highest bit set.\n\nWhen this is the case, we can reuse the existing filter without having\nto compute a new one. This is done by marking trees which are known to\nhave (or not have) any such paths. When a commit's root tree is verified\nto not have any such paths, we mark it as such and declare that the\ncommit's Bloom filter is reusable.\n\nNote that this heuristic only goes in one direction. If neither a commit\nnor its first parent have any paths in their trees with non-ASCII\ncharacters, then we know for certain that a path with non-ASCII\ncharacters will not appear in a tree-diff against that commit's first\nparent. The reverse isn't necessarily true: just because the tree-diff\ndoesn't contain any such paths does not imply that no such paths exist\nin either tree.\n\nSo we end up recomputing some Bloom filters that we don't strictly have\nto (i.e. their bits are the same no matter which version of murmur3 we\nuse). But culling these out is impossible, since we'd have to perform\nthe full tree-diff, which is the same effort as computing the Bloom\nfilter from scratch.\n\nBut because we can cache our results in each tree's flag bits, we can\noften avoid recomputing many filters, thereby reducing the time it takes\nto run\n\n $ git commit-graph write --changed-paths --reachable\n\nwhen upgrading from v1 to v2 Bloom filters.\n\nTo benchmark this, let's generate a commit-graph in linux.git with v1\nchanged-paths in generation order[^1]:\n\n $ git clone git@github.com:torvalds/linux.git\n $ cd linux\n $ git commit-graph write --reachable --changed-paths\n $ graph=\".git/objects/info/commit-graph\"\n $ mv $graph{,.bak}\n\nThen let's time how long it takes to go from v1 to v2 filters (with and\nwithout the upgrade path enabled), resetting the state of the\ncommit-graph each time:\n\n $ git config commitGraph.changedPathsVersion 2\n $ hyperfine -p 'cp -f $graph.bak $graph' -L v 0,1 \\\n 'GIT_TEST_UPGRADE_BLOOM_FILTERS={v} git.compile commit-graph write --reachable --changed-paths'\n\nOn linux.git (where there aren't any non-ASCII paths), the timings\nindicate that this patch represents a speed-up over recomputing all\nBloom filters from scratch:\n\n Benchmark 1: GIT_TEST_UPGRADE_BLOOM_FILTERS=0 git.compile commit-graph write --reachable --changed-paths\n Time (mean ± σ): 124.873 s ± 0.316 s [User: 124.081 s, System: 0.643 s]\n Range (min … max): 124.621 s … 125.227 s 3 runs\n\n Benchmark 2: GIT_TEST_UPGRADE_BLOOM_FILTERS=1 git.compile commit-graph write --reachable --changed-paths\n Time (mean ± σ): 79.271 s ± 0.163 s [User: 74.611 s, System: 4.521 s]\n Range (min … max): 79.112 s … 79.437 s 3 runs\n\n Summary\n 'GIT_TEST_UPGRADE_BLOOM_FILTERS=1 git.compile commit-graph write --reachable --changed-paths' ran\n 1.58 ± 0.01 times faster than 'GIT_TEST_UPGRADE_BLOOM_FILTERS=0 git.compile commit-graph write --reachable --changed-paths'\n\nOn git.git, we do have some non-ASCII paths, giving us a more modest\nimprovement from 4.163 seconds to 3.348 seconds, for a 1.24x speed-up.\nOn my machine, the stats for git.git are:\n\n - 8,285 Bloom filters computed from scratch\n - 10 Bloom filters generated as empty\n - 4 Bloom filters generated as truncated due to too many changed paths\n - 65,114 Bloom filters were reused when transitioning from v1 to v2.\n\n[^1]: Note that this is is important, since `--stdin-packs` or\n `--stdin-commits` orders commits in the commit-graph by their pack\n position (with `--stdin-packs`) or in the raw input (with\n `--stdin-commits`).\n\n Since we compute Bloom filters in the same order that commits appear\n in the graph, we must see a commit's (first) parent before we process\n the commit itself. This is only guaranteed to happen when sorting\n commits by their generation number.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 90 ++++++++++++++++++++++++++++++++++++++++++--\n bloom.h | 1 +\n commit-graph.c | 5 +++\n object.h | 1 +\n t/t4216-log-bloom.sh | 35 ++++++++++++++++-\n 5 files changed, 127 insertions(+), 5 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 739fa093ba..24dd874e46 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -7,6 +7,9 @@\n #include \"commit-graph.h\"\n #include \"commit.h\"\n #include \"commit-slab.h\"\n+#include \"tree.h\"\n+#include \"tree-walk.h\"\n+#include \"config.h\"\n \n define_commit_slab(bloom_filter_slab, struct bloom_filter);\n \n@@ -250,6 +253,73 @@ static void init_truncated_large_filter(struct bloom_filter *filter,\n \tfilter->version = version;\n }\n \n+#define VISITED (1u<<21)\n+#define HIGH_BITS (1u<<22)\n+\n+static int has_entries_with_high_bit(struct repository *r, struct tree *t)\n+{\n+\tif (parse_tree(t))\n+\t\treturn 1;\n+\n+\tif (!(t->object.flags & VISITED)) {\n+\t\tstruct tree_desc desc;\n+\t\tstruct name_entry entry;\n+\n+\t\tinit_tree_desc(&desc, t->buffer, t->size);\n+\t\twhile (tree_entry(&desc, &entry)) {\n+\t\t\tsize_t i;\n+\t\t\tfor (i = 0; i < entry.pathlen; i++) {\n+\t\t\t\tif (entry.path[i] & 0x80) {\n+\t\t\t\t\tt->object.flags |= HIGH_BITS;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t\tif (S_ISDIR(entry.mode)) {\n+\t\t\t\tstruct tree *sub = lookup_tree(r, &entry.oid);\n+\t\t\t\tif (sub && has_entries_with_high_bit(r, sub)) {\n+\t\t\t\t\tt->object.flags |= HIGH_BITS;\n+\t\t\t\t\tgoto done;\n+\t\t\t\t}\n+\t\t\t}\n+\n+\t\t}\n+\n+done:\n+\t\tt->object.flags |= VISITED;\n+\t}\n+\n+\treturn !!(t->object.flags & HIGH_BITS);\n+}\n+\n+static int commit_tree_has_high_bit_paths(struct repository *r,\n+\t\t\t\t\t struct commit *c)\n+{\n+\tstruct tree *t;\n+\tif (repo_parse_commit(r, c))\n+\t\treturn 1;\n+\tt = repo_get_commit_tree(r, c);\n+\tif (!t)\n+\t\treturn 1;\n+\treturn has_entries_with_high_bit(r, t);\n+}\n+\n+static struct bloom_filter *upgrade_filter(struct repository *r, struct commit *c,\n+\t\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t\t int hash_version)\n+{\n+\tstruct commit_list *p = c->parents;\n+\tif (commit_tree_has_high_bit_paths(r, c))\n+\t\treturn NULL;\n+\n+\tif (p && commit_tree_has_high_bit_paths(r, p->item))\n+\t\treturn NULL;\n+\n+\tfilter->version = hash_version;\n+\n+\treturn filter;\n+}\n+\n struct bloom_filter *get_bloom_filter(struct repository *r, struct commit *c)\n {\n \tstruct bloom_filter *filter;\n@@ -292,9 +362,23 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t filter, graph_pos);\n \t}\n \n-\tif ((filter->data && filter->len) &&\n-\t (!settings || settings->hash_version == filter->version))\n-\t\treturn filter;\n+\tif (filter->data && filter->len) {\n+\t\tstruct bloom_filter *upgrade;\n+\t\tif (!settings || settings->hash_version == filter->version)\n+\t\t\treturn filter;\n+\n+\t\t/* version mismatch, see if we can upgrade */\n+\t\tif (compute_if_not_present &&\n+\t\t git_env_bool(\"GIT_TEST_UPGRADE_BLOOM_FILTERS\", 1)) {\n+\t\t\tupgrade = upgrade_filter(r, c, filter,\n+\t\t\t\t\t\t settings->hash_version);\n+\t\t\tif (upgrade) {\n+\t\t\t\tif (computed)\n+\t\t\t\t\t*computed |= BLOOM_UPGRADED;\n+\t\t\t\treturn upgrade;\n+\t\t\t}\n+\t\t}\n+\t}\n \tif (!compute_if_not_present)\n \t\treturn NULL;\n \ndiff --git a/bloom.h b/bloom.h\nindex bfe389e29c..e3a9b68905 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -102,6 +102,7 @@ enum bloom_filter_computed {\n \tBLOOM_COMPUTED = (1 << 1),\n \tBLOOM_TRUNC_LARGE = (1 << 2),\n \tBLOOM_TRUNC_EMPTY = (1 << 3),\n+\tBLOOM_UPGRADED = (1 << 4),\n };\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e0fc62e110..571f38335a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1109,6 +1109,7 @@ struct write_commit_graph_context {\n \tint count_bloom_filter_not_computed;\n \tint count_bloom_filter_trunc_empty;\n \tint count_bloom_filter_trunc_large;\n+\tint count_bloom_filter_upgraded;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1716,6 +1717,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \t\t\t ctx->count_bloom_filter_trunc_empty);\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-large\",\n \t\t\t ctx->count_bloom_filter_trunc_large);\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-upgraded\",\n+\t\t\t ctx->count_bloom_filter_upgraded);\n }\n \n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n@@ -1757,6 +1760,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\tctx->count_bloom_filter_trunc_empty++;\n \t\t\tif (computed & BLOOM_TRUNC_LARGE)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n+\t\t} else if (computed & BLOOM_UPGRADED) {\n+\t\t\tctx->count_bloom_filter_upgraded++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n \t\t\tctx->count_bloom_filter_not_computed++;\n \t\tctx->total_bloom_filter_data_size += filter\ndiff --git a/object.h b/object.h\nindex db25714b4e..2e5e08725f 100644\n--- a/object.h\n+++ b/object.h\n@@ -75,6 +75,7 @@ void object_array_init(struct object_array *array);\n * commit-reach.c: 16-----19\n * sha1-name.c: 20\n * list-objects-filter.c: 21\n+ * bloom.c: 2122\n * builtin/fsck.c: 0--3\n * builtin/gc.c: 0\n * builtin/index-pack.c: 2021\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 8f8b5d4966..a321d8d713 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -221,6 +221,10 @@ test_filter_trunc_large () {\n \tgrep \"\\\"key\\\":\\\"filter-trunc-large\\\",\\\"value\\\":\\\"$1\\\"\" $2\n }\n \n+test_filter_upgraded () {\n+\tgrep \"\\\"key\\\":\\\"filter-upgraded\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n test_expect_success 'correctly report changes over limit' '\n \tgit init limits &&\n \t(\n@@ -629,10 +633,19 @@ test_expect_success 'when writing another commit graph, preserve existing versio\n test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n \tgit init doublewrite &&\n \ttest_commit -C doublewrite c \"$CENT\" &&\n+\n \tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n-\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 1 trace2.txt &&\n+\ttest_filter_upgraded 0 trace2.txt &&\n+\n \tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n-\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 1 trace2.txt &&\n+\ttest_filter_upgraded 0 trace2.txt &&\n+\n \t(\n \t\tcd doublewrite &&\n \t\techo \"c01f\" >expect &&\n@@ -641,4 +654,22 @@ test_expect_success 'when writing commit graph, do not reuse changed-path of ano\n \t)\n '\n \n+test_expect_success 'when writing commit graph, reuse changed-path of another version where possible' '\n+\tgit init upgrade &&\n+\n+\ttest_commit -C upgrade base no-high-bits &&\n+\n+\tgit -C upgrade config --add commitgraph.changedPathsVersion 1 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C upgrade commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 1 trace2.txt &&\n+\ttest_filter_upgraded 0 trace2.txt &&\n+\n+\tgit -C upgrade config --add commitgraph.changedPathsVersion 2 &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tgit -C upgrade commit-graph write --reachable --changed-paths &&\n+\ttest_filter_computed 0 trace2.txt &&\n+\ttest_filter_upgraded 1 trace2.txt\n+'\n+\n test_done\n-- \n2.42.0.342.g8bb3a896ee\n\n"},{"id":"483021","messageId":"ca362408d59f903f0c8094e5290fdd03e54ec195.1696969995.git.me@ttaylorr.com","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"[PATCH v3 17/17] bloom: introduce `deinit_bloom_filters()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:34:11Z","receivedAt":"2023-10-10T20:34:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"After we are done using Bloom filters, we do not currently clean up any\nmemory allocated by the commit slab used to store those filters in the\nfirst place.\n\nBesides the bloom_filter structures themselves, there is mostly nothing\nto free() in the first place, since in the read-only path all Bloom\nfilter's `data` members point to a memory mapped region in the\ncommit-graph file itself.\n\nBut when generating Bloom filters from scratch (or initializing\ntruncated filters) we allocate additional memory to store the filter's\ndata.\n\nKeep track of when we need to free() this additional chunk of memory by\nusing an extra pointer `to_free`. Most of the time this will be NULL\n(indicating that we are representing an existing Bloom filter stored in\na memory mapped region). When it is non-NULL, free it before discarding\nthe Bloom filters slab.\n\nSuggested-by: Jonathan Tan \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \nSigned-off-by: Taylor Blau \n---\n bloom.c | 16 +++++++++++++++-\n bloom.h | 3 +++\n commit-graph.c | 4 ++++\n 3 files changed, 22 insertions(+), 1 deletion(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 24dd874e46..ff131893cd 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -59,6 +59,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n \t\t\t\t\tsizeof(unsigned char) * start_index +\n \t\t\t\t\tBLOOMDATA_CHUNK_HEADER_SIZE);\n \tfilter->version = g->bloom_filter_settings->hash_version;\n+\tfilter->to_free = NULL;\n \n \treturn 1;\n }\n@@ -231,6 +232,18 @@ void init_bloom_filters(void)\n \tinit_bloom_filter_slab(&bloom_filters);\n }\n \n+static void free_one_bloom_filter(struct bloom_filter *filter)\n+{\n+\tif (!filter)\n+\t\treturn;\n+\tfree(filter->to_free);\n+}\n+\n+void deinit_bloom_filters(void)\n+{\n+\tdeep_clear_bloom_filter_slab(&bloom_filters, free_one_bloom_filter);\n+}\n+\n static int pathmap_cmp(const void *hashmap_cmp_fn_data UNUSED,\n \t\t const struct hashmap_entry *eptr,\n \t\t const struct hashmap_entry *entry_or_key,\n@@ -247,7 +260,7 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data UNUSED,\n static void init_truncated_large_filter(struct bloom_filter *filter,\n \t\t\t\t\tint version)\n {\n-\tfilter->data = xmalloc(1);\n+\tfilter->data = filter->to_free = xmalloc(1);\n \tfilter->data[0] = 0xFF;\n \tfilter->len = 1;\n \tfilter->version = version;\n@@ -449,6 +462,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tfilter->len = 1;\n \t\t}\n \t\tCALLOC_ARRAY(filter->data, filter->len);\n+\t\tfilter->to_free = filter->data;\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n \t\t\tstruct bloom_key key;\ndiff --git a/bloom.h b/bloom.h\nindex e3a9b68905..d20e64bfbb 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -56,6 +56,8 @@ struct bloom_filter {\n \tunsigned char *data;\n \tsize_t len;\n \tint version;\n+\n+\tvoid *to_free;\n };\n \n /*\n@@ -96,6 +98,7 @@ void add_key_to_filter(const struct bloom_key *key,\n \t\t const struct bloom_filter_settings *settings);\n \n void init_bloom_filters(void);\n+void deinit_bloom_filters(void);\n \n enum bloom_filter_computed {\n \tBLOOM_NOT_COMPUTED = (1 << 0),\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 571f38335a..7ccec429cb 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -787,6 +787,7 @@ static void close_commit_graph_one(struct commit_graph *g)\n void close_commit_graph(struct raw_object_store *o)\n {\n \tclose_commit_graph_one(o->commit_graph);\n+\tdeinit_bloom_filters();\n \to->commit_graph = NULL;\n }\n \n@@ -2588,6 +2589,9 @@ int write_commit_graph(struct object_directory *odb,\n \n \tres = write_commit_graph_file(ctx);\n \n+\tif (ctx->changed_paths)\n+\t\tdeinit_bloom_filters();\n+\n \tif (ctx->split)\n \t\tmark_commit_graphs(ctx);\n \n-- \n2.42.0.342.g8bb3a896ee\n"},{"id":"483022","messageId":"ZSW1asgEDWMSQkm+@nand.local","threadId":"60142","inReplyTo":"xmqqr0m3y3o2.fsf@gitster.g","subject":"Re: [PATCH 07/15] commit-graph: new filter ver. that fixes murmur3","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-10T20:34:50Z","receivedAt":"2023-10-10T20:35:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Oct 09, 2023 at 12:52:13PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > However, I am not entirely sure I agree with you that this is a \"new\"\n> > issue. At least in the sense that Git (on 'master') does not currently\n> > know how to deal with Bloom filters that have different settings across\n> > commit-graph layers.\n> >\n> > IOW, you could produce this problem today using the test you wrote in\n> > <20201015132147.GB24954@szeder.dev> using different values of the\n> > GIT_BLOOM_SETTINGS environment variables as a proxy for different values\n> > of the commitGraph.changedPathsVersion configuration variable introduced\n> > in this series.\n> >\n> > So I think that this series makes it easier to fall into that trap, but\n> > the trap itself is not new. I think a reasonable stopgap (which IIUC you\n> > have suggested earlier) is to prevent writing a new commit-graph layer\n> > with a different hash version than the previous layer.\n>\n> What we probably want more urgently than that stopgap is to perhaps\n> teach the code pretend as if commit-graph did not exist when we\n> detect multiple layers use different hash versions (or perhaps only\n> use the base layer and ignore the rest as an anti-pessimization), to\n> protect correctness first, no?\n\nVery good suggestion, thanks.\n\nThanks,\nTaylor\n"},{"id":"483332","messageId":"ZS5JkyBuFzW09LXH@tanuki","threadId":"60142","inReplyTo":"2ecc0a2d58432b149d73a3e2abfa948eb1f0aa0b.1696969994.git.me@ttaylorr.com","subject":"Re: [PATCH v3 03/17] commit-graph: ensure Bloom filters are read with consistent settings","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-17T08:45:07Z","receivedAt":"2023-10-17T08:45:17Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 10, 2023 at 04:33:26PM -0400, Taylor Blau wrote:\n> The changed-path Bloom filter mechanism is parameterized by a couple of\n> variables, notably the number of bits per hash (typically \"m\" in Bloom\n> filter literature) and the number of hashes themselves (typically \"k\").\n> \n> It is critically important that filters are read with the Bloom filter\n> settings that they were written with. Failing to do so would mean that\n> each query is liable to compute different fingerprints, meaning that the\n> filter itself could return a false negative. This goes against a basic\n> assumption of using Bloom filters (that they may return false positives,\n> but never false negatives) and can lead to incorrect results.\n> \n> We have some existing logic to carry forward existing Bloom filter\n> settings from one layer to the next. In `write_commit_graph()`, we have\n> something like:\n> \n> if (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\n> struct commit_graph *g = ctx->r->objects->commit_graph;\n> \n> /* We have changed-paths already. Keep them in the next graph */\n> if (g && g->chunk_bloom_data) {\n> ctx->changed_paths = 1;\n> ctx->bloom_settings = g->bloom_filter_settings;\n> }\n> }\n> \n> , which drags forward Bloom filter settings across adjacent layers.\n> \n> This doesn't quite address all cases, however, since it is possible for\n> intermediate layers to contain no Bloom filters at all. For example,\n> suppose we have two layers in a commit-graph chain, say, {G1, G2}. If G1\n> contains Bloom filters, but G2 doesn't, a new G3 (whose base graph is\n> G2) may be written with arbitrary Bloom filter settings, because we only\n> check the immediately adjacent layer's settings for compatibility.\n> \n> This behavior has existed since the introduction of changed-path Bloom\n> filters. But in practice, this is not such a big deal, since the only\n> way up until this point to modify the Bloom filter settings at write\n> time is with the undocumented environment variables:\n> \n> - GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\n> - GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\n> - GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\n> \n> (it is still possible to tweak MAX_CHANGED_PATHS between layers, but\n> this does not affect reads, so is allowed to differ across multiple\n> graph layers).\n> \n> But in future commits, we will introduce another parameter to change the\n> hash algorithm used to compute Bloom fingerprints itself. This will be\n> exposed via a configuration setting, making this foot-gun easier to use.\n> \n> To prevent this potential issue, validate that all layers of a split\n> commit-graph have compatible settings with the newest layer which\n> contains Bloom filters.\n> \n> Reported-by: SZEDER Gábor \n> Original-test-by: SZEDER Gábor \n> Signed-off-by: Taylor Blau \n> ---\n> commit-graph.c | 25 +++++++++++++++++\n> t/t4216-log-bloom.sh | 64 ++++++++++++++++++++++++++++++++++++++++++++\n> 2 files changed, 89 insertions(+)\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index 1a56efcf69..ae0902f7f4 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -498,6 +498,30 @@ static int validate_mixed_generation_chain(struct commit_graph *g)\n> \treturn 0;\n> }\n> \n> +static void validate_mixed_bloom_settings(struct commit_graph *g)\n> +{\n> +\tstruct bloom_filter_settings *settings = NULL;\n> +\tfor (; g; g = g->base_graph) {\n> +\t\tif (!g->bloom_filter_settings)\n> +\t\t\tcontinue;\n> +\t\tif (!settings) {\n> +\t\t\tsettings = g->bloom_filter_settings;\n> +\t\t\tcontinue;\n> +\t\t}\n> +\n> +\t\tif (g->bloom_filter_settings->bits_per_entry != settings->bits_per_entry ||\n> +\t\t g->bloom_filter_settings->num_hashes != settings->num_hashes) {\n> +\t\t\tg->chunk_bloom_indexes = NULL;\n> +\t\t\tg->chunk_bloom_data = NULL;\n> +\t\t\tFREE_AND_NULL(g->bloom_filter_settings);\n> +\n> +\t\t\twarning(_(\"disabling Bloom filters for commit-graph \"\n> +\t\t\t\t \"layer '%s' due to incompatible settings\"),\n> +\t\t\t\toid_to_hex(&g->oid));\n> +\t\t}\n> +\t}\n> +}\n> +\n> static int add_graph_to_chain(struct commit_graph *g,\n> \t\t\t struct commit_graph *chain,\n> \t\t\t struct object_id *oids,\n> @@ -614,6 +638,7 @@ struct commit_graph *load_commit_graph_chain_fd_st(struct repository *r,\n> \t}\n> \n> \tvalidate_mixed_generation_chain(graph_chain);\n> +\tvalidate_mixed_bloom_settings(graph_chain);\n> \n> \tfree(oids);\n> \tfclose(fp);\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index 322640feeb..f49a8f2fbf 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -420,4 +420,68 @@ test_expect_success 'Bloom generation backfills empty commits' '\n> \t)\n> '\n> \n> +graph=.git/objects/info/commit-graph\n> +graphdir=.git/objects/info/commit-graphs\n> +chain=$graphdir/commit-graph-chain\n> +\n> +test_expect_success 'setup for mixed Bloom setting tests' '\n> +\trepo=mixed-bloom-settings &&\n> +\n> +\tgit init $repo &&\n> +\tfor i in one two three\n> +\tdo\n> +\t\ttest_commit -C $repo $i file || return 1\n> +\tdone\n> +'\n> +\n> +test_expect_success 'split' '\n> +\t# Compute Bloom filters with \"unusual\" settings.\n> +\tgit -C $repo rev-parse one >in &&\n> +\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=3 git -C $repo commit-graph write \\\n> +\t\t--stdin-commits --changed-paths --split +\tlayer=$(head -n 1 $repo/$chain) &&\n> +\n> +\t# A commit-graph layer without Bloom filters \"hides\" the layers\n> +\t# below ...\n> +\tgit -C $repo rev-parse two >in &&\n> +\tgit -C $repo commit-graph write --stdin-commits --no-changed-paths \\\n> +\t\t--split=no-merge +\n> +\t# Another commit-graph layer that has Bloom filters, but with\n> +\t# standard settings, and is thus incompatible with the base\n> +\t# layer written above.\n> +\tgit -C $repo rev-parse HEAD >in &&\n> +\tgit -C $repo commit-graph write --stdin-commits --changed-paths \\\n> +\t\t--split=no-merge +\n> +\ttest_line_count = 3 $repo/$chain &&\n> +\n> +\t# Ensure that incompatible Bloom filters are ignored.\n> +\tgit -C $repo -c core.commitGraph=false log --oneline --no-decorate -- file \\\n> +\t\t>expect 2>err &&\n> +\tgit -C $repo log --oneline --no-decorate -- file >actual 2>err &&\n> +\ttest_cmp expect actual &&\n> +\tgrep \"disabling Bloom filters for commit-graph layer .$layer.\" err\n> +'\n\nUp to this point everything looks sensible to me.\n\n> +test_expect_success 'merge graph layers with incompatible Bloom settings' '\n> +\t# Ensure that incompatible Bloom filters are ignored when\n> +\t# generating new layers.\n> +\tgit -C $repo commit-graph write --reachable --changed-paths 2>err &&\n> +\tgrep \"disabling Bloom filters for commit-graph layer .$layer.\" err &&\n> +\n> +\ttest_path_is_file $repo/$graph &&\n> +\ttest_dir_is_empty $repo/$graphdir &&\n> +\n> +\t# ...and merging existing ones.\n> +\tgit -C $repo -c core.commitGraph=false log --oneline --no-decorate -- file \\\n> +\t\t>expect 2>err &&\n> +\tGIT_TRACE2_PERF=\"$(pwd)/trace.perf\" \\\n> +\t\tgit -C $repo log --oneline --no-decorate -- file >actual 2>err &&\n\nBut this test is a bit confusing to me, to be honest, also because the\ncomment for the second block here reads funny. We don't really merge\nanything, do we? We only generate logs and compare that the log with and\nwithout the resulting merged commit graph is the same. The actual logic\nhappened before.\n\n> +\ttest_cmp expect actual && cat err &&\n\nThe `cat err` looks like a leftover from debugging.\n\n> +\tgrep \"statistics:{\\\"filter_not_present\\\":0\" trace.perf &&\n\nAlso, why should the filter not be present here? If we merge the\ncommit-graphs with `--changed-paths` I'd have expected that we either\ncarry over bloom filters from preexisting commit graphs if compatible,\nor otherwise generate them if they are either incompatible or don't\nexist.\n\nI feel like I'm missing something obvious, so this may be me just\nmissing the bigger picture.\n\n> +\t! grep \"disabling Bloom filters\" err\n\nCan we make this assertion stricter and verify that `err` is empty? I\nalways think that `! grep` is quite a fragile pattern as it is quite\nprone to becoming stale, e.g. when the error message itself would\nchange.\n\nPatrick\n\n> +'\n> +\n> test_done\n> -- \n> 2.42.0.342.g8bb3a896ee\n> \n"},{"id":"483333","messageId":"ZS5JmQib3onjirC6@tanuki","threadId":"60142","inReplyTo":"cba766f224b0d2b4fd952b11bef8068c07dfcf88.1696969994.git.me@ttaylorr.com","subject":"Re: [PATCH v3 08/17] t4216: test changed path filters with high bit paths","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-17T08:45:13Z","receivedAt":"2023-10-17T08:45:19Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 10, 2023 at 04:33:42PM -0400, Taylor Blau wrote:\n> From: Jonathan Tan \n> \n> Subsequent commits will teach Git another version of changed path\n> filter that has different behavior with paths that contain at least\n> one character with its high bit set, so test the existing behavior as\n> a baseline.\n> \n> Signed-off-by: Jonathan Tan \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n\nNit: the signoffs are still funny here.\n\n> ---\n> t/t4216-log-bloom.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++\n> 1 file changed, 52 insertions(+)\n> \n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index f49a8f2fbf..da67c40134 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -484,4 +484,56 @@ test_expect_success 'merge graph layers with incompatible Bloom settings' '\n> \t! grep \"disabling Bloom filters\" err\n> '\n> \n> +get_first_changed_path_filter () {\n> +\ttest-tool read-graph bloom-filters >filters.dat &&\n> +\thead -n 1 filters.dat\n> +}\n> +\n> +# chosen to be the same under all Unicode normalization forms\n> +CENT=$(printf \"\\302\\242\")\n> +\n> +test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n> +\tgit init highbit1 &&\n> +\ttest_commit -C highbit1 c1 \"$CENT\" &&\n> +\tgit -C highbit1 commit-graph write --reachable --changed-paths\n> +'\n> +\n> +test_expect_success 'setup check value of version 1 changed-path' '\n> +\t(\n> +\t\tcd highbit1 &&\n> +\t\techo \"52a9\" >expect &&\n> +\t\tget_first_changed_path_filter >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +# expect will not match actual if char is unsigned by default. Write the test\n> +# in this way, so that a user running this test script can still see if the two\n> +# files match. (It will appear as an ordinary success if they match, and a skip\n> +# if not.)\n> +if test_cmp highbit1/expect highbit1/actual\n> +then\n> +\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n> +fi\n> +test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n> +\t# Only the prereq matters for this test.\n> +\ttrue\n> +'\n\nDoesn't this mean that the preceding test where we `test_cmp expect\nactual` can fail on some platforms depending on the signedness of\n`char`?\n\nPatrick\n\n> +test_expect_success 'setup make another commit' '\n> +\t# \"git log\" does not use Bloom filters for root commits - see how, in\n> +\t# revision.c, rev_compare_tree() (the only code path that eventually calls\n> +\t# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit\n> +\t# has one parent. Therefore, make another commit so that we perform the tests on\n> +\t# a non-root commit.\n> +\ttest_commit -C highbit1 anotherc1 \"another$CENT\"\n> +'\n> +\n> +test_expect_success 'version 1 changed-path used when version 1 requested' '\n> +\t(\n> +\t\tcd highbit1 &&\n> +\t\ttest_bloom_filters_used \"-- another$CENT\"\n> +\t)\n> +'\n> +\n> test_done\n> -- \n> 2.42.0.342.g8bb3a896ee\n> \n"},{"id":"483334","messageId":"ZS5JnkybxvetTUzu@tanuki","threadId":"60142","inReplyTo":"94552abf455c6d341a0811333ae4edb4a8cea259.1696969994.git.me@ttaylorr.com","subject":"Re: [PATCH v3 05/17] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-17T08:45:18Z","receivedAt":"2023-10-17T08:45:25Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 10, 2023 at 04:33:33PM -0400, Taylor Blau wrote:\n> Prepare for the 'read-graph' test helper to perform other tasks besides\n> dumping high-level information about the commit-graph by extracting its\n> main routine into a separate function.\n> \n> Signed-off-by: Taylor Blau \n> Signed-off-by: Jonathan Tan \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n\nNit: your signoff is duplicated here. This is also still the case for\nsome of the other commits.\n\nPatrick\n\n> ---\n> t/helper/test-read-graph.c | 31 ++++++++++++++++++-------------\n> 1 file changed, 18 insertions(+), 13 deletions(-)\n> \n> diff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\n> index 8c7a83f578..3375392f6c 100644\n> --- a/t/helper/test-read-graph.c\n> +++ b/t/helper/test-read-graph.c\n> @@ -5,20 +5,8 @@\n> #include \"bloom.h\"\n> #include \"setup.h\"\n> \n> -int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n> +static void dump_graph_info(struct commit_graph *graph)\n> {\n> -\tstruct commit_graph *graph = NULL;\n> -\tstruct object_directory *odb;\n> -\n> -\tsetup_git_directory();\n> -\todb = the_repository->objects->odb;\n> -\n> -\tprepare_repo_settings(the_repository);\n> -\n> -\tgraph = read_commit_graph_one(the_repository, odb);\n> -\tif (!graph)\n> -\t\treturn 1;\n> -\n> \tprintf(\"header: %08x %d %d %d %d\\n\",\n> \t\tntohl(*(uint32_t*)graph->data),\n> \t\t*(unsigned char*)(graph->data + 4),\n> @@ -57,6 +45,23 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n> \tif (graph->topo_levels)\n> \t\tprintf(\" topo_levels\");\n> \tprintf(\"\\n\");\n> +}\n> +\n> +int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n> +{\n> +\tstruct commit_graph *graph = NULL;\n> +\tstruct object_directory *odb;\n> +\n> +\tsetup_git_directory();\n> +\todb = the_repository->objects->odb;\n> +\n> +\tprepare_repo_settings(the_repository);\n> +\n> +\tgraph = read_commit_graph_one(the_repository, odb);\n> +\tif (!graph)\n> +\t\treturn 1;\n> +\n> +\tdump_graph_info(graph);\n> \n> \tUNLEAK(graph);\n> \n> -- \n> 2.42.0.342.g8bb3a896ee\n> \n"},{"id":"483335","messageId":"ZS5JpJCw-MY256xo@tanuki","threadId":"60142","inReplyTo":"61d44519a5ffaf2c040198cf8d80d05a09de5de5.1696969994.git.me@ttaylorr.com","subject":"Re: [PATCH v3 10/17] commit-graph: new filter ver. that fixes murmur3","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-17T08:45:24Z","receivedAt":"2023-10-17T08:45:32Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 10, 2023 at 04:33:49PM -0400, Taylor Blau wrote:\n> From: Jonathan Tan \n> \n> The murmur3 implementation in bloom.c has a bug when converting series\n> of 4 bytes into network-order integers when char is signed (which is\n> controllable by a compiler option, and the default signedness of char is\n> platform-specific). When a string contains characters with the high bit\n> set, this bug causes results that, although internally consistent within\n> Git, does not accord with other implementations of murmur3 (thus,\n> the changed path filters wouldn't be readable by other off-the-shelf\n> implementatios of murmur3) and even with Git binaries that were compiled\n> with different signedness of char. This bug affects both how Git writes\n> changed path filters to disk and how Git interprets changed path filters\n> on disk.\n> \n> Therefore, introduce a new version (2) of changed path filters that\n> corrects this problem. The existing version (1) is still supported and\n> is still the default, but users should migrate away from it as soon\n> as possible.\n> \n> Because this bug only manifests with characters that have the high bit\n> set, it may be possible that some (or all) commits in a given repo would\n> have the same changed path filter both before and after this fix is\n> applied. However, in order to determine whether this is the case, the\n> changed paths would first have to be computed, at which point it is not\n> much more expensive to just compute a new changed path filter.\n> \n> So this patch does not include any mechanism to \"salvage\" changed path\n> filters from repositories. There is also no \"mixed\" mode - for each\n> invocation of Git, reading and writing changed path filters are done\n> with the same version number; this version number may be explicitly\n> stated (typically if the user knows which version they need) or\n> automatically determined from the version of the existing changed path\n> filters in the repository.\n> \n> There is a change in write_commit_graph(). graph_read_bloom_data()\n> makes it possible for chunk_bloom_data to be non-NULL but\n> bloom_filter_settings to be NULL, which causes a segfault later on. I\n> produced such a segfault while developing this patch, but couldn't find\n> a way to reproduce it neither after this complete patch (or before),\n> but in any case it seemed like a good thing to include that might help\n> future patch authors.\n> \n> The value in t0095 was obtained from another murmur3 implementation\n> using the following Go source code:\n> \n> package main\n> \n> import \"fmt\"\n> import \"github.com/spaolacci/murmur3\"\n> \n> func main() {\n> fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n> fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n> }\n> \n> Signed-off-by: Jonathan Tan \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/config/commitgraph.txt | 5 +-\n> bloom.c | 69 +++++++++++++++++-\n> bloom.h | 8 +-\n> commit-graph.c | 32 ++++++--\n> t/helper/test-bloom.c | 9 ++-\n> t/t0095-bloom.sh | 8 ++\n> t/t4216-log-bloom.sh | 105 +++++++++++++++++++++++++++\n> 7 files changed, 223 insertions(+), 13 deletions(-)\n> \n> diff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\n> index 2dc9170622..acc74a2f27 100644\n> --- a/Documentation/config/commitgraph.txt\n> +++ b/Documentation/config/commitgraph.txt\n> @@ -15,7 +15,7 @@ commitGraph.readChangedPaths::\n> \n> commitGraph.changedPathsVersion::\n> \tSpecifies the version of the changed-path Bloom filters that Git will read and\n> -\twrite. May be -1, 0 or 1.\n> +\twrite. May be -1, 0, 1, or 2.\n> +\n> Defaults to -1.\n> +\n> @@ -28,4 +28,7 @@ filters when instructed to write.\n> If 1, Git will only read version 1 Bloom filters, and will write version 1\n> Bloom filters.\n> +\n> +If 2, Git will only read version 2 Bloom filters, and will write version 2\n> +Bloom filters.\n> ++\n> See linkgit:git-commit-graph[1] for more information.\n> diff --git a/bloom.c b/bloom.c\n> index 3e78cfe79d..ebef5cfd2f 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -66,7 +66,64 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n> * Not considered to be cryptographically secure.\n> * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n> */\n> -uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n> +uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n> +{\n> +\tconst uint32_t c1 = 0xcc9e2d51;\n> +\tconst uint32_t c2 = 0x1b873593;\n> +\tconst uint32_t r1 = 15;\n> +\tconst uint32_t r2 = 13;\n> +\tconst uint32_t m = 5;\n> +\tconst uint32_t n = 0xe6546b64;\n> +\tint i;\n> +\tuint32_t k1 = 0;\n> +\tconst char *tail;\n> +\n> +\tint len4 = len / sizeof(uint32_t);\n> +\n> +\tuint32_t k;\n> +\tfor (i = 0; i < len4; i++) {\n> +\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n> +\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n> +\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n> +\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n> +\t\tk = byte1 | byte2 | byte3 | byte4;\n> +\t\tk *= c1;\n> +\t\tk = rotate_left(k, r1);\n> +\t\tk *= c2;\n> +\n> +\t\tseed ^= k;\n> +\t\tseed = rotate_left(seed, r2) * m + n;\n> +\t}\n> +\n> +\ttail = (data + len4 * sizeof(uint32_t));\n> +\n> +\tswitch (len & (sizeof(uint32_t) - 1)) {\n> +\tcase 3:\n> +\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n> +\t\t/*-fallthrough*/\n> +\tcase 2:\n> +\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n> +\t\t/*-fallthrough*/\n> +\tcase 1:\n> +\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n> +\t\tk1 *= c1;\n> +\t\tk1 = rotate_left(k1, r1);\n> +\t\tk1 *= c2;\n> +\t\tseed ^= k1;\n> +\t\tbreak;\n> +\t}\n> +\n> +\tseed ^= (uint32_t)len;\n> +\tseed ^= (seed >> 16);\n> +\tseed *= 0x85ebca6b;\n> +\tseed ^= (seed >> 13);\n> +\tseed *= 0xc2b2ae35;\n> +\tseed ^= (seed >> 16);\n> +\n> +\treturn seed;\n> +}\n> +\n> +static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n> {\n> \tconst uint32_t c1 = 0xcc9e2d51;\n> \tconst uint32_t c2 = 0x1b873593;\n> @@ -131,8 +188,14 @@ void fill_bloom_key(const char *data,\n> \tint i;\n> \tconst uint32_t seed0 = 0x293ae76f;\n> \tconst uint32_t seed1 = 0x7e646e2c;\n> -\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n> -\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n> +\tuint32_t hash0, hash1;\n> +\tif (settings->hash_version == 2) {\n> +\t\thash0 = murmur3_seeded_v2(seed0, data, len);\n> +\t\thash1 = murmur3_seeded_v2(seed1, data, len);\n> +\t} else {\n> +\t\thash0 = murmur3_seeded_v1(seed0, data, len);\n> +\t\thash1 = murmur3_seeded_v1(seed1, data, len);\n> +\t}\n> \n> \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n> \tfor (i = 0; i < settings->num_hashes; i++)\n> diff --git a/bloom.h b/bloom.h\n> index 1e4f612d2c..138d57a86b 100644\n> --- a/bloom.h\n> +++ b/bloom.h\n> @@ -8,9 +8,11 @@ struct commit_graph;\n> struct bloom_filter_settings {\n> \t/*\n> \t * The version of the hashing technique being used.\n> -\t * We currently only support version = 1 which is\n> +\t * The newest version is 2, which is\n> \t * the seeded murmur3 hashing technique implemented\n> -\t * in bloom.c.\n> +\t * in bloom.c. Bloom filters of version 1 were created\n> +\t * with prior versions of Git, which had a bug in the\n> +\t * implementation of the hash function.\n> \t */\n> \tuint32_t hash_version;\n> \n> @@ -80,7 +82,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n> * Not considered to be cryptographically secure.\n> * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n> */\n> -uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n> +uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n> \n> void fill_bloom_key(const char *data,\n> \t\t size_t len,\n> diff --git a/commit-graph.c b/commit-graph.c\n> index ea677c87fb..db623afd09 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -314,17 +314,26 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n> \treturn 0;\n> }\n> \n> +struct graph_read_bloom_data_context {\n> +\tstruct commit_graph *g;\n> +\tint *commit_graph_changed_paths_version;\n> +};\n> +\n> static int graph_read_bloom_data(const unsigned char *chunk_start,\n> \t\t\t\t size_t chunk_size, void *data)\n> {\n> -\tstruct commit_graph *g = data;\n> +\tstruct graph_read_bloom_data_context *c = data;\n> +\tstruct commit_graph *g = c->g;\n> \tuint32_t hash_version;\n> -\tg->chunk_bloom_data = chunk_start;\n> \thash_version = get_be32(chunk_start);\n> \n> -\tif (hash_version != 1)\n> +\tif (*c->commit_graph_changed_paths_version == -1) {\n> +\t\t*c->commit_graph_changed_paths_version = hash_version;\n> +\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n> \t\treturn 0;\n> +\t}\n\nIn case we have `c->commit_graph_changed_paths_version == -1` we lose\nthe check that the hash version is something that we know and support,\ndon't we? And while we do start to handle `-1` in the writing path, I\nthink we don't in the reading path unless I missed something.\n\n> +\tg->chunk_bloom_data = chunk_start;\n> \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n> \tg->bloom_filter_settings->hash_version = hash_version;\n> \tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n> @@ -412,10 +421,14 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n> \t}\n> \n> \tif (s->commit_graph_changed_paths_version) {\n> +\t\tstruct graph_read_bloom_data_context context = {\n> +\t\t\t.g = graph,\n> +\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n> +\t\t};\n> \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n> \t\t\t &graph->chunk_bloom_indexes);\n> \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n> -\t\t\t graph_read_bloom_data, graph);\n> +\t\t\t graph_read_bloom_data, &context);\n> \t}\n> \n> \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n> @@ -2441,6 +2454,13 @@ int write_commit_graph(struct object_directory *odb,\n> \t}\n> \tif (!commit_graph_compatible(r))\n> \t\treturn 0;\n> +\tif (r->settings.commit_graph_changed_paths_version < -1\n> +\t || r->settings.commit_graph_changed_paths_version > 2) {\n> +\t\twarning(_(\"attempting to write a commit-graph, but \"\n> +\t\t\t \"'commitgraph.changedPathsVersion' (%d) is not supported\"),\n> +\t\t\tr->settings.commit_graph_changed_paths_version);\n> +\t\treturn 0;\n> +\t}\n> \n> \tCALLOC_ARRAY(ctx, 1);\n> \tctx->r = r;\n> @@ -2453,6 +2473,8 @@ int write_commit_graph(struct object_directory *odb,\n> \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n> \tctx->num_generation_data_overflows = 0;\n> \n> +\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n> +\t\t? 2 : 1;\n> \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n> \t\t\t\t\t\t bloom_settings.bits_per_entry);\n> \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n> @@ -2482,7 +2504,7 @@ int write_commit_graph(struct object_directory *odb,\n> \t\tg = ctx->r->objects->commit_graph;\n> \n> \t\t/* We have changed-paths already. Keep them in the next graph */\n> -\t\tif (g && g->chunk_bloom_data) {\n> +\t\tif (g && g->bloom_filter_settings) {\n> \t\t\tctx->changed_paths = 1;\n> \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n> \t\t}\n> diff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\n> index aabe31d724..3cbc0a5b50 100644\n> --- a/t/helper/test-bloom.c\n> +++ b/t/helper/test-bloom.c\n> @@ -50,6 +50,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n> \n> static const char *bloom_usage = \"\\n\"\n> \" test-tool bloom get_murmur3 \\n\"\n> +\" test-tool bloom get_murmur3_seven_highbit\\n\"\n> \" test-tool bloom generate_filter [...]\\n\"\n> \" test-tool bloom get_filter_for_commit \\n\";\n> \n> @@ -64,7 +65,13 @@ int cmd__bloom(int argc, const char **argv)\n> \t\tuint32_t hashed;\n> \t\tif (argc < 3)\n> \t\t\tusage(bloom_usage);\n> -\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n> +\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n> +\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n> +\t}\n> +\n> +\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n> +\t\tuint32_t hashed;\n> +\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n> \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n> \t}\n> \n> diff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\n> index b567383eb8..c8d84ab606 100755\n> --- a/t/t0095-bloom.sh\n> +++ b/t/t0095-bloom.sh\n> @@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n> \ttest_cmp expect actual\n> '\n> \n> +test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n> +\tcat >expect <<-\\EOF &&\n> +\tMurmur3 Hash with seed=0:0xa183ccfd\n> +\tEOF\n> +\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n> +\ttest_cmp expect actual\n> +'\n> +\n> test_expect_success 'compute bloom key for empty string' '\n> \tcat >expect <<-\\EOF &&\n> \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index da67c40134..8f8b5d4966 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -536,4 +536,109 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n> \t)\n> '\n> \n> +test_expect_success 'version 1 changed-path not used when version 2 requested' '\n> +\t(\n> +\t\tcd highbit1 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n> +\t\ttest_bloom_filters_not_used \"-- another$CENT\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'version 1 changed-path used when autodetect requested' '\n> +\t(\n> +\t\tcd highbit1 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n> +\t\ttest_bloom_filters_used \"-- another$CENT\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n> +\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n> +\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n> +\t(\n> +\t\tcd highbit1 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n> +\t\techo \"options: bloom(1,10,7) read_generation_data\" >expect &&\n> +\t\ttest-tool read-graph >full &&\n> +\t\tgrep options full >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n> +\tgit init highbit2 &&\n> +\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n> +\ttest_commit -C highbit2 c2 \"$CENT\" &&\n> +\tgit -C highbit2 commit-graph write --reachable --changed-paths\n> +'\n> +\n> +test_expect_success 'check value of version 2 changed-path' '\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\techo \"c01f\" >expect &&\n> +\t\tget_first_changed_path_filter >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'setup make another commit' '\n> +\t# \"git log\" does not use Bloom filters for root commits - see how, in\n> +\t# revision.c, rev_compare_tree() (the only code path that eventually calls\n> +\t# get_bloom_filter()) is only called by try_to_simplify_commit() when the commit\n> +\t# has one parent. Therefore, make another commit so that we perform the tests on\n> +\t# a non-root commit.\n> +\ttest_commit -C highbit2 anotherc2 \"another$CENT\"\n> +'\n> +\n> +test_expect_success 'version 2 changed-path used when version 2 requested' '\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\ttest_bloom_filters_used \"-- another$CENT\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'version 2 changed-path not used when version 1 requested' '\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n> +\t\ttest_bloom_filters_not_used \"-- another$CENT\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'version 2 changed-path used when autodetect requested' '\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n> +\t\ttest_bloom_filters_used \"-- another$CENT\"\n> +\t)\n> +'\n> +\n> +test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n> +\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n> +\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n> +\t(\n> +\t\tcd highbit2 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n> +\t\techo \"options: bloom(2,10,7) read_generation_data\" >expect &&\n> +\t\ttest-tool read-graph >full &&\n> +\t\tgrep options full >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n> +test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n> +\tgit init doublewrite &&\n> +\ttest_commit -C doublewrite c \"$CENT\" &&\n> +\tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n> +\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n> +\tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n> +\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n> +\t(\n> +\t\tcd doublewrite &&\n> +\t\techo \"c01f\" >expect &&\n> +\t\tget_first_changed_path_filter >actual &&\n> +\t\ttest_cmp expect actual\n> +\t)\n> +'\n> +\n\nWith the supposedly missing check in mind, should we also add tests for\ncurrently unknown versions like 3 or -2?\n\nPatrick\n\n> test_done\n> -- \n> 2.42.0.342.g8bb3a896ee\n> \n"},{"id":"483336","messageId":"ZS5JqdUmUWZ8OMoo@tanuki","threadId":"60142","inReplyTo":"09d8669c3a074e7a2ace9d650a345244b2362f7e.1696969994.git.me@ttaylorr.com","subject":"Re: [PATCH v3 13/17] commit-graph.c: unconditionally load Bloom filters","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-17T08:45:29Z","receivedAt":"2023-10-17T08:45:35Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 10, 2023 at 04:33:59PM -0400, Taylor Blau wrote:\n> In 9e4df4da07 (commit-graph: new filter ver. that fixes murmur3,\n\nNit: It's a bit funny to read this reference to a commit ID when the\ncommit in question is part of the same series. Isn't it likely to grow\nstale?\n\n> 2023-08-01), we began ignoring the Bloom data (\"BDAT\") chunk for\n> commit-graphs whose Bloom filters were computed using a hash version\n> incompatible with the value of `commitGraph.changedPathVersion`.\n> \n> Now that the Bloom API has been hardened to discard these incompatible\n> filters (with the exception of low-level APIs), we can safely load these\n> Bloom filters unconditionally.\n> \n> We no longer want to return early from `graph_read_bloom_data()`, and\n> similarly do not want to set the bloom_settings' `hash_version` field as\n> a side-effect. The latter is because we want to wait until we know which\n> Bloom settings we're using (either the defaults, from the GIT_TEST\n> variables, or from the previous commit-graph layer) before deciding what\n> hash_version to use.\n> \n> If we detect an existing BDAT chunk, we'll infer the rest of the\n> settings (e.g., number of hashes, bits per entry, and maximum number of\n> changed paths) from the earlier graph layer. The hash_version will be\n> inferred from the previous layer as well, unless one has already been\n> specified via configuration.\n> \n> Once all of that is done, we normalize the value of the hash_version to\n> either \"1\" or \"2\".\n> \n> Signed-off-by: Taylor Blau \n> ---\n> commit-graph.c | 19 ++++++++++---------\n> 1 file changed, 10 insertions(+), 9 deletions(-)\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index db623afd09..fa3b58e762 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -327,12 +327,6 @@ static int graph_read_bloom_data(const unsigned char *chunk_start,\n> \tuint32_t hash_version;\n> \thash_version = get_be32(chunk_start);\n> \n> -\tif (*c->commit_graph_changed_paths_version == -1) {\n> -\t\t*c->commit_graph_changed_paths_version = hash_version;\n> -\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n> -\t\treturn 0;\n> -\t}\n> -\n> \tg->chunk_bloom_data = chunk_start;\n> \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n> \tg->bloom_filter_settings->hash_version = hash_version;\n> @@ -2473,8 +2467,7 @@ int write_commit_graph(struct object_directory *odb,\n> \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n> \tctx->num_generation_data_overflows = 0;\n> \n> -\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n> -\t\t? 2 : 1;\n> +\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version;\n> \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n> \t\t\t\t\t\t bloom_settings.bits_per_entry);\n> \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n> @@ -2506,10 +2499,18 @@ int write_commit_graph(struct object_directory *odb,\n> \t\t/* We have changed-paths already. Keep them in the next graph */\n> \t\tif (g && g->bloom_filter_settings) {\n> \t\t\tctx->changed_paths = 1;\n> -\t\t\tctx->bloom_settings = g->bloom_filter_settings;\n> +\n> +\t\t\t/* don't propagate the hash_version unless unspecified */\n> +\t\t\tif (bloom_settings.hash_version == -1)\n> +\t\t\t\tbloom_settings.hash_version = g->bloom_filter_settings->hash_version;\n> +\t\t\tbloom_settings.bits_per_entry = g->bloom_filter_settings->bits_per_entry;\n> +\t\t\tbloom_settings.num_hashes = g->bloom_filter_settings->num_hashes;\n> +\t\t\tbloom_settings.max_changed_paths = g->bloom_filter_settings->max_changed_paths;\n> \t\t}\n> \t}\n> \n> +\tbloom_settings.hash_version = bloom_settings.hash_version == 2 ? 2 : 1;\n> +\n\nWhat if there is a future version of Git that writes Bloom filters with\nhash version 3? Should we really normalize that to `1`?\n\nPatrick\n\n> \tif (ctx->split) {\n> \t\tstruct commit_graph *g = ctx->r->objects->commit_graph;\n> \n> -- \n> 2.42.0.342.g8bb3a896ee\n> \n"},{"id":"483337","messageId":"ZS5JsDKk8RioQfOA@tanuki","threadId":"60142","inReplyTo":"cover.1696969994.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/17] bloom: changed-path Bloom filters v2 (& sundries)","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-10-17T08:45:36Z","receivedAt":"2023-10-17T08:45:45Z","isPatch":true,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Tue, Oct 10, 2023 at 04:33:17PM -0400, Taylor Blau wrote:\n> (Rebased onto the tip of 'master', which is 3a06386e31 (The fifteenth\n> batch, 2023-10-04), at the time of writing).\n> \n> This series is a reroll of the combined efforts of [1] and [2] to\n> introduce the v2 changed-path Bloom filters, which fixes a bug in our\n> existing implementation of murmur3 paths with non-ASCII characters (when\n> the \"char\" type is signed).\n> \n> In large part, this is the same as the previous round. But this round\n> includes some extra bits that address issues pointed out by SZEDER\n> Gábor, which are:\n> \n> - not reading Bloom filters for root commits\n> - corrupting Bloom filter reads by tweaking the filter settings\n> between layers.\n> \n> These issues were discussed in (among other places) [3], and [4],\n> respectively.\n> \n> Thanks to Jonathan, Peff, and SZEDER who have helped a great deal in\n> assembling these patches. As usual, a range-diff is included below.\n> Thanks in advance for your\n> review!\n\nAs this patch series has been sitting around without reviews for a week\nI've tried my best to give it a go. Note though that this area is mostly\noutside of my own comfort zone, so some of the questions and suggestions\nmight ultimately not apply.\n\nPatrick\n\n> [1]: https://lore.kernel.org/git/cover.1684790529.git.jonathantanmy@google.com/\n> [2]: https://lore.kernel.org/git/cover.1691426160.git.me@ttaylorr.com/\n> [3]: https://public-inbox.org/git/20201015132147.GB24954@szeder.dev/\n> [4]: https://lore.kernel.org/git/20230830200218.GA5147@szeder.dev/\n> \n> Jonathan Tan (4):\n> gitformat-commit-graph: describe version 2 of BDAT\n> t4216: test changed path filters with high bit paths\n> repo-settings: introduce commitgraph.changedPathsVersion\n> commit-graph: new filter ver. that fixes murmur3\n> \n> Taylor Blau (13):\n> t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`\n> revision.c: consult Bloom filters for root commits\n> commit-graph: ensure Bloom filters are read with consistent settings\n> t/helper/test-read-graph.c: extract `dump_graph_info()`\n> bloom.h: make `load_bloom_filter_from_graph()` public\n> t/helper/test-read-graph: implement `bloom-filters` mode\n> bloom: annotate filters with hash version\n> bloom: prepare to discard incompatible Bloom filters\n> commit-graph.c: unconditionally load Bloom filters\n> commit-graph: drop unnecessary `graph_read_bloom_data_context`\n> object.h: fix mis-aligned flag bits table\n> commit-graph: reuse existing Bloom filters where possible\n> bloom: introduce `deinit_bloom_filters()`\n> \n> Documentation/config/commitgraph.txt | 26 ++-\n> Documentation/gitformat-commit-graph.txt | 9 +-\n> bloom.c | 208 ++++++++++++++++-\n> bloom.h | 38 +++-\n> commit-graph.c | 61 ++++-\n> object.h | 3 +-\n> oss-fuzz/fuzz-commit-graph.c | 2 +-\n> repo-settings.c | 6 +-\n> repository.h | 2 +-\n> revision.c | 26 ++-\n> t/helper/test-bloom.c | 9 +-\n> t/helper/test-read-graph.c | 67 ++++--\n> t/t0095-bloom.sh | 8 +\n> t/t4216-log-bloom.sh | 272 ++++++++++++++++++++++-\n> 14 files changed, 682 insertions(+), 55 deletions(-)\n> \n> Range-diff against v2:\n> 10: 002a06d1e9 ! 1: fe671d616c t/t4216-log-bloom.sh: harden `test_bloom_filters_not_used()`\n> @@ Commit message\n> indicating that no filters were used.\n> \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## t/t4216-log-bloom.sh ##\n> @@ t/t4216-log-bloom.sh: test_bloom_filters_used () {\n> -: ---------- > 2: 7d0fa93543 revision.c: consult Bloom filters for root commits\n> -: ---------- > 3: 2ecc0a2d58 commit-graph: ensure Bloom filters are read with consistent settings\n> 1: 5fa681b58e ! 4: 17703ed89a gitformat-commit-graph: describe version 2 of BDAT\n> @@ Commit message\n> Signed-off-by: Jonathan Tan \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## Documentation/gitformat-commit-graph.txt ##\n> @@ Documentation/gitformat-commit-graph.txt: All multi-byte numbers are in network byte order.\n> 2: 623d840575 ! 5: 94552abf45 t/helper/test-read-graph.c: extract `dump_graph_info()`\n> @@ Commit message\n> Signed-off-by: Jonathan Tan \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## t/helper/test-read-graph.c ##\n> @@\n> 3: bc9d77ae60 ! 6: 3d81efa27b bloom.h: make `load_bloom_filter_from_graph()` public\n> @@ Commit message\n> Signed-off-by: Jonathan Tan \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## bloom.c ##\n> @@ bloom.c: static inline unsigned char get_bitmask(uint32_t pos)\n> 4: ac7008aed3 ! 7: d23cd89037 t/helper/test-read-graph: implement `bloom-filters` mode\n> @@ Commit message\n> Signed-off-by: Jonathan Tan \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## t/helper/test-read-graph.c ##\n> @@ t/helper/test-read-graph.c: static void dump_graph_info(struct commit_graph *graph)\n> @@ t/helper/test-read-graph.c: int cmd__read_graph(int argc UNUSED, const char **ar\n> -\treturn 0;\n> +\treturn ret;\n> }\n> ++\n> ++\n> 5: 71755ba856 ! 8: cba766f224 t4216: test changed path filters with high bit paths\n> @@ Commit message\n> Signed-off-by: Taylor Blau \n> \n> ## t/t4216-log-bloom.sh ##\n> -@@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty commits' '\n> - \t)\n> +@@ t/t4216-log-bloom.sh: test_expect_success 'merge graph layers with incompatible Bloom settings' '\n> + \t! grep \"disabling Bloom filters\" err\n> '\n> \n> +get_first_changed_path_filter () {\n> 6: 9768d92c0f ! 9: a08a961f41 repo-settings: introduce commitgraph.changedPathsVersion\n> @@ Commit message\n> Signed-off-by: Jonathan Tan \n> Signed-off-by: Junio C Hamano \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## Documentation/config/commitgraph.txt ##\n> @@ Documentation/config/commitgraph.txt: commitGraph.maxNewFilters::\n> 7: f911b4bfab = 10: 61d44519a5 commit-graph: new filter ver. that fixes murmur3\n> 8: 35009900df ! 11: a8c10f8de8 bloom: annotate filters with hash version\n> @@ Commit message\n> Bloom filter.\n> \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## bloom.c ##\n> @@ bloom.c: int load_bloom_filter_from_graph(struct commit_graph *g,\n> 9: 138bc16905 ! 12: 2ba10a4b4b bloom: prepare to discard incompatible Bloom filters\n> @@ Commit message\n> `get_or_compute_bloom_filter()`.\n> \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## bloom.c ##\n> @@ bloom.c: static void init_truncated_large_filter(struct bloom_filter *filter,\n> 11: 2437e62813 ! 13: 09d8669c3a commit-graph.c: unconditionally load Bloom filters\n> @@ Commit message\n> either \"1\" or \"2\".\n> \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## commit-graph.c ##\n> @@ commit-graph.c: static int graph_read_bloom_data(const unsigned char *chunk_start,\n> 12: fe8fb2f5fe ! 14: 0d4f9dc4ee commit-graph: drop unnecessary `graph_read_bloom_data_context`\n> @@ Commit message\n> \n> Noticed-by: Jonathan Tan \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## commit-graph.c ##\n> @@ commit-graph.c: static int graph_read_oid_lookup(const unsigned char *chunk_start,\n> 13: 825af91e11 ! 15: 1f7f27bc47 object.h: fix mis-aligned flag bits table\n> @@ Commit message\n> Bit position 23 is one column too far to the left.\n> \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## object.h ##\n> @@ object.h: void object_array_init(struct object_array *array);\n> 14: 593b317192 ! 16: abbef95ae8 commit-graph: reuse existing Bloom filters where possible\n> @@ Commit message\n> commits by their generation number.\n> \n> Signed-off-by: Taylor Blau \n> - Signed-off-by: Junio C Hamano \n> - Signed-off-by: Taylor Blau \n> \n> ## bloom.c ##\n> @@\n> 15: 8bf2c9cf98 = 17: ca362408d5 bloom: introduce `deinit_bloom_filters()`\n> -- \n> 2.42.0.342.g8bb3a896ee\n"},{"id":"483421","messageId":"ZTAXyMWeQLRWcpuu@nand.local","threadId":"60142","inReplyTo":"ZS5JnkybxvetTUzu@tanuki","subject":"Re: [PATCH v3 05/17] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-18T17:37:12Z","receivedAt":"2023-10-18T17:37:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Oct 17, 2023 at 10:45:18AM +0200, Patrick Steinhardt wrote:\n> On Tue, Oct 10, 2023 at 04:33:33PM -0400, Taylor Blau wrote:\n> > Prepare for the 'read-graph' test helper to perform other tasks besides\n> > dumping high-level information about the commit-graph by extracting its\n> > main routine into a separate function.\n> >\n> > Signed-off-by: Taylor Blau \n> > Signed-off-by: Jonathan Tan \n> > Signed-off-by: Junio C Hamano \n> > Signed-off-by: Taylor Blau \n>\n> Nit: your signoff is duplicated here. This is also still the case for\n> some of the other commits.\n\nYeah, this is an artifact of having tossed these patches back and forth\n(originally Jonathan sent some of these, then I sent another round, then\nJonathan, now me again). It's a little verbose, but accurately tracks\nthe DCO across multiple rounds.\n\nThanks,\nTaylor\n"},{"id":"483422","messageId":"ZTAYvKBeqUyU7o4W@nand.local","threadId":"60142","inReplyTo":"ZS5JmQib3onjirC6@tanuki","subject":"Re: [PATCH v3 08/17] t4216: test changed path filters with high bit paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-18T17:41:16Z","receivedAt":"2023-10-18T17:41:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Oct 17, 2023 at 10:45:13AM +0200, Patrick Steinhardt wrote:\n> > +test_expect_success 'setup check value of version 1 changed-path' '\n> > +\t(\n> > +\t\tcd highbit1 &&\n> > +\t\techo \"52a9\" >expect &&\n> > +\t\tget_first_changed_path_filter >actual &&\n> > +\t\ttest_cmp expect actual\n> > +\t)\n> > +'\n> > +\n> > +# expect will not match actual if char is unsigned by default. Write the test\n> > +# in this way, so that a user running this test script can still see if the two\n> > +# files match. (It will appear as an ordinary success if they match, and a skip\n> > +# if not.)\n> > +if test_cmp highbit1/expect highbit1/actual\n> > +then\n> > +\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n> > +fi\n> > +test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n> > +\t# Only the prereq matters for this test.\n> > +\ttrue\n> > +'\n>\n> Doesn't this mean that the preceding test where we `test_cmp expect\n> actual` can fail on some platforms depending on the signedness of\n> `char`?\n\nGreat catch, I am surprised this slipped by in earlier rounds. This\nshould do the trick, since we don't actually care about conditioning\nthat test's passing on test_cmp coming up clean. We check that in the if\nstatement you pointed out here, so:\n\n--- 8< ---\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 114672e904..400dce2193 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -502,8 +502,7 @@ test_expect_success 'setup check value of version 1 changed-path' '\n \t(\n \t\tcd highbit1 &&\n \t\techo \"52a9\" >expect &&\n-\t\tget_first_changed_path_filter >actual &&\n-\t\ttest_cmp expect actual\n+\t\tget_first_changed_path_filter >actual\n \t)\n '\n--- >8 ---\n\nThanks,\nTaylor\n"},{"id":"483423","messageId":"ZTAaEffz2MkkTjzD@nand.local","threadId":"60142","inReplyTo":"ZS5JpJCw-MY256xo@tanuki","subject":"Re: [PATCH v3 10/17] commit-graph: new filter ver. that fixes murmur3","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-18T17:46:57Z","receivedAt":"2023-10-18T17:47:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Oct 17, 2023 at 10:45:24AM +0200, Patrick Steinhardt wrote:\n> > @@ -314,17 +314,26 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n> > \treturn 0;\n> > }\n> >\n> > +struct graph_read_bloom_data_context {\n> > +\tstruct commit_graph *g;\n> > +\tint *commit_graph_changed_paths_version;\n> > +};\n> > +\n> > static int graph_read_bloom_data(const unsigned char *chunk_start,\n> > \t\t\t\t size_t chunk_size, void *data)\n> > {\n> > -\tstruct commit_graph *g = data;\n> > +\tstruct graph_read_bloom_data_context *c = data;\n> > +\tstruct commit_graph *g = c->g;\n> > \tuint32_t hash_version;\n> > -\tg->chunk_bloom_data = chunk_start;\n> > \thash_version = get_be32(chunk_start);\n> >\n> > -\tif (hash_version != 1)\n> > +\tif (*c->commit_graph_changed_paths_version == -1) {\n> > +\t\t*c->commit_graph_changed_paths_version = hash_version;\n> > +\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n> > \t\treturn 0;\n> > +\t}\n>\n> In case we have `c->commit_graph_changed_paths_version == -1` we lose\n> the check that the hash version is something that we know and support,\n> don't we? And while we do start to handle `-1` in the writing path, I\n> think we don't in the reading path unless I missed something.\n\nWe don't have to deal with c->commit_graph_changed_paths_version being\n-1 here, since we normalize it when reading the BDAT chunk. See\ncommit-graph.c::graph_read_bloom_data(), particularly:\n\n if (*c->commit_graph_changed_paths_version == -1)\n *c->commit_graph_changed_paths_version = hash_version;\n else if (hash_version != *c->commit_graph_changed_paths_version)\n return 0;\n\n> > +test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n> > +\tgit init doublewrite &&\n> > +\ttest_commit -C doublewrite c \"$CENT\" &&\n> > +\tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n> > +\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n> > +\tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n> > +\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n> > +\t(\n> > +\t\tcd doublewrite &&\n> > +\t\techo \"c01f\" >expect &&\n> > +\t\tget_first_changed_path_filter >actual &&\n> > +\t\ttest_cmp expect actual\n> > +\t)\n> > +'\n> > +\n>\n> With the supposedly missing check in mind, should we also add tests for\n> currently unknown versions like 3 or -2?\n\nGood idea, I'll update the test to reflect.\n\nThanks,\nTaylor\n"},{"id":"483424","messageId":"ZTAaO5Jg8CGr/pJL@nand.local","threadId":"60142","inReplyTo":"ZS5JsDKk8RioQfOA@tanuki","subject":"Re: [PATCH v3 00/17] bloom: changed-path Bloom filters v2 (& sundries)","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-10-18T17:47:39Z","receivedAt":"2023-10-18T17:47:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Oct 17, 2023 at 10:45:36AM +0200, Patrick Steinhardt wrote:\n> > Thanks to Jonathan, Peff, and SZEDER who have helped a great deal in\n> > assembling these patches. As usual, a range-diff is included below.\n> > Thanks in advance for your\n> > review!\n>\n> As this patch series has been sitting around without reviews for a week\n> I've tried my best to give it a go. Note though that this area is mostly\n> outside of my own comfort zone, so some of the questions and suggestions\n> might ultimately not apply.\n\nThanks for giving it a look! I generated a few small tweaks on top of\nwhat I already had here based on your review, so I'll send a reroll\nshortly.\n\nThanks,\nTaylor\n"},{"id":"483465","messageId":"xmqqh6mnzdrb.fsf@gitster.g","threadId":"60142","inReplyTo":"ZTAXyMWeQLRWcpuu@nand.local","subject":"Re: [PATCH v3 05/17] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-18T23:56:08Z","receivedAt":"2023-10-18T23:56:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Tue, Oct 17, 2023 at 10:45:18AM +0200, Patrick Steinhardt wrote:\n>> On Tue, Oct 10, 2023 at 04:33:33PM -0400, Taylor Blau wrote:\n>> > Prepare for the 'read-graph' test helper to perform other tasks besides\n>> > dumping high-level information about the commit-graph by extracting its\n>> > main routine into a separate function.\n>> >\n>> > Signed-off-by: Taylor Blau \n>> > Signed-off-by: Jonathan Tan \n>> > Signed-off-by: Junio C Hamano \n>> > Signed-off-by: Taylor Blau \n>>\n>> Nit: your signoff is duplicated here. This is also still the case for\n>> some of the other commits.\n>\n> Yeah, this is an artifact of having tossed these patches back and forth\n> (originally Jonathan sent some of these, then I sent another round, then\n> Jonathan, now me again). It's a little verbose, but accurately tracks\n> the DCO across multiple rounds.\n\nWell, between you and Jonathan Tan, that might be true, but my\ninvolvement in the chain is only that I happened to have looked at\nan earlier round and without doing anything else. Even if you\nrefetched from my tree and based the new round on that version, it\nwould have been the same if you started from what you sent out\nearlier without even looking at my tree. So I do not think that\nchain of DCO adds very little value by recording my sign-off there.\n\n\n"}]}