{"thread":{"id":"59781","subject":"[PATCH 0/2] Changed path filter hash fix and version bump","startedAt":"2023-05-22T21:48:19Z","lastAt":"2023-08-03T18:47:40Z","messageCount":116,"participants":["Jonathan Tan","Junio C Hamano","Derrick Stolee","Ramsay Jones","Eric Sunshine","Taylor Blau"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"477656","messageId":"cover.1684790529.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":null,"subject":"[PATCH 0/2] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-22T21:48:05Z","receivedAt":"2023-05-22T21:48:19Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Following the conversation in [1], here are patches to fix the murmur3\nhash function used in creating (and interpreting) changed path filters,\nand also to bump the version number to 2.\n\nThis is I think the simplest way to do this (invalidating all existing\nchanged path filters). The resource-consuming part of creating a changed\npath filter is in computing the changed paths (thus, reading trees and\ncalculating changes), and to check if a changed path filter could be\nreused, one would need to compute the changed paths anyway in order to\ndetermine if any of them have high-bit strings, so I did not pursue this\nfurther. Server operators might be able to reuse changed path filters\nif, for example, they have a more efficient way to determine that no\npaths in a repo have the high bit set, but I think that this is out of\nscope for the Git project.\n\nIn patch 2, I couldn't figure out how to make Bash pass high-bit strings\nas a CLI argument for some reason, so I hardcoded the string I wanted\nin the test helper instead. If anyone knows how to pass such strings,\nplease let me know.\n\n[1] https://lore.kernel.org/git/20230511224101.972442-1-jonathantanmy@google.com/\n\nJonathan Tan (2):\n  t4216: test wrong bloom filter version rejection\n  commit-graph: fix murmur3, bump filter ver. to 2\n\n bloom.c               | 14 +++++++-------\n bloom.h               |  9 ++++++---\n commit-graph.c        |  4 ++--\n t/helper/test-bloom.c |  7 +++++++\n t/t0095-bloom.sh      |  8 ++++++++\n t/t4216-log-bloom.sh  | 36 +++++++++++++++++++++++++++++++++---\n 6 files changed, 63 insertions(+), 15 deletions(-)\n\n-- \n2.40.1.698.g37aff9b760-goog\n\n"},{"id":"477657","messageId":"2e154c9088e83fa9433ebe3e9ddf99dc30155d94.1684790529.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1684790529.git.jonathantanmy@google.com","subject":"[PATCH 1/2] t4216: test wrong bloom filter version rejection","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-22T21:48:06Z","receivedAt":"2023-05-22T21:48:24Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Add a test that checks that Git does not make use of changed path\nfilters that have an unrecognized version.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n t/t4216-log-bloom.sh | 30 ++++++++++++++++++++++++++++++\n 1 file changed, 30 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..f14cc1c1f1 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -85,6 +85,36 @@ test_bloom_filters_not_used () {\n \ttest_cmp log_wo_bloom log_w_bloom\n }\n \n+get_bdat_offset () {\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"N\", \"$1\") if /BDAT\\0\\0\\0\\0(....)/ or exit 1' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n+test_expect_success 'incompatible bloom filter versions are not used' '\n+\tcp .git/objects/info/commit-graph old-commit-graph &&\n+\ttest_when_finished \"mv old-commit-graph .git/objects/info/commit-graph\" &&\n+\n+\tBDAT_OFFSET=$(get_bdat_offset) &&\n+\n+\t# Write an arbitrary number to the least significant byte of the\n+\t# version field in the BDAT chunk\n+\tcat old-commit-graph >new-commit-graph &&\n+\tprintf \"\\aa\" |\n+\t\tdd of=new-commit-graph bs=1 count=1 \\\n+\t\t\tseek=$((BDAT_OFFSET + 3)) conv=notrunc &&\n+\tmv new-commit-graph .git/objects/info/commit-graph &&\n+\ttest_bloom_filters_not_used \"-- A\" &&\n+\n+\t# But the correct version number works\n+\tcat old-commit-graph >new-commit-graph &&\n+\tprintf \"\\01\" |\n+\t\tdd of=new-commit-graph bs=1 count=1 \\\n+\t\t\tseek=$((BDAT_OFFSET + 3)) conv=notrunc &&\n+\tmv new-commit-graph .git/objects/info/commit-graph &&\n+\ttest_bloom_filters_used \"-- A\"\n+'\n+\n for path in A A/B A/B/C A/file1 A/B/file2 A/B/C/file3 file4 file5 file5_renamed file_to_be_deleted\n do\n \tfor option in \"\" \\\n-- \n2.40.1.698.g37aff9b760-goog\n\n"},{"id":"477658","messageId":"1cb0ee10253ab38b194c6554ecc68a5267e21145.1684790529.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1684790529.git.jonathantanmy@google.com","subject":"[PATCH 2/2] commit-graph: fix murmur3, bump filter ver. to 2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-22T21:48:07Z","receivedAt":"2023-05-22T21:48:26Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 and even with\nGit binaries that were compiled with different signedness of char. This\nbug affects both how Git writes changed path filters to disk and how Git\ninterprets changed path filters on disk.\n\nTherefore, fix this bug. And because changed path filters on disk might\nno longer be compatible, teach Git to write \"2\" as the version when\nwriting changed path filters (instead of \"1\" currently), and only accept\n\"2\" as the version when reading them (instead of \"1\" currently).\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter. So this\npatch does not include any mechanism to \"salvage\" changed path filters\nfrom repositories.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in the test was obtained from another murmur3 implementation\nusing the following Go source code:\n\n  package main\n\n  import \"fmt\"\n  import \"github.com/spaolacci/murmur3\"\n\n  func main() {\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n  }\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n bloom.c               | 14 +++++++-------\n bloom.h               |  9 ++++++---\n commit-graph.c        |  4 ++--\n t/helper/test-bloom.c |  7 +++++++\n t/t0095-bloom.sh      |  8 ++++++++\n t/t4216-log-bloom.sh  |  8 ++++----\n 6 files changed, 34 insertions(+), 16 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex aef6b5fea2..fec243b2f1 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -82,10 +82,10 @@ uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n \n \tuint32_t k;\n \tfor (i = 0; i < len4; i++) {\n-\t\tuint32_t byte1 = (uint32_t)data[4*i];\n-\t\tuint32_t byte2 = ((uint32_t)data[4*i + 1]) << 8;\n-\t\tuint32_t byte3 = ((uint32_t)data[4*i + 2]) << 16;\n-\t\tuint32_t byte4 = ((uint32_t)data[4*i + 3]) << 24;\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n \t\tk = byte1 | byte2 | byte3 | byte4;\n \t\tk *= c1;\n \t\tk = rotate_left(k, r1);\n@@ -99,13 +99,13 @@ uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n \n \tswitch (len & (sizeof(uint32_t) - 1)) {\n \tcase 3:\n-\t\tk1 ^= ((uint32_t)tail[2]) << 16;\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n \t\t/*-fallthrough*/\n \tcase 2:\n-\t\tk1 ^= ((uint32_t)tail[1]) << 8;\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n \t\t/*-fallthrough*/\n \tcase 1:\n-\t\tk1 ^= ((uint32_t)tail[0]) << 0;\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n \t\tk1 *= c1;\n \t\tk1 = rotate_left(k1, r1);\n \t\tk1 *= c2;\ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..8526fa948c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -7,9 +7,11 @@ struct repository;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * We currently only support version = 2 which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -38,8 +40,9 @@ struct bloom_filter_settings {\n \tuint32_t max_changed_paths;\n };\n \n+#define BLOOM_HASH_VERSION 2\n #define DEFAULT_BLOOM_MAX_CHANGES 512\n-#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10, DEFAULT_BLOOM_MAX_CHANGES }\n+#define DEFAULT_BLOOM_FILTER_SETTINGS { BLOOM_HASH_VERSION, 7, 10, DEFAULT_BLOOM_MAX_CHANGES }\n #define BITS_PER_WORD 8\n #define BLOOMDATA_CHUNK_HEADER_SIZE 3 * sizeof(uint32_t)\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 843bdb458d..2eb9b781f4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -314,7 +314,7 @@ static int graph_read_bloom_data(const unsigned char *chunk_start,\n \tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n+\tif (hash_version != BLOOM_HASH_VERSION)\n \t\treturn 0;\n \n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n@@ -2402,7 +2402,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex aabe31d724..5624e4d2db 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -50,6 +50,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \"  test-tool bloom get_murmur3 <string>\\n\"\n+\"  test-tool bloom get_murmur3_seven_highbit\\n\"\n \"  test-tool bloom generate_filter <string> [<string>...]\\n\"\n \"  test-tool bloom get_filter_for_commit <commit-hex>\\n\";\n \n@@ -68,6 +69,12 @@ int cmd__bloom(int argc, const char **argv)\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n \tif (!strcmp(argv[1], \"generate_filter\")) {\n \t\tstruct bloom_filter filter;\n \t\tint i = 2;\ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex f14cc1c1f1..7a193aa143 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -48,7 +48,7 @@ graph_read_expect () {\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n \tchunks: oid_fanout oid_lookup commit_metadata generation_data bloom_indexes bloom_data\n-\toptions: bloom(1,10,7) read_generation_data\n+\toptions: bloom(2,10,7) read_generation_data\n \tEOF\n \ttest-tool read-graph >actual &&\n \ttest_cmp expect actual\n@@ -108,7 +108,7 @@ test_expect_success 'incompatible bloom filter versions are not used' '\n \n \t# But the correct version number works\n \tcat old-commit-graph >new-commit-graph &&\n-\tprintf \"\\01\" |\n+\tprintf \"\\02\" |\n \t\tdd of=new-commit-graph bs=1 count=1 \\\n \t\t\tseek=$((BDAT_OFFSET + 3)) conv=notrunc &&\n \tmv new-commit-graph .git/objects/info/commit-graph &&\n@@ -209,10 +209,10 @@ test_expect_success 'persist filter settings' '\n \t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n \t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2.txt &&\n+\tgrep \"{\\\"hash_version\\\":2,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2-auto.txt\n+\tgrep \"{\\\"hash_version\\\":2,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2-auto.txt\n '\n \n test_max_changed_paths () {\n-- \n2.40.1.698.g37aff9b760-goog\n\n"},{"id":"477660","messageId":"xmqqttw3irpp.fsf@gitster.g","threadId":"59781","inReplyTo":"cover.1684790529.git.jonathantanmy@google.com","subject":"Re: [PATCH 0/2] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-05-23T04:42:26Z","receivedAt":"2023-05-23T04:42:31Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Following the conversation in [1], here are patches to fix the murmur3\n> hash function used in creating (and interpreting) changed path filters,\n> and also to bump the version number to 2.\n\nWonderful.  Thanks for a quick update.  Will take a look when I come\nback to the keyboard (I'm on half-vacation right now).\n\n>\n> This is I think the simplest way to do this (invalidating all existing\n> changed path filters). The resource-consuming part of creating a changed\n> path filter is in computing the changed paths (thus, reading trees and\n> calculating changes), and to check if a changed path filter could be\n> reused, one would need to compute the changed paths anyway in order to\n> determine if any of them have high-bit strings, so I did not pursue this\n> further. Server operators might be able to reuse changed path filters\n> if, for example, they have a more efficient way to determine that no\n> paths in a repo have the high bit set, but I think that this is out of\n> scope for the Git project.\n>\n> In patch 2, I couldn't figure out how to make Bash pass high-bit strings\n> as a CLI argument for some reason, so I hardcoded the string I wanted\n> in the test helper instead. If anyone knows how to pass such strings,\n> please let me know.\n>\n> [1] https://lore.kernel.org/git/20230511224101.972442-1-jonathantanmy@google.com/\n>\n> Jonathan Tan (2):\n>   t4216: test wrong bloom filter version rejection\n>   commit-graph: fix murmur3, bump filter ver. to 2\n>\n>  bloom.c               | 14 +++++++-------\n>  bloom.h               |  9 ++++++---\n>  commit-graph.c        |  4 ++--\n>  t/helper/test-bloom.c |  7 +++++++\n>  t/t0095-bloom.sh      |  8 ++++++++\n>  t/t4216-log-bloom.sh  | 36 +++++++++++++++++++++++++++++++++---\n>  6 files changed, 63 insertions(+), 15 deletions(-)\n"},{"id":"477663","messageId":"def26c71-0fbb-58a3-f1cd-f8e532b67503@github.com","threadId":"59781","inReplyTo":"1cb0ee10253ab38b194c6554ecc68a5267e21145.1684790529.git.jonathantanmy@google.com","subject":"Re: [PATCH 2/2] commit-graph: fix murmur3, bump filter ver. to 2","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-05-23T13:00:59Z","receivedAt":"2023-05-23T13:01:11Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/22/2023 5:48 PM, Jonathan Tan wrote:\n> The murmur3 implementation in bloom.c has a bug when converting series\n> of 4 bytes into network-order integers when char is signed (which is\n> controllable by a compiler option, and the default signedness of char is\n> platform-specific). When a string contains characters with the high bit\n> set, this bug causes results that, although internally consistent within\n> Git, does not accord with other implementations of murmur3 and even with\n> Git binaries that were compiled with different signedness of char. This\n> bug affects both how Git writes changed path filters to disk and how Git\n> interprets changed path filters on disk.\n> \n> Therefore, fix this bug. And because changed path filters on disk might\n> no longer be compatible, teach Git to write \"2\" as the version when\n> writing changed path filters (instead of \"1\" currently), and only accept\n> \"2\" as the version when reading them (instead of \"1\" currently).\n\nI appreciate that you discovered and are presenting a way out of this\nproblem, however the current approach does not preserve compatibility\nenough.\n\n> @@ -82,10 +82,10 @@ uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n>  \n>  \tuint32_t k;\n>  \tfor (i = 0; i < len4; i++) {\n> -\t\tuint32_t byte1 = (uint32_t)data[4*i];\n> -\t\tuint32_t byte2 = ((uint32_t)data[4*i + 1]) << 8;\n> -\t\tuint32_t byte3 = ((uint32_t)data[4*i + 2]) << 16;\n> -\t\tuint32_t byte4 = ((uint32_t)data[4*i + 3]) << 24;\n> +\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n> +\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n> +\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n> +\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n>  \t\tk = byte1 | byte2 | byte3 | byte4;\n>  \t\tk *= c1;\n>  \t\tk = rotate_left(k, r1);\n\nBy changing this algorithm directly (instead of making an \"unsigned\" version,\nor renaming this one to the \"maybe signed\" version), you are making it\nimpossible for us to ship a version that can read version 1 Bloom filters,\nso all read-only history operations will immediately slow down (because they\nwill ignore v1 chunks, better than incorrectly parsing v1 chunks).\n\n> @@ -314,7 +314,7 @@ static int graph_read_bloom_data(const unsigned char *chunk_start,\n>  \tg->chunk_bloom_data = chunk_start;\n>  \thash_version = get_be32(chunk_start);\n>  \n> -\tif (hash_version != 1)\n> +\tif (hash_version != BLOOM_HASH_VERSION)\n>  \t\treturn 0;\n  \nHere's where we would ignore v1 filters, instead of continuing to read them\n(with all the risks involved).\n\nIn order for this to be something we can ship safely to environments that depend\non changed-path Bloom filters, we need to be able to parse v1 filters. It would\nbe even better if we didn't write v2 filters by default, but instead hid it\nbehind a config option that is off by default for at least one major release.\n\nI notice that you didn't update the commit-graph format docs, which seems like\na valuable place to describe the new version number, as well as any plans to\ncompletely deprecate v1. For instance, describing the v1 implementation as\nhaving inconsistent application of murmur3 is a valuable thing to have, but\nthen describe the plans for deprecating it as an unsafe format.\n\nHere is a potential plan to consider:\n\n 1. v2.42.0 includes writing v2 format, off by default.\n 2. v2.43.0 writes v2 format by default.\n 3. v2.44.0 no longer parses v1 format (ignored without error).\n\nThanks,\n-Stolee\n"},{"id":"477677","messageId":"20230523230049.1225412-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"def26c71-0fbb-58a3-f1cd-f8e532b67503@github.com","subject":"Re: [PATCH 2/2] commit-graph: fix murmur3, bump filter ver. to 2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-23T23:00:49Z","receivedAt":"2023-05-23T23:00:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Derrick Stolee <derrickstolee@github.com> writes:\n> I notice that you didn't update the commit-graph format docs,\n\nAh, thanks for the reminder.\n\n> Here is a potential plan to consider:\n> \n>  1. v2.42.0 includes writing v2 format, off by default.\n>  2. v2.43.0 writes v2 format by default.\n>  3. v2.44.0 no longer parses v1 format (ignored without error).\n\nFirst of all, thanks for your comments on the migration process - that\nis indeed the most complicated part of this.\n\nThe code change to support 2 versions seems not too hard (duplicate\nmurmur3_seeded() and modify one so that we have one version 1 and\none version 2, and teach fill_bloom_key() to call the appropriate one\nbased on the struct bloom_filter_settings) but this requires both the\ncode author and reviewer(s) to check that we don't have cases in which\nwe read or write one version when we're supposed to do it with the\nother. And the benefit of doing this seems to just be giving server\nadministrators an opportunity to perform the migration at a more relaxed\npace, which I think there are other ways to accomplish if we really\nwanted to do this, so I wanted to avoid having 2 versions in the Git\ncodebase.\n"},{"id":"477681","messageId":"xmqqv8gihahz.fsf@gitster.g","threadId":"59781","inReplyTo":"def26c71-0fbb-58a3-f1cd-f8e532b67503@github.com","subject":"Re: [PATCH 2/2] commit-graph: fix murmur3, bump filter ver. to 2","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-05-23T23:51:52Z","receivedAt":"2023-05-23T23:52:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <derrickstolee@github.com> writes:\n\n> I appreciate that you discovered and are presenting a way out of this\n> problem, however the current approach does not preserve compatibility\n> enough.\n> ...\n> By changing this algorithm directly (instead of making an \"unsigned\" version,\n> or renaming this one to the \"maybe signed\" version), you are making it\n> impossible for us to ship a version that can read version 1 Bloom filters,\n> so all read-only history operations will immediately slow down (because they\n> will ignore v1 chunks, better than incorrectly parsing v1 chunks).\n>\n> Here's where we would ignore v1 filters, instead of continuing to read them\n> (with all the risks involved).\n\nI do not know the \"all the risks involved\" comment.  Is the risk\nsomething we can mitigate by still reading v1 data but be careful\nabout when not to apply the filters?\n\nI may be misremembering the original discussion, but wasn't the\nconclusion that v1 data is salvageable in the sense that it can\nstill reliably say that, given a pathname without bytes with\nhigh-bit set, it cannot possibly belong to the set of changed paths,\neven though, because the filter is contaminated with \"signed\" data,\nits false-positive rate may be higher than using \"unsigned\" version?\nAnd based on that observation, we can still read v1 data but only\nuse the Bloom filters when the queried paths have no byte with\nhigh-bit set.\n\nAlso if we are operating in such an environment then would it be\npossible to first compute as if we were going to generate v2 data,\nbut write it as v1 after reading all the path and realizing there\nare no problematic paths?  IOW, even if the version of Git is\ncapable of writing and reading v2, it does not have to use v2,\nright?  To put it the other way around, we will have to and can keep\nsupporting data that is labeled as v1, no?\n\n> In order for this to be something we can ship safely to environments that depend\n> on changed-path Bloom filters, we need to be able to parse v1 filters. It would\n> be even better if we didn't write v2 filters by default, but instead hid it\n> behind a config option that is off by default for at least one major release.\n\nIs the concern that we will double the chunk size because both v1\nand v2 will be written?  Or is it that we will stop writing v1 if we\nstart writing v2 and switching too early will mean the repositories\nwill become slower for older implementations that haven't died out?\n\nThanks.\n"},{"id":"477699","messageId":"20230524212657.1348621-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"xmqqv8gihahz.fsf@gitster.g","subject":"Re: [PATCH 2/2] commit-graph: fix murmur3, bump filter ver. to 2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-24T21:26:57Z","receivedAt":"2023-05-24T21:27:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> I may be misremembering the original discussion, but wasn't the\n> conclusion that v1 data is salvageable in the sense that it can\n> still reliably say that, given a pathname without bytes with\n> high-bit set, it cannot possibly belong to the set of changed paths,\n> even though, because the filter is contaminated with \"signed\" data,\n> its false-positive rate may be higher than using \"unsigned\" version?\n> And based on that observation, we can still read v1 data but only\n> use the Bloom filters when the queried paths have no byte with\n> high-bit set.\n\nThere are at least 3 ways of salvaging the data that we've discussed:\n\n- Enumerating all of a repo's paths and if none of them have a high bit,\n  retain the existing filters.\n- Walking all of a repo's trees (so that we know which tree corresponds\n  to which commit) and if for a commit, all its trees have no high bit,\n  retain the filter for that tree (otherwise recompute it).\n- Keep using a version 1 filter but only when the sought-for path has no\n  high bit (as you describe here).\n\n(The first 2 is my interpretation of what Taylor described [1].)\n\nI'm not sure if we want to keep version 1 filters around at all -\nthis would work with Git as long as it is not compiled with different\nsignedness of char, but would not work with other implementations of\nGit (unless they replicate the hashing bug). There is also the issue of\nhow we're going to indicate that in a commit graph file, some filters\nare version 1 and some filters are version 2 (unless the plan is to\ncompletely rewrite the filters in this case, but then we'll run into\nthe issue that computing these filters en-masse is expensive, as Taylor\ndescribes also in [1]).\n\n> Also if we are operating in such an environment then would it be\n> possible to first compute as if we were going to generate v2 data,\n> but write it as v1 after reading all the path and realizing there\n> are no problematic paths?  \n\nI think in this case, we would want to write it as v2 anyway, because\nthere's no way to distinguish a v1 that has high bits and is written\nincorrectly versus a v1 that happens to have no high bits and therefore\nis identical under v2.\n\n> IOW, even if the version of Git is\n> capable of writing and reading v2, it does not have to use v2,\n> right?  To put it the other way around, we will have to and can keep\n> supporting data that is labeled as v1, no?\n\nI think this is the main point - whether we want to continue supporting\ndata labeled as v1. I personally think that we should migrate to v2\nas quickly as possible. But if the consensus is that we should support\nboth, at least for a few releases of Git, I'll go with that.\n\n[1] https://lore.kernel.org/git/ZF116EDcmAy7XEbC@nand.local/\n"},{"id":"477733","messageId":"a4d22518-d3e6-76aa-fe94-2ec77a1de96a@github.com","threadId":"59781","inReplyTo":"20230524212657.1348621-1-jonathantanmy@google.com","subject":"Re: [PATCH 2/2] commit-graph: fix murmur3, bump filter ver. to 2","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-05-26T13:19:41Z","receivedAt":"2023-05-26T13:19:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 5/24/2023 5:26 PM, Jonathan Tan wrote:\n> Junio C Hamano <gitster@pobox.com> writes:\n\n>> IOW, even if the version of Git is\n>> capable of writing and reading v2, it does not have to use v2,\n>> right?  To put it the other way around, we will have to and can keep\n>> supporting data that is labeled as v1, no?\n> \n> I think this is the main point - whether we want to continue supporting\n> data labeled as v1. I personally think that we should migrate to v2\n> as quickly as possible. But if the consensus is that we should support\n> both, at least for a few releases of Git, I'll go with that.\n\nI agree on migrating quickly as possible, within basic safety guidelines.\n\nShipping a Git change that suddenly is unable to use on-disk data that\nit has previously relied upon is not a safe change. And that is the\nabsolute minimum amount of safety required. The other side is to not\nmake a Git change that suddenly changes the on-disk format without a\nswitch to disable it.\n\nThink about it this way: if there was a bug in the code, could we\nsafely roll it back? If we are immediately writing v2 filters after\nthe deployment, then rolling it back will cause the previous version\nto not recognize those filters, leading to a delayed recovery.\n\nI'd be willing to modify my suggested steps:\n\n>>> 1. v2.42.0 includes writing v2 format, off by default.\n>>> 2. v2.43.0 writes v2 format by default.\n>>> 3. v2.44.0 no longer parses v1 format (ignored without error).\n\nto something simpler:\n\n 1. v2.42.0 writes v2 format by default, but can be disabled by config.\n 2. v2.43.0 no longer parses or writes v1 format.\n\nWith this, we could proactively set the config value that disables the\nv2 format in our production environment, then slowly re-enable that\nconfig after the binaries have deployed. This allows us to limit the\nblast radius if something goes wrong, which is really important.\n\nFurther, I'm describing an environment where we control all of the Git\nversions that are interacting with the repositories. Other environments\ndon't have that luxury, such as typical client users.\n\nEven the three-version plan is an accelerated deprecation plan, based\non previous examples in Git.\n\nThanks,\n-Stolee\n"},{"id":"477802","messageId":"20230530172646.2781197-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"a4d22518-d3e6-76aa-fe94-2ec77a1de96a@github.com","subject":"Re: [PATCH 2/2] commit-graph: fix murmur3, bump filter ver. to 2","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-30T17:26:46Z","receivedAt":"2023-05-30T17:27:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Derrick Stolee <derrickstolee@github.com> writes:\n> I'd be willing to modify my suggested steps:\n> \n> >>> 1. v2.42.0 includes writing v2 format, off by default.\n> >>> 2. v2.43.0 writes v2 format by default.\n> >>> 3. v2.44.0 no longer parses v1 format (ignored without error).\n> \n> to something simpler:\n> \n>  1. v2.42.0 writes v2 format by default, but can be disabled by config.\n>  2. v2.43.0 no longer parses or writes v1 format.\n> \n> With this, we could proactively set the config value that disables the\n> v2 format in our production environment, then slowly re-enable that\n> config after the binaries have deployed. This allows us to limit the\n> blast radius if something goes wrong, which is really important.\n> \n> Further, I'm describing an environment where we control all of the Git\n> versions that are interacting with the repositories. Other environments\n> don't have that luxury, such as typical client users.\n> \n> Even the three-version plan is an accelerated deprecation plan, based\n> on previous examples in Git.\n> \n> Thanks,\n> -Stolee\n\nOK, let me take a look and see what this (having at least a version\nof Git that supports both versions of hash functions) would look like.\nIf we're going to have this, we might as well roll it out as safely\nas possible, so I'll aim for your original step 1 of 3 (write v2, off\nby default).\n"},{"id":"477841","messageId":"cover.1685574402.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1684790529.git.jonathantanmy@google.com","subject":"[PATCH v2 0/3] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-31T23:12:34Z","receivedAt":"2023-05-31T23:12:50Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Here's a new version. With this, Git can function with both version\n1 (incorrect murmur3) and version 2 (correct murmur3) changed path\nfilters, but not at the same time: the user can set a config variable to\nchoose which one, and Git will ignore existing changed path filters of\nthe wrong version (and always write the version that the config variable\nsays).\n\nIn patch 1, the test assumes that char is signed. I'm not sure if it's\nworth asserting on the contents of the filter, since it depends on\nwhether char is signed, but I've included it anyway (since it's easy\nto remove).\n\nJonathan Tan (3):\n  t4216: test changed path filters with high bit paths\n  repo-settings: introduce commitgraph.changedPathsVersion\n  commit-graph: new filter ver. that fixes murmur3\n\n Documentation/config/commitgraph.txt | 16 +++++--\n bloom.c                              | 65 ++++++++++++++++++++++++++--\n bloom.h                              |  8 ++--\n commit-graph.c                       | 29 ++++++++++---\n oss-fuzz/fuzz-commit-graph.c         |  2 +-\n repo-settings.c                      |  6 ++-\n repository.h                         |  2 +-\n t/helper/test-bloom.c                |  9 +++-\n t/t0095-bloom.sh                     |  8 ++++\n t/t4216-log-bloom.sh                 | 65 ++++++++++++++++++++++++++++\n 10 files changed, 192 insertions(+), 18 deletions(-)\n\nRange-diff against v1:\n1:  3a5d53d3c0 < -:  ---------- t4216: test wrong bloom filter version rejection\n2:  5a91f9682b < -:  ---------- commit-graph: fix murmur3, bump filter ver. to 2\n-:  ---------- > 1:  175dc912fe t4216: test changed path filters with high bit paths\n-:  ---------- > 2:  4a7553f3fb repo-settings: introduce commitgraph.changedPathsVersion\n-:  ---------- > 3:  f5c3f6080a commit-graph: new filter ver. that fixes murmur3\n-- \n2.41.0.rc0.172.g3f132b7071-goog\n\n"},{"id":"477842","messageId":"175dc912fe4cc7f79c5106875cc9e8c30618ab7a.1685574402.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1685574402.git.jonathantanmy@google.com","subject":"[PATCH v2 1/3] t4216: test changed path filters with high bit paths","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-31T23:12:35Z","receivedAt":"2023-05-31T23:12:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Subsequent commits will teach Git another version of changed path\nfilter that has different behavior with paths that contain at least\none character with its high bit set, so test the existing behavior as\na baseline.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n t/t4216-log-bloom.sh | 34 ++++++++++++++++++++++++++++++++++\n 1 file changed, 34 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..2ec5b5b5e7 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -404,4 +404,38 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n+get_bdat_offset () {\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"N\", \"$1\") if /BDAT\\0\\0\\0\\0(....)/ or exit 1' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n+get_first_changed_path_filter () {\n+\tBDAT_OFFSET=$(get_bdat_offset) &&\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"H*\", substr($_, '$BDAT_OFFSET' + 12, 2))' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n+# chosen to be the same under all Unicode normalization forms\n+CENT=$(printf \"\\xc2\\xa2\")\n+\n+test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n+\tgit init highbit1 &&\n+\ttest_commit -C highbit1 c1 \"$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'check value of version 1 changed-path' '\n+\t(cd highbit1 &&\n+\t\tprintf \"52a9\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+test_expect_success 'version 1 changed-path used when version 1 requested' '\n+\t(cd highbit1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n test_done\n-- \n2.41.0.rc0.172.g3f132b7071-goog\n\n"},{"id":"477843","messageId":"4a7553f3fb69b91c80535632c80181b51fd2d015.1685574402.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1685574402.git.jonathantanmy@google.com","subject":"[PATCH v2 2/3] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-31T23:12:36Z","receivedAt":"2023-05-31T23:12:53Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A subsequent commit will introduce another version of the changed-path\nfilter in the commit graph file. In order to control which version is\nto be accepted when read (and which version to write), a config variable\nis needed.\n\nTherefore, introduce this config variable. For forwards compatibility,\nteach Git to not read commit graphs when the config variable\nis set to an unsupported version. Because we teach Git this,\ncommitgraph.readChangedPaths is now redundant, so deprecate it and\ndefine its behavior in terms of the config variable we introduce.\n\nThis commit does not change the behavior of writing (Git writes changed\npath filters when explicitly instructed regardless of any config\nvariable), but a subsequent commit will restrict Git such that it will\nonly write when commitgraph.changedPathsVersion is 0, 1, or 2.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/config/commitgraph.txt | 16 +++++++++++++---\n commit-graph.c                       |  2 +-\n oss-fuzz/fuzz-commit-graph.c         |  2 +-\n repo-settings.c                      |  6 +++++-\n repository.h                         |  2 +-\n 5 files changed, 21 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c..eaa10bf232 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -9,6 +9,16 @@ commitGraph.maxNewFilters::\n \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n \n commitGraph.readChangedPaths::\n-\tIf true, then git will use the changed-path Bloom filters in the\n-\tcommit-graph file (if it exists, and they are present). Defaults to\n-\ttrue. See linkgit:git-commit-graph[1] for more information.\n+\tDeprecated. Equivalent to changedPathsVersion=1 if true, and\n+\tchangedPathsVersion=0 if false.\n+\n+commitGraph.changedPathsVersion::\n+\tSpecifies the version of the changed-path Bloom filters that Git will read and\n+\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n+\tmatch the version set in this config variable will be ignored.\n++\n+Defaults to 1.\n++\n+If 0, git will write version 1 Bloom filters when instructed to write.\n++\n+See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c11b59f28b..bd448047f1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -399,7 +399,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_read_changed_paths) {\n+\tif (s->commit_graph_changed_paths_version == 1) {\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\ndiff --git a/oss-fuzz/fuzz-commit-graph.c b/oss-fuzz/fuzz-commit-graph.c\nindex 914026f5d8..b56731f51a 100644\n--- a/oss-fuzz/fuzz-commit-graph.c\n+++ b/oss-fuzz/fuzz-commit-graph.c\n@@ -18,7 +18,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \t * possible.\n \t */\n \tthe_repository->settings.commit_graph_generation_version = 2;\n-\tthe_repository->settings.commit_graph_read_changed_paths = 1;\n+\tthe_repository->settings.commit_graph_changed_paths_version = 1;\n \tg = parse_commit_graph(&the_repository->settings, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 3dbd3f0e2e..6cbe02681b 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n \tint value;\n \tconst char *strval;\n \tint manyfiles;\n+\tint readChangedPaths;\n \n \tif (!r->gitdir)\n \t\tBUG(\"Cannot add settings for uninitialized repository\");\n@@ -54,7 +55,10 @@ void prepare_repo_settings(struct repository *r)\n \t/* Commit graph config or default, does not cascade (simple) */\n \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n-\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &readChangedPaths, 1);\n+\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n+\t\t     &r->settings.commit_graph_changed_paths_version,\n+\t\t     readChangedPaths ? 1 : 0);\n \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n \ndiff --git a/repository.h b/repository.h\nindex e8c67ffe16..1f1c32a6dd 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -32,7 +32,7 @@ struct repo_settings {\n \n \tint core_commit_graph;\n \tint commit_graph_generation_version;\n-\tint commit_graph_read_changed_paths;\n+\tint commit_graph_changed_paths_version;\n \tint gc_write_commit_graph;\n \tint gc_cruft_packs;\n \tint fetch_write_commit_graph;\n-- \n2.41.0.rc0.172.g3f132b7071-goog\n\n"},{"id":"477844","messageId":"f5c3f6080af540b004cc140b97c62a13f6ee163a.1685574402.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1685574402.git.jonathantanmy@google.com","subject":"[PATCH v2 3/3] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-05-31T23:12:37Z","receivedAt":"2023-05-31T23:13:00Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 and even with\nGit binaries that were compiled with different signedness of char. This\nbug affects both how Git writes changed path filters to disk and how Git\ninterprets changed path filters on disk.\n\nTherefore, introduce a new version (2) of changed path filters that\ncorrects this problem. The existing version (1) is still supported and\nis still the default, but users should migrate away from it as soon\nas possible.\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter.\n\nSo this patch does not include any mechanism to \"salvage\" changed path\nfilters from repositories. There is also no \"mixed\" mode - for each\ninvocation of Git, reading and writing changed path filters are done\nwith the same version number.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in t0095 was obtained from another murmur3 implementation\nusing the following Go source code:\n\n  package main\n\n  import \"fmt\"\n  import \"github.com/spaolacci/murmur3\"\n\n  func main() {\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n  }\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/config/commitgraph.txt |  2 +-\n bloom.c                              | 65 ++++++++++++++++++++++++++--\n bloom.h                              |  8 ++--\n commit-graph.c                       | 29 ++++++++++---\n t/helper/test-bloom.c                |  9 +++-\n t/t0095-bloom.sh                     |  8 ++++\n t/t4216-log-bloom.sh                 | 31 +++++++++++++\n 7 files changed, 139 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex eaa10bf232..c64ee4f459 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -14,7 +14,7 @@ commitGraph.readChangedPaths::\n \n commitGraph.changedPathsVersion::\n \tSpecifies the version of the changed-path Bloom filters that Git will read and\n-\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n+\twrite. May be 0, 1, or 2. Any changed-path Bloom filters on disk that do not\n \tmatch the version set in this config variable will be ignored.\n +\n Defaults to 1.\ndiff --git a/bloom.c b/bloom.c\nindex d0730525da..915d8e5a31 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -65,7 +65,64 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n+{\n+\tconst uint32_t c1 = 0xcc9e2d51;\n+\tconst uint32_t c2 = 0x1b873593;\n+\tconst uint32_t r1 = 15;\n+\tconst uint32_t r2 = 13;\n+\tconst uint32_t m = 5;\n+\tconst uint32_t n = 0xe6546b64;\n+\tint i;\n+\tuint32_t k1 = 0;\n+\tconst char *tail;\n+\n+\tint len4 = len / sizeof(uint32_t);\n+\n+\tuint32_t k;\n+\tfor (i = 0; i < len4; i++) {\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n+\t\tk = byte1 | byte2 | byte3 | byte4;\n+\t\tk *= c1;\n+\t\tk = rotate_left(k, r1);\n+\t\tk *= c2;\n+\n+\t\tseed ^= k;\n+\t\tseed = rotate_left(seed, r2) * m + n;\n+\t}\n+\n+\ttail = (data + len4 * sizeof(uint32_t));\n+\n+\tswitch (len & (sizeof(uint32_t) - 1)) {\n+\tcase 3:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n+\t\t/*-fallthrough*/\n+\tcase 2:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n+\t\t/*-fallthrough*/\n+\tcase 1:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n+\t\tk1 *= c1;\n+\t\tk1 = rotate_left(k1, r1);\n+\t\tk1 *= c2;\n+\t\tseed ^= k1;\n+\t\tbreak;\n+\t}\n+\n+\tseed ^= (uint32_t)len;\n+\tseed ^= (seed >> 16);\n+\tseed *= 0x85ebca6b;\n+\tseed ^= (seed >> 13);\n+\tseed *= 0xc2b2ae35;\n+\tseed ^= (seed >> 16);\n+\n+\treturn seed;\n+}\n+\n+static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n {\n \tconst uint32_t c1 = 0xcc9e2d51;\n \tconst uint32_t c2 = 0x1b873593;\n@@ -130,8 +187,10 @@ void fill_bloom_key(const char *data,\n \tint i;\n \tconst uint32_t seed0 = 0x293ae76f;\n \tconst uint32_t seed1 = 0x7e646e2c;\n-\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n-\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n+\tconst uint32_t hash0 = (settings->hash_version == 2\n+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed0, data, len);\n+\tconst uint32_t hash1 = (settings->hash_version == 2\n+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed1, data, len);\n \n \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n \tfor (i = 0; i < settings->num_hashes; i++)\ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..0c33ae282c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -7,9 +7,11 @@ struct repository;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * The newest version is 2, which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -75,7 +77,7 @@ struct bloom_key {\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n \n void fill_bloom_key(const char *data,\n \t\t    size_t len,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex bd448047f1..36e6d09e74 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -302,15 +302,21 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct graph_read_bloom_data_data {\n+\tstruct commit_graph *g;\n+\tint commit_graph_changed_paths_version;\n+};\n+\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t  size_t chunk_size, void *data)\n {\n-\tstruct commit_graph *g = data;\n+\tstruct graph_read_bloom_data_data *d = data;\n+\tstruct commit_graph *g = d->g;\n \tuint32_t hash_version;\n \tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n+\tif (hash_version != d->commit_graph_changed_paths_version)\n \t\treturn 0;\n \n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n@@ -399,11 +405,16 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_changed_paths_version == 1) {\n+\tif (s->commit_graph_changed_paths_version == 1\n+\t    || s->commit_graph_changed_paths_version == 2) {\n+\t\tstruct graph_read_bloom_data_data data = {\n+\t\t\t.g = graph,\n+\t\t\t.commit_graph_changed_paths_version = s->commit_graph_changed_paths_version\n+\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t   graph_read_bloom_data, graph);\n+\t\t\t   graph_read_bloom_data, &data);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -2302,6 +2313,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tif (r->settings.commit_graph_changed_paths_version < 0\n+\t    || r->settings.commit_graph_changed_paths_version > 2) {\n+\t\twarning(_(\"attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported\"),\n+\t\t\tr->settings.commit_graph_changed_paths_version);\n+\t\treturn 0;\n+\t}\n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n+\t\t? 2 : 1;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t      bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2331,7 +2350,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 6c900ca668..34b8dd9164 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -48,6 +48,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \"  test-tool bloom get_murmur3 <string>\\n\"\n+\"  test-tool bloom get_murmur3_seven_highbit\\n\"\n \"  test-tool bloom generate_filter <string> [<string>...]\\n\"\n \"  test-tool bloom get_filter_for_commit <commit-hex>\\n\";\n \n@@ -62,7 +63,13 @@ int cmd__bloom(int argc, const char **argv)\n \t\tuint32_t hashed;\n \t\tif (argc < 3)\n \t\t\tusage(bloom_usage);\n-\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n+\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 2ec5b5b5e7..764c6dee0f 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -438,4 +438,35 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n \t\ttest_bloom_filters_used \"-- $CENT\")\n '\n \n+test_expect_success 'version 1 changed-path not used when version 2 requested' '\n+\t(cd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n+\tgit init highbit2 &&\n+\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n+\ttest_commit -C highbit2 c2 \"$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'check value of version 2 changed-path' '\n+\t(cd highbit2 &&\n+\t\tprintf \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+test_expect_success 'version 2 changed-path used when version 2 requested' '\n+\t(cd highbit2 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'version 2 changed-path not used when version 1 requested' '\n+\t(cd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n test_done\n-- \n2.41.0.rc0.172.g3f132b7071-goog\n\n"},{"id":"477996","messageId":"xmqq5y858ikd.fsf@gitster.g","threadId":"59781","inReplyTo":"cover.1685574402.git.jonathantanmy@google.com","subject":"Re: [PATCH v2 0/3] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-03T01:01:54Z","receivedAt":"2023-06-03T01:01:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Here's a new version. With this, Git can function with both version\n> 1 (incorrect murmur3) and version 2 (correct murmur3) changed path\n> filters, but not at the same time: the user can set a config variable to\n> choose which one, and Git will ignore existing changed path filters of\n> the wrong version (and always write the version that the config variable\n> says).\n\nHmph.  On a system with unsigned char, we should be able to keep\nusing version 1 without losing correctness, I suspect, but probably\nthey are in the minority we do not have to care about?  I can see\nthe desire to simplify the migration plan (i.e. essentially have no\nmigration---this will give us just a flag day per repository), but\nI'll let others to comment.\n\n> In patch 1, the test assumes that char is signed. I'm not sure if it's\n> worth asserting on the contents of the filter, since it depends on\n> whether char is signed, but I've included it anyway (since it's easy\n> to remove).\n\nSo, on a system with unsigned char, would these tests fail?  Do we\nneed a prereq to skip them?\n\nThanks.\n"},{"id":"478010","messageId":"xmqqa5xh705s.fsf@gitster.g","threadId":"59781","inReplyTo":"xmqq5y858ikd.fsf@gitster.g","subject":"Re: [PATCH v2 0/3] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-03T02:24:47Z","receivedAt":"2023-06-03T02:25:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> Jonathan Tan <jonathantanmy@google.com> writes:\n>\n>> Here's a new version. With this, Git can function with both version\n>> 1 (incorrect murmur3) and version 2 (correct murmur3) changed path\n>> filters, but not at the same time: the user can set a config variable to\n>> choose which one, and Git will ignore existing changed path filters of\n>> the wrong version (and always write the version that the config variable\n>> says).\n\nSeems to break t4216 when merged to 'seen' to replace the previous\nround.  Could you take a look?\n"},{"id":"478146","messageId":"20230607163056.1602219-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"xmqqa5xh705s.fsf@gitster.g","subject":"Re: [PATCH v2 0/3] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-07T16:30:56Z","receivedAt":"2023-06-07T16:31:42Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> Junio C Hamano <gitster@pobox.com> writes:\n> \n> > Jonathan Tan <jonathantanmy@google.com> writes:\n> >\n> >> Here's a new version. With this, Git can function with both version\n> >> 1 (incorrect murmur3) and version 2 (correct murmur3) changed path\n> >> filters, but not at the same time: the user can set a config variable to\n> >> choose which one, and Git will ignore existing changed path filters of\n> >> the wrong version (and always write the version that the config variable\n> >> says).\n> \n> Seems to break t4216 when merged to 'seen' to replace the previous\n> round.  Could you take a look?\n\nOK - I see that it fails CI when I upload the merge to GitHub (although\nit passes locally). I'll take a look.\n"},{"id":"478153","messageId":"20230607213712.2589505-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"20230607163056.1602219-1-jonathantanmy@google.com","subject":"Re: [PATCH v2 0/3] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-07T21:37:12Z","receivedAt":"2023-06-07T21:37:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n> Junio C Hamano <gitster@pobox.com> writes:\n> > Seems to break t4216 when merged to 'seen' to replace the previous\n> > round.  Could you take a look?\n> \n> OK - I see that it fails CI when I upload the merge to GitHub (although\n> it passes locally). I'll take a look.\n\nHmm...so when the test writes a file with a high bit filename, the\nfilename written is different. Here's the output from my machine\n(Linux):\n\n        ++ git -C highbit1/ commit -m c1                                                                                                                          \n        [main (root-commit) 808e481] c1                                                                                                                           \n         Author: A U Thor <author@example.com>                                                                                                                    \n         1 file changed, 1 insertion(+)                                                                                                                           \n         create mode 100644 \"\\302\\242\"                                                                                                                            \n        ++ case \"$tag\" in                                                                                                                                         \n        ++ git -C highbit1/ tag c1                                                                                                                                \n        ++ touch $'\\302\\242\\302\\242'                                                                                                                              \n        ++ ls                                                                                                                                                     \n         A        expect   file5_renamed   limits        log_wo_bloom   trace2-auto.txt  ''$'\\302\\242\\302\\242'                                                    \n         actual   file4    highbit1        log_w_bloom   trace.perf     trace2.txt                                                                                \n        ++ git -C highbit1 commit-graph write --reachable --changed-paths                                                                                         \n        bloom calculating -62,-94 version 1                                                                                                                       \n        ok 141 - set up repo with high bit path, version 1 changed-path       \n\nand here's the output from the CI on GitHub on linux-gcc-default:\n\n        [main (root-commit) 3f37a43] c1\n         Author: A U Thor <author@example.com>\n         1 file changed, 1 insertion(+)\n         create mode 100644 \"\\\\xc2\\\\xa2\"\n        + git -C highbit1/ tag c1\n        + touch \\xc2\\xa2\\xc2\\xa2\n        + ls\n        A\n        \\xc2\\xa2\\xc2\\xa2\n        actual\n        expect\n        file4\n        file5_renamed\n        highbit1\n        limits\n        log_w_bloom\n        log_wo_bloom\n        trace.perf\n        trace2-auto.txt\n        trace2.txt\n        + git -C highbit1 commit-graph write --reachable --changed-paths\n        bloom calculating 92,120 version 1\n\n        ok 141 - set up repo with high bit path, version 1 changed-path\n\nThese were run with some extra code, here for completeness:\n\n        diff --git a/bloom.c b/bloom.c\n        index dea883d8d6..ccc3e0ce80 100644\n        --- a/bloom.c\n        +++ b/bloom.c\n        @@ -192,6 +192,8 @@ void fill_bloom_key(const char *data,\n                        ? murmur3_seeded_v2 : murmur3_seeded_v1)(seed0, data, len);\n                const uint32_t hash1 = (settings->hash_version == 2\n                        ? murmur3_seeded_v2 : murmur3_seeded_v1)(seed1, data, len);\n        +       if (len > 0)\n        +               fprintf(stderr, \"bloom calculating %d,%d version %d\\n\", data[0], data[1], settings->hash_version);\n \n                key->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n                for (i = 0; i < settings->num_hashes; i++)\n        diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n        index 764c6dee0f..1121e7d4cd 100755\n        --- a/t/t4216-log-bloom.sh\n        +++ b/t/t4216-log-bloom.sh\n        @@ -423,6 +423,8 @@ CENT=$(printf \"\\xc2\\xa2\")\n         test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n                git init highbit1 &&\n                test_commit -C highbit1 c1 \"$CENT\" &&\n        +       touch $CENT$CENT &&\n        +       ls &&\n                git -C highbit1 commit-graph write --reachable --changed-paths\n         '\n\nNotice the different filename after \"create mode\", and the different\n\"ls\" output. I'll investigate some more, but if anyone offhand knows\nwhat's going on (and/or knows how to write a high-bit filename portably,\neven across  Linux), please let me know.\n\nAn alternative is to drop the tests from these patches - so, leave them\nin during the review period and reviewers would see that the tests pass\nthe CI jobs for Windows and Mac OS X pass, and then before we merge it,\ndelete the tests from the patches. This also solves needing to prevent\nunsigned char platforms from running the version 1 tests - there's no\nprereq for that yet and we would have to investigate making one.\n"},{"id":"478173","messageId":"cover.1686251688.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1684790529.git.jonathantanmy@google.com","subject":"[PATCH v3 0/4] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-08T19:21:21Z","receivedAt":"2023-06-08T19:22:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Here's an updated version with changes only in the tests:\n - resilience against unsigned-by-default (a \"skip\" will be shown)\n - for systems that have different quoting behavior, the tests\n   will be skipped (see patch 2 for more information)\n\nSome of the test changes may seem a bit of a hack, so if you have\na better way of doing things, please let me know.\n\nI've also included a change to the file format specification. To\nreviewers, if you generally agree that we need a version 2 but are still\nunsure about how we should migrate, consider saying so and then perhaps\nwe can merge the first patch while the rest remain under review. This\nwill give other projects, like JGit, more certainty as to the direction\nthat the Git project wants to take.\n\nJonathan Tan (4):\n  gitformat-commit-graph: describe version 2 of BDAT\n  t4216: test changed path filters with high bit paths\n  repo-settings: introduce commitgraph.changedPathsVersion\n  commit-graph: new filter ver. that fixes murmur3\n\n Documentation/config/commitgraph.txt     | 16 ++++-\n Documentation/gitformat-commit-graph.txt |  9 ++-\n bloom.c                                  | 65 +++++++++++++++++-\n bloom.h                                  |  8 ++-\n commit-graph.c                           | 29 ++++++--\n oss-fuzz/fuzz-commit-graph.c             |  2 +-\n repo-settings.c                          |  6 +-\n repository.h                             |  2 +-\n t/helper/test-bloom.c                    |  9 ++-\n t/t0095-bloom.sh                         |  8 +++\n t/t4216-log-bloom.sh                     | 86 ++++++++++++++++++++++++\n 11 files changed, 219 insertions(+), 21 deletions(-)\n\nRange-diff against v2:\n-:  ---------- > 1:  d4b63945f6 gitformat-commit-graph: describe version 2 of BDAT\n1:  c587eb3470 ! 2:  aa4535776e t4216: test changed path filters with high bit paths\n    @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm\n     +# chosen to be the same under all Unicode normalization forms\n     +CENT=$(printf \"\\xc2\\xa2\")\n     +\n    -+test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n    ++# Some systems (in particular, Linux on the CI running on GitHub at the time of\n    ++# writing) store into CENT a literal backslash, then \"x\", and so on (instead of\n    ++# the high-bit characters needed). In these systems, do not run the following\n    ++# tests.\n    ++if test \"$(printf $CENT | perl -0777 -ne 'no utf8; print ord($_)')\" = \"194\"\n    ++then\n    ++\ttest_set_prereq HIGH_BIT\n    ++fi\n    ++\n    ++test_expect_success HIGH_BIT 'set up repo with high bit path, version 1 changed-path' '\n     +\tgit init highbit1 &&\n     +\ttest_commit -C highbit1 c1 \"$CENT\" &&\n     +\tgit -C highbit1 commit-graph write --reachable --changed-paths\n     +'\n     +\n    -+test_expect_success 'check value of version 1 changed-path' '\n    ++test_expect_success HIGH_BIT 'setup check value of version 1 changed-path' '\n     +\t(cd highbit1 &&\n     +\t\tprintf \"52a9\" >expect &&\n    -+\t\tget_first_changed_path_filter >actual &&\n    -+\t\ttest_cmp expect actual)\n    ++\t\tget_first_changed_path_filter >actual)\n    ++'\n    ++\n    ++# expect will not match actual if int is unsigned by default. Write the test\n    ++# in this way, so that a user running this test script can still see if the two\n    ++# files match. (It will appear as an ordinary success if they match, and a skip\n    ++# if not.)\n    ++if test_cmp highbit1/expect highbit1/actual\n    ++then\n    ++\ttest_set_prereq SIGNED_INT_BY_DEFAULT\n    ++fi\n    ++test_expect_success SIGNED_INT_BY_DEFAULT 'check value of version 1 changed-path' '\n    ++\t# Only the prereq matters for this test.\n    ++\ttrue\n     +'\n     +\n    -+test_expect_success 'version 1 changed-path used when version 1 requested' '\n    ++test_expect_success HIGH_BIT 'version 1 changed-path used when version 1 requested' '\n     +\t(cd highbit1 &&\n     +\t\ttest_bloom_filters_used \"-- $CENT\")\n     +'\n2:  d0e5dd20dc = 3:  d6982268a4 repo-settings: introduce commitgraph.changedPathsVersion\n3:  eb19f8a35b ! 4:  e879483c42 commit-graph: new filter ver. that fixes murmur3\n    @@ t/t0095-bloom.sh: test_expect_success 'compute unseeded murmur3 hash for test st\n      \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\n     \n      ## t/t4216-log-bloom.sh ##\n    -@@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when version 1 requested' '\n    +@@ t/t4216-log-bloom.sh: test_expect_success HIGH_BIT 'version 1 changed-path used when version 1 request\n      \t\ttest_bloom_filters_used \"-- $CENT\")\n      '\n      \n    -+test_expect_success 'version 1 changed-path not used when version 2 requested' '\n    ++test_expect_success HIGH_BIT 'version 1 changed-path not used when version 2 requested' '\n     +\t(cd highbit1 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n     +\t\ttest_bloom_filters_not_used \"-- $CENT\")\n     +'\n     +\n    -+test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n    ++test_expect_success HIGH_BIT 'set up repo with high bit path, version 2 changed-path' '\n     +\tgit init highbit2 &&\n     +\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n     +\ttest_commit -C highbit2 c2 \"$CENT\" &&\n     +\tgit -C highbit2 commit-graph write --reachable --changed-paths\n     +'\n     +\n    -+test_expect_success 'check value of version 2 changed-path' '\n    ++test_expect_success HIGH_BIT 'check value of version 2 changed-path' '\n     +\t(cd highbit2 &&\n     +\t\tprintf \"c01f\" >expect &&\n     +\t\tget_first_changed_path_filter >actual &&\n     +\t\ttest_cmp expect actual)\n     +'\n     +\n    -+test_expect_success 'version 2 changed-path used when version 2 requested' '\n    ++test_expect_success HIGH_BIT 'version 2 changed-path used when version 2 requested' '\n     +\t(cd highbit2 &&\n     +\t\ttest_bloom_filters_used \"-- $CENT\")\n     +'\n     +\n    -+test_expect_success 'version 2 changed-path not used when version 1 requested' '\n    ++test_expect_success HIGH_BIT 'version 2 changed-path not used when version 1 requested' '\n     +\t(cd highbit2 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n     +\t\ttest_bloom_filters_not_used \"-- $CENT\")\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478174","messageId":"d4b63945f6d52e68a4760393f787acde0fccc312.1686251688.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1686251688.git.jonathantanmy@google.com","subject":"[PATCH v3 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-08T19:21:22Z","receivedAt":"2023-06-08T19:22:17Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The code change to Git to support version 2 will be done in subsequent\ncommits.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/gitformat-commit-graph.txt | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\nindex 31cad585e2..9dab222830 100644\n--- a/Documentation/gitformat-commit-graph.txt\n+++ b/Documentation/gitformat-commit-graph.txt\n@@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n \n ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n     * It starts with header consisting of three unsigned 32-bit integers:\n-      - Version of the hash algorithm being used. We currently only support\n-\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n+      - Version of the hash algorithm being used. We currently support\n+\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n \timplemented exactly as described in\n \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n-\tin Probabilistic Verification\"\n+\tin Probabilistic Verification\". Version 1 bloom filters have a bug that appears\n+\twhen int is signed and the repository has path names that have characters >=\n+\t0x80; Git supports reading and writing them, but this ability will be removed\n+\tin a future version of Git.\n       - The number of times a path is hashed and hence the number of bit positions\n \t      that cumulatively determine whether a file is present in the commit.\n       - The minimum number of bits 'b' per entry in the Bloom filter. If the filter\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478175","messageId":"aa4535776e08004e86e2a3061e4a45c466d89024.1686251688.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1686251688.git.jonathantanmy@google.com","subject":"[PATCH v3 2/4] t4216: test changed path filters with high bit paths","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-08T19:21:23Z","receivedAt":"2023-06-08T19:22:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Subsequent commits will teach Git another version of changed path\nfilter that has different behavior with paths that contain at least\none character with its high bit set, so test the existing behavior as\na baseline.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n t/t4216-log-bloom.sh | 55 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 55 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..f68df24bd5 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -404,4 +404,59 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n+get_bdat_offset () {\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"N\", \"$1\") if /BDAT\\0\\0\\0\\0(....)/ or exit 1' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n+get_first_changed_path_filter () {\n+\tBDAT_OFFSET=$(get_bdat_offset) &&\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"H*\", substr($_, '$BDAT_OFFSET' + 12, 2))' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n+# chosen to be the same under all Unicode normalization forms\n+CENT=$(printf \"\\xc2\\xa2\")\n+\n+# Some systems (in particular, Linux on the CI running on GitHub at the time of\n+# writing) store into CENT a literal backslash, then \"x\", and so on (instead of\n+# the high-bit characters needed). In these systems, do not run the following\n+# tests.\n+if test \"$(printf $CENT | perl -0777 -ne 'no utf8; print ord($_)')\" = \"194\"\n+then\n+\ttest_set_prereq HIGH_BIT\n+fi\n+\n+test_expect_success HIGH_BIT 'set up repo with high bit path, version 1 changed-path' '\n+\tgit init highbit1 &&\n+\ttest_commit -C highbit1 c1 \"$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success HIGH_BIT 'setup check value of version 1 changed-path' '\n+\t(cd highbit1 &&\n+\t\tprintf \"52a9\" >expect &&\n+\t\tget_first_changed_path_filter >actual)\n+'\n+\n+# expect will not match actual if int is unsigned by default. Write the test\n+# in this way, so that a user running this test script can still see if the two\n+# files match. (It will appear as an ordinary success if they match, and a skip\n+# if not.)\n+if test_cmp highbit1/expect highbit1/actual\n+then\n+\ttest_set_prereq SIGNED_INT_BY_DEFAULT\n+fi\n+test_expect_success SIGNED_INT_BY_DEFAULT 'check value of version 1 changed-path' '\n+\t# Only the prereq matters for this test.\n+\ttrue\n+'\n+\n+test_expect_success HIGH_BIT 'version 1 changed-path used when version 1 requested' '\n+\t(cd highbit1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n test_done\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478176","messageId":"d6982268a427de9d2e56f4a4dcdd200ea5adcdcc.1686251688.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1686251688.git.jonathantanmy@google.com","subject":"[PATCH v3 3/4] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-08T19:21:24Z","receivedAt":"2023-06-08T19:22:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A subsequent commit will introduce another version of the changed-path\nfilter in the commit graph file. In order to control which version is\nto be accepted when read (and which version to write), a config variable\nis needed.\n\nTherefore, introduce this config variable. For forwards compatibility,\nteach Git to not read commit graphs when the config variable\nis set to an unsupported version. Because we teach Git this,\ncommitgraph.readChangedPaths is now redundant, so deprecate it and\ndefine its behavior in terms of the config variable we introduce.\n\nThis commit does not change the behavior of writing (Git writes changed\npath filters when explicitly instructed regardless of any config\nvariable), but a subsequent commit will restrict Git such that it will\nonly write when commitgraph.changedPathsVersion is 0, 1, or 2.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt | 16 +++++++++++++---\n commit-graph.c                       |  2 +-\n oss-fuzz/fuzz-commit-graph.c         |  2 +-\n repo-settings.c                      |  6 +++++-\n repository.h                         |  2 +-\n 5 files changed, 21 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c..eaa10bf232 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -9,6 +9,16 @@ commitGraph.maxNewFilters::\n \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n \n commitGraph.readChangedPaths::\n-\tIf true, then git will use the changed-path Bloom filters in the\n-\tcommit-graph file (if it exists, and they are present). Defaults to\n-\ttrue. See linkgit:git-commit-graph[1] for more information.\n+\tDeprecated. Equivalent to changedPathsVersion=1 if true, and\n+\tchangedPathsVersion=0 if false.\n+\n+commitGraph.changedPathsVersion::\n+\tSpecifies the version of the changed-path Bloom filters that Git will read and\n+\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n+\tmatch the version set in this config variable will be ignored.\n++\n+Defaults to 1.\n++\n+If 0, git will write version 1 Bloom filters when instructed to write.\n++\n+See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c11b59f28b..bd448047f1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -399,7 +399,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_read_changed_paths) {\n+\tif (s->commit_graph_changed_paths_version == 1) {\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\ndiff --git a/oss-fuzz/fuzz-commit-graph.c b/oss-fuzz/fuzz-commit-graph.c\nindex 914026f5d8..b56731f51a 100644\n--- a/oss-fuzz/fuzz-commit-graph.c\n+++ b/oss-fuzz/fuzz-commit-graph.c\n@@ -18,7 +18,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \t * possible.\n \t */\n \tthe_repository->settings.commit_graph_generation_version = 2;\n-\tthe_repository->settings.commit_graph_read_changed_paths = 1;\n+\tthe_repository->settings.commit_graph_changed_paths_version = 1;\n \tg = parse_commit_graph(&the_repository->settings, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 3dbd3f0e2e..6cbe02681b 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n \tint value;\n \tconst char *strval;\n \tint manyfiles;\n+\tint readChangedPaths;\n \n \tif (!r->gitdir)\n \t\tBUG(\"Cannot add settings for uninitialized repository\");\n@@ -54,7 +55,10 @@ void prepare_repo_settings(struct repository *r)\n \t/* Commit graph config or default, does not cascade (simple) */\n \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n-\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &readChangedPaths, 1);\n+\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n+\t\t     &r->settings.commit_graph_changed_paths_version,\n+\t\t     readChangedPaths ? 1 : 0);\n \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n \ndiff --git a/repository.h b/repository.h\nindex e8c67ffe16..1f1c32a6dd 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -32,7 +32,7 @@ struct repo_settings {\n \n \tint core_commit_graph;\n \tint commit_graph_generation_version;\n-\tint commit_graph_read_changed_paths;\n+\tint commit_graph_changed_paths_version;\n \tint gc_write_commit_graph;\n \tint gc_cruft_packs;\n \tint fetch_write_commit_graph;\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478177","messageId":"e879483c42de77509740e38c1caa76a08e56a55f.1686251688.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1686251688.git.jonathantanmy@google.com","subject":"[PATCH v3 4/4] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-08T19:21:25Z","receivedAt":"2023-06-08T19:22:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 and even with\nGit binaries that were compiled with different signedness of char. This\nbug affects both how Git writes changed path filters to disk and how Git\ninterprets changed path filters on disk.\n\nTherefore, introduce a new version (2) of changed path filters that\ncorrects this problem. The existing version (1) is still supported and\nis still the default, but users should migrate away from it as soon\nas possible.\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter.\n\nSo this patch does not include any mechanism to \"salvage\" changed path\nfilters from repositories. There is also no \"mixed\" mode - for each\ninvocation of Git, reading and writing changed path filters are done\nwith the same version number.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in t0095 was obtained from another murmur3 implementation\nusing the following Go source code:\n\n  package main\n\n  import \"fmt\"\n  import \"github.com/spaolacci/murmur3\"\n\n  func main() {\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n  }\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt |  2 +-\n bloom.c                              | 65 ++++++++++++++++++++++++++--\n bloom.h                              |  8 ++--\n commit-graph.c                       | 29 ++++++++++---\n t/helper/test-bloom.c                |  9 +++-\n t/t0095-bloom.sh                     |  8 ++++\n t/t4216-log-bloom.sh                 | 31 +++++++++++++\n 7 files changed, 139 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex eaa10bf232..c64ee4f459 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -14,7 +14,7 @@ commitGraph.readChangedPaths::\n \n commitGraph.changedPathsVersion::\n \tSpecifies the version of the changed-path Bloom filters that Git will read and\n-\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n+\twrite. May be 0, 1, or 2. Any changed-path Bloom filters on disk that do not\n \tmatch the version set in this config variable will be ignored.\n +\n Defaults to 1.\ndiff --git a/bloom.c b/bloom.c\nindex d0730525da..915d8e5a31 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -65,7 +65,64 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n+{\n+\tconst uint32_t c1 = 0xcc9e2d51;\n+\tconst uint32_t c2 = 0x1b873593;\n+\tconst uint32_t r1 = 15;\n+\tconst uint32_t r2 = 13;\n+\tconst uint32_t m = 5;\n+\tconst uint32_t n = 0xe6546b64;\n+\tint i;\n+\tuint32_t k1 = 0;\n+\tconst char *tail;\n+\n+\tint len4 = len / sizeof(uint32_t);\n+\n+\tuint32_t k;\n+\tfor (i = 0; i < len4; i++) {\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n+\t\tk = byte1 | byte2 | byte3 | byte4;\n+\t\tk *= c1;\n+\t\tk = rotate_left(k, r1);\n+\t\tk *= c2;\n+\n+\t\tseed ^= k;\n+\t\tseed = rotate_left(seed, r2) * m + n;\n+\t}\n+\n+\ttail = (data + len4 * sizeof(uint32_t));\n+\n+\tswitch (len & (sizeof(uint32_t) - 1)) {\n+\tcase 3:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n+\t\t/*-fallthrough*/\n+\tcase 2:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n+\t\t/*-fallthrough*/\n+\tcase 1:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n+\t\tk1 *= c1;\n+\t\tk1 = rotate_left(k1, r1);\n+\t\tk1 *= c2;\n+\t\tseed ^= k1;\n+\t\tbreak;\n+\t}\n+\n+\tseed ^= (uint32_t)len;\n+\tseed ^= (seed >> 16);\n+\tseed *= 0x85ebca6b;\n+\tseed ^= (seed >> 13);\n+\tseed *= 0xc2b2ae35;\n+\tseed ^= (seed >> 16);\n+\n+\treturn seed;\n+}\n+\n+static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n {\n \tconst uint32_t c1 = 0xcc9e2d51;\n \tconst uint32_t c2 = 0x1b873593;\n@@ -130,8 +187,10 @@ void fill_bloom_key(const char *data,\n \tint i;\n \tconst uint32_t seed0 = 0x293ae76f;\n \tconst uint32_t seed1 = 0x7e646e2c;\n-\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n-\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n+\tconst uint32_t hash0 = (settings->hash_version == 2\n+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed0, data, len);\n+\tconst uint32_t hash1 = (settings->hash_version == 2\n+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed1, data, len);\n \n \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n \tfor (i = 0; i < settings->num_hashes; i++)\ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..0c33ae282c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -7,9 +7,11 @@ struct repository;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * The newest version is 2, which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -75,7 +77,7 @@ struct bloom_key {\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n \n void fill_bloom_key(const char *data,\n \t\t    size_t len,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex bd448047f1..36e6d09e74 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -302,15 +302,21 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct graph_read_bloom_data_data {\n+\tstruct commit_graph *g;\n+\tint commit_graph_changed_paths_version;\n+};\n+\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t  size_t chunk_size, void *data)\n {\n-\tstruct commit_graph *g = data;\n+\tstruct graph_read_bloom_data_data *d = data;\n+\tstruct commit_graph *g = d->g;\n \tuint32_t hash_version;\n \tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n+\tif (hash_version != d->commit_graph_changed_paths_version)\n \t\treturn 0;\n \n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n@@ -399,11 +405,16 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_changed_paths_version == 1) {\n+\tif (s->commit_graph_changed_paths_version == 1\n+\t    || s->commit_graph_changed_paths_version == 2) {\n+\t\tstruct graph_read_bloom_data_data data = {\n+\t\t\t.g = graph,\n+\t\t\t.commit_graph_changed_paths_version = s->commit_graph_changed_paths_version\n+\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t   graph_read_bloom_data, graph);\n+\t\t\t   graph_read_bloom_data, &data);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -2302,6 +2313,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tif (r->settings.commit_graph_changed_paths_version < 0\n+\t    || r->settings.commit_graph_changed_paths_version > 2) {\n+\t\twarning(_(\"attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported\"),\n+\t\t\tr->settings.commit_graph_changed_paths_version);\n+\t\treturn 0;\n+\t}\n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n+\t\t? 2 : 1;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t      bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2331,7 +2350,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 6c900ca668..34b8dd9164 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -48,6 +48,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \"  test-tool bloom get_murmur3 <string>\\n\"\n+\"  test-tool bloom get_murmur3_seven_highbit\\n\"\n \"  test-tool bloom generate_filter <string> [<string>...]\\n\"\n \"  test-tool bloom get_filter_for_commit <commit-hex>\\n\";\n \n@@ -62,7 +63,13 @@ int cmd__bloom(int argc, const char **argv)\n \t\tuint32_t hashed;\n \t\tif (argc < 3)\n \t\t\tusage(bloom_usage);\n-\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n+\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex f68df24bd5..4c8f039eb4 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -459,4 +459,35 @@ test_expect_success HIGH_BIT 'version 1 changed-path used when version 1 request\n \t\ttest_bloom_filters_used \"-- $CENT\")\n '\n \n+test_expect_success HIGH_BIT 'version 1 changed-path not used when version 2 requested' '\n+\t(cd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n+test_expect_success HIGH_BIT 'set up repo with high bit path, version 2 changed-path' '\n+\tgit init highbit2 &&\n+\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n+\ttest_commit -C highbit2 c2 \"$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success HIGH_BIT 'check value of version 2 changed-path' '\n+\t(cd highbit2 &&\n+\t\tprintf \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+test_expect_success HIGH_BIT 'version 2 changed-path used when version 2 requested' '\n+\t(cd highbit2 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n+test_expect_success HIGH_BIT 'version 2 changed-path not used when version 1 requested' '\n+\t(cd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n test_done\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478179","messageId":"1d31a8a6-1e45-bc42-2485-ff39f3cea24b@ramsayjones.plus.com","threadId":"59781","inReplyTo":"d4b63945f6d52e68a4760393f787acde0fccc312.1686251688.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2023-06-08T19:52:57Z","receivedAt":"2023-06-08T19:53:07Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 08/06/2023 20:21, Jonathan Tan wrote:\n> The code change to Git to support version 2 will be done in subsequent\n> commits.\n> \n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  Documentation/gitformat-commit-graph.txt | 9 ++++++---\n>  1 file changed, 6 insertions(+), 3 deletions(-)\n> \n> diff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\n> index 31cad585e2..9dab222830 100644\n> --- a/Documentation/gitformat-commit-graph.txt\n> +++ b/Documentation/gitformat-commit-graph.txt\n> @@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n>  \n>  ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n>      * It starts with header consisting of three unsigned 32-bit integers:\n> -      - Version of the hash algorithm being used. We currently only support\n> -\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n> +      - Version of the hash algorithm being used. We currently support\n> +\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n>  \timplemented exactly as described in\n>  \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n>  \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n>  \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n> -\tin Probabilistic Verification\"\n> +\tin Probabilistic Verification\". Version 1 bloom filters have a bug that appears\n> +\twhen int is signed and the repository has path names that have characters >=\n\ns/int is signed/char is signed/ ?\n\nATB,\nRamsay Jones\n\n> +\t0x80; Git supports reading and writing them, but this ability will be removed\n> +\tin a future version of Git.\n>        - The number of times a path is hashed and hence the number of bit positions\n>  \t      that cumulatively determine whether a file is present in the commit.\n>        - The minimum number of bits 'b' per entry in the Bloom filter. If the filter\n"},{"id":"478180","messageId":"3ff7acfc-0fd2-88a8-3e31-2c6405a03f59@ramsayjones.plus.com","threadId":"59781","inReplyTo":"cover.1686251688.git.jonathantanmy@google.com","subject":"Re: [PATCH v3 0/4] Changed path filter hash fix and version bump","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2023-06-08T19:50:56Z","receivedAt":"2023-06-08T19:54:03Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 08/06/2023 20:21, Jonathan Tan wrote:\n> Here's an updated version with changes only in the tests:\n>  - resilience against unsigned-by-default (a \"skip\" will be shown)\n>  - for systems that have different quoting behavior, the tests\n>    will be skipped (see patch 2 for more information)\n> \n> Some of the test changes may seem a bit of a hack, so if you have\n> a better way of doing things, please let me know.\n> \n> I've also included a change to the file format specification. To\n> reviewers, if you generally agree that we need a version 2 but are still\n> unsure about how we should migrate, consider saying so and then perhaps\n> we can merge the first patch while the rest remain under review. This\n> will give other projects, like JGit, more certainty as to the direction\n> that the Git project wants to take.\n> \n> Jonathan Tan (4):\n>   gitformat-commit-graph: describe version 2 of BDAT\n>   t4216: test changed path filters with high bit paths\n>   repo-settings: introduce commitgraph.changedPathsVersion\n>   commit-graph: new filter ver. that fixes murmur3\n> \n>  Documentation/config/commitgraph.txt     | 16 ++++-\n>  Documentation/gitformat-commit-graph.txt |  9 ++-\n>  bloom.c                                  | 65 +++++++++++++++++-\n>  bloom.h                                  |  8 ++-\n>  commit-graph.c                           | 29 ++++++--\n>  oss-fuzz/fuzz-commit-graph.c             |  2 +-\n>  repo-settings.c                          |  6 +-\n>  repository.h                             |  2 +-\n>  t/helper/test-bloom.c                    |  9 ++-\n>  t/t0095-bloom.sh                         |  8 +++\n>  t/t4216-log-bloom.sh                     | 86 ++++++++++++++++++++++++\n>  11 files changed, 219 insertions(+), 21 deletions(-)\n> \n> Range-diff against v2:\n> -:  ---------- > 1:  d4b63945f6 gitformat-commit-graph: describe version 2 of BDAT\n> 1:  c587eb3470 ! 2:  aa4535776e t4216: test changed path filters with high bit paths\n>     @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm\n>      +# chosen to be the same under all Unicode normalization forms\n>      +CENT=$(printf \"\\xc2\\xa2\")\n\nI think the only change you need to make here (because /usr/bin/sh\non Ubuntu is usually 'dash' not 'bash') is to use octal rather than\nhexadecimal. ie: CENT=$(printf \"\\302\\242\")\n\nHTH\n\nATB,\nRamsay Jones\n\n"},{"id":"478181","messageId":"20230609000828.1762551-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"3ff7acfc-0fd2-88a8-3e31-2c6405a03f59@ramsayjones.plus.com","subject":"Re: [PATCH v3 0/4] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-09T00:08:28Z","receivedAt":"2023-06-09T00:08:34Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Ramsay Jones <ramsay@ramsayjones.plus.com> writes:\n> >      +CENT=$(printf \"\\xc2\\xa2\")\n> \n> I think the only change you need to make here (because /usr/bin/sh\n> on Ubuntu is usually 'dash' not 'bash') is to use octal rather than\n> hexadecimal. ie: CENT=$(printf \"\\302\\242\")\n> \n> HTH\n> \n> ATB,\n> Ramsay Jones\n\nThanks! Yes, it works. I've checked that it passes CI [1] and will wait\nfor other reviews before sending out a new version to the list.\n\n[1] https://github.com/jonathantanmy/git/actions/runs/5216513514\n"},{"id":"478308","messageId":"xmqqh6rc8j9u.fsf@gitster.g","threadId":"59781","inReplyTo":"1d31a8a6-1e45-bc42-2485-ff39f3cea24b@ramsayjones.plus.com","subject":"Re: [PATCH v3 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-12T21:26:05Z","receivedAt":"2023-06-12T21:27:30Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramsay Jones <ramsay@ramsayjones.plus.com> writes:\n\n>>  \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n>> -\tin Probabilistic Verification\"\n>> +\tin Probabilistic Verification\". Version 1 bloom filters have a bug that appears\n>> +\twhen int is signed and the repository has path names that have characters >=\n>\n> s/int is signed/char is signed/ ?\n\nIndeed.  Thanks for sharp eyes.  Will tweak while queuing.\n"},{"id":"478309","messageId":"xmqqa5x48j0j.fsf@gitster.g","threadId":"59781","inReplyTo":"3ff7acfc-0fd2-88a8-3e31-2c6405a03f59@ramsayjones.plus.com","subject":"Re: [PATCH v3 0/4] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-12T21:31:40Z","receivedAt":"2023-06-12T21:31:47Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ramsay Jones <ramsay@ramsayjones.plus.com> writes:\n\n> I think the only change you need to make here (because /usr/bin/sh\n> on Ubuntu is usually 'dash' not 'bash') is to use octal rather than\n> hexadecimal. ie: CENT=$(printf \"\\302\\242\")\n\nPerhaps an addition to Documentation/CodingGuidelines is in order?\n\n Documentation/CodingGuidelines | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git c/Documentation/CodingGuidelines w/Documentation/CodingGuidelines\nindex 9d5c27807a..78bc60665d 100644\n--- c/Documentation/CodingGuidelines\n+++ w/Documentation/CodingGuidelines\n@@ -188,6 +188,15 @@ For shell scripts specifically (not exhaustive):\n    hopefully nobody starts using \"local\" before they are reimplemented\n    in C ;-)\n \n+ - In 'printf' format string, do not use hexadecimals, as they are not\n+   portable.  Write \n+\n+     CENT=$(printf \"\\302\\242\")\n+\n+   not\n+\n+     CENT=$(printf \"\\xc2\\xa2\")\n+\n \n For C programs:\n \n\n\n"},{"id":"478323","messageId":"20230613171634.15985-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"xmqqa5x48j0j.fsf@gitster.g","subject":"Re: [PATCH v3 0/4] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-13T17:16:33Z","receivedAt":"2023-06-13T17:16:50Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> Perhaps an addition to Documentation/CodingGuidelines is in order?\n> \n>  Documentation/CodingGuidelines | 9 +++++++++\n>  1 file changed, 9 insertions(+)\n> \n> diff --git c/Documentation/CodingGuidelines w/Documentation/CodingGuidelines\n> index 9d5c27807a..78bc60665d 100644\n> --- c/Documentation/CodingGuidelines\n> +++ w/Documentation/CodingGuidelines\n> @@ -188,6 +188,15 @@ For shell scripts specifically (not exhaustive):\n>     hopefully nobody starts using \"local\" before they are reimplemented\n>     in C ;-)\n>  \n> + - In 'printf' format string, do not use hexadecimals, as they are not\n> +   portable.  Write \n> +\n> +     CENT=$(printf \"\\302\\242\")\n> +\n> +   not\n> +\n> +     CENT=$(printf \"\\xc2\\xa2\")\n\nI've checked with \"dash\" and this applies to any quoted string, not just\nwhen passed to printf. I'll prepare a patch describing this.\n"},{"id":"478324","messageId":"20230613172927.19019-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"20230613171634.15985-1-jonathantanmy@google.com","subject":"[PATCH] CodingGuidelines: use octal escapes, not hex","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-13T17:29:26Z","receivedAt":"2023-06-13T17:30:49Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Hexadecimal escapes in shell scripts are not portable across shells (in\nparticular, \"dash\" does not support them). Write in the CodingGuidelines\ndocument that we should be using octal escapes instead.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/CodingGuidelines | 3 +++\n 1 file changed, 3 insertions(+)\n\ndiff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines\nindex 003393ed16..1c54abd7c5 100644\n--- a/Documentation/CodingGuidelines\n+++ b/Documentation/CodingGuidelines\n@@ -188,6 +188,9 @@ For shell scripts specifically (not exhaustive):\n    hopefully nobody starts using \"local\" before they are reimplemented\n    in C ;-)\n \n+ - Use octal escape sequences (e.g. \"\\302\\242\"), not hexadecimal (e.g.\n+   \"\\xc2\\xa2\"), as the latter is not portable.\n+\n \n For C programs:\n \n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478325","messageId":"f095e2b486489290bb6da2fc23e9caa7da4af40d.1686677910.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1686677910.git.jonathantanmy@google.com","subject":"[PATCH v4 2/4] t4216: test changed path filters with high bit paths","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-13T17:39:56Z","receivedAt":"2023-06-13T17:40:23Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Subsequent commits will teach Git another version of changed path\nfilter that has different behavior with paths that contain at least\none character with its high bit set, so test the existing behavior as\na baseline.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n t/t4216-log-bloom.sh | 46 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 46 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..baa0c48897 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -404,4 +404,50 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n+get_bdat_offset () {\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"N\", \"$1\") if /BDAT\\0\\0\\0\\0(....)/ or exit 1' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n+get_first_changed_path_filter () {\n+\tBDAT_OFFSET=$(get_bdat_offset) &&\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"H*\", substr($_, '$BDAT_OFFSET' + 12, 2))' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n+# chosen to be the same under all Unicode normalization forms\n+CENT=$(printf \"\\302\\242\")\n+\n+test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n+\tgit init highbit1 &&\n+\ttest_commit -C highbit1 c1 \"$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'setup check value of version 1 changed-path' '\n+\t(cd highbit1 &&\n+\t\tprintf \"52a9\" >expect &&\n+\t\tget_first_changed_path_filter >actual)\n+'\n+\n+# expect will not match actual if char is unsigned by default. Write the test\n+# in this way, so that a user running this test script can still see if the two\n+# files match. (It will appear as an ordinary success if they match, and a skip\n+# if not.)\n+if test_cmp highbit1/expect highbit1/actual\n+then\n+\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n+fi\n+test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n+\t# Only the prereq matters for this test.\n+\ttrue\n+'\n+\n+test_expect_success 'version 1 changed-path used when version 1 requested' '\n+\t(cd highbit1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n test_done\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478326","messageId":"a3b52af4c96db9a8164119e8ace0541db47b6815.1686677910.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1686677910.git.jonathantanmy@google.com","subject":"[PATCH v4 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-13T17:39:55Z","receivedAt":"2023-06-13T17:40:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The code change to Git to support version 2 will be done in subsequent\ncommits.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/gitformat-commit-graph.txt | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\nindex 31cad585e2..112e6d36a6 100644\n--- a/Documentation/gitformat-commit-graph.txt\n+++ b/Documentation/gitformat-commit-graph.txt\n@@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n \n ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n     * It starts with header consisting of three unsigned 32-bit integers:\n-      - Version of the hash algorithm being used. We currently only support\n-\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n+      - Version of the hash algorithm being used. We currently support\n+\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n \timplemented exactly as described in\n \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n-\tin Probabilistic Verification\"\n+\tin Probabilistic Verification\". Version 1 bloom filters have a bug that appears\n+\twhen char is signed and the repository has path names that have characters >=\n+\t0x80; Git supports reading and writing them, but this ability will be removed\n+\tin a future version of Git.\n       - The number of times a path is hashed and hence the number of bit positions\n \t      that cumulatively determine whether a file is present in the commit.\n       - The minimum number of bits 'b' per entry in the Bloom filter. If the filter\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478327","messageId":"6adfa53dafc8c34f94a017fdc08460911f53d310.1686677910.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1686677910.git.jonathantanmy@google.com","subject":"[PATCH v4 3/4] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-13T17:39:57Z","receivedAt":"2023-06-13T17:40:27Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A subsequent commit will introduce another version of the changed-path\nfilter in the commit graph file. In order to control which version is\nto be accepted when read (and which version to write), a config variable\nis needed.\n\nTherefore, introduce this config variable. For forwards compatibility,\nteach Git to not read commit graphs when the config variable\nis set to an unsupported version. Because we teach Git this,\ncommitgraph.readChangedPaths is now redundant, so deprecate it and\ndefine its behavior in terms of the config variable we introduce.\n\nThis commit does not change the behavior of writing (Git writes changed\npath filters when explicitly instructed regardless of any config\nvariable), but a subsequent commit will restrict Git such that it will\nonly write when commitgraph.changedPathsVersion is 0, 1, or 2.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt | 16 +++++++++++++---\n commit-graph.c                       |  2 +-\n oss-fuzz/fuzz-commit-graph.c         |  2 +-\n repo-settings.c                      |  6 +++++-\n repository.h                         |  2 +-\n 5 files changed, 21 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c..eaa10bf232 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -9,6 +9,16 @@ commitGraph.maxNewFilters::\n \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n \n commitGraph.readChangedPaths::\n-\tIf true, then git will use the changed-path Bloom filters in the\n-\tcommit-graph file (if it exists, and they are present). Defaults to\n-\ttrue. See linkgit:git-commit-graph[1] for more information.\n+\tDeprecated. Equivalent to changedPathsVersion=1 if true, and\n+\tchangedPathsVersion=0 if false.\n+\n+commitGraph.changedPathsVersion::\n+\tSpecifies the version of the changed-path Bloom filters that Git will read and\n+\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n+\tmatch the version set in this config variable will be ignored.\n++\n+Defaults to 1.\n++\n+If 0, git will write version 1 Bloom filters when instructed to write.\n++\n+See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c11b59f28b..bd448047f1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -399,7 +399,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_read_changed_paths) {\n+\tif (s->commit_graph_changed_paths_version == 1) {\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\ndiff --git a/oss-fuzz/fuzz-commit-graph.c b/oss-fuzz/fuzz-commit-graph.c\nindex 914026f5d8..b56731f51a 100644\n--- a/oss-fuzz/fuzz-commit-graph.c\n+++ b/oss-fuzz/fuzz-commit-graph.c\n@@ -18,7 +18,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \t * possible.\n \t */\n \tthe_repository->settings.commit_graph_generation_version = 2;\n-\tthe_repository->settings.commit_graph_read_changed_paths = 1;\n+\tthe_repository->settings.commit_graph_changed_paths_version = 1;\n \tg = parse_commit_graph(&the_repository->settings, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 3dbd3f0e2e..6cbe02681b 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n \tint value;\n \tconst char *strval;\n \tint manyfiles;\n+\tint readChangedPaths;\n \n \tif (!r->gitdir)\n \t\tBUG(\"Cannot add settings for uninitialized repository\");\n@@ -54,7 +55,10 @@ void prepare_repo_settings(struct repository *r)\n \t/* Commit graph config or default, does not cascade (simple) */\n \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n-\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &readChangedPaths, 1);\n+\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n+\t\t     &r->settings.commit_graph_changed_paths_version,\n+\t\t     readChangedPaths ? 1 : 0);\n \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n \ndiff --git a/repository.h b/repository.h\nindex e8c67ffe16..1f1c32a6dd 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -32,7 +32,7 @@ struct repo_settings {\n \n \tint core_commit_graph;\n \tint commit_graph_generation_version;\n-\tint commit_graph_read_changed_paths;\n+\tint commit_graph_changed_paths_version;\n \tint gc_write_commit_graph;\n \tint gc_cruft_packs;\n \tint fetch_write_commit_graph;\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478328","messageId":"5c65bf8a22d657c58b1109928b4298e95ccb56bf.1686677910.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1686677910.git.jonathantanmy@google.com","subject":"[PATCH v4 4/4] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-13T17:39:58Z","receivedAt":"2023-06-13T17:40:29Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 and even with\nGit binaries that were compiled with different signedness of char. This\nbug affects both how Git writes changed path filters to disk and how Git\ninterprets changed path filters on disk.\n\nTherefore, introduce a new version (2) of changed path filters that\ncorrects this problem. The existing version (1) is still supported and\nis still the default, but users should migrate away from it as soon\nas possible.\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter.\n\nSo this patch does not include any mechanism to \"salvage\" changed path\nfilters from repositories. There is also no \"mixed\" mode - for each\ninvocation of Git, reading and writing changed path filters are done\nwith the same version number.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in t0095 was obtained from another murmur3 implementation\nusing the following Go source code:\n\n  package main\n\n  import \"fmt\"\n  import \"github.com/spaolacci/murmur3\"\n\n  func main() {\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n  }\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt |  2 +-\n bloom.c                              | 65 ++++++++++++++++++++++++++--\n bloom.h                              |  8 ++--\n commit-graph.c                       | 29 ++++++++++---\n t/helper/test-bloom.c                |  9 +++-\n t/t0095-bloom.sh                     |  8 ++++\n t/t4216-log-bloom.sh                 | 31 +++++++++++++\n 7 files changed, 139 insertions(+), 13 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex eaa10bf232..c64ee4f459 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -14,7 +14,7 @@ commitGraph.readChangedPaths::\n \n commitGraph.changedPathsVersion::\n \tSpecifies the version of the changed-path Bloom filters that Git will read and\n-\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n+\twrite. May be 0, 1, or 2. Any changed-path Bloom filters on disk that do not\n \tmatch the version set in this config variable will be ignored.\n +\n Defaults to 1.\ndiff --git a/bloom.c b/bloom.c\nindex d0730525da..915d8e5a31 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -65,7 +65,64 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n+{\n+\tconst uint32_t c1 = 0xcc9e2d51;\n+\tconst uint32_t c2 = 0x1b873593;\n+\tconst uint32_t r1 = 15;\n+\tconst uint32_t r2 = 13;\n+\tconst uint32_t m = 5;\n+\tconst uint32_t n = 0xe6546b64;\n+\tint i;\n+\tuint32_t k1 = 0;\n+\tconst char *tail;\n+\n+\tint len4 = len / sizeof(uint32_t);\n+\n+\tuint32_t k;\n+\tfor (i = 0; i < len4; i++) {\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n+\t\tk = byte1 | byte2 | byte3 | byte4;\n+\t\tk *= c1;\n+\t\tk = rotate_left(k, r1);\n+\t\tk *= c2;\n+\n+\t\tseed ^= k;\n+\t\tseed = rotate_left(seed, r2) * m + n;\n+\t}\n+\n+\ttail = (data + len4 * sizeof(uint32_t));\n+\n+\tswitch (len & (sizeof(uint32_t) - 1)) {\n+\tcase 3:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n+\t\t/*-fallthrough*/\n+\tcase 2:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n+\t\t/*-fallthrough*/\n+\tcase 1:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n+\t\tk1 *= c1;\n+\t\tk1 = rotate_left(k1, r1);\n+\t\tk1 *= c2;\n+\t\tseed ^= k1;\n+\t\tbreak;\n+\t}\n+\n+\tseed ^= (uint32_t)len;\n+\tseed ^= (seed >> 16);\n+\tseed *= 0x85ebca6b;\n+\tseed ^= (seed >> 13);\n+\tseed *= 0xc2b2ae35;\n+\tseed ^= (seed >> 16);\n+\n+\treturn seed;\n+}\n+\n+static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n {\n \tconst uint32_t c1 = 0xcc9e2d51;\n \tconst uint32_t c2 = 0x1b873593;\n@@ -130,8 +187,10 @@ void fill_bloom_key(const char *data,\n \tint i;\n \tconst uint32_t seed0 = 0x293ae76f;\n \tconst uint32_t seed1 = 0x7e646e2c;\n-\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n-\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n+\tconst uint32_t hash0 = (settings->hash_version == 2\n+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed0, data, len);\n+\tconst uint32_t hash1 = (settings->hash_version == 2\n+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed1, data, len);\n \n \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n \tfor (i = 0; i < settings->num_hashes; i++)\ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..0c33ae282c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -7,9 +7,11 @@ struct repository;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * The newest version is 2, which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -75,7 +77,7 @@ struct bloom_key {\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n \n void fill_bloom_key(const char *data,\n \t\t    size_t len,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex bd448047f1..36e6d09e74 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -302,15 +302,21 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct graph_read_bloom_data_data {\n+\tstruct commit_graph *g;\n+\tint commit_graph_changed_paths_version;\n+};\n+\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t  size_t chunk_size, void *data)\n {\n-\tstruct commit_graph *g = data;\n+\tstruct graph_read_bloom_data_data *d = data;\n+\tstruct commit_graph *g = d->g;\n \tuint32_t hash_version;\n \tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n+\tif (hash_version != d->commit_graph_changed_paths_version)\n \t\treturn 0;\n \n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n@@ -399,11 +405,16 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_changed_paths_version == 1) {\n+\tif (s->commit_graph_changed_paths_version == 1\n+\t    || s->commit_graph_changed_paths_version == 2) {\n+\t\tstruct graph_read_bloom_data_data data = {\n+\t\t\t.g = graph,\n+\t\t\t.commit_graph_changed_paths_version = s->commit_graph_changed_paths_version\n+\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t   graph_read_bloom_data, graph);\n+\t\t\t   graph_read_bloom_data, &data);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -2302,6 +2313,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tif (r->settings.commit_graph_changed_paths_version < 0\n+\t    || r->settings.commit_graph_changed_paths_version > 2) {\n+\t\twarning(_(\"attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported\"),\n+\t\t\tr->settings.commit_graph_changed_paths_version);\n+\t\treturn 0;\n+\t}\n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n+\t\t? 2 : 1;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t      bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2331,7 +2350,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 6c900ca668..34b8dd9164 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -48,6 +48,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \"  test-tool bloom get_murmur3 <string>\\n\"\n+\"  test-tool bloom get_murmur3_seven_highbit\\n\"\n \"  test-tool bloom generate_filter <string> [<string>...]\\n\"\n \"  test-tool bloom get_filter_for_commit <commit-hex>\\n\";\n \n@@ -62,7 +63,13 @@ int cmd__bloom(int argc, const char **argv)\n \t\tuint32_t hashed;\n \t\tif (argc < 3)\n \t\t\tusage(bloom_usage);\n-\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n+\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex baa0c48897..5cb35c3a3e 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -450,4 +450,35 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n \t\ttest_bloom_filters_used \"-- $CENT\")\n '\n \n+test_expect_success 'version 1 changed-path not used when version 2 requested' '\n+\t(cd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n+\tgit init highbit2 &&\n+\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n+\ttest_commit -C highbit2 c2 \"$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'check value of version 2 changed-path' '\n+\t(cd highbit2 &&\n+\t\tprintf \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+test_expect_success 'version 2 changed-path used when version 2 requested' '\n+\t(cd highbit2 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'version 2 changed-path not used when version 1 requested' '\n+\t(cd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n test_done\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478329","messageId":"cover.1686677910.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1684790529.git.jonathantanmy@google.com","subject":"[PATCH v4 0/4] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-13T17:39:54Z","receivedAt":"2023-06-13T17:40:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks Ramsay for spotting the errors and mentioning that I can use\noctal escapes. Here's an update taking into account their comments.\n\nJonathan Tan (4):\n  gitformat-commit-graph: describe version 2 of BDAT\n  t4216: test changed path filters with high bit paths\n  repo-settings: introduce commitgraph.changedPathsVersion\n  commit-graph: new filter ver. that fixes murmur3\n\n Documentation/config/commitgraph.txt     | 16 ++++-\n Documentation/gitformat-commit-graph.txt |  9 ++-\n bloom.c                                  | 65 +++++++++++++++++++-\n bloom.h                                  |  8 ++-\n commit-graph.c                           | 29 +++++++--\n oss-fuzz/fuzz-commit-graph.c             |  2 +-\n repo-settings.c                          |  6 +-\n repository.h                             |  2 +-\n t/helper/test-bloom.c                    |  9 ++-\n t/t0095-bloom.sh                         |  8 +++\n t/t4216-log-bloom.sh                     | 77 ++++++++++++++++++++++++\n 11 files changed, 210 insertions(+), 21 deletions(-)\n\nRange-diff against v3:\n1:  d4b63945f6 ! 1:  a3b52af4c9 gitformat-commit-graph: describe version 2 of BDAT\n    @@ Documentation/gitformat-commit-graph.txt: All multi-byte numbers are in network\n      \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n     -\tin Probabilistic Verification\"\n     +\tin Probabilistic Verification\". Version 1 bloom filters have a bug that appears\n    -+\twhen int is signed and the repository has path names that have characters >=\n    ++\twhen char is signed and the repository has path names that have characters >=\n     +\t0x80; Git supports reading and writing them, but this ability will be removed\n     +\tin a future version of Git.\n            - The number of times a path is hashed and hence the number of bit positions\n2:  aa4535776e ! 2:  f095e2b486 t4216: test changed path filters with high bit paths\n    @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm\n     +}\n     +\n     +# chosen to be the same under all Unicode normalization forms\n    -+CENT=$(printf \"\\xc2\\xa2\")\n    ++CENT=$(printf \"\\302\\242\")\n     +\n    -+# Some systems (in particular, Linux on the CI running on GitHub at the time of\n    -+# writing) store into CENT a literal backslash, then \"x\", and so on (instead of\n    -+# the high-bit characters needed). In these systems, do not run the following\n    -+# tests.\n    -+if test \"$(printf $CENT | perl -0777 -ne 'no utf8; print ord($_)')\" = \"194\"\n    -+then\n    -+\ttest_set_prereq HIGH_BIT\n    -+fi\n    -+\n    -+test_expect_success HIGH_BIT 'set up repo with high bit path, version 1 changed-path' '\n    ++test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n     +\tgit init highbit1 &&\n     +\ttest_commit -C highbit1 c1 \"$CENT\" &&\n     +\tgit -C highbit1 commit-graph write --reachable --changed-paths\n     +'\n     +\n    -+test_expect_success HIGH_BIT 'setup check value of version 1 changed-path' '\n    ++test_expect_success 'setup check value of version 1 changed-path' '\n     +\t(cd highbit1 &&\n     +\t\tprintf \"52a9\" >expect &&\n     +\t\tget_first_changed_path_filter >actual)\n     +'\n     +\n    -+# expect will not match actual if int is unsigned by default. Write the test\n    ++# expect will not match actual if char is unsigned by default. Write the test\n     +# in this way, so that a user running this test script can still see if the two\n     +# files match. (It will appear as an ordinary success if they match, and a skip\n     +# if not.)\n     +if test_cmp highbit1/expect highbit1/actual\n     +then\n    -+\ttest_set_prereq SIGNED_INT_BY_DEFAULT\n    ++\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n     +fi\n    -+test_expect_success SIGNED_INT_BY_DEFAULT 'check value of version 1 changed-path' '\n    ++test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n     +\t# Only the prereq matters for this test.\n     +\ttrue\n     +'\n     +\n    -+test_expect_success HIGH_BIT 'version 1 changed-path used when version 1 requested' '\n    ++test_expect_success 'version 1 changed-path used when version 1 requested' '\n     +\t(cd highbit1 &&\n     +\t\ttest_bloom_filters_used \"-- $CENT\")\n     +'\n3:  d6982268a4 = 3:  6adfa53daf repo-settings: introduce commitgraph.changedPathsVersion\n4:  e879483c42 ! 4:  5c65bf8a22 commit-graph: new filter ver. that fixes murmur3\n    @@ t/t0095-bloom.sh: test_expect_success 'compute unseeded murmur3 hash for test st\n      \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\n     \n      ## t/t4216-log-bloom.sh ##\n    -@@ t/t4216-log-bloom.sh: test_expect_success HIGH_BIT 'version 1 changed-path used when version 1 request\n    +@@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when version 1 requested' '\n      \t\ttest_bloom_filters_used \"-- $CENT\")\n      '\n      \n    -+test_expect_success HIGH_BIT 'version 1 changed-path not used when version 2 requested' '\n    ++test_expect_success 'version 1 changed-path not used when version 2 requested' '\n     +\t(cd highbit1 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n     +\t\ttest_bloom_filters_not_used \"-- $CENT\")\n     +'\n     +\n    -+test_expect_success HIGH_BIT 'set up repo with high bit path, version 2 changed-path' '\n    ++test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n     +\tgit init highbit2 &&\n     +\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n     +\ttest_commit -C highbit2 c2 \"$CENT\" &&\n     +\tgit -C highbit2 commit-graph write --reachable --changed-paths\n     +'\n     +\n    -+test_expect_success HIGH_BIT 'check value of version 2 changed-path' '\n    ++test_expect_success 'check value of version 2 changed-path' '\n     +\t(cd highbit2 &&\n     +\t\tprintf \"c01f\" >expect &&\n     +\t\tget_first_changed_path_filter >actual &&\n     +\t\ttest_cmp expect actual)\n     +'\n     +\n    -+test_expect_success HIGH_BIT 'version 2 changed-path used when version 2 requested' '\n    ++test_expect_success 'version 2 changed-path used when version 2 requested' '\n     +\t(cd highbit2 &&\n     +\t\ttest_bloom_filters_used \"-- $CENT\")\n     +'\n     +\n    -+test_expect_success HIGH_BIT 'version 2 changed-path not used when version 1 requested' '\n    ++test_expect_success 'version 2 changed-path not used when version 1 requested' '\n     +\t(cd highbit2 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n     +\t\ttest_bloom_filters_not_used \"-- $CENT\")\n-- \n2.41.0.162.gfafddb0af9-goog\n\n"},{"id":"478330","messageId":"CAPig+cRS=cpV1e4QoSy5uypA+-hECyPe2rEg3scV8LVBfzZ5hA@mail.gmail.com","threadId":"59781","inReplyTo":"20230613172927.19019-1-jonathantanmy@google.com","subject":"Re: [PATCH] CodingGuidelines: use octal escapes, not hex","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2023-06-13T18:16:15Z","receivedAt":"2023-06-13T18:16:35Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Jun 13, 2023 at 1:44 PM Jonathan Tan <jonathantanmy@google.com> wrote:\n> Hexadecimal escapes in shell scripts are not portable across shells (in\n> particular, \"dash\" does not support them). Write in the CodingGuidelines\n> document that we should be using octal escapes instead.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n> diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines\n> @@ -188,6 +188,9 @@ For shell scripts specifically (not exhaustive):\n> + - Use octal escape sequences (e.g. \"\\302\\242\"), not hexadecimal (e.g.\n> +   \"\\xc2\\xa2\"), as the latter is not portable.\n\nThe shell itself doesn't interpret these sequences, so this\ndescription feels too generic. Perhaps it would make more sense to\ncite specific tools for which octal sequences are needed for\nportability reasons, such as `printf`, `sed`, `tr`, etc.\n"},{"id":"478332","messageId":"20230613184304.36482-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"CAPig+cRS=cpV1e4QoSy5uypA+-hECyPe2rEg3scV8LVBfzZ5hA@mail.gmail.com","subject":"Re: [PATCH] CodingGuidelines: use octal escapes, not hex","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-13T18:43:03Z","receivedAt":"2023-06-13T18:43:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Eric Sunshine <sunshine@sunshineco.com> writes:\n> On Tue, Jun 13, 2023 at 1:44 PM Jonathan Tan <jonathantanmy@google.com> wrote:\n> > Hexadecimal escapes in shell scripts are not portable across shells (in\n> > particular, \"dash\" does not support them). Write in the CodingGuidelines\n> > document that we should be using octal escapes instead.\n> >\n> > Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> > ---\n> > diff --git a/Documentation/CodingGuidelines b/Documentation/CodingGuidelines\n> > @@ -188,6 +188,9 @@ For shell scripts specifically (not exhaustive):\n> > + - Use octal escape sequences (e.g. \"\\302\\242\"), not hexadecimal (e.g.\n> > +   \"\\xc2\\xa2\"), as the latter is not portable.\n> \n> The shell itself doesn't interpret these sequences, so this\n> description feels too generic. Perhaps it would make more sense to\n> cite specific tools for which octal sequences are needed for\n> portability reasons, such as `printf`, `sed`, `tr`, etc.\n\nAh...good point. I checked with \"echo\" in \"dash\" and assumed that it\nwas \"dash\" that was interpreting the escapes, but indeed it is the\n\"echo\" (and \"printf\") builtins in \"dash\" that are actually interpreting\nthem. What do you think of the following in the commit message:\n\n  Hexadecimal escapes in shell scripts are not portable across shell builtins (in\n  particular, the \"printf\" of \"dash\" does not support them). Write in the CodingGuidelines\n  document that we should be using octal escapes instead.\n\nand in the CodingGuidelines doc:\n\n+ - Use octal escape sequences (e.g. \"\\302\\242\"), not hexadecimal (e.g.\n+   \"\\xc2\\xa2\"), as the latter is not portable across some shell builtins like printf.\n"},{"id":"478333","messageId":"CAPig+cQnyL8-9EHVe5P_oRbUdsXA9WBiK_W4LHq7HOF4-ek5Pg@mail.gmail.com","threadId":"59781","inReplyTo":"20230613184304.36482-1-jonathantanmy@google.com","subject":"Re: [PATCH] CodingGuidelines: use octal escapes, not hex","fromName":"Eric Sunshine","fromEmail":"sunshine@sunshineco.com","sentAt":"2023-06-13T19:15:14Z","receivedAt":"2023-06-13T19:15:31Z","isPatch":true,"sender":{"key":"sunshine@sunshineco.com","avatar":"https://avatars.githubusercontent.com/u/163641?v=4"},"body":"On Tue, Jun 13, 2023 at 2:43 PM Jonathan Tan <jonathantanmy@google.com> wrote:\n> Eric Sunshine <sunshine@sunshineco.com> writes:\n> > On Tue, Jun 13, 2023 at 1:44 PM Jonathan Tan <jonathantanmy@google.com> wrote:\n> > > + - Use octal escape sequences (e.g. \"\\302\\242\"), not hexadecimal (e.g.\n> > > +   \"\\xc2\\xa2\"), as the latter is not portable.\n> >\n> > The shell itself doesn't interpret these sequences, so this\n> > description feels too generic. Perhaps it would make more sense to\n> > cite specific tools for which octal sequences are needed for\n> > portability reasons, such as `printf`, `sed`, `tr`, etc.\n>\n> Ah...good point. I checked with \"echo\" in \"dash\" and assumed that it\n> was \"dash\" that was interpreting the escapes, but indeed it is the\n> \"echo\" (and \"printf\") builtins in \"dash\" that are actually interpreting\n> them. What do you think of the following in the commit message:\n>\n>   Hexadecimal escapes in shell scripts are not portable across shell builtins (in\n>   particular, the \"printf\" of \"dash\" does not support them). Write in the CodingGuidelines\n>   document that we should be using octal escapes instead.\n>\n> and in the CodingGuidelines doc:\n>\n> + - Use octal escape sequences (e.g. \"\\302\\242\"), not hexadecimal (e.g.\n> +   \"\\xc2\\xa2\"), as the latter is not portable across some shell builtins like printf.\n\nThe portability concern is not specific to a certain shell or whether\na command is builtin, so it feels misleading to single out \"dash\" and\nbuiltins. The same portability problems can crop up, as well, with\nolder (non-\"dash\") shells, and with commands which may or may not be\nbuiltins (such as \"echo\" which, historically, was not always a\nbuiltin), and non-builtins commands, such as \"sed\" and \"tr\".\n\nSo, for the commit message, perhaps simply:\n\n    Extend the shell-scripting section of CodingGuidelines to suggest\n    octal escape sequences (e.g. \"\\302\\242\") over hexadecimal\n    (e.g. \"\\xc2\\xa2\") since the latter can be a source of portability\n    problems.\n\nAs for the change to CodingGuidelines, this would probably be sufficient:\n\n    Use octal escape sequences (e.g. \"\\302\\242\"), not hexadecimal\n    (e.g. \"\\xc2\\xa2\"), since the latter is not portable across some\n    commands, such as `printf`, `sed`, `tr`, etc.\n"},{"id":"478334","messageId":"xmqqa5x35g1e.fsf@gitster.g","threadId":"59781","inReplyTo":"20230613171634.15985-1-jonathantanmy@google.com","subject":"Re: [PATCH v3 0/4] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-13T19:16:29Z","receivedAt":"2023-06-13T19:16:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n>> + - In 'printf' format string, do not use hexadecimals, as they are not\n>> +   portable.  Write \n>> +\n>> +     CENT=$(printf \"\\302\\242\")\n>> +\n>> +   not\n>> +\n>> +     CENT=$(printf \"\\xc2\\xa2\")\n>\n> I've checked with \"dash\" and this applies to any quoted string, not just\n> when passed to printf. I'll prepare a patch describing this.\n\nWhat do you mean by \"any quoted string\"?\n\nI think built-in 'echo' of dash takes \"\\302\\242\" and emits the\ncent-sign (but /bin/echo may not), but I do not think it is \"any\nquoted string\".  To wit:\n\n    dash$ echo '\\302\\242'\n    ¢\n\nThe echo built into dash shows the cent-sign.  The example does not\nlet us tell if is the shell (i.e. the 'echo' command sees the\ncent-sign in its argv[1]) or if it is the command (i.e. the 'echo'\nsees 8-byte string \"bs three zero two bs two four two\" in its\nargv[1], and shows that as the cent-sign), though.\n\n    dash$ /bin/echo '\\302\\242'\n    \\302\\242\n\nThis shows that shell is not doing anything fancy.  /bin/echo gets\nthe 8-byte string in its argv[1] and emits that as-is.\n\n    dash$ /bin/echo \"\\302\\242\"\n    \\302\\242\n\nAgain this shows the same; I added this example to demonstrate that\nit is _not_ like the shell interprets the backslashed octal strings\ndepending on how they are quoted.\n\n    dash$ printf \"%s\\n\" '\\302\\242'\n    \\302\\242\n    dash$ printf \"%s\\n\" \"\\302\\242\"\n    \\302\\242\n\nAnd these demonstrates that argv[2] given to printf in these cases\nare 8-byte string \"bs three zero two bs two four two\" and the shell\nis not doing anything fancy.\n\nSo, I would suggest not saying \"any quoted string\".  In addition,\neven though dash's built-in echo that recognizes \"\\0num\" seems to be\nconformant to what POSIX specifies (cf. [*1*]), GNU requires \"-e\" in\norder to do so in both standalone 'echo' binary and one built into\n'bash', so we cannot rely on this POSIX behaviour when writing a\nportable script.  Hence, I would recommend us to focus on giving a\npiece of advice on use of printf in this part of the documentation.\n\n\n[References]\n\n*1* https://pubs.opengroup.org/onlinepubs/9699919799/utilities/echo.html\n*2* https://www.gnu.org/software/coreutils/manual/html_node/echo-invocation.html\n"},{"id":"478335","messageId":"xmqq5y7r5fsx.fsf@gitster.g","threadId":"59781","inReplyTo":"cover.1686677910.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 0/4] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-13T19:21:34Z","receivedAt":"2023-06-13T19:21:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Thanks Ramsay for spotting the errors and mentioning that I can use\n> octal escapes. Here's an update taking into account their comments.\n\nThe changes look good.  Will queue.\n\nStolee, you had comments on an earlier round---how does this one\nlook?\n\nThanks.\n"},{"id":"478336","messageId":"xmqq1qif5fgf.fsf@gitster.g","threadId":"59781","inReplyTo":"CAPig+cQnyL8-9EHVe5P_oRbUdsXA9WBiK_W4LHq7HOF4-ek5Pg@mail.gmail.com","subject":"Re: [PATCH] CodingGuidelines: use octal escapes, not hex","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-13T19:29:04Z","receivedAt":"2023-06-13T19:29:09Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Eric Sunshine <sunshine@sunshineco.com> writes:\n\n> So, for the commit message, perhaps simply:\n>\n>     Extend the shell-scripting section of CodingGuidelines to suggest\n>     octal escape sequences (e.g. \"\\302\\242\") over hexadecimal\n>     (e.g. \"\\xc2\\xa2\") since the latter can be a source of portability\n>     problems.\n>\n> As for the change to CodingGuidelines, this would probably be sufficient:\n>\n>     Use octal escape sequences (e.g. \"\\302\\242\"), not hexadecimal\n>     (e.g. \"\\xc2\\xa2\"), since the latter is not portable across some\n>     commands, such as `printf`, `sed`, `tr`, etc.\n\nI'd prefer singling out `printf`, actually, and not talking about\n\"across some commands\".\n\nAs I said in a separate message, we certainly do *not* want to rely\non `echo` interpreting bs-escaped octal sequences without '-e', even\nthough it may be expected on a POSIX systems, because it is not\nportable across systems our users commonly encounter.\n\nAnd `printf` has been what we chose to turn bs-escaped octal\nsequence into binary.  I'd prefer not having to even worry about\n`sed`, `tr`, etc. behaving differently and not allowing to expect\nthese other commands to be usable for turning bs-escaped octal\nsequence into binary would be one way to achieve that goal.\n\nThanks.\n"},{"id":"478339","messageId":"xmqqlegn3tz3.fsf@gitster.g","threadId":"59781","inReplyTo":"a3b52af4c96db9a8164119e8ace0541db47b6815.1686677910.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-13T21:58:24Z","receivedAt":"2023-06-13T21:59:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> The code change to Git to support version 2 will be done in subsequent\n> commits.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  Documentation/gitformat-commit-graph.txt | 9 ++++++---\n>  1 file changed, 6 insertions(+), 3 deletions(-)\n>\n> diff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\n> index 31cad585e2..112e6d36a6 100644\n> --- a/Documentation/gitformat-commit-graph.txt\n> +++ b/Documentation/gitformat-commit-graph.txt\n> @@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n>  \n>  ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n>      * It starts with header consisting of three unsigned 32-bit integers:\n> -      - Version of the hash algorithm being used. We currently only support\n> -\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n> +      - Version of the hash algorithm being used. We currently support\n> +\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n>  \timplemented exactly as described in\n>  \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n>  \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n>  \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n> -\tin Probabilistic Verification\"\n> +\tin Probabilistic Verification\". Version 1 bloom filters have a bug that appears\n\n\"bloom\" -> \"Bloom\", probably, as the name comes from the name of its\ninventor (just like we spell \"Boolean\", not \"boolean\").\n\n> +\twhen char is signed and the repository has path names that have characters >=\n> +\t0x80; Git supports reading and writing them, but this ability will be removed\n> +\tin a future version of Git.\n\nMakes sense.\n\nI wonder if we want to mention what the undesired misbehaviour the\n\"bug\" causes and what we do to avoid getting affected by the bug\nhere.  If we can say something like \"When querying for a pathname\nwith a byte with high-bit set, the buggy filter may produce false\nnegative, making the filter unusable, but asking for a pathname\nwithout such a byte produces no false negatives (even though we may\nget false positives).  When Git reads version 1 filter data, it\nrefrains from using it for processing paths with high-bit set to\navoid triggering the bug\", then it would be ideal.  Or \"When the\nrepository has even a single pathname with high-bit set anywhere in\nits history, version 1 Bloom can give false negative when querying\nany paths and becomes unusable.  You can use $THIS configuration\nvariable to disable use of Bloom filter data in such a case\" would\nalso be fine.  The point is to give actionable piece of information\nto the readers.\n\nThanks.\n"},{"id":"478548","messageId":"6ab75484-5893-691b-e40d-b4305c889081@github.com","threadId":"59781","inReplyTo":"xmqqlegn3tz3.fsf@gitster.g","subject":"Re: [PATCH v4 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-06-20T13:22:51Z","receivedAt":"2023-06-20T13:23:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/13/2023 5:58 PM, Junio C Hamano wrote:\n> Jonathan Tan <jonathantanmy@google.com> writes:\n\n>> +\tin Probabilistic Verification\". Version 1 bloom filters have a bug that appears\n> \n> \"bloom\" -> \"Bloom\", probably, as the name comes from the name of its\n> inventor (just like we spell \"Boolean\", not \"boolean\").\n\nThe ultimate recognition comes from when the term named after you\nbecomes lower-case (\"boolean\" is sometimes in this category, but\ndefinitely \"abelian\" is an example).\n\nIn this case, you are right that we should capitalize Bloom.\n\n>> +\twhen char is signed and the repository has path names that have characters >=\n>> +\t0x80; Git supports reading and writing them, but this ability will be removed\n>> +\tin a future version of Git.\n> \n> Makes sense.\n\nI also like how you organized this: \"We support version 2. 1 is\nstill around but not for long.\"\n \n> I wonder if we want to mention what the undesired misbehaviour the\n> \"bug\" causes and what we do to avoid getting affected by the bug\n> here.  If we can say something like \"When querying for a pathname\n> with a byte with high-bit set, the buggy filter may produce false\n> negative, making the filter unusable, but asking for a pathname\n> without such a byte produces no false negatives (even though we may\n> get false positives).  When Git reads version 1 filter data, it\n> refrains from using it for processing paths with high-bit set to\n> avoid triggering the bug\", then it would be ideal.  Or \"When the\n> repository has even a single pathname with high-bit set anywhere in\n> its history, version 1 Bloom can give false negative when querying\n> any paths and becomes unusable.  You can use $THIS configuration\n> variable to disable use of Bloom filter data in such a case\" would\n> also be fine.  The point is to give actionable piece of information\n> to the readers.\n\nThis is definitely helpful, but if someone is having issues we\nwould say \"try version 2 and see if it still happens\" and not\nover-index on the underlying reason.\n\nThat's to say, I'm OK with the shorter description of the problem.\nFeel free to expand if you're interested, though.\n\nThanks,\n-Stolee\n"},{"id":"478549","messageId":"ceb8eb58-6cbe-1846-14e4-33c4967e3024@github.com","threadId":"59781","inReplyTo":"6adfa53dafc8c34f94a017fdc08460911f53d310.1686677910.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 3/4] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-06-20T13:28:02Z","receivedAt":"2023-06-20T13:28:42Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/13/2023 1:39 PM, Jonathan Tan wrote:\n> A subsequent commit will introduce another version of the changed-path\n> filter in the commit graph file. In order to control which version is\n> to be accepted when read (and which version to write), a config variable\n> is needed.\n> \n> Therefore, introduce this config variable. For forwards compatibility,\n> teach Git to not read commit graphs when the config variable\n> is set to an unsupported version. Because we teach Git this,\n> commitgraph.readChangedPaths is now redundant, so deprecate it and\n> define its behavior in terms of the config variable we introduce.\n\nI'm late to the party, but I support this change. The safety valve\nof \"I want to turn this off if something goes wrong\" is long overdue\nfor deletion (it would help someone in this high-bit situation).\n\nHaving a new replacement is a good way to preserve the safety valve\nbehavior while also promoting the new versioning scheme.\n \n> This commit does not change the behavior of writing (Git writes changed\n> path filters when explicitly instructed regardless of any config\n> variable), but a subsequent commit will restrict Git such that it will\n> only write when commitgraph.changedPathsVersion is 0, 1, or 2.\n\n>  commitGraph.readChangedPaths::\n> -\tIf true, then git will use the changed-path Bloom filters in the\n> -\tcommit-graph file (if it exists, and they are present). Defaults to\n> -\ttrue. See linkgit:git-commit-graph[1] for more information.\n> +\tDeprecated. Equivalent to changedPathsVersion=1 if true, and\n> +\tchangedPathsVersion=0 if false.\n\nThis defaulted to true before...\n\n> +commitGraph.changedPathsVersion::\n> +\tSpecifies the version of the changed-path Bloom filters that Git will read and\n> +\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n> +\tmatch the version set in this config variable will be ignored.\n> ++\n> +Defaults to 1.\n\nSo this version defaults to 1. Good.\n\n> ++\n> +If 0, git will write version 1 Bloom filters when instructed to write.\n\n> -\tif (s->commit_graph_read_changed_paths) {\n> +\tif (s->commit_graph_changed_paths_version == 1) {\n>  \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n>  \t\t\t   &graph->chunk_bloom_indexes);\n>  \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n\n\n>  \t/* Commit graph config or default, does not cascade (simple) */\n>  \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n>  \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n> -\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n> +\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &readChangedPaths, 1);\n> +\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n> +\t\t     &r->settings.commit_graph_changed_paths_version,\n> +\t\t     readChangedPaths ? 1 : 0);\n\nAnd here we default to 'true' and '1', respectively. This allows \n'commitGraph.readChangedPaths=false' to override\n'commitGraph.changedPathsVersion=1'. Should that implication be\ndocumented?\n\nThanks,\n-Stolee\n"},{"id":"478550","messageId":"4c5cadf2-bd70-c56e-3935-fb24b604f411@github.com","threadId":"59781","inReplyTo":"5c65bf8a22d657c58b1109928b4298e95ccb56bf.1686677910.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 4/4] commit-graph: new filter ver. that fixes murmur3","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-06-20T13:39:09Z","receivedAt":"2023-06-20T13:39:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/13/2023 1:39 PM, Jonathan Tan wrote:\n\n>  commitGraph.changedPathsVersion::\n>  \tSpecifies the version of the changed-path Bloom filters that Git will read and\n> -\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n> +\twrite. May be 0, 1, or 2. Any changed-path Bloom filters on disk that do not\n>  \tmatch the version set in this config variable will be ignored.\n>  +\n>  Defaults to 1.\n\nIs this a good place to document the planned modification of this default in\nfuture versions of Git?\n\n> +static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n>  {\n>  \tconst uint32_t c1 = 0xcc9e2d51;\n>  \tconst uint32_t c2 = 0x1b873593;\n> @@ -130,8 +187,10 @@ void fill_bloom_key(const char *data,\n>  \tint i;\n>  \tconst uint32_t seed0 = 0x293ae76f;\n>  \tconst uint32_t seed1 = 0x7e646e2c;\n> -\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n> -\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n> +\tconst uint32_t hash0 = (settings->hash_version == 2\n> +\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed0, data, len);\n> +\tconst uint32_t hash1 = (settings->hash_version == 2\n> +\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed1, data, len);\n\nThis is the critical step. I've not seen this ternary trick within\na function name choice, but it makes for a compact check. Nice.\n\nHowever, I think the 'settings->hash_version' is the wrong place\nto look for the condition. We should be getting this value from the\ncommit-graph we are reading. (More on this later.)\n\n>  struct bloom_filter_settings {\n>  \t/*\n>  \t * The version of the hashing technique being used.\n> -\t * We currently only support version = 1 which is\n> +\t * The newest version is 2, which is\n>  \t * the seeded murmur3 hashing technique implemented\n> -\t * in bloom.c.\n> +\t * in bloom.c. Bloom filters of version 1 were created\n> +\t * with prior versions of Git, which had a bug in the\n> +\t * implementation of the hash function.\n\n...\n\n> +struct graph_read_bloom_data_data {\n> +\tstruct commit_graph *g;\n> +\tint commit_graph_changed_paths_version;\n> +};\n> +\n>  static int graph_read_bloom_data(const unsigned char *chunk_start,\n>  \t\t\t\t  size_t chunk_size, void *data)\n>  {\n> -\tstruct commit_graph *g = data;\n> +\tstruct graph_read_bloom_data_data *d = data;\n> +\tstruct commit_graph *g = d->g;\n>  \tuint32_t hash_version;\n>  \tg->chunk_bloom_data = chunk_start;\n>  \thash_version = get_be32(chunk_start);\n>  \n> -\tif (hash_version != 1)\n> +\tif (hash_version != d->commit_graph_changed_paths_version)\n>  \t\treturn 0;\n\nThis makes it appear like we cannot read a commit-graph that has\na Bloom filter version that doesn't match the configured version.\n\nThis seems incorrect. If we want to configure to _write_ v2, we\nshould still be able to _read_ v1 concurrently until those v2\nfilters are written.\n\nThis check should be:\n\n\tif (hash_version <= 0 || hash_version > 2)\n\t\treturn 0;\n\nand then \n\n\tg->filter_hash_version = hash_version;\n\nto store this hash version somewhere in the graph. This way, we\ncan read any commit-graph file and will not suffer performance\nproblems in the time between setting the config value and writing\nthe new commit-graph file.\n\n> -\tif (s->commit_graph_changed_paths_version == 1) {\n> +\tif (s->commit_graph_changed_paths_version == 1\n> +\t    || s->commit_graph_changed_paths_version == 2) {\n\nPerhaps this could just be\n\n\tif (s->commit_graph_changed_paths_version) {\n\nto say \"not zero means we still read the filters\". Though, since\nthis config _should_ mean \"which version do we _write_?\" it might\nbe good to go back on the \"unifying the two config options\".\n\n> +\t\tstruct graph_read_bloom_data_data data = {\n> +\t\t\t.g = graph,\n> +\t\t\t.commit_graph_changed_paths_version = s->commit_graph_changed_paths_version\n> +\t\t};\n>  \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n>  \t\t\t   &graph->chunk_bloom_indexes);\n>  \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n> -\t\t\t   graph_read_bloom_data, graph);\n> +\t\t\t   graph_read_bloom_data, &data);\n>  \t}\n\nMuch of this block will not be necessary as we don't need to\nsend the repo settings into the read_chunk() method.\n\n> +\tif (r->settings.commit_graph_changed_paths_version < 0\n> +\t    || r->settings.commit_graph_changed_paths_version > 2) {\n> +\t\twarning(_(\"attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported\"),\n> +\t\t\tr->settings.commit_graph_changed_paths_version);\n> +\t\treturn 0;\n\nI see the \"< 0\" means we aren't considering the case of disabling \nwrites with the zero value. We should exit early if we see a zero\nhere, for extra safety.\n\n> +\t}\n> +\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n> +\t\t? 2 : 1;\n\nOnce we've checked that this value is not zero, we can do this:\n\n\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version;\n\n\n>  \t\t/* We have changed-paths already. Keep them in the next graph */\n> -\t\tif (g && g->chunk_bloom_data) {\n> +\t\tif (g && g->bloom_filter_settings) {\n>  \t\t\tctx->changed_paths = 1;\n>  \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n>  \t\t}\n\n\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -450,4 +450,35 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n>  \t\ttest_bloom_filters_used \"-- $CENT\")\n>  '\n>  \n> +test_expect_success 'version 1 changed-path not used when version 2 requested' '\n> +\t(cd highbit1 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n> +\t\ttest_bloom_filters_not_used \"-- $CENT\")\n> +'\n\nI think this test is the wrong behavior. We should be able to use version 1\nwhen version 2 is requested.\n\nInstead, start with version 1, and _upgrade_ to version 2 and then check\nwhich version exists in the file.\n\nWe should only _not_ use the filters if the version is 0 (or\ncommitGraph.readChangedPaths=false). I think this might be a good enough reason\nto \n\n> +\n> +test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n> +\tgit init highbit2 &&\n> +\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n> +\ttest_commit -C highbit2 c2 \"$CENT\" &&\n> +\tgit -C highbit2 commit-graph write --reachable --changed-paths\n> +'\n> +\n> +test_expect_success 'check value of version 2 changed-path' '\n> +\t(cd highbit2 &&\n> +\t\tprintf \"c01f\" >expect &&\n> +\t\tget_first_changed_path_filter >actual &&\n> +\t\ttest_cmp expect actual)\n> +'\n> +\n> +test_expect_success 'version 2 changed-path used when version 2 requested' '\n> +\t(cd highbit2 &&\n> +\t\ttest_bloom_filters_used \"-- $CENT\")\n> +'\n> +\n> +test_expect_success 'version 2 changed-path not used when version 1 requested' '\n> +\t(cd highbit2 &&\n> +\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n> +\t\ttest_bloom_filters_not_used \"-- $CENT\")\n> +'\n\nAgain, this is also the wrong situation.\n\nThanks,\n-Stolee\n"},{"id":"478551","messageId":"c7b66d2c-cdc3-1f0f-60a0-a2ee21c277bf@github.com","threadId":"59781","inReplyTo":"xmqq5y7r5fsx.fsf@gitster.g","subject":"Re: [PATCH v4 0/4] Changed path filter hash fix and version bump","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-06-20T13:43:46Z","receivedAt":"2023-06-20T13:43:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/13/2023 3:21 PM, Junio C Hamano wrote:\n> Jonathan Tan <jonathantanmy@google.com> writes:\n> \n>> Thanks Ramsay for spotting the errors and mentioning that I can use\n>> octal escapes. Here's an update taking into account their comments.\n> \n> The changes look good.  Will queue.\n> \n> Stolee, you had comments on an earlier round---how does this one\n> look?\n\nI'm sorry I'm so late to this. I've been meaning to get to it, but\nit's been a crazy couple of weeks.\n\nThis version is not ready. The backwards compatibility story is\nincomplete.\n\nWhen commitGraph.changedPathsVersion is set, it does not allow\nreading a previous filter version, leaving us in a poor performance\nstate until the commit-graph file can be rewritten.\n\nWhile I was reviewing, it seemed reasonable to deprecate\ncommitGraph.readChangedPaths, but this use of \"also restrict writes\nto this version\" didn't make sense to me at the time. Instead, it\nwould be good to have this clarity between the config options:\n\n commitGraph.readChangedPaths: should we read and use the filters\n that exist on disk? Defaults to 'true'.\n\n commitGraph.changedPathsVersion: Which version should we _write_\n when writing a new commit-graph? Defaults to '1' but will default\n to '2' in the next major verion, then '1' will no longer be an\n accepted value in the version after that.\n\nThe tricky part is that during the commit-graph write, you will\nneed to check the existing filter value to see if it matches. If\nnot, the filters will need to be recomputed from scratch. This\nwill change patch 4 a bit, but it's the right thing to do.\n\nThanks,\n-Stolee\n\n"},{"id":"478572","messageId":"xmqqilbim14d.fsf@gitster.g","threadId":"59781","inReplyTo":"4c5cadf2-bd70-c56e-3935-fb24b604f411@github.com","subject":"Re: [PATCH v4 4/4] commit-graph: new filter ver. that fixes murmur3","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-20T18:37:06Z","receivedAt":"2023-06-20T18:37:14Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee <derrickstolee@github.com> writes:\n\n> However, I think the 'settings->hash_version' is the wrong place\n> to look for the condition. We should be getting this value from the\n> commit-graph we are reading. (More on this later.)\n> ...\n>> -\tif (hash_version != 1)\n>> +\tif (hash_version != d->commit_graph_changed_paths_version)\n>>  \t\treturn 0;\n>\n> This makes it appear like we cannot read a commit-graph that has\n> a Bloom filter version that doesn't match the configured version.\n>\n> This seems incorrect. If we want to configure to _write_ v2, we\n> should still be able to _read_ v1 concurrently until those v2\n> filters are written.\n\nGood eyes.  Thanks for carefully reading.\n\n\n"},{"id":"478604","messageId":"20230620215619.1398374-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"c7b66d2c-cdc3-1f0f-60a0-a2ee21c277bf@github.com","subject":"Re: [PATCH v4 0/4] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-20T21:56:19Z","receivedAt":"2023-06-20T21:56:26Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Derrick Stolee <derrickstolee@github.com> writes:\n> When commitGraph.changedPathsVersion is set, it does not allow\n> reading a previous filter version, leaving us in a poor performance\n> state until the commit-graph file can be rewritten.\n> \n> While I was reviewing, it seemed reasonable to deprecate\n> commitGraph.readChangedPaths, but this use of \"also restrict writes\n> to this version\" didn't make sense to me at the time. Instead, it\n> would be good to have this clarity between the config options:\n> \n>  commitGraph.readChangedPaths: should we read and use the filters\n>  that exist on disk? Defaults to 'true'.\n> \n>  commitGraph.changedPathsVersion: Which version should we _write_\n>  when writing a new commit-graph? Defaults to '1' but will default\n>  to '2' in the next major verion, then '1' will no longer be an\n>  accepted value in the version after that.\n> \n> The tricky part is that during the commit-graph write, you will\n> need to check the existing filter value to see if it matches. If\n> not, the filters will need to be recomputed from scratch. This\n> will change patch 4 a bit, but it's the right thing to do.\n> \n> Thanks,\n> -Stolee\n\nOK - this sounds reasonable. I'll take a look.\n"},{"id":"478636","messageId":"ZJLoSDcETsPMyONL@nand.local","threadId":"59781","inReplyTo":"xmqqlegn3tz3.fsf@gitster.g","subject":"Re: [PATCH v4 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T12:08:40Z","receivedAt":"2023-06-21T12:08:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 13, 2023 at 02:58:24PM -0700, Junio C Hamano wrote:\n> \"bloom\" -> \"Bloom\", probably, as the name comes from the name of its\n> inventor (just like we spell \"Boolean\", not \"boolean\").\n\nIndeed.\n\n> > +\twhen char is signed and the repository has path names that have characters >=\n> > +\t0x80; Git supports reading and writing them, but this ability will be removed\n> > +\tin a future version of Git.\n>\n> Makes sense.\n>\n> I wonder if we want to mention what the undesired misbehaviour the\n> \"bug\" causes and what we do to avoid getting affected by the bug\n> here.  If we can say something like \"When querying for a pathname\n> with a byte with high-bit set, the buggy filter may produce false\n> negative, making the filter unusable, but asking for a pathname\n> without such a byte produces no false negatives (even though we may\n> get false positives).  When Git reads version 1 filter data, it\n> refrains from using it for processing paths with high-bit set to\n> avoid triggering the bug\", then it would be ideal.\n\nYour description of the bug matches my understanding of the issue, that\na corrupt filter would produce false negatives and thus be unusable.\n\nI skimmed through the rest of the series, and couldn't find a spot where\nwe do the latter, i.e. still use v1 filters as long as we don't have any\ncharacters in the path with high-order bits set.\n\nI think this would be as simple as modifying the Bloom filter query\nfunction to return \"maybe\" before even trying to hash a path with at\nleast one character with its high-bit set.\n\nApologies if this functionality is implemented and I just missed it.\n\nThanks,\nTaylor\n"},{"id":"478638","messageId":"ZJLprjbzaSvwkNTd@nand.local","threadId":"59781","inReplyTo":"6adfa53dafc8c34f94a017fdc08460911f53d310.1686677910.git.jonathantanmy@google.com","subject":"Re: [PATCH v4 3/4] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T12:14:38Z","receivedAt":"2023-06-21T12:14:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 13, 2023 at 10:39:57AM -0700, Jonathan Tan wrote:\n> diff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\n> index 30604e4a4c..eaa10bf232 100644\n> --- a/Documentation/config/commitgraph.txt\n> +++ b/Documentation/config/commitgraph.txt\n> @@ -9,6 +9,16 @@ commitGraph.maxNewFilters::\n>  \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n>\n>  commitGraph.readChangedPaths::\n> -\tIf true, then git will use the changed-path Bloom filters in the\n> -\tcommit-graph file (if it exists, and they are present). Defaults to\n> -\ttrue. See linkgit:git-commit-graph[1] for more information.\n> +\tDeprecated. Equivalent to changedPathsVersion=1 if true, and\n> +\tchangedPathsVersion=0 if false.\n> +\n> +commitGraph.changedPathsVersion::\n> +\tSpecifies the version of the changed-path Bloom filters that Git will read and\n> +\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n> +\tmatch the version set in this config variable will be ignored.\n> ++\n> +Defaults to 1.\n> ++\n> +If 0, git will write version 1 Bloom filters when instructed to write.\n> ++\n> +See linkgit:git-commit-graph[1] for more information.\n\nHmm. I'm a little confused: we should still be able to use the old\nbroken filters if (and only if) the paths we're querying don't have any\nbytes with their high-order bit set, no?\n\nThat should be true with the caveat that querying such a path would need\nto result in our querying function returning \"maybe\" instead of\n\"definitely not\" to protect against the false-negatives described\nearlier.\n\nAs I read this, it seems to imply that as soon as this change lands that\nwe'll stop reading old Bloom filters altogether. Is that the case?\n\nIf so, I wonder if we can do this with without needing this\nconfiguration setting at all (by writing the newest version of Bloom\nfilters possible, and working around the existing ones with the\naforementioned workaround).\n\nThanks,\nTaylor\n"},{"id":"478639","messageId":"ZJLq0XE3lSVzQ28k@nand.local","threadId":"59781","inReplyTo":"c7b66d2c-cdc3-1f0f-60a0-a2ee21c277bf@github.com","subject":"Re: [PATCH v4 0/4] Changed path filter hash fix and version bump","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-06-21T12:19:29Z","receivedAt":"2023-06-21T12:19:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Jun 20, 2023 at 09:43:46AM -0400, Derrick Stolee wrote:\n> This version is not ready. The backwards compatibility story is\n> incomplete.\n\nI'm also late to the party, but I agree with Stolee here, having come to\nthe same conclusion about needing to support reading older (corrupt) Bloom\nfilters when possible (i.e. when paths contain no bytes which have their\nhigh-order bits set), and assuming the filter contains all paths\notherwise.\n\n>  commitGraph.changedPathsVersion: Which version should we _write_\n>  when writing a new commit-graph? Defaults to '1' but will default\n>  to '2' in the next major verion, then '1' will no longer be an\n>  accepted value in the version after that.\n\nI am not sure if there's a situation where we'd ever want to not write\nthe newer versions when starting a new commit-graph (or chain) from\nscratch.\n\nI think that follows from what you and I are both suggesting w.r.t\nbackwards compatibility. If that's the case, I think that we could in\ntheory drop this configuration setting altogether.\n\nOr, at the very least, we should be able to change it change only what\nversion we *write*, not read. I think this is what you are suggesting\nabove, but I am not 100% sure, so apologies if I'm just repeating what\nyou've already suggested.\n\n> The tricky part is that during the commit-graph write, you will\n> need to check the existing filter value to see if it matches. If\n> not, the filters will need to be recomputed from scratch. This\n> will change patch 4 a bit, but it's the right thing to do.\n\nYup, good suggestion.\n\nThanks,\nTaylor\n"},{"id":"478650","messageId":"0fd02ca8-bd5d-4143-956f-2c456897b87a@github.com","threadId":"59781","inReplyTo":"ZJLq0XE3lSVzQ28k@nand.local","subject":"Re: [PATCH v4 0/4] Changed path filter hash fix and version bump","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-06-21T17:53:12Z","receivedAt":"2023-06-21T17:55:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/21/2023 8:19 AM, Taylor Blau wrote:\n> On Tue, Jun 20, 2023 at 09:43:46AM -0400, Derrick Stolee wrote:\n\n>>  commitGraph.changedPathsVersion: Which version should we _write_\n>>  when writing a new commit-graph? Defaults to '1' but will default\n>>  to '2' in the next major verion, then '1' will no longer be an\n>>  accepted value in the version after that.\n> \n> I am not sure if there's a situation where we'd ever want to not write\n> the newer versions when starting a new commit-graph (or chain) from\n> scratch.\n\nI'd rather have the choice to start writing the new filter mode be\nmade by config rather than a change to the Git binary. Makes for a\nmore gradual rollout to be sure there aren't issues with the new\nversion.\n\nSo please keep the configuration value, but have it indicate the\nmode used when writing filters.\n\nThanks,\n-Stolee\n"},{"id":"478714","messageId":"20230622222637.2045705-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"ZJLoSDcETsPMyONL@nand.local","subject":"Re: [PATCH v4 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-22T22:26:37Z","receivedAt":"2023-06-22T22:26:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau <me@ttaylorr.com> writes:\n> > I wonder if we want to mention what the undesired misbehaviour the\n> > \"bug\" causes and what we do to avoid getting affected by the bug\n> > here.  If we can say something like \"When querying for a pathname\n> > with a byte with high-bit set, the buggy filter may produce false\n> > negative, making the filter unusable, but asking for a pathname\n> > without such a byte produces no false negatives (even though we may\n> > get false positives).  When Git reads version 1 filter data, it\n> > refrains from using it for processing paths with high-bit set to\n> > avoid triggering the bug\", then it would be ideal.\n> \n> Your description of the bug matches my understanding of the issue, that\n> a corrupt filter would produce false negatives and thus be unusable.\n> \n> I skimmed through the rest of the series, and couldn't find a spot where\n> we do the latter, i.e. still use v1 filters as long as we don't have any\n> characters in the path with high-order bits set.\n> \n> I think this would be as simple as modifying the Bloom filter query\n> function to return \"maybe\" before even trying to hash a path with at\n> least one character with its high-bit set.\n> \n> Apologies if this functionality is implemented and I just missed it.\n> \n> Thanks,\n> Taylor\n\nThanks for the suggestion - yeah, this might work.\n"},{"id":"478715","messageId":"20230622222756.2046051-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"0fd02ca8-bd5d-4143-956f-2c456897b87a@github.com","subject":"Re: [PATCH v4 0/4] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-06-22T22:27:56Z","receivedAt":"2023-06-22T22:28:01Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Derrick Stolee <derrickstolee@github.com> writes:\n> On 6/21/2023 8:19 AM, Taylor Blau wrote:\n> > On Tue, Jun 20, 2023 at 09:43:46AM -0400, Derrick Stolee wrote:\n> \n> >>  commitGraph.changedPathsVersion: Which version should we _write_\n> >>  when writing a new commit-graph? Defaults to '1' but will default\n> >>  to '2' in the next major verion, then '1' will no longer be an\n> >>  accepted value in the version after that.\n> > \n> > I am not sure if there's a situation where we'd ever want to not write\n> > the newer versions when starting a new commit-graph (or chain) from\n> > scratch.\n> \n> I'd rather have the choice to start writing the new filter mode be\n> made by config rather than a change to the Git binary. Makes for a\n> more gradual rollout to be sure there aren't issues with the new\n> version.\n> \n> So please keep the configuration value, but have it indicate the\n> mode used when writing filters.\n> \n> Thanks,\n> -Stolee\n\nIt looks like we can't avoid writing both versions (we need to write\nversion 1 so that we can reuse existing Bloom filters when writing, if\nthe repo has version 1 Bloom filters) so a config that tells us which to\nwrite sounds doable. I'll take a look.\n"},{"id":"478730","messageId":"e57b2272-b269-b705-3d42-d32e0b410f03@github.com","threadId":"59781","inReplyTo":"20230622222637.2045705-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-06-23T13:05:51Z","receivedAt":"2023-06-23T13:06:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/22/2023 6:26 PM, Jonathan Tan wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n>>> I wonder if we want to mention what the undesired misbehaviour the\n>>> \"bug\" causes and what we do to avoid getting affected by the bug\n>>> here.  If we can say something like \"When querying for a pathname\n>>> with a byte with high-bit set, the buggy filter may produce false\n>>> negative, making the filter unusable, but asking for a pathname\n>>> without such a byte produces no false negatives (even though we may\n>>> get false positives).  When Git reads version 1 filter data, it\n>>> refrains from using it for processing paths with high-bit set to\n>>> avoid triggering the bug\", then it would be ideal.\n>>\n>> Your description of the bug matches my understanding of the issue, that\n>> a corrupt filter would produce false negatives and thus be unusable.\n>>\n>> I skimmed through the rest of the series, and couldn't find a spot where\n>> we do the latter, i.e. still use v1 filters as long as we don't have any\n>> characters in the path with high-order bits set.\n>>\n>> I think this would be as simple as modifying the Bloom filter query\n>> function to return \"maybe\" before even trying to hash a path with at\n>> least one character with its high-bit set.\n>>\n>> Apologies if this functionality is implemented and I just missed it.\n>>\n>> Thanks,\n>> Taylor\n> \n> Thanks for the suggestion - yeah, this might work.\n\nIf I understand the situation correctly, the high bits can make the\nhashes \"not very random\" but they are still effective at identifying\nthe \"maybe\" case consistently for the inputs it is given (it would not\npresent a \"no\" when it should not, but it might say \"maybe\" more often\nthan it should). The behavior is only incorrect if the same commit-graph\nfile is used with two different Git versions that were compiled with\ndifferent signed-ness.\n\nIf that is the case, then ignoring the Bloom filters when you see a\nhigh bit would change the performance implication from \"probably\nslower\" to \"definitely slower\" but not affect the correctness in a\nsystem that doesn't have competing Git versions with different\ncompiler semantics.\n\nThat is to say, doing this extra work doesn't seem to be critical to\nmaking this change. The ROI seems too low.\n\nThanks,\n-Stolee\n"},{"id":"478731","messageId":"6510d9ec-2dc6-223d-3578-c6b05bc08f3c@github.com","threadId":"59781","inReplyTo":"20230622222756.2046051-1-jonathantanmy@google.com","subject":"Re: [PATCH v4 0/4] Changed path filter hash fix and version bump","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-06-23T13:18:37Z","receivedAt":"2023-06-23T13:18:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/22/2023 6:27 PM, Jonathan Tan wrote:\n\n> It looks like we can't avoid writing both versions (we need to write\n> version 1 so that we can reuse existing Bloom filters when writing, if\n> the repo has version 1 Bloom filters) so a config that tells us which to\n> write sounds doable. I'll take a look.\n\nI don't fully understand what you're saying here.\n\nWe need to be able to write both versions (not simultaneously, but\ntoggled via config) so we can roll out this change carefully instead\nof suddenly due to the Git executable changing.\n\nBut we don't need to be able to write version 1 just so we can reuse\nversion 1 filters. In fact, we should be able to upgrade to version 2\nif the config points at that, but we should _not_ re-use the filters\nin that case.\n\nThis does present an interesting challenge for the upgrade: we have\nthe commitGraph.maxNewFilters option to limit the amount of new filters\nwe write at a given time. When shifting to version 2, we will start\nfrom scratch, so that could have some effect. I will consider how to\nhandle this, perhaps by raising the number temporarily.\n\nThanks,\n-Stolee\n"},{"id":"479484","messageId":"cover.1689283789.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1684790529.git.jonathantanmy@google.com","subject":"[PATCH v5 0/4] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-13T21:42:07Z","receivedAt":"2023-07-13T21:43:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Sorry it took me a while to get back to this. Looking at the existing\ncode, Bloom filters are passed around a lot without context, especially\nwhen writing - they are generated into a commit slab and then when it is\ntime to write them to disk, they are taken from that commit slab. And\nrather than annotating where they are passed around, I thought it better\nto stick to the single-version approach in version 4 (per Git invocation\nand per repo, only one version), which also sidesteps what happens if\nthere so happens to be multiple commit graphs each with their own Bloom\nfilter version (not possible to be generated by Git but possible with\na hex editor) and what happens if we want to write a different version\nthan what is currently stored in the commit slab. But with an auto-\ndetection of that version, I think we have what we need; in regular\noperation, Git will run with whatever the version on disk is, and when\nit is time to migrate, the user can explicitly specify the version.\n\nI did not implement the mitigation of not using the Bloom filters when\na high-bit path is sought because, as Stolee says, this is useful only\nwhen mixing Git implementations and will slow down operations (without\nany increase in correctness) in the absence of such a mix [1]. But I can\nimplement this if need be.\n\n[1] https://lore.kernel.org/git/e57b2272-b269-b705-3d42-d32e0b410f03@github.com/\n\nJonathan Tan (4):\n  gitformat-commit-graph: describe version 2 of BDAT\n  t4216: test changed path filters with high bit paths\n  repo-settings: introduce commitgraph.changedPathsVersion\n  commit-graph: new filter ver. that fixes murmur3\n\n Documentation/config/commitgraph.txt     |  19 +++-\n Documentation/gitformat-commit-graph.txt |   9 +-\n bloom.c                                  |  65 ++++++++++++-\n bloom.h                                  |   8 +-\n commit-graph.c                           |  33 +++++--\n oss-fuzz/fuzz-commit-graph.c             |   2 +-\n repo-settings.c                          |   6 +-\n repository.h                             |   2 +-\n t/helper/test-bloom.c                    |   9 +-\n t/t0095-bloom.sh                         |   8 ++\n t/t4216-log-bloom.sh                     | 117 +++++++++++++++++++++++\n 11 files changed, 256 insertions(+), 22 deletions(-)\n\nRange-diff against v4:\n1:  a5955cda3d ! 1:  52e281eef0 gitformat-commit-graph: describe version 2 of BDAT\n    @@ Documentation/gitformat-commit-graph.txt: All multi-byte numbers are in network\n      \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n      \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n     -\tin Probabilistic Verification\"\n    -+\tin Probabilistic Verification\". Version 1 bloom filters have a bug that appears\n    ++\tin Probabilistic Verification\". Version 1 Bloom filters have a bug that appears\n     +\twhen char is signed and the repository has path names that have characters >=\n     +\t0x80; Git supports reading and writing them, but this ability will be removed\n     +\tin a future version of Git.\n2:  68732120f9 ! 2:  94a4c7af38 t4216: test changed path filters with high bit paths\n    @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm\n     +test_expect_success 'setup check value of version 1 changed-path' '\n     +\t(cd highbit1 &&\n     +\t\tprintf \"52a9\" >expect &&\n    -+\t\tget_first_changed_path_filter >actual)\n    ++\t\tget_first_changed_path_filter >actual &&\n    ++\t\ttest_cmp expect actual)\n     +'\n     +\n     +# expect will not match actual if char is unsigned by default. Write the test\n3:  44cbcc6a69 ! 3:  131095666d repo-settings: introduce commitgraph.changedPathsVersion\n    @@ Commit message\n         repo-settings: introduce commitgraph.changedPathsVersion\n     \n         A subsequent commit will introduce another version of the changed-path\n    -    filter in the commit graph file. In order to control which version is\n    -    to be accepted when read (and which version to write), a config variable\n    -    is needed.\n    +    filter in the commit graph file. In order to control which version to\n    +    write (and read), a config variable is needed.\n     \n         Therefore, introduce this config variable. For forwards compatibility,\n         teach Git to not read commit graphs when the config variable\n    @@ Commit message\n         This commit does not change the behavior of writing (Git writes changed\n         path filters when explicitly instructed regardless of any config\n         variable), but a subsequent commit will restrict Git such that it will\n    -    only write when commitgraph.changedPathsVersion is 0, 1, or 2.\n    +    only write when commitgraph.changedPathsVersion is a recognized value.\n     \n         Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n         Signed-off-by: Junio C Hamano <gitster@pobox.com>\n    @@ Documentation/config/commitgraph.txt: commitGraph.maxNewFilters::\n     -\tIf true, then git will use the changed-path Bloom filters in the\n     -\tcommit-graph file (if it exists, and they are present). Defaults to\n     -\ttrue. See linkgit:git-commit-graph[1] for more information.\n    -+\tDeprecated. Equivalent to changedPathsVersion=1 if true, and\n    ++\tDeprecated. Equivalent to changedPathsVersion=-1 if true, and\n     +\tchangedPathsVersion=0 if false.\n     +\n     +commitGraph.changedPathsVersion::\n     +\tSpecifies the version of the changed-path Bloom filters that Git will read and\n    -+\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n    ++\twrite. May be -1, 0 or 1. Any changed-path Bloom filters on disk that do not\n     +\tmatch the version set in this config variable will be ignored.\n     ++\n    -+Defaults to 1.\n    ++Defaults to -1.\n    +++\n    ++If -1, Git will use the version of the changed-path Bloom filters in the\n    ++repository, defaulting to 1 if there are none.\n     ++\n     +If 0, git will write version 1 Bloom filters when instructed to write.\n     ++\n    @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repo_settings *s,\n      \t}\n      \n     -\tif (s->commit_graph_read_changed_paths) {\n    -+\tif (s->commit_graph_changed_paths_version == 1) {\n    ++\tif (s->commit_graph_changed_paths_version != 0) {\n      \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n      \t\t\t   &graph->chunk_bloom_indexes);\n      \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n    @@ repo-settings.c: void prepare_repo_settings(struct repository *r)\n     +\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &readChangedPaths, 1);\n     +\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n     +\t\t     &r->settings.commit_graph_changed_paths_version,\n    -+\t\t     readChangedPaths ? 1 : 0);\n    ++\t\t     readChangedPaths ? -1 : 0);\n      \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n      \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n      \n4:  6dee3bfa70 ! 4:  47ba89c565 commit-graph: new filter ver. that fixes murmur3\n    @@ Commit message\n         So this patch does not include any mechanism to \"salvage\" changed path\n         filters from repositories. There is also no \"mixed\" mode - for each\n         invocation of Git, reading and writing changed path filters are done\n    -    with the same version number.\n    +    with the same version number; this version number may be explicitly\n    +    stated (typically if the user knows which version they need) or\n    +    automatically determined from the version of the existing changed path\n    +    filters in the repository.\n     \n         There is a change in write_commit_graph(). graph_read_bloom_data()\n         makes it possible for chunk_bloom_data to be non-NULL but\n    @@ Documentation/config/commitgraph.txt: commitGraph.readChangedPaths::\n      \n      commitGraph.changedPathsVersion::\n      \tSpecifies the version of the changed-path Bloom filters that Git will read and\n    --\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n    -+\twrite. May be 0, 1, or 2. Any changed-path Bloom filters on disk that do not\n    +-\twrite. May be -1, 0 or 1. Any changed-path Bloom filters on disk that do not\n    ++\twrite. May be -1, 0, 1, or 2. Any changed-path Bloom filters on disk that do not\n      \tmatch the version set in this config variable will be ignored.\n      +\n    - Defaults to 1.\n    + Defaults to -1.\n     \n      ## bloom.c ##\n     @@ bloom.c: static int load_bloom_filter_from_graph(struct commit_graph *g,\n    @@ commit-graph.c: static int graph_read_oid_lookup(const unsigned char *chunk_star\n      \n     +struct graph_read_bloom_data_data {\n     +\tstruct commit_graph *g;\n    -+\tint commit_graph_changed_paths_version;\n    ++\tint *commit_graph_changed_paths_version;\n     +};\n     +\n      static int graph_read_bloom_data(const unsigned char *chunk_start,\n    @@ commit-graph.c: static int graph_read_oid_lookup(const unsigned char *chunk_star\n      \thash_version = get_be32(chunk_start);\n      \n     -\tif (hash_version != 1)\n    -+\tif (hash_version != d->commit_graph_changed_paths_version)\n    - \t\treturn 0;\n    +-\t\treturn 0;\n    ++\tif (*d->commit_graph_changed_paths_version == -1) {\n    ++\t\t*d->commit_graph_changed_paths_version = hash_version;\n    ++\t} else if (hash_version != *d->commit_graph_changed_paths_version) {\n    ++ \t\treturn 0;\n    ++\t}\n      \n      \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n    + \tg->bloom_filter_settings->hash_version = hash_version;\n     @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repo_settings *s,\n    - \t\t\tgraph->read_generation_data = 1;\n      \t}\n      \n    --\tif (s->commit_graph_changed_paths_version == 1) {\n    -+\tif (s->commit_graph_changed_paths_version == 1\n    -+\t    || s->commit_graph_changed_paths_version == 2) {\n    + \tif (s->commit_graph_changed_paths_version != 0) {\n     +\t\tstruct graph_read_bloom_data_data data = {\n     +\t\t\t.g = graph,\n    -+\t\t\t.commit_graph_changed_paths_version = s->commit_graph_changed_paths_version\n    ++\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n     +\t\t};\n      \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n      \t\t\t   &graph->chunk_bloom_indexes);\n    @@ commit-graph.c: int write_commit_graph(struct object_directory *odb,\n      \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n      \tctx->num_generation_data_overflows = 0;\n      \n    -+\tif (r->settings.commit_graph_changed_paths_version < 0\n    ++\tif (r->settings.commit_graph_changed_paths_version < -1\n     +\t    || r->settings.commit_graph_changed_paths_version > 2) {\n     +\t\twarning(_(\"attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported\"),\n     +\t\t\tr->settings.commit_graph_changed_paths_version);\n    @@ t/t0095-bloom.sh: test_expect_success 'compute unseeded murmur3 hash for test st\n      \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\n     \n      ## t/t4216-log-bloom.sh ##\n    +@@ t/t4216-log-bloom.sh: get_bdat_offset () {\n    + \t\t.git/objects/info/commit-graph\n    + }\n    + \n    ++get_changed_path_filter_version () {\n    ++\tBDAT_OFFSET=$(get_bdat_offset) &&\n    ++\tperl -0777 -ne \\\n    ++\t\t'print unpack(\"H*\", substr($_, '$BDAT_OFFSET', 4))' \\\n    ++\t\t.git/objects/info/commit-graph\n    ++}\n    ++\n    + get_first_changed_path_filter () {\n    + \tBDAT_OFFSET=$(get_bdat_offset) &&\n    + \tperl -0777 -ne \\\n    +@@ t/t4216-log-bloom.sh: test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n    + \tgit -C highbit1 commit-graph write --reachable --changed-paths\n    + '\n    + \n    +-test_expect_success 'setup check value of version 1 changed-path' '\n    ++test_expect_success 'check value of version 1 changed-path' '\n    + \t(cd highbit1 &&\n    + \t\tprintf \"52a9\" >expect &&\n    + \t\tget_first_changed_path_filter >actual &&\n     @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when version 1 requested' '\n      \t\ttest_bloom_filters_used \"-- $CENT\")\n      '\n    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n     +\t\ttest_bloom_filters_not_used \"-- $CENT\")\n     +'\n     +\n    ++test_expect_success 'version 1 changed-path used when autodetect requested' '\n    ++\t(cd highbit1 &&\n    ++\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n    ++\t\ttest_bloom_filters_used \"-- $CENT\")\n    ++'\n    ++\n    ++test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n    ++\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n    ++\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n    ++\t(cd highbit1 &&\n    ++\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n    ++\t\tprintf \"00000001\" >expect &&\n    ++\t\tget_changed_path_filter_version >actual &&\n    ++\t\ttest_cmp expect actual)\n    ++'\n    ++\n     +test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n     +\tgit init highbit2 &&\n     +\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n     +\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n     +\t\ttest_bloom_filters_not_used \"-- $CENT\")\n     +'\n    ++\n    ++test_expect_success 'version 2 changed-path used when autodetect requested' '\n    ++\t(cd highbit2 &&\n    ++\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n    ++\t\ttest_bloom_filters_used \"-- $CENT\")\n    ++'\n    ++\n    ++test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n    ++\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n    ++\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n    ++\t(cd highbit2 &&\n    ++\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n    ++\t\tprintf \"00000002\" >expect &&\n    ++\t\tget_changed_path_filter_version >actual &&\n    ++\t\ttest_cmp expect actual)\n    ++'\n     +\n      test_done\n-- \n2.41.0.255.g8b1d071c50-goog\n\n"},{"id":"479485","messageId":"131095666d414ff68416526112c8f8deb31ac9e1.1689283789.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689283789.git.jonathantanmy@google.com","subject":"[PATCH v5 3/4] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-13T21:42:10Z","receivedAt":"2023-07-13T21:43:14Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A subsequent commit will introduce another version of the changed-path\nfilter in the commit graph file. In order to control which version to\nwrite (and read), a config variable is needed.\n\nTherefore, introduce this config variable. For forwards compatibility,\nteach Git to not read commit graphs when the config variable\nis set to an unsupported version. Because we teach Git this,\ncommitgraph.readChangedPaths is now redundant, so deprecate it and\ndefine its behavior in terms of the config variable we introduce.\n\nThis commit does not change the behavior of writing (Git writes changed\npath filters when explicitly instructed regardless of any config\nvariable), but a subsequent commit will restrict Git such that it will\nonly write when commitgraph.changedPathsVersion is a recognized value.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt | 19 ++++++++++++++++---\n commit-graph.c                       |  2 +-\n oss-fuzz/fuzz-commit-graph.c         |  2 +-\n repo-settings.c                      |  6 +++++-\n repository.h                         |  2 +-\n 5 files changed, 24 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c..07f3799e05 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -9,6 +9,19 @@ commitGraph.maxNewFilters::\n \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n \n commitGraph.readChangedPaths::\n-\tIf true, then git will use the changed-path Bloom filters in the\n-\tcommit-graph file (if it exists, and they are present). Defaults to\n-\ttrue. See linkgit:git-commit-graph[1] for more information.\n+\tDeprecated. Equivalent to changedPathsVersion=-1 if true, and\n+\tchangedPathsVersion=0 if false.\n+\n+commitGraph.changedPathsVersion::\n+\tSpecifies the version of the changed-path Bloom filters that Git will read and\n+\twrite. May be -1, 0 or 1. Any changed-path Bloom filters on disk that do not\n+\tmatch the version set in this config variable will be ignored.\n++\n+Defaults to -1.\n++\n+If -1, Git will use the version of the changed-path Bloom filters in the\n+repository, defaulting to 1 if there are none.\n++\n+If 0, git will write version 1 Bloom filters when instructed to write.\n++\n+See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c11b59f28b..9b72319450 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -399,7 +399,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_read_changed_paths) {\n+\tif (s->commit_graph_changed_paths_version != 0) {\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\ndiff --git a/oss-fuzz/fuzz-commit-graph.c b/oss-fuzz/fuzz-commit-graph.c\nindex 914026f5d8..b56731f51a 100644\n--- a/oss-fuzz/fuzz-commit-graph.c\n+++ b/oss-fuzz/fuzz-commit-graph.c\n@@ -18,7 +18,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \t * possible.\n \t */\n \tthe_repository->settings.commit_graph_generation_version = 2;\n-\tthe_repository->settings.commit_graph_read_changed_paths = 1;\n+\tthe_repository->settings.commit_graph_changed_paths_version = 1;\n \tg = parse_commit_graph(&the_repository->settings, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 3dbd3f0e2e..e3b6565ffc 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n \tint value;\n \tconst char *strval;\n \tint manyfiles;\n+\tint readChangedPaths;\n \n \tif (!r->gitdir)\n \t\tBUG(\"Cannot add settings for uninitialized repository\");\n@@ -54,7 +55,10 @@ void prepare_repo_settings(struct repository *r)\n \t/* Commit graph config or default, does not cascade (simple) */\n \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n-\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &readChangedPaths, 1);\n+\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n+\t\t     &r->settings.commit_graph_changed_paths_version,\n+\t\t     readChangedPaths ? -1 : 0);\n \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n \ndiff --git a/repository.h b/repository.h\nindex e8c67ffe16..1f1c32a6dd 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -32,7 +32,7 @@ struct repo_settings {\n \n \tint core_commit_graph;\n \tint commit_graph_generation_version;\n-\tint commit_graph_read_changed_paths;\n+\tint commit_graph_changed_paths_version;\n \tint gc_write_commit_graph;\n \tint gc_cruft_packs;\n \tint fetch_write_commit_graph;\n-- \n2.41.0.255.g8b1d071c50-goog\n\n"},{"id":"479486","messageId":"94a4c7af38525d04f3effc035084e661fe382dcd.1689283789.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689283789.git.jonathantanmy@google.com","subject":"[PATCH v5 2/4] t4216: test changed path filters with high bit paths","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-13T21:42:09Z","receivedAt":"2023-07-13T21:43:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Subsequent commits will teach Git another version of changed path\nfilter that has different behavior with paths that contain at least\none character with its high bit set, so test the existing behavior as\na baseline.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n t/t4216-log-bloom.sh | 47 ++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 47 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..0cf208fdf5 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -404,4 +404,51 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n+get_bdat_offset () {\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"N\", \"$1\") if /BDAT\\0\\0\\0\\0(....)/ or exit 1' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n+get_first_changed_path_filter () {\n+\tBDAT_OFFSET=$(get_bdat_offset) &&\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"H*\", substr($_, '$BDAT_OFFSET' + 12, 2))' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n+# chosen to be the same under all Unicode normalization forms\n+CENT=$(printf \"\\302\\242\")\n+\n+test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n+\tgit init highbit1 &&\n+\ttest_commit -C highbit1 c1 \"$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'setup check value of version 1 changed-path' '\n+\t(cd highbit1 &&\n+\t\tprintf \"52a9\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+# expect will not match actual if char is unsigned by default. Write the test\n+# in this way, so that a user running this test script can still see if the two\n+# files match. (It will appear as an ordinary success if they match, and a skip\n+# if not.)\n+if test_cmp highbit1/expect highbit1/actual\n+then\n+\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n+fi\n+test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n+\t# Only the prereq matters for this test.\n+\ttrue\n+'\n+\n+test_expect_success 'version 1 changed-path used when version 1 requested' '\n+\t(cd highbit1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n test_done\n-- \n2.41.0.255.g8b1d071c50-goog\n\n"},{"id":"479487","messageId":"52e281eef0e857de93e5bdc6e0ef50065adaab2c.1689283789.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689283789.git.jonathantanmy@google.com","subject":"[PATCH v5 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-13T21:42:08Z","receivedAt":"2023-07-13T21:43:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The code change to Git to support version 2 will be done in subsequent\ncommits.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/gitformat-commit-graph.txt | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\nindex 31cad585e2..3e906e8030 100644\n--- a/Documentation/gitformat-commit-graph.txt\n+++ b/Documentation/gitformat-commit-graph.txt\n@@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n \n ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n     * It starts with header consisting of three unsigned 32-bit integers:\n-      - Version of the hash algorithm being used. We currently only support\n-\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n+      - Version of the hash algorithm being used. We currently support\n+\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n \timplemented exactly as described in\n \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n-\tin Probabilistic Verification\"\n+\tin Probabilistic Verification\". Version 1 Bloom filters have a bug that appears\n+\twhen char is signed and the repository has path names that have characters >=\n+\t0x80; Git supports reading and writing them, but this ability will be removed\n+\tin a future version of Git.\n       - The number of times a path is hashed and hence the number of bit positions\n \t      that cumulatively determine whether a file is present in the commit.\n       - The minimum number of bits 'b' per entry in the Bloom filter. If the filter\n-- \n2.41.0.255.g8b1d071c50-goog\n\n"},{"id":"479488","messageId":"47ba89c565d3383a8a14272fe52ac274be82d0be.1689283789.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689283789.git.jonathantanmy@google.com","subject":"[PATCH v5 4/4] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-13T21:42:11Z","receivedAt":"2023-07-13T21:43:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 and even with\nGit binaries that were compiled with different signedness of char. This\nbug affects both how Git writes changed path filters to disk and how Git\ninterprets changed path filters on disk.\n\nTherefore, introduce a new version (2) of changed path filters that\ncorrects this problem. The existing version (1) is still supported and\nis still the default, but users should migrate away from it as soon\nas possible.\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter.\n\nSo this patch does not include any mechanism to \"salvage\" changed path\nfilters from repositories. There is also no \"mixed\" mode - for each\ninvocation of Git, reading and writing changed path filters are done\nwith the same version number; this version number may be explicitly\nstated (typically if the user knows which version they need) or\nautomatically determined from the version of the existing changed path\nfilters in the repository.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in t0095 was obtained from another murmur3 implementation\nusing the following Go source code:\n\n  package main\n\n  import \"fmt\"\n  import \"github.com/spaolacci/murmur3\"\n\n  func main() {\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n  }\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt |  2 +-\n bloom.c                              | 65 +++++++++++++++++++++++--\n bloom.h                              |  8 ++--\n commit-graph.c                       | 31 ++++++++++--\n t/helper/test-bloom.c                |  9 +++-\n t/t0095-bloom.sh                     |  8 ++++\n t/t4216-log-bloom.sh                 | 72 +++++++++++++++++++++++++++-\n 7 files changed, 181 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 07f3799e05..b93ccfba8f 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -14,7 +14,7 @@ commitGraph.readChangedPaths::\n \n commitGraph.changedPathsVersion::\n \tSpecifies the version of the changed-path Bloom filters that Git will read and\n-\twrite. May be -1, 0 or 1. Any changed-path Bloom filters on disk that do not\n+\twrite. May be -1, 0, 1, or 2. Any changed-path Bloom filters on disk that do not\n \tmatch the version set in this config variable will be ignored.\n +\n Defaults to -1.\ndiff --git a/bloom.c b/bloom.c\nindex d0730525da..915d8e5a31 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -65,7 +65,64 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n+{\n+\tconst uint32_t c1 = 0xcc9e2d51;\n+\tconst uint32_t c2 = 0x1b873593;\n+\tconst uint32_t r1 = 15;\n+\tconst uint32_t r2 = 13;\n+\tconst uint32_t m = 5;\n+\tconst uint32_t n = 0xe6546b64;\n+\tint i;\n+\tuint32_t k1 = 0;\n+\tconst char *tail;\n+\n+\tint len4 = len / sizeof(uint32_t);\n+\n+\tuint32_t k;\n+\tfor (i = 0; i < len4; i++) {\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n+\t\tk = byte1 | byte2 | byte3 | byte4;\n+\t\tk *= c1;\n+\t\tk = rotate_left(k, r1);\n+\t\tk *= c2;\n+\n+\t\tseed ^= k;\n+\t\tseed = rotate_left(seed, r2) * m + n;\n+\t}\n+\n+\ttail = (data + len4 * sizeof(uint32_t));\n+\n+\tswitch (len & (sizeof(uint32_t) - 1)) {\n+\tcase 3:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n+\t\t/*-fallthrough*/\n+\tcase 2:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n+\t\t/*-fallthrough*/\n+\tcase 1:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n+\t\tk1 *= c1;\n+\t\tk1 = rotate_left(k1, r1);\n+\t\tk1 *= c2;\n+\t\tseed ^= k1;\n+\t\tbreak;\n+\t}\n+\n+\tseed ^= (uint32_t)len;\n+\tseed ^= (seed >> 16);\n+\tseed *= 0x85ebca6b;\n+\tseed ^= (seed >> 13);\n+\tseed *= 0xc2b2ae35;\n+\tseed ^= (seed >> 16);\n+\n+\treturn seed;\n+}\n+\n+static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n {\n \tconst uint32_t c1 = 0xcc9e2d51;\n \tconst uint32_t c2 = 0x1b873593;\n@@ -130,8 +187,10 @@ void fill_bloom_key(const char *data,\n \tint i;\n \tconst uint32_t seed0 = 0x293ae76f;\n \tconst uint32_t seed1 = 0x7e646e2c;\n-\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n-\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n+\tconst uint32_t hash0 = (settings->hash_version == 2\n+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed0, data, len);\n+\tconst uint32_t hash1 = (settings->hash_version == 2\n+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed1, data, len);\n \n \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n \tfor (i = 0; i < settings->num_hashes; i++)\ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..0c33ae282c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -7,9 +7,11 @@ struct repository;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * The newest version is 2, which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -75,7 +77,7 @@ struct bloom_key {\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n \n void fill_bloom_key(const char *data,\n \t\t    size_t len,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 9b72319450..c50107eed5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -302,16 +302,25 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct graph_read_bloom_data_data {\n+\tstruct commit_graph *g;\n+\tint *commit_graph_changed_paths_version;\n+};\n+\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t  size_t chunk_size, void *data)\n {\n-\tstruct commit_graph *g = data;\n+\tstruct graph_read_bloom_data_data *d = data;\n+\tstruct commit_graph *g = d->g;\n \tuint32_t hash_version;\n \tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n-\t\treturn 0;\n+\tif (*d->commit_graph_changed_paths_version == -1) {\n+\t\t*d->commit_graph_changed_paths_version = hash_version;\n+\t} else if (hash_version != *d->commit_graph_changed_paths_version) {\n+ \t\treturn 0;\n+\t}\n \n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n \tg->bloom_filter_settings->hash_version = hash_version;\n@@ -400,10 +409,14 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t}\n \n \tif (s->commit_graph_changed_paths_version != 0) {\n+\t\tstruct graph_read_bloom_data_data data = {\n+\t\t\t.g = graph,\n+\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n+\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t   graph_read_bloom_data, graph);\n+\t\t\t   graph_read_bloom_data, &data);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -2302,6 +2315,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tif (r->settings.commit_graph_changed_paths_version < -1\n+\t    || r->settings.commit_graph_changed_paths_version > 2) {\n+\t\twarning(_(\"attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported\"),\n+\t\t\tr->settings.commit_graph_changed_paths_version);\n+\t\treturn 0;\n+\t}\n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n+\t\t? 2 : 1;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t      bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2331,7 +2352,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 6c900ca668..34b8dd9164 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -48,6 +48,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \"  test-tool bloom get_murmur3 <string>\\n\"\n+\"  test-tool bloom get_murmur3_seven_highbit\\n\"\n \"  test-tool bloom generate_filter <string> [<string>...]\\n\"\n \"  test-tool bloom get_filter_for_commit <commit-hex>\\n\";\n \n@@ -62,7 +63,13 @@ int cmd__bloom(int argc, const char **argv)\n \t\tuint32_t hashed;\n \t\tif (argc < 3)\n \t\t\tusage(bloom_usage);\n-\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n+\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 0cf208fdf5..6ff26e5af5 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -410,6 +410,13 @@ get_bdat_offset () {\n \t\t.git/objects/info/commit-graph\n }\n \n+get_changed_path_filter_version () {\n+\tBDAT_OFFSET=$(get_bdat_offset) &&\n+\tperl -0777 -ne \\\n+\t\t'print unpack(\"H*\", substr($_, '$BDAT_OFFSET', 4))' \\\n+\t\t.git/objects/info/commit-graph\n+}\n+\n get_first_changed_path_filter () {\n \tBDAT_OFFSET=$(get_bdat_offset) &&\n \tperl -0777 -ne \\\n@@ -426,7 +433,7 @@ test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n \tgit -C highbit1 commit-graph write --reachable --changed-paths\n '\n \n-test_expect_success 'setup check value of version 1 changed-path' '\n+test_expect_success 'check value of version 1 changed-path' '\n \t(cd highbit1 &&\n \t\tprintf \"52a9\" >expect &&\n \t\tget_first_changed_path_filter >actual &&\n@@ -451,4 +458,67 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n \t\ttest_bloom_filters_used \"-- $CENT\")\n '\n \n+test_expect_success 'version 1 changed-path not used when version 2 requested' '\n+\t(cd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'version 1 changed-path used when autodetect requested' '\n+\t(cd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n+\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n+\t(cd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\tprintf \"00000001\" >expect &&\n+\t\tget_changed_path_filter_version >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n+\tgit init highbit2 &&\n+\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n+\ttest_commit -C highbit2 c2 \"$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'check value of version 2 changed-path' '\n+\t(cd highbit2 &&\n+\t\tprintf \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+test_expect_success 'version 2 changed-path used when version 2 requested' '\n+\t(cd highbit2 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'version 2 changed-path not used when version 1 requested' '\n+\t(cd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'version 2 changed-path used when autodetect requested' '\n+\t(cd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n+\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n+\t(cd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\tprintf \"00000002\" >expect &&\n+\t\tget_changed_path_filter_version >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n test_done\n-- \n2.41.0.255.g8b1d071c50-goog\n\n"},{"id":"479493","messageId":"xmqqlefjlanl.fsf@gitster.g","threadId":"59781","inReplyTo":"cover.1689283789.git.jonathantanmy@google.com","subject":"Re: [PATCH v5 0/4] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-13T22:16:46Z","receivedAt":"2023-07-13T22:16:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> I did not implement the mitigation of not using the Bloom filters when\n> a high-bit path is sought because, as Stolee says, this is useful only\n> when mixing Git implementations and will slow down operations (without\n> any increase in correctness) in the absence of such a mix [1].\n\nSensible, I guess.\n\n>     @@ Commit message\n>          This commit does not change the behavior of writing (Git writes changed\n>          path filters when explicitly instructed regardless of any config\n>          variable), but a subsequent commit will restrict Git such that it will\n>     -    only write when commitgraph.changedPathsVersion is 0, 1, or 2.\n>     +    only write when commitgraph.changedPathsVersion is a recognized value.\n\nThis is nicer.\n\n>          Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n>          Signed-off-by: Junio C Hamano <gitster@pobox.com>\n>     @@ Documentation/config/commitgraph.txt: commitGraph.maxNewFilters::\n>      -\tIf true, then git will use the changed-path Bloom filters in the\n>      -\tcommit-graph file (if it exists, and they are present). Defaults to\n>      -\ttrue. See linkgit:git-commit-graph[1] for more information.\n>     -+\tDeprecated. Equivalent to changedPathsVersion=1 if true, and\n>     ++\tDeprecated. Equivalent to changedPathsVersion=-1 if true, and\n>      +\tchangedPathsVersion=0 if false.\n\nI forgot to comment on this part earlier, but does the context make\nit clear enough that these `changedPathsVersion` references are\nabout `commitGraph.changedPathsVersion` configuration variable\nwithout fully spelled out?  They sit next to each other right now,\nso it may not be too bad.  If they appeared across more distance,\nI would be worried, though.\n\n>      +commitGraph.changedPathsVersion::\n>      +\tSpecifies the version of the changed-path Bloom filters that Git will read and\n>     -+\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n>     ++\twrite. May be -1, 0 or 1. Any changed-path Bloom filters on disk that do not\n>      +\tmatch the version set in this config variable will be ignored.\n\nSo, any time the user configures this to a different value, we will\nstart to ignore the existing changed-path-filters data in the\nrepository, and when we are told to write commit-graph, we will\nconstruct changed-path-filters data using the new version?\n\n>      ++\n>     -+Defaults to 1.\n>     ++Defaults to -1.\n>     +++\n>     ++If -1, Git will use the version of the changed-path Bloom filters in the\n>     ++repository, defaulting to 1 if there are none.\n\nOK, that was misleading.  The configuration can say \"-1\" and it does\nnot mean \"I'll ignore anything other than version -1\"---it means\n\"I'll read anything\".  The earlier statement should be toned down so\nthat we do not surprise readers, perhaps\n\n    When set to a positive integer value, any changed-path Bloom\n    filters on disk whose version is different from the value are\n    ignored.\n\nto signal that 0 and negative are special.  Then the readers can\nanticipate that special cases are described next.\n\n    When set to -1, then ...\n    When set to 0, then ...\n    Defaults to -1.\n    \nWhen set to the special value -1, what version will we write?\n\n>      +If 0, git will write version 1 Bloom filters when instructed to write.\n\nAnd we will only read 0 and refuse to read 1?  Or we will read both\n0 and 1?\n\nThanks.\n"},{"id":"479496","messageId":"xmqq4jm7l92q.fsf@gitster.g","threadId":"59781","inReplyTo":"94a4c7af38525d04f3effc035084e661fe382dcd.1689283789.git.jonathantanmy@google.com","subject":"Re: [PATCH v5 2/4] t4216: test changed path filters with high bit paths","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-13T22:50:53Z","receivedAt":"2023-07-13T22:51:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Subsequent commits will teach Git another version of changed path\n> filter that has different behavior with paths that contain at least\n> one character with its high bit set, so test the existing behavior as\n> a baseline.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> Signed-off-by: Junio C Hamano <gitster@pobox.com>\n> ---\n>  t/t4216-log-bloom.sh | 47 ++++++++++++++++++++++++++++++++++++++++++++\n>  1 file changed, 47 insertions(+)\n>\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index fa9d32facf..0cf208fdf5 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -404,4 +404,51 @@ test_expect_success 'Bloom generation backfills empty commits' '\n>  \t)\n>  '\n>  \n> +get_bdat_offset () {\n> +\tperl -0777 -ne \\\n> +\t\t'print unpack(\"N\", \"$1\") if /BDAT\\0\\0\\0\\0(....)/ or exit 1' \\\n> +\t\t.git/objects/info/commit-graph\n> +}\n\nHopefully the 8-byte anchoring pattern is unique enough at least for\nthe purpose of this test script.\n"},{"id":"479497","messageId":"xmqqwmz3ju4c.fsf@gitster.g","threadId":"59781","inReplyTo":"xmqqlefjlanl.fsf@gitster.g","subject":"Re: [PATCH v5 0/4] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-13T22:59:15Z","receivedAt":"2023-07-13T22:59:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n>>      +If 0, git will write version 1 Bloom filters when instructed to write.\n>\n> And we will only read 0 and refuse to read 1?  Or we will read both\n> 0 and 1?\n\nAnswering to myself (only this part).  As setting the \"version\"\nvariable to 0 is equivalent to setting \"read\" variable to \"false\",\nwe will refuse to read anything.\n\n"},{"id":"479527","messageId":"20230714184843.1360071-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"xmqqlefjlanl.fsf@gitster.g","subject":"Re: [PATCH v5 0/4] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-14T18:48:43Z","receivedAt":"2023-07-14T18:48:53Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> >          Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> >          Signed-off-by: Junio C Hamano <gitster@pobox.com>\n> >     @@ Documentation/config/commitgraph.txt: commitGraph.maxNewFilters::\n> >      -\tIf true, then git will use the changed-path Bloom filters in the\n> >      -\tcommit-graph file (if it exists, and they are present). Defaults to\n> >      -\ttrue. See linkgit:git-commit-graph[1] for more information.\n> >     -+\tDeprecated. Equivalent to changedPathsVersion=1 if true, and\n> >     ++\tDeprecated. Equivalent to changedPathsVersion=-1 if true, and\n> >      +\tchangedPathsVersion=0 if false.\n> \n> I forgot to comment on this part earlier, but does the context make\n> it clear enough that these `changedPathsVersion` references are\n> about `commitGraph.changedPathsVersion` configuration variable\n> without fully spelled out?  They sit next to each other right now,\n> so it may not be too bad.  If they appeared across more distance,\n> I would be worried, though.\n\nAh, probably better to spell it out. I'll change it.\n\n> >      +commitGraph.changedPathsVersion::\n> >      +\tSpecifies the version of the changed-path Bloom filters that Git will read and\n> >     -+\twrite. May be 0 or 1. Any changed-path Bloom filters on disk that do not\n> >     ++\twrite. May be -1, 0 or 1. Any changed-path Bloom filters on disk that do not\n> >      +\tmatch the version set in this config variable will be ignored.\n> \n> So, any time the user configures this to a different value, we will\n> start to ignore the existing changed-path-filters data in the\n> repository, and when we are told to write commit-graph, we will\n> construct changed-path-filters data using the new version?\n\nYes.\n\n> >     -+Defaults to 1.\n> >     ++Defaults to -1.\n> >     +++\n> >     ++If -1, Git will use the version of the changed-path Bloom filters in the\n> >     ++repository, defaulting to 1 if there are none.\n> \n> OK, that was misleading.  The configuration can say \"-1\" and it does\n> not mean \"I'll ignore anything other than version -1\"---it means\n> \"I'll read anything\".  The earlier statement should be toned down so\n> that we do not surprise readers, perhaps\n\nAh, good point. Will do.\n\n>     When set to a positive integer value, any changed-path Bloom\n>     filters on disk whose version is different from the value are\n>     ignored.\n> \n> to signal that 0 and negative are special.  Then the readers can\n> anticipate that special cases are described next.\n> \n>     When set to -1, then ...\n>     When set to 0, then ...\n>     Defaults to -1.\n>     \n> When set to the special value -1, what version will we write?\n> \n> >      +If 0, git will write version 1 Bloom filters when instructed to write.\n> \n> And we will only read 0 and refuse to read 1?  Or we will read both\n> 0 and 1?\n> \n> Thanks.\n\nCurrently, there is only version 1 (no version 0) and after all the\npatches in this patch set are applied, there will be version 1 and\nversion 2. I think that with your suggestions above, it will be clearer\nto the reader.\n \n"},{"id":"479646","messageId":"ZLgcfvIrV3TapMAp@nand.local","threadId":"59781","inReplyTo":"52e281eef0e857de93e5bdc6e0ef50065adaab2c.1689283789.git.jonathantanmy@google.com","subject":"Re: [PATCH v5 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-19T17:25:18Z","receivedAt":"2023-07-19T17:25:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 13, 2023 at 02:42:08PM -0700, Jonathan Tan wrote:\n> The code change to Git to support version 2 will be done in subsequent\n> commits.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> Signed-off-by: Junio C Hamano <gitster@pobox.com>\n> ---\n>  Documentation/gitformat-commit-graph.txt | 9 ++++++---\n>  1 file changed, 6 insertions(+), 3 deletions(-)\n>\n> diff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\n> index 31cad585e2..3e906e8030 100644\n> --- a/Documentation/gitformat-commit-graph.txt\n> +++ b/Documentation/gitformat-commit-graph.txt\n> @@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n>\n>  ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n\nThis is a little beyond the scope of your series, but since we're\nchanging the on-disk format here a little bit, I think that it might be\nworth it to consider whether there are any other changes that we'd like\nto perform at the same time.\n\nOne that comes to mind is serializing the `max_changed_paths` value of\nthe Bloom filter settings, which is currently hard-coded as a constant,\nc.f. 97ffa4fab50 (commit-graph.c: store maximum changed paths,\n2020-09-17).\n\nWe always assume that the value there is 512, or the environment\nvariable GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS, if it is set. But it\nmight be nice to write it to disk, since it would allow us to do\nsomething like:\n\n--- 8< ---\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facfb..a42b0b03cfb 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -178,11 +178,12 @@ test_expect_success 'persist filter settings' '\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n \t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n \t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=513 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2.txt &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":513\" trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2-auto.txt\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":513\" trace2-auto.txt\n '\n\n test_max_changed_paths () {\n--- >8 ---\n\nWhich is currently not possible (the second grep assertion will fail,\nsince Git has no way to remember what the value of max_changed_paths is\nfrom the existing commit-graph).\n\n> +\tin Probabilistic Verification\". Version 1 Bloom filters have a bug that appears\n> +\twhen char is signed and the repository has path names that have characters >=\n> +\t0x80; Git supports reading and writing them, but this ability will be removed\n> +\tin a future version of Git.\n\nMakes sense.\n\nThanks,\nTaylor\n"},{"id":"479647","messageId":"ZLgc5q0FrL1s7j0j@nand.local","threadId":"59781","inReplyTo":"xmqq4jm7l92q.fsf@gitster.g","subject":"Re: [PATCH v5 2/4] t4216: test changed path filters with high bit paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-19T17:27:02Z","receivedAt":"2023-07-19T17:27:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 13, 2023 at 03:50:53PM -0700, Junio C Hamano wrote:\n> > @@ -404,4 +404,51 @@ test_expect_success 'Bloom generation backfills empty commits' '\n> >  \t)\n> >  '\n> >\n> > +get_bdat_offset () {\n> > +\tperl -0777 -ne \\\n> > +\t\t'print unpack(\"N\", \"$1\") if /BDAT\\0\\0\\0\\0(....)/ or exit 1' \\\n> > +\t\t.git/objects/info/commit-graph\n> > +}\n>\n> Hopefully the 8-byte anchoring pattern is unique enough at least for\n> the purpose of this test script.\n\nI had the same thought myself. I wonder if this would be more\nstraightforward to implement in a test-helper.\n\nThanks,\nTaylor\n"},{"id":"479648","messageId":"cover.1689789226.git.me@ttaylorr.com","threadId":"59781","inReplyTo":"ZLgc5q0FrL1s7j0j@nand.local","subject":"[PATCH 0/4] commit-graph: avoid looking at Bloom filter data directly","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-19T17:55:41Z","receivedAt":"2023-07-19T17:55:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Jonathan,\n\nHere's a few commits (and one fixup!) that could go before the second\npatch of your series, with the fixup! getting squashed into the second\npatch itself.\n\nThe first three are preparatory, but the fourth patch should allow us to\ndrop the Perl hackery necessary to dump the raw contents of arbitrary\nBloom filters.\n\nFeel free to pick up these patches (or not), just wanted to get these\nout there as a possible suggestion.\n\nTaylor Blau (4):\n  t/helper/test-read-graph.c: extract `dump_graph_info()`\n  bloom.h: make `load_bloom_filter_from_graph()` public\n  t/helper/test-read-graph: implement `bloom-filters` mode\n  fixup! t4216: test changed path filters with high bit paths\n\n bloom.c                    |  6 ++--\n bloom.h                    |  5 +++\n t/helper/test-read-graph.c | 67 ++++++++++++++++++++++++++++++--------\n t/t4216-log-bloom.sh       | 14 ++------\n 4 files changed, 64 insertions(+), 28 deletions(-)\n\n--\n2.41.0.366.g215419bf3c2.dirty\n"},{"id":"479649","messageId":"215419bf3c21ce630ce6a8b06e5c1034c6d878f5.1689789226.git.me@ttaylorr.com","threadId":"59781","inReplyTo":"cover.1689789226.git.me@ttaylorr.com","subject":"[PATCH 1/4] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-19T17:55:44Z","receivedAt":"2023-07-19T17:55:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for the 'read-graph' test helper to perform other tasks besides\ndumping high-level information about the commit-graph by extracting its\nmain routine into a separate function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-read-graph.c | 33 ++++++++++++++++++++-------------\n 1 file changed, 20 insertions(+), 13 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 8c7a83f578f..c6649284123 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -5,20 +5,8 @@\n #include \"bloom.h\"\n #include \"setup.h\"\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_info(struct commit_graph *graph)\n {\n-\tstruct commit_graph *graph = NULL;\n-\tstruct object_directory *odb;\n-\n-\tsetup_git_directory();\n-\todb = the_repository->objects->odb;\n-\n-\tprepare_repo_settings(the_repository);\n-\n-\tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n@@ -57,8 +45,27 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tif (graph->topo_levels)\n \t\tprintf(\" topo_levels\");\n \tprintf(\"\\n\");\n+}\n+\n+int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tstruct object_directory *odb;\n+\n+\tsetup_git_directory();\n+\todb = the_repository->objects->odb;\n+\n+\tprepare_repo_settings(the_repository);\n+\n+\tgraph = read_commit_graph_one(the_repository, odb);\n+\tif (!graph)\n+\t\treturn 1;\n+\n+\tdump_graph_info(graph);\n \n \tUNLEAK(graph);\n \n \treturn 0;\n }\n+\n+\n-- \n2.41.0.366.g215419bf3c2.dirty\n\n"},{"id":"479650","messageId":"cc56e0afda2ea8ad501d58d207ce94f802342efa.1689789226.git.me@ttaylorr.com","threadId":"59781","inReplyTo":"cover.1689789226.git.me@ttaylorr.com","subject":"[PATCH 2/4] bloom.h: make `load_bloom_filter_from_graph()` public","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-19T17:55:46Z","receivedAt":"2023-07-19T17:55:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Prepare for a future commit to use the load_bloom_filter_from_graph()\nfunction directly to load specific Bloom filters out of the commit-graph\nfor manual inspection (to be used during tests).\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n bloom.c | 6 +++---\n bloom.h | 5 +++++\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex aef6b5fea2d..3e78cfe79d4 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -29,9 +29,9 @@ static inline unsigned char get_bitmask(uint32_t pos)\n \treturn ((unsigned char)1) << (pos & (BITS_PER_WORD - 1));\n }\n \n-static int load_bloom_filter_from_graph(struct commit_graph *g,\n-\t\t\t\t\tstruct bloom_filter *filter,\n-\t\t\t\t\tuint32_t graph_pos)\n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos)\n {\n \tuint32_t lex_pos, start_index, end_index;\n \ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe212..1e4f612d2c2 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -3,6 +3,7 @@\n \n struct commit;\n struct repository;\n+struct commit_graph;\n \n struct bloom_filter_settings {\n \t/*\n@@ -68,6 +69,10 @@ struct bloom_key {\n \tuint32_t *hashes;\n };\n \n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos);\n+\n /*\n  * Calculate the murmur3 32-bit hash value for the given data\n  * using the given seed.\n-- \n2.41.0.366.g215419bf3c2.dirty\n\n"},{"id":"479651","messageId":"51463636135842c1041af54ea460a54e2cf3055a.1689789226.git.me@ttaylorr.com","threadId":"59781","inReplyTo":"cover.1689789226.git.me@ttaylorr.com","subject":"[PATCH 3/4] t/helper/test-read-graph: implement `bloom-filters` mode","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-19T17:55:49Z","receivedAt":"2023-07-19T17:55:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Implement a mode of the \"read-graph\" test helper to dump out the\nhexadecimal contents of the Bloom filter(s) contained in a commit-graph.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-read-graph.c | 42 +++++++++++++++++++++++++++++++++-----\n 1 file changed, 37 insertions(+), 5 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex c6649284123..da9ac8584da 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -47,10 +47,32 @@ static void dump_graph_info(struct commit_graph *graph)\n \tprintf(\"\\n\");\n }\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_bloom_filters(struct commit_graph *graph)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < graph->num_commits + graph->num_commits_in_base; i++) {\n+\t\tstruct bloom_filter filter = { 0 };\n+\t\tsize_t j;\n+\n+\t\tif (load_bloom_filter_from_graph(graph, &filter, i) < 0) {\n+\t\t\tfprintf(stderr, \"missing Bloom filter for graph \"\n+\t\t\t\t\"position %\"PRIu32\"\\n\", i);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (j = 0; j < filter.len; j++)\n+\t\t\tprintf(\"%02x\", filter.data[j]);\n+\t\tif (filter.len)\n+\t\t\tprintf(\"\\n\");\n+\t}\n+}\n+\n+int cmd__read_graph(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n \tstruct object_directory *odb;\n+\tint ret = 0;\n \n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n@@ -58,14 +80,24 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tprepare_repo_settings(the_repository);\n \n \tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n+\tif (!graph) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n \n-\tdump_graph_info(graph);\n+\tif (argc <= 1)\n+\t\tdump_graph_info(graph);\n+\telse if (!strcmp(argv[1], \"bloom-filters\"))\n+\t\tdump_graph_bloom_filters(graph);\n+\telse {\n+\t\tfprintf(stderr, \"unknown sub-command: '%s'\\n\", argv[1]);\n+\t\tret = 1;\n+\t}\n \n+done:\n \tUNLEAK(graph);\n \n-\treturn 0;\n+\treturn ret;\n }\n \n \n-- \n2.41.0.366.g215419bf3c2.dirty\n\n"},{"id":"479652","messageId":"5bac3bc2e5b85f553e39b9a334ead372d97eb753.1689789226.git.me@ttaylorr.com","threadId":"59781","inReplyTo":"cover.1689789226.git.me@ttaylorr.com","subject":"[PATCH 4/4] fixup! t4216: test changed path filters with high bit paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-19T17:55:52Z","receivedAt":"2023-07-19T17:56:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Signed-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/t4216-log-bloom.sh | 14 +++-----------\n 1 file changed, 3 insertions(+), 11 deletions(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 0cf208fdf55..c49528b947a 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -404,17 +404,9 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n-get_bdat_offset () {\n-\tperl -0777 -ne \\\n-\t\t'print unpack(\"N\", \"$1\") if /BDAT\\0\\0\\0\\0(....)/ or exit 1' \\\n-\t\t.git/objects/info/commit-graph\n-}\n-\n get_first_changed_path_filter () {\n-\tBDAT_OFFSET=$(get_bdat_offset) &&\n-\tperl -0777 -ne \\\n-\t\t'print unpack(\"H*\", substr($_, '$BDAT_OFFSET' + 12, 2))' \\\n-\t\t.git/objects/info/commit-graph\n+\ttest-tool read-graph bloom-filters >filters.dat &&\n+\thead -n 1 filters.dat\n }\n \n # chosen to be the same under all Unicode normalization forms\n@@ -428,7 +420,7 @@ test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n \n test_expect_success 'setup check value of version 1 changed-path' '\n \t(cd highbit1 &&\n-\t\tprintf \"52a9\" >expect &&\n+\t\techo \"52a9\" >expect &&\n \t\tget_first_changed_path_filter >actual &&\n \t\ttest_cmp expect actual)\n '\n-- \n2.41.0.366.g215419bf3c2.dirty\n"},{"id":"479653","messageId":"ZLgnJdo3o9WA5sNy@nand.local","threadId":"59781","inReplyTo":"131095666d414ff68416526112c8f8deb31ac9e1.1689283789.git.jonathantanmy@google.com","subject":"Re: [PATCH v5 3/4] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-19T18:10:45Z","receivedAt":"2023-07-19T18:10:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 13, 2023 at 02:42:10PM -0700, Jonathan Tan wrote:\n> diff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\n> index 30604e4a4c..07f3799e05 100644\n> --- a/Documentation/config/commitgraph.txt\n> +++ b/Documentation/config/commitgraph.txt\n> @@ -9,6 +9,19 @@ commitGraph.maxNewFilters::\n>  \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n>\n>  commitGraph.readChangedPaths::\n> -\tIf true, then git will use the changed-path Bloom filters in the\n> -\tcommit-graph file (if it exists, and they are present). Defaults to\n> -\ttrue. See linkgit:git-commit-graph[1] for more information.\n> +\tDeprecated. Equivalent to changedPathsVersion=-1 if true, and\n> +\tchangedPathsVersion=0 if false.\n\nWhat happens if we have a combination of the two, like:\n\n    [commitGraph]\n        readChangedPaths = false\n        changedPathsVersion = 1\n\n? From reading the implementation below, I think the answer is that\nchangedPathsVersion would win out. I think that's fine behavior to\nimplement (the more modern configuration option taking precedence over\nthe deprecated one). But I think that we should either (a) note that\nprecedence in the documentation here, or (b) issue a warning() when both\nare set.\n\nFor my $.02, I think that doing just (a) would be sufficient.\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index c11b59f28b..9b72319450 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -399,7 +399,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n>  \t\t\tgraph->read_generation_data = 1;\n>  \t}\n>\n> -\tif (s->commit_graph_read_changed_paths) {\n> +\tif (s->commit_graph_changed_paths_version != 0) {\n>  \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n>  \t\t\t   &graph->chunk_bloom_indexes);\n>  \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n\nJust a small note, but writing this as\n\n    if (!s->commit_graph_changed_paths_version)\n\nwould probably be more in line with our coding guidelines.\n\n> diff --git a/repo-settings.c b/repo-settings.c\n> index 3dbd3f0e2e..e3b6565ffc 100644\n> --- a/repo-settings.c\n> +++ b/repo-settings.c\n> @@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n>  \tint value;\n>  \tconst char *strval;\n>  \tint manyfiles;\n> +\tint readChangedPaths;\n\nSmall note: this should be snake-cased like \"read_changed_paths\".\n\n> diff --git a/repository.h b/repository.h\n> index e8c67ffe16..1f1c32a6dd 100644\n> --- a/repository.h\n> +++ b/repository.h\n> @@ -32,7 +32,7 @@ struct repo_settings {\n>\n>  \tint core_commit_graph;\n>  \tint commit_graph_generation_version;\n> -\tint commit_graph_read_changed_paths;\n\nNice, I'm glad that we're getting rid of this variable and replacing it\nwith commit_graph_changed_paths_version instead.\n\n> +\tint commit_graph_changed_paths_version;\n\nLooking good.\n\nThanks,\nTaylor\n"},{"id":"479655","messageId":"ZLgqbB2JG8+HPoHN@nand.local","threadId":"59781","inReplyTo":"47ba89c565d3383a8a14272fe52ac274be82d0be.1689283789.git.jonathantanmy@google.com","subject":"Re: [PATCH v5 4/4] commit-graph: new filter ver. that fixes murmur3","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-19T18:24:44Z","receivedAt":"2023-07-19T18:24:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 13, 2023 at 02:42:11PM -0700, Jonathan Tan wrote:\n> The murmur3 implementation in bloom.c has a bug when converting series\n> of 4 bytes into network-order integers when char is signed (which is\n> controllable by a compiler option, and the default signedness of char is\n> platform-specific). When a string contains characters with the high bit\n> set, this bug causes results that, although internally consistent within\n> Git, does not accord with other implementations of murmur3 and even with\n> Git binaries that were compiled with different signedness of char. This\n> bug affects both how Git writes changed path filters to disk and how Git\n> interprets changed path filters on disk.\n\nI think that you make a worthwhile point that the existing\nimplementation is internally consistent, but doesn't actually implement\na conventional murmur3. I wonder if it's worth being explicit where you\nmention its internal consistency to say that the existing implementation\nwould never cause us to produce wrong results, but wouldn't be readable\nby other off-the-shelf implementations of murmur3.\n\n(To be clear, I think that you already make this point, I'm just\nsuggesting that it may be worth spelling it out even more explicitly\nthan what is written above).\n\n> Therefore, introduce a new version (2) of changed path filters that\n> corrects this problem. The existing version (1) is still supported and\n> is still the default, but users should migrate away from it as soon\n> as possible.\n\nMakes sense.\n\n> Because this bug only manifests with characters that have the high bit\n> set, it may be possible that some (or all) commits in a given repo would\n> have the same changed path filter both before and after this fix is\n> applied. However, in order to determine whether this is the case, the\n> changed paths would first have to be computed, at which point it is not\n> much more expensive to just compute a new changed path filter.\n\nHmm. I think in the general case that is true, but I wonder if there's a\nshortcut we could take for trees that are known to not have *any*\ncharacters with their high-order bits set. That is, if we scan both of\nthe first parent's trees and determine that no such paths exist, the\ncontents of the Bloom filter would be the same in either version, right?\n\nI think that that would be faster than recomputing all filters from\nscratch. In either case, we have to load the whole tree. But if we can\nquickly scan (and cache our results by setting some bit--indicating the\nabsence of paths with characters having their highest bit set--on the tree\nobjects' `flags` field), then we should be able to copy forward the\nexisting representation of the filter.\n\nI think the early checks would be more expensive, since in the worst\ncase you have to walk the entire tree, only to realize that you actually\nwanted to compute a first-parent tree diff, meaning you have to\nessentially repeat the whole walk over again. But for repositories that\nhave few or no commits whose Bloom filters need computing, I think it\nwould be significantly faster, since many of the sub-trees wouldn't need\nto be visited again.\n\n> There is a change in write_commit_graph(). graph_read_bloom_data()\n> makes it possible for chunk_bloom_data to be non-NULL but\n> bloom_filter_settings to be NULL, which causes a segfault later on. I\n> produced such a segfault while developing this patch, but couldn't find\n> a way to reproduce it neither after this complete patch (or before),\n> but in any case it seemed like a good thing to include that might help\n> future patch authors.\n\nHmm. Interesting, I'd love to know more about what you were doing that\nproduced the segfault. I think it would be worth it to prove to\nourselves that this segfault can't occur in the wild. Or if it can, it\nwould be worth it to understand the bug and prevent it from happening.\n\n> +static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n>  {\n>  \tconst uint32_t c1 = 0xcc9e2d51;\n>  \tconst uint32_t c2 = 0x1b873593;\n> @@ -130,8 +187,10 @@ void fill_bloom_key(const char *data,\n>  \tint i;\n>  \tconst uint32_t seed0 = 0x293ae76f;\n>  \tconst uint32_t seed1 = 0x7e646e2c;\n> -\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n> -\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n> +\tconst uint32_t hash0 = (settings->hash_version == 2\n> +\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed0, data, len);\n> +\tconst uint32_t hash1 = (settings->hash_version == 2\n> +\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed1, data, len);\n\nI do admire the ternary operator over the function being called, as I\nthink that Stolee pointed out earlier in this series. But I worry that\nthese two checks could fall out of sync with each other, causing us to\npick incosistent values for hash0, and hash1, respectively.\n\nFWIW, I would probably write this as:\n\n    const uint32_t hash0, hash1;\n    if (settings->hash_version == 2) {\n        hash0 = murmur3_seeded_v2(seed0, data, len);\n        hash1 = murmur3_seeded_v2(seed1, data, len);\n    } else {\n        hash0 = murmur3_seeded_v1(seed0, data, len);\n        hash1 = murmur3_seeded_v1(seed1, data, len);\n    }\n\nI suppose that there isn't anything keeping the calls within each arm of\nthe if-statement above in sync with each other (i.e., I could call\nmurmur3_seeded_v1() immediately before dispatching a call to\nmurmur3_seeded_v2()). So in that sense I think that this is no more or\nless safe than what's already written.\n\nBut IMHO I think this one reads more cleanly, so I might prefer it over\nwhat you have in this patch. But I don't feel so strongly about it\neither way.\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 9b72319450..c50107eed5 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -302,16 +302,25 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n>  \treturn 0;\n>  }\n>\n> +struct graph_read_bloom_data_data {\n> +\tstruct commit_graph *g;\n> +\tint *commit_graph_changed_paths_version;\n> +};\n> +\n\nNit: maybe `graph_read_bloom_data_context`, to avoid repeating \"data\"? I\ndon't have strong feelings here, FWIW.\n\nThe rest of the implementation and tests look good to me.\n\nThanks,\nTaylor\n"},{"id":"479656","messageId":"xmqqwmyvaemr.fsf@gitster.g","threadId":"59781","inReplyTo":"cover.1689789226.git.me@ttaylorr.com","subject":"Re: [PATCH 0/4] commit-graph: avoid looking at Bloom filter data directly","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-19T19:24:28Z","receivedAt":"2023-07-19T19:24:58Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> The first three are preparatory, but the fourth patch should allow us to\n> drop the Perl hackery necessary to dump the raw contents of arbitrary\n> Bloom filters.\n>\n> Feel free to pick up these patches (or not), just wanted to get these\n> out there as a possible suggestion.\n\nThanks, these seem like a good thing to do.  Would love to see an\nupdated version with them rolled in.\n\nThanks both for working so well together.\n\n>\n> Taylor Blau (4):\n>   t/helper/test-read-graph.c: extract `dump_graph_info()`\n>   bloom.h: make `load_bloom_filter_from_graph()` public\n>   t/helper/test-read-graph: implement `bloom-filters` mode\n>   fixup! t4216: test changed path filters with high bit paths\n>\n>  bloom.c                    |  6 ++--\n>  bloom.h                    |  5 +++\n>  t/helper/test-read-graph.c | 67 ++++++++++++++++++++++++++++++--------\n>  t/t4216-log-bloom.sh       | 14 ++------\n>  4 files changed, 64 insertions(+), 28 deletions(-)\n>\n> --\n> 2.41.0.366.g215419bf3c2.dirty\n"},{"id":"479679","messageId":"20230720202006.3815802-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"ZLgcfvIrV3TapMAp@nand.local","subject":"Re: [PATCH v5 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T20:20:06Z","receivedAt":"2023-07-20T20:20:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau <me@ttaylorr.com> writes:\n> This is a little beyond the scope of your series, but since we're\n> changing the on-disk format here a little bit, I think that it might be\n> worth it to consider whether there are any other changes that we'd like\n> to perform at the same time.\n> \n> One that comes to mind is serializing the `max_changed_paths` value of\n> the Bloom filter settings, which is currently hard-coded as a constant,\n> c.f. 97ffa4fab50 (commit-graph.c: store maximum changed paths,\n> 2020-09-17).\n> \n> We always assume that the value there is 512, or the environment\n> variable GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS, if it is set. But it\n> might be nice to write it to disk, since it would allow us to do\n> something like:\n\nThat's true...I do think it makes sense to either have both\nbits_per_entry and max_changed_paths (because if we are reusing Bloom\nfilters when writing, presumably we would want to make sure that the\nexisting ones were generated using the same settings), or have neither\n(since we don't need them for reading).\n\nHaving said that, I am inclined to not change this, so that the offset\ncalculations are the same for both versions (e.g. in the test tool\ntoo), and as far as I know, we haven't had problems with this. But I can\nchange it if people want.\n\n> > +\tin Probabilistic Verification\". Version 1 Bloom filters have a bug that appears\n> > +\twhen char is signed and the repository has path names that have characters >=\n> > +\t0x80; Git supports reading and writing them, but this ability will be removed\n> > +\tin a future version of Git.\n> \n> Makes sense.\n> \n> Thanks,\n> Taylor\n\nThanks for taking a look.\n"},{"id":"479680","messageId":"20230720202203.3816309-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689789226.git.me@ttaylorr.com","subject":"Re: [PATCH 0/4] commit-graph: avoid looking at Bloom filter data directly","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T20:22:03Z","receivedAt":"2023-07-20T20:22:09Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau <me@ttaylorr.com> writes:\n> Hi Jonathan,\n> \n> Here's a few commits (and one fixup!) that could go before the second\n> patch of your series, with the fixup! getting squashed into the second\n> patch itself.\n> \n> The first three are preparatory, but the fourth patch should allow us to\n> drop the Perl hackery necessary to dump the raw contents of arbitrary\n> Bloom filters.\n> \n> Feel free to pick up these patches (or not), just wanted to get these\n> out there as a possible suggestion.\n\nAh, thanks! I'm on the fence myself about whether this really needs to\nbe more rigorous (the Perl part just gives us an offset; we still check\nthat what's at the offset is correct) but since you have written the\npatches already, I'll just include them.\n \n"},{"id":"479682","messageId":"20230720204232.1328482-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"ZLgnJdo3o9WA5sNy@nand.local","subject":"Re: [PATCH v5 3/4] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T20:42:32Z","receivedAt":"2023-07-20T20:42:39Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau <me@ttaylorr.com> writes:\n> What happens if we have a combination of the two, like:\n> \n>     [commitGraph]\n>         readChangedPaths = false\n>         changedPathsVersion = 1\n> \n> ? From reading the implementation below, I think the answer is that\n> changedPathsVersion would win out. I think that's fine behavior to\n> implement (the more modern configuration option taking precedence over\n> the deprecated one). But I think that we should either (a) note that\n> precedence in the documentation here, or (b) issue a warning() when both\n> are set.\n> \n> For my $.02, I think that doing just (a) would be sufficient.\n\nThanks. I added a note to the documentation.\n\n> > -\tif (s->commit_graph_read_changed_paths) {\n> > +\tif (s->commit_graph_changed_paths_version != 0) {\n> >  \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n> >  \t\t\t   &graph->chunk_bloom_indexes);\n> >  \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n> \n> Just a small note, but writing this as\n> \n>     if (!s->commit_graph_changed_paths_version)\n> \n> would probably be more in line with our coding guidelines.\n\nAh, yes. Changed it (without the exclamation mark).\n\n> > diff --git a/repo-settings.c b/repo-settings.c\n> > index 3dbd3f0e2e..e3b6565ffc 100644\n> > --- a/repo-settings.c\n> > +++ b/repo-settings.c\n> > @@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n> >  \tint value;\n> >  \tconst char *strval;\n> >  \tint manyfiles;\n> > +\tint readChangedPaths;\n> \n> Small note: this should be snake-cased like \"read_changed_paths\".\n\nWhoops...thanks for the catch.\n\n \n"},{"id":"479684","messageId":"ZLmg/58O9EsHi7Rn@nand.local","threadId":"59781","inReplyTo":"20230720204232.1328482-1-jonathantanmy@google.com","subject":"Re: [PATCH v5 3/4] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-20T21:02:55Z","receivedAt":"2023-07-20T21:03:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 20, 2023 at 01:42:32PM -0700, Jonathan Tan wrote:\n> > > -\tif (s->commit_graph_read_changed_paths) {\n> > > +\tif (s->commit_graph_changed_paths_version != 0) {\n> > >  \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n> > >  \t\t\t   &graph->chunk_bloom_indexes);\n> > >  \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n> >\n> > Just a small note, but writing this as\n> >\n> >     if (!s->commit_graph_changed_paths_version)\n> >\n> > would probably be more in line with our coding guidelines.\n>\n> Ah, yes. Changed it (without the exclamation mark).\n\nHah, whoops -- thanks for spotting my mistake there. :-)\n\nThanks,\nTaylor\n"},{"id":"479685","messageId":"20230720212753.2008505-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"ZLgqbB2JG8+HPoHN@nand.local","subject":"Re: [PATCH v5 4/4] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T21:27:53Z","receivedAt":"2023-07-20T21:27:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau <me@ttaylorr.com> writes:\n> On Thu, Jul 13, 2023 at 02:42:11PM -0700, Jonathan Tan wrote:\n> > The murmur3 implementation in bloom.c has a bug when converting series\n> > of 4 bytes into network-order integers when char is signed (which is\n> > controllable by a compiler option, and the default signedness of char is\n> > platform-specific). When a string contains characters with the high bit\n> > set, this bug causes results that, although internally consistent within\n> > Git, does not accord with other implementations of murmur3 and even with\n> > Git binaries that were compiled with different signedness of char. This\n> > bug affects both how Git writes changed path filters to disk and how Git\n> > interprets changed path filters on disk.\n> \n> I think that you make a worthwhile point that the existing\n> implementation is internally consistent, but doesn't actually implement\n> a conventional murmur3. I wonder if it's worth being explicit where you\n> mention its internal consistency to say that the existing implementation\n> would never cause us to produce wrong results, but wouldn't be readable\n> by other off-the-shelf implementations of murmur3.\n> \n> (To be clear, I think that you already make this point, I'm just\n> suggesting that it may be worth spelling it out even more explicitly\n> than what is written above).\n\nOK, I've added some more text describing this.\n\n> > Because this bug only manifests with characters that have the high bit\n> > set, it may be possible that some (or all) commits in a given repo would\n> > have the same changed path filter both before and after this fix is\n> > applied. However, in order to determine whether this is the case, the\n> > changed paths would first have to be computed, at which point it is not\n> > much more expensive to just compute a new changed path filter.\n> \n> Hmm. I think in the general case that is true, but I wonder if there's a\n> shortcut we could take for trees that are known to not have *any*\n> characters with their high-order bits set. That is, if we scan both of\n> the first parent's trees and determine that no such paths exist, the\n> contents of the Bloom filter would be the same in either version, right?\n> \n> I think that that would be faster than recomputing all filters from\n> scratch. In either case, we have to load the whole tree. But if we can\n> quickly scan (and cache our results by setting some bit--indicating the\n> absence of paths with characters having their highest bit set--on the tree\n> objects' `flags` field), then we should be able to copy forward the\n> existing representation of the filter.\n> \n> I think the early checks would be more expensive, since in the worst\n> case you have to walk the entire tree, only to realize that you actually\n> wanted to compute a first-parent tree diff, meaning you have to\n> essentially repeat the whole walk over again. But for repositories that\n> have few or no commits whose Bloom filters need computing, I think it\n> would be significantly faster, since many of the sub-trees wouldn't need\n> to be visited again.\n\nSo for repositories that need little-to-no recomputation of Bloom\nfilters, your idea likely means that each tree needs to be read once,\nas compared to recomputing everything in which, I think, each tree needs\nto be read roughly twice (once when computing the Bloom filter for the\ncommit that introduces it, and once for the commit that substitutes a\ndifferent tree in place).\n\nI could change the text of the commit message to discuss this (instead\nof the blanket statement that it would be too hard), although I think\nthat an implementation of this can be done after this patchset. What do\nyou think?\n\n> > There is a change in write_commit_graph(). graph_read_bloom_data()\n> > makes it possible for chunk_bloom_data to be non-NULL but\n> > bloom_filter_settings to be NULL, which causes a segfault later on. I\n> > produced such a segfault while developing this patch, but couldn't find\n> > a way to reproduce it neither after this complete patch (or before),\n> > but in any case it seemed like a good thing to include that might help\n> > future patch authors.\n> \n> Hmm. Interesting, I'd love to know more about what you were doing that\n> produced the segfault. I think it would be worth it to prove to\n> ourselves that this segfault can't occur in the wild. Or if it can, it\n> would be worth it to understand the bug and prevent it from happening.\n\nIf I remember correctly, I changed the version in\nDEFAULT_BLOOM_FILTER_SETTINGS to 2 and ran it to see what broke. This\nwas a while back so I don't remember it exactly, though.\n\n> > @@ -130,8 +187,10 @@ void fill_bloom_key(const char *data,\n> >  \tint i;\n> >  \tconst uint32_t seed0 = 0x293ae76f;\n> >  \tconst uint32_t seed1 = 0x7e646e2c;\n> > -\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n> > -\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n> > +\tconst uint32_t hash0 = (settings->hash_version == 2\n> > +\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed0, data, len);\n> > +\tconst uint32_t hash1 = (settings->hash_version == 2\n> > +\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed1, data, len);\n> \n> I do admire the ternary operator over the function being called, as I\n> think that Stolee pointed out earlier in this series. But I worry that\n> these two checks could fall out of sync with each other, causing us to\n> pick incosistent values for hash0, and hash1, respectively.\n> \n> FWIW, I would probably write this as:\n> \n>     const uint32_t hash0, hash1;\n>     if (settings->hash_version == 2) {\n>         hash0 = murmur3_seeded_v2(seed0, data, len);\n>         hash1 = murmur3_seeded_v2(seed1, data, len);\n>     } else {\n>         hash0 = murmur3_seeded_v1(seed0, data, len);\n>         hash1 = murmur3_seeded_v1(seed1, data, len);\n>     }\n> \n> I suppose that there isn't anything keeping the calls within each arm of\n> the if-statement above in sync with each other (i.e., I could call\n> murmur3_seeded_v1() immediately before dispatching a call to\n> murmur3_seeded_v2()). So in that sense I think that this is no more or\n> less safe than what's already written.\n> \n> But IMHO I think this one reads more cleanly, so I might prefer it over\n> what you have in this patch. But I don't feel so strongly about it\n> either way.\n\nI'm OK either way, so I'll go with your approach (except the \"const\",\nsince my compiler doesn't like that).\n\n> > @@ -302,16 +302,25 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n> >  \treturn 0;\n> >  }\n> >\n> > +struct graph_read_bloom_data_data {\n> > +\tstruct commit_graph *g;\n> > +\tint *commit_graph_changed_paths_version;\n> > +};\n> > +\n> \n> Nit: maybe `graph_read_bloom_data_context`, to avoid repeating \"data\"? I\n> don't have strong feelings here, FWIW.\n\nI'll use \"context\".\n\n> The rest of the implementation and tests look good to me.\n> \n> Thanks,\n> Taylor\n\nThanks for the review.\n"},{"id":"479687","messageId":"cover.1689889382.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1684790529.git.jonathantanmy@google.com","subject":"[PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T21:46:33Z","receivedAt":"2023-07-20T21:46:54Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks, Junio and Taylor, for your reviews. I have included Taylor's\npatches in this patch set.\n\nThere seemed to be some merge conflicts when I tried to apply the\npatches Taylor provided on the base that I built my patches on (that is,\nthe base of jt/path-filter-fix, namely, maint-2.40), so I have rebased\nall my patches onto latest master.\n\nJonathan Tan (4):\n  gitformat-commit-graph: describe version 2 of BDAT\n  t4216: test changed path filters with high bit paths\n  repo-settings: introduce commitgraph.changedPathsVersion\n  commit-graph: new filter ver. that fixes murmur3\n\nTaylor Blau (3):\n  t/helper/test-read-graph.c: extract `dump_graph_info()`\n  bloom.h: make `load_bloom_filter_from_graph()` public\n  t/helper/test-read-graph: implement `bloom-filters` mode\n\n Documentation/config/commitgraph.txt     |  26 +++++-\n Documentation/gitformat-commit-graph.txt |   9 +-\n bloom.c                                  |  75 ++++++++++++++--\n bloom.h                                  |  13 ++-\n commit-graph.c                           |  33 +++++--\n oss-fuzz/fuzz-commit-graph.c             |   2 +-\n repo-settings.c                          |   6 +-\n repository.h                             |   2 +-\n t/helper/test-bloom.c                    |   9 +-\n t/helper/test-read-graph.c               |  67 ++++++++++++---\n t/t0095-bloom.sh                         |   8 ++\n t/t4216-log-bloom.sh                     | 104 +++++++++++++++++++++++\n 12 files changed, 315 insertions(+), 39 deletions(-)\n\nRange-diff against v5:\n1:  efe7f40fed = 1:  3ce6090a4d gitformat-commit-graph: describe version 2 of BDAT\n-:  ---------- > 2:  1955734d1f t/helper/test-read-graph.c: extract `dump_graph_info()`\n-:  ---------- > 3:  4cf7c2f634 bloom.h: make `load_bloom_filter_from_graph()` public\n-:  ---------- > 4:  47b55758e6 t/helper/test-read-graph: implement `bloom-filters` mode\n2:  f684d07971 ! 5:  5276e6a90e t4216: test changed path filters with high bit paths\n    @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm\n      \t)\n      '\n      \n    -+get_bdat_offset () {\n    -+\tperl -0777 -ne \\\n    -+\t\t'print unpack(\"N\", \"$1\") if /BDAT\\0\\0\\0\\0(....)/ or exit 1' \\\n    -+\t\t.git/objects/info/commit-graph\n    -+}\n    -+\n     +get_first_changed_path_filter () {\n    -+\tBDAT_OFFSET=$(get_bdat_offset) &&\n    -+\tperl -0777 -ne \\\n    -+\t\t'print unpack(\"H*\", substr($_, '$BDAT_OFFSET' + 12, 2))' \\\n    -+\t\t.git/objects/info/commit-graph\n    ++\ttest-tool read-graph bloom-filters >filters.dat &&\n    ++\thead -n 1 filters.dat\n     +}\n     +\n     +# chosen to be the same under all Unicode normalization forms\n    @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm\n     +\n     +test_expect_success 'setup check value of version 1 changed-path' '\n     +\t(cd highbit1 &&\n    -+\t\tprintf \"52a9\" >expect &&\n    ++\t\techo \"52a9\" >expect &&\n     +\t\tget_first_changed_path_filter >actual &&\n     +\t\ttest_cmp expect actual)\n     +'\n3:  2fadd87063 ! 6:  dc3f6d2d4f repo-settings: introduce commitgraph.changedPathsVersion\n    @@ Documentation/config/commitgraph.txt: commitGraph.maxNewFilters::\n     -\tIf true, then git will use the changed-path Bloom filters in the\n     -\tcommit-graph file (if it exists, and they are present). Defaults to\n     -\ttrue. See linkgit:git-commit-graph[1] for more information.\n    -+\tDeprecated. Equivalent to changedPathsVersion=-1 if true, and\n    -+\tchangedPathsVersion=0 if false.\n    ++\tDeprecated. Equivalent to commitGraph.changedPathsVersion=-1 if true, and\n    ++\tcommitGraph.changedPathsVersion=0 if false. (If commitGraph.changedPathVersion\n    ++\tis also set, commitGraph.changedPathsVersion takes precedence.)\n     +\n     +commitGraph.changedPathsVersion::\n     +\tSpecifies the version of the changed-path Bloom filters that Git will read and\n    -+\twrite. May be -1, 0 or 1. Any changed-path Bloom filters on disk that do not\n    -+\tmatch the version set in this config variable will be ignored.\n    ++\twrite. May be -1, 0 or 1.\n     ++\n     +Defaults to -1.\n     ++\n     +If -1, Git will use the version of the changed-path Bloom filters in the\n     +repository, defaulting to 1 if there are none.\n     ++\n    -+If 0, git will write version 1 Bloom filters when instructed to write.\n    ++If 0, Git will not read any Bloom filters, and will write version 1 Bloom\n    ++filters when instructed to write.\n    +++\n    ++If 1, Git will only read version 1 Bloom filters, and will write version 1\n    ++Bloom filters.\n     ++\n     +See linkgit:git-commit-graph[1] for more information.\n     \n    @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repo_settings *s,\n      \t}\n      \n     -\tif (s->commit_graph_read_changed_paths) {\n    -+\tif (s->commit_graph_changed_paths_version != 0) {\n    ++\tif (s->commit_graph_changed_paths_version) {\n      \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n      \t\t\t   &graph->chunk_bloom_indexes);\n      \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n    @@ repo-settings.c: void prepare_repo_settings(struct repository *r)\n      \tint value;\n      \tconst char *strval;\n      \tint manyfiles;\n    -+\tint readChangedPaths;\n    ++\tint read_changed_paths;\n      \n      \tif (!r->gitdir)\n      \t\tBUG(\"Cannot add settings for uninitialized repository\");\n    @@ repo-settings.c: void prepare_repo_settings(struct repository *r)\n      \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n      \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n     -\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n    -+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &readChangedPaths, 1);\n    ++\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &read_changed_paths, 1);\n     +\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n     +\t\t     &r->settings.commit_graph_changed_paths_version,\n    -+\t\t     readChangedPaths ? -1 : 0);\n    ++\t\t     read_changed_paths ? -1 : 0);\n      \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n      \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n      \n    @@ repository.h: struct repo_settings {\n     -\tint commit_graph_read_changed_paths;\n     +\tint commit_graph_changed_paths_version;\n      \tint gc_write_commit_graph;\n    - \tint gc_cruft_packs;\n      \tint fetch_write_commit_graph;\n    + \tint command_requires_full_index;\n4:  e31711ae85 ! 7:  6e2d797406 commit-graph: new filter ver. that fixes murmur3\n    @@ Commit message\n         controllable by a compiler option, and the default signedness of char is\n         platform-specific). When a string contains characters with the high bit\n         set, this bug causes results that, although internally consistent within\n    -    Git, does not accord with other implementations of murmur3 and even with\n    -    Git binaries that were compiled with different signedness of char. This\n    -    bug affects both how Git writes changed path filters to disk and how Git\n    -    interprets changed path filters on disk.\n    +    Git, does not accord with other implementations of murmur3 (thus,\n    +    the changed path filters wouldn't be readable by other off-the-shelf\n    +    implementatios of murmur3) and even with Git binaries that were compiled\n    +    with different signedness of char. This bug affects both how Git writes\n    +    changed path filters to disk and how Git interprets changed path filters\n    +    on disk.\n     \n         Therefore, introduce a new version (2) of changed path filters that\n         corrects this problem. The existing version (1) is still supported and\n    @@ Documentation/config/commitgraph.txt: commitGraph.readChangedPaths::\n      \n      commitGraph.changedPathsVersion::\n      \tSpecifies the version of the changed-path Bloom filters that Git will read and\n    --\twrite. May be -1, 0 or 1. Any changed-path Bloom filters on disk that do not\n    -+\twrite. May be -1, 0, 1, or 2. Any changed-path Bloom filters on disk that do not\n    - \tmatch the version set in this config variable will be ignored.\n    +-\twrite. May be -1, 0 or 1.\n    ++\twrite. May be -1, 0, 1, or 2.\n      +\n      Defaults to -1.\n    + +\n    +@@ Documentation/config/commitgraph.txt: filters when instructed to write.\n    + If 1, Git will only read version 1 Bloom filters, and will write version 1\n    + Bloom filters.\n    + +\n    ++If 2, Git will only read version 2 Bloom filters, and will write version 2\n    ++Bloom filters.\n    +++\n    + See linkgit:git-commit-graph[1] for more information.\n     \n      ## bloom.c ##\n    -@@ bloom.c: static int load_bloom_filter_from_graph(struct commit_graph *g,\n    +@@ bloom.c: int load_bloom_filter_from_graph(struct commit_graph *g,\n       * Not considered to be cryptographically secure.\n       * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n       */\n    @@ bloom.c: void fill_bloom_key(const char *data,\n      \tconst uint32_t seed1 = 0x7e646e2c;\n     -\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n     -\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n    -+\tconst uint32_t hash0 = (settings->hash_version == 2\n    -+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed0, data, len);\n    -+\tconst uint32_t hash1 = (settings->hash_version == 2\n    -+\t\t? murmur3_seeded_v2 : murmur3_seeded_v1)(seed1, data, len);\n    ++\tuint32_t hash0, hash1;\n    ++\tif (settings->hash_version == 2) {\n    ++\t\thash0 = murmur3_seeded_v2(seed0, data, len);\n    ++\t\thash1 = murmur3_seeded_v2(seed1, data, len);\n    ++\t} else {\n    ++\t\thash0 = murmur3_seeded_v1(seed0, data, len);\n    ++\t\thash1 = murmur3_seeded_v1(seed1, data, len);\n    ++\t}\n      \n      \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n      \tfor (i = 0; i < settings->num_hashes; i++)\n     \n      ## bloom.h ##\n    -@@ bloom.h: struct repository;\n    +@@ bloom.h: struct commit_graph;\n      struct bloom_filter_settings {\n      \t/*\n      \t * The version of the hashing technique being used.\n    @@ bloom.h: struct repository;\n      \t */\n      \tuint32_t hash_version;\n      \n    -@@ bloom.h: struct bloom_key {\n    +@@ bloom.h: int load_bloom_filter_from_graph(struct commit_graph *g,\n       * Not considered to be cryptographically secure.\n       * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n       */\n    @@ commit-graph.c: static int graph_read_oid_lookup(const unsigned char *chunk_star\n      \treturn 0;\n      }\n      \n    -+struct graph_read_bloom_data_data {\n    ++struct graph_read_bloom_data_context {\n     +\tstruct commit_graph *g;\n     +\tint *commit_graph_changed_paths_version;\n     +};\n    @@ commit-graph.c: static int graph_read_oid_lookup(const unsigned char *chunk_star\n      \t\t\t\t  size_t chunk_size, void *data)\n      {\n     -\tstruct commit_graph *g = data;\n    -+\tstruct graph_read_bloom_data_data *d = data;\n    -+\tstruct commit_graph *g = d->g;\n    ++\tstruct graph_read_bloom_data_context *c = data;\n    ++\tstruct commit_graph *g = c->g;\n      \tuint32_t hash_version;\n      \tg->chunk_bloom_data = chunk_start;\n      \thash_version = get_be32(chunk_start);\n      \n     -\tif (hash_version != 1)\n    -+\tif (*d->commit_graph_changed_paths_version == -1) {\n    -+\t\t*d->commit_graph_changed_paths_version = hash_version;\n    -+\t} else if (hash_version != *d->commit_graph_changed_paths_version) {\n    - \t\treturn 0;\n    +-\t\treturn 0;\n    ++\tif (*c->commit_graph_changed_paths_version == -1) {\n    ++\t\t*c->commit_graph_changed_paths_version = hash_version;\n    ++\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n    ++ \t\treturn 0;\n     +\t}\n      \n      \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n    @@ commit-graph.c: static int graph_read_oid_lookup(const unsigned char *chunk_star\n     @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repo_settings *s,\n      \t}\n      \n    - \tif (s->commit_graph_changed_paths_version != 0) {\n    -+\t\tstruct graph_read_bloom_data_data data = {\n    + \tif (s->commit_graph_changed_paths_version) {\n    ++\t\tstruct graph_read_bloom_data_context context = {\n     +\t\t\t.g = graph,\n     +\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n     +\t\t};\n    @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repo_settings *s,\n      \t\t\t   &graph->chunk_bloom_indexes);\n      \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n     -\t\t\t   graph_read_bloom_data, graph);\n    -+\t\t\t   graph_read_bloom_data, &data);\n    ++\t\t\t   graph_read_bloom_data, &context);\n      \t}\n      \n      \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n    @@ t/t0095-bloom.sh: test_expect_success 'compute unseeded murmur3 hash for test st\n      \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\n     \n      ## t/t4216-log-bloom.sh ##\n    -@@ t/t4216-log-bloom.sh: get_bdat_offset () {\n    - \t\t.git/objects/info/commit-graph\n    - }\n    - \n    -+get_changed_path_filter_version () {\n    -+\tBDAT_OFFSET=$(get_bdat_offset) &&\n    -+\tperl -0777 -ne \\\n    -+\t\t'print unpack(\"H*\", substr($_, '$BDAT_OFFSET', 4))' \\\n    -+\t\t.git/objects/info/commit-graph\n    -+}\n    -+\n    - get_first_changed_path_filter () {\n    - \tBDAT_OFFSET=$(get_bdat_offset) &&\n    - \tperl -0777 -ne \\\n     @@ t/t4216-log-bloom.sh: test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n      \tgit -C highbit1 commit-graph write --reachable --changed-paths\n      '\n    @@ t/t4216-log-bloom.sh: test_expect_success 'set up repo with high bit path, versi\n     -test_expect_success 'setup check value of version 1 changed-path' '\n     +test_expect_success 'check value of version 1 changed-path' '\n      \t(cd highbit1 &&\n    - \t\tprintf \"52a9\" >expect &&\n    + \t\techo \"52a9\" >expect &&\n      \t\tget_first_changed_path_filter >actual &&\n     @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when version 1 requested' '\n      \t\ttest_bloom_filters_used \"-- $CENT\")\n    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n     +\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n     +\t(cd highbit1 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n    -+\t\tprintf \"00000001\" >expect &&\n    -+\t\tget_changed_path_filter_version >actual &&\n    ++\t\techo \"options: bloom(1,10,7) read_generation_data\" >expect &&\n    ++\t\ttest-tool read-graph >full &&\n    ++\t\tgrep options full >actual &&\n     +\t\ttest_cmp expect actual)\n     +'\n     +\n    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n     +\n     +test_expect_success 'check value of version 2 changed-path' '\n     +\t(cd highbit2 &&\n    -+\t\tprintf \"c01f\" >expect &&\n    ++\t\techo \"c01f\" >expect &&\n     +\t\tget_first_changed_path_filter >actual &&\n     +\t\ttest_cmp expect actual)\n     +'\n    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n     +\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n     +\t(cd highbit2 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n    -+\t\tprintf \"00000002\" >expect &&\n    -+\t\tget_changed_path_filter_version >actual &&\n    ++\t\techo \"options: bloom(2,10,7) read_generation_data\" >expect &&\n    ++\t\ttest-tool read-graph >full &&\n    ++\t\tgrep options full >actual &&\n     +\t\ttest_cmp expect actual)\n     +'\n     +\n-- \n2.41.0.487.g6d72f3e995-goog\n\n"},{"id":"479686","messageId":"3ce6090a4d3634b2e947d57f06714182207bb33a.1689889382.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689889382.git.jonathantanmy@google.com","subject":"[PATCH v6 1/7] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T21:46:34Z","receivedAt":"2023-07-20T21:46:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The code change to Git to support version 2 will be done in subsequent\ncommits.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/gitformat-commit-graph.txt | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\nindex 31cad585e2..3e906e8030 100644\n--- a/Documentation/gitformat-commit-graph.txt\n+++ b/Documentation/gitformat-commit-graph.txt\n@@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n \n ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n     * It starts with header consisting of three unsigned 32-bit integers:\n-      - Version of the hash algorithm being used. We currently only support\n-\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n+      - Version of the hash algorithm being used. We currently support\n+\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n \timplemented exactly as described in\n \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n-\tin Probabilistic Verification\"\n+\tin Probabilistic Verification\". Version 1 Bloom filters have a bug that appears\n+\twhen char is signed and the repository has path names that have characters >=\n+\t0x80; Git supports reading and writing them, but this ability will be removed\n+\tin a future version of Git.\n       - The number of times a path is hashed and hence the number of bit positions\n \t      that cumulatively determine whether a file is present in the commit.\n       - The minimum number of bits 'b' per entry in the Bloom filter. If the filter\n-- \n2.41.0.487.g6d72f3e995-goog\n\n"},{"id":"479688","messageId":"1955734d1f7385ccc1bd3bb189b72e36d8a9b6ec.1689889382.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689889382.git.jonathantanmy@google.com","subject":"[PATCH v6 2/7] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T21:46:35Z","receivedAt":"2023-07-20T21:46:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau <me@ttaylorr.com>\n\nPrepare for the 'read-graph' test helper to perform other tasks besides\ndumping high-level information about the commit-graph by extracting its\nmain routine into a separate function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-read-graph.c | 33 ++++++++++++++++++++-------------\n 1 file changed, 20 insertions(+), 13 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 8c7a83f578..c664928412 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -5,20 +5,8 @@\n #include \"bloom.h\"\n #include \"setup.h\"\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_info(struct commit_graph *graph)\n {\n-\tstruct commit_graph *graph = NULL;\n-\tstruct object_directory *odb;\n-\n-\tsetup_git_directory();\n-\todb = the_repository->objects->odb;\n-\n-\tprepare_repo_settings(the_repository);\n-\n-\tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n@@ -57,8 +45,27 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tif (graph->topo_levels)\n \t\tprintf(\" topo_levels\");\n \tprintf(\"\\n\");\n+}\n+\n+int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tstruct object_directory *odb;\n+\n+\tsetup_git_directory();\n+\todb = the_repository->objects->odb;\n+\n+\tprepare_repo_settings(the_repository);\n+\n+\tgraph = read_commit_graph_one(the_repository, odb);\n+\tif (!graph)\n+\t\treturn 1;\n+\n+\tdump_graph_info(graph);\n \n \tUNLEAK(graph);\n \n \treturn 0;\n }\n+\n+\n-- \n2.41.0.487.g6d72f3e995-goog\n\n"},{"id":"479689","messageId":"4cf7c2f634c09b0912963fb2360aa31eb1d8dbba.1689889382.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689889382.git.jonathantanmy@google.com","subject":"[PATCH v6 3/7] bloom.h: make `load_bloom_filter_from_graph()` public","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T21:46:36Z","receivedAt":"2023-07-20T21:47:02Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau <me@ttaylorr.com>\n\nPrepare for a future commit to use the load_bloom_filter_from_graph()\nfunction directly to load specific Bloom filters out of the commit-graph\nfor manual inspection (to be used during tests).\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n bloom.c | 6 +++---\n bloom.h | 5 +++++\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex aef6b5fea2..3e78cfe79d 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -29,9 +29,9 @@ static inline unsigned char get_bitmask(uint32_t pos)\n \treturn ((unsigned char)1) << (pos & (BITS_PER_WORD - 1));\n }\n \n-static int load_bloom_filter_from_graph(struct commit_graph *g,\n-\t\t\t\t\tstruct bloom_filter *filter,\n-\t\t\t\t\tuint32_t graph_pos)\n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos)\n {\n \tuint32_t lex_pos, start_index, end_index;\n \ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..1e4f612d2c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -3,6 +3,7 @@\n \n struct commit;\n struct repository;\n+struct commit_graph;\n \n struct bloom_filter_settings {\n \t/*\n@@ -68,6 +69,10 @@ struct bloom_key {\n \tuint32_t *hashes;\n };\n \n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos);\n+\n /*\n  * Calculate the murmur3 32-bit hash value for the given data\n  * using the given seed.\n-- \n2.41.0.487.g6d72f3e995-goog\n\n"},{"id":"479690","messageId":"5276e6a90ecb84cbca3b14a27dc055cf81d27ac3.1689889382.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689889382.git.jonathantanmy@google.com","subject":"[PATCH v6 5/7] t4216: test changed path filters with high bit paths","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T21:46:38Z","receivedAt":"2023-07-20T21:47:04Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Subsequent commits will teach Git another version of changed path\nfilter that has different behavior with paths that contain at least\none character with its high bit set, so test the existing behavior as\na baseline.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n t/t4216-log-bloom.sh | 39 +++++++++++++++++++++++++++++++++++++++\n 1 file changed, 39 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..c49528b947 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -404,4 +404,43 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n+get_first_changed_path_filter () {\n+\ttest-tool read-graph bloom-filters >filters.dat &&\n+\thead -n 1 filters.dat\n+}\n+\n+# chosen to be the same under all Unicode normalization forms\n+CENT=$(printf \"\\302\\242\")\n+\n+test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n+\tgit init highbit1 &&\n+\ttest_commit -C highbit1 c1 \"$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'setup check value of version 1 changed-path' '\n+\t(cd highbit1 &&\n+\t\techo \"52a9\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+# expect will not match actual if char is unsigned by default. Write the test\n+# in this way, so that a user running this test script can still see if the two\n+# files match. (It will appear as an ordinary success if they match, and a skip\n+# if not.)\n+if test_cmp highbit1/expect highbit1/actual\n+then\n+\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n+fi\n+test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n+\t# Only the prereq matters for this test.\n+\ttrue\n+'\n+\n+test_expect_success 'version 1 changed-path used when version 1 requested' '\n+\t(cd highbit1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n test_done\n-- \n2.41.0.487.g6d72f3e995-goog\n\n"},{"id":"479691","messageId":"47b55758e603debbcecdc817d016714ab98e2b54.1689889382.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689889382.git.jonathantanmy@google.com","subject":"[PATCH v6 4/7] t/helper/test-read-graph: implement `bloom-filters` mode","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T21:46:37Z","receivedAt":"2023-07-20T21:47:06Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau <me@ttaylorr.com>\n\nImplement a mode of the \"read-graph\" test helper to dump out the\nhexadecimal contents of the Bloom filter(s) contained in a commit-graph.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\n---\n t/helper/test-read-graph.c | 42 +++++++++++++++++++++++++++++++++-----\n 1 file changed, 37 insertions(+), 5 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex c664928412..da9ac8584d 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -47,10 +47,32 @@ static void dump_graph_info(struct commit_graph *graph)\n \tprintf(\"\\n\");\n }\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_bloom_filters(struct commit_graph *graph)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < graph->num_commits + graph->num_commits_in_base; i++) {\n+\t\tstruct bloom_filter filter = { 0 };\n+\t\tsize_t j;\n+\n+\t\tif (load_bloom_filter_from_graph(graph, &filter, i) < 0) {\n+\t\t\tfprintf(stderr, \"missing Bloom filter for graph \"\n+\t\t\t\t\"position %\"PRIu32\"\\n\", i);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (j = 0; j < filter.len; j++)\n+\t\t\tprintf(\"%02x\", filter.data[j]);\n+\t\tif (filter.len)\n+\t\t\tprintf(\"\\n\");\n+\t}\n+}\n+\n+int cmd__read_graph(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n \tstruct object_directory *odb;\n+\tint ret = 0;\n \n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n@@ -58,14 +80,24 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tprepare_repo_settings(the_repository);\n \n \tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n+\tif (!graph) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n \n-\tdump_graph_info(graph);\n+\tif (argc <= 1)\n+\t\tdump_graph_info(graph);\n+\telse if (!strcmp(argv[1], \"bloom-filters\"))\n+\t\tdump_graph_bloom_filters(graph);\n+\telse {\n+\t\tfprintf(stderr, \"unknown sub-command: '%s'\\n\", argv[1]);\n+\t\tret = 1;\n+\t}\n \n+done:\n \tUNLEAK(graph);\n \n-\treturn 0;\n+\treturn ret;\n }\n \n \n-- \n2.41.0.487.g6d72f3e995-goog\n\n"},{"id":"479692","messageId":"dc3f6d2d4fc5b785f86a2cfe14269c5847afad92.1689889382.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689889382.git.jonathantanmy@google.com","subject":"[PATCH v6 6/7] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T21:46:39Z","receivedAt":"2023-07-20T21:47:08Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A subsequent commit will introduce another version of the changed-path\nfilter in the commit graph file. In order to control which version to\nwrite (and read), a config variable is needed.\n\nTherefore, introduce this config variable. For forwards compatibility,\nteach Git to not read commit graphs when the config variable\nis set to an unsupported version. Because we teach Git this,\ncommitgraph.readChangedPaths is now redundant, so deprecate it and\ndefine its behavior in terms of the config variable we introduce.\n\nThis commit does not change the behavior of writing (Git writes changed\npath filters when explicitly instructed regardless of any config\nvariable), but a subsequent commit will restrict Git such that it will\nonly write when commitgraph.changedPathsVersion is a recognized value.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt | 23 ++++++++++++++++++++---\n commit-graph.c                       |  2 +-\n oss-fuzz/fuzz-commit-graph.c         |  2 +-\n repo-settings.c                      |  6 +++++-\n repository.h                         |  2 +-\n 5 files changed, 28 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c..2dc9170622 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -9,6 +9,23 @@ commitGraph.maxNewFilters::\n \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n \n commitGraph.readChangedPaths::\n-\tIf true, then git will use the changed-path Bloom filters in the\n-\tcommit-graph file (if it exists, and they are present). Defaults to\n-\ttrue. See linkgit:git-commit-graph[1] for more information.\n+\tDeprecated. Equivalent to commitGraph.changedPathsVersion=-1 if true, and\n+\tcommitGraph.changedPathsVersion=0 if false. (If commitGraph.changedPathVersion\n+\tis also set, commitGraph.changedPathsVersion takes precedence.)\n+\n+commitGraph.changedPathsVersion::\n+\tSpecifies the version of the changed-path Bloom filters that Git will read and\n+\twrite. May be -1, 0 or 1.\n++\n+Defaults to -1.\n++\n+If -1, Git will use the version of the changed-path Bloom filters in the\n+repository, defaulting to 1 if there are none.\n++\n+If 0, Git will not read any Bloom filters, and will write version 1 Bloom\n+filters when instructed to write.\n++\n+If 1, Git will only read version 1 Bloom filters, and will write version 1\n+Bloom filters.\n++\n+See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex efc697e437..1f26c07de4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -401,7 +401,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_read_changed_paths) {\n+\tif (s->commit_graph_changed_paths_version) {\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\ndiff --git a/oss-fuzz/fuzz-commit-graph.c b/oss-fuzz/fuzz-commit-graph.c\nindex 2992079dd9..325c0b991a 100644\n--- a/oss-fuzz/fuzz-commit-graph.c\n+++ b/oss-fuzz/fuzz-commit-graph.c\n@@ -19,7 +19,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \t * possible.\n \t */\n \tthe_repository->settings.commit_graph_generation_version = 2;\n-\tthe_repository->settings.commit_graph_read_changed_paths = 1;\n+\tthe_repository->settings.commit_graph_changed_paths_version = 1;\n \tg = parse_commit_graph(&the_repository->settings, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 525f69c0c7..db8fe817f3 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n \tint value;\n \tconst char *strval;\n \tint manyfiles;\n+\tint read_changed_paths;\n \n \tif (!r->gitdir)\n \t\tBUG(\"Cannot add settings for uninitialized repository\");\n@@ -54,7 +55,10 @@ void prepare_repo_settings(struct repository *r)\n \t/* Commit graph config or default, does not cascade (simple) */\n \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n-\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &read_changed_paths, 1);\n+\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n+\t\t     &r->settings.commit_graph_changed_paths_version,\n+\t\t     read_changed_paths ? -1 : 0);\n \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n \ndiff --git a/repository.h b/repository.h\nindex 5f18486f64..f71154e12c 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -29,7 +29,7 @@ struct repo_settings {\n \n \tint core_commit_graph;\n \tint commit_graph_generation_version;\n-\tint commit_graph_read_changed_paths;\n+\tint commit_graph_changed_paths_version;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \tint command_requires_full_index;\n-- \n2.41.0.487.g6d72f3e995-goog\n\n"},{"id":"479693","messageId":"6e2d797406cb5ab691451d85f446e87c826c6f3a.1689889382.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1689889382.git.jonathantanmy@google.com","subject":"[PATCH v6 7/7] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-20T21:46:40Z","receivedAt":"2023-07-20T21:47:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 (thus,\nthe changed path filters wouldn't be readable by other off-the-shelf\nimplementatios of murmur3) and even with Git binaries that were compiled\nwith different signedness of char. This bug affects both how Git writes\nchanged path filters to disk and how Git interprets changed path filters\non disk.\n\nTherefore, introduce a new version (2) of changed path filters that\ncorrects this problem. The existing version (1) is still supported and\nis still the default, but users should migrate away from it as soon\nas possible.\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter.\n\nSo this patch does not include any mechanism to \"salvage\" changed path\nfilters from repositories. There is also no \"mixed\" mode - for each\ninvocation of Git, reading and writing changed path filters are done\nwith the same version number; this version number may be explicitly\nstated (typically if the user knows which version they need) or\nautomatically determined from the version of the existing changed path\nfilters in the repository.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in t0095 was obtained from another murmur3 implementation\nusing the following Go source code:\n\n  package main\n\n  import \"fmt\"\n  import \"github.com/spaolacci/murmur3\"\n\n  func main() {\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n  }\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt |  5 +-\n bloom.c                              | 69 ++++++++++++++++++++++++++--\n bloom.h                              |  8 ++--\n commit-graph.c                       | 31 +++++++++++--\n t/helper/test-bloom.c                |  9 +++-\n t/t0095-bloom.sh                     |  8 ++++\n t/t4216-log-bloom.sh                 | 67 ++++++++++++++++++++++++++-\n 7 files changed, 183 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 2dc9170622..acc74a2f27 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -15,7 +15,7 @@ commitGraph.readChangedPaths::\n \n commitGraph.changedPathsVersion::\n \tSpecifies the version of the changed-path Bloom filters that Git will read and\n-\twrite. May be -1, 0 or 1.\n+\twrite. May be -1, 0, 1, or 2.\n +\n Defaults to -1.\n +\n@@ -28,4 +28,7 @@ filters when instructed to write.\n If 1, Git will only read version 1 Bloom filters, and will write version 1\n Bloom filters.\n +\n+If 2, Git will only read version 2 Bloom filters, and will write version 2\n+Bloom filters.\n++\n See linkgit:git-commit-graph[1] for more information.\ndiff --git a/bloom.c b/bloom.c\nindex 3e78cfe79d..ebef5cfd2f 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -66,7 +66,64 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n+{\n+\tconst uint32_t c1 = 0xcc9e2d51;\n+\tconst uint32_t c2 = 0x1b873593;\n+\tconst uint32_t r1 = 15;\n+\tconst uint32_t r2 = 13;\n+\tconst uint32_t m = 5;\n+\tconst uint32_t n = 0xe6546b64;\n+\tint i;\n+\tuint32_t k1 = 0;\n+\tconst char *tail;\n+\n+\tint len4 = len / sizeof(uint32_t);\n+\n+\tuint32_t k;\n+\tfor (i = 0; i < len4; i++) {\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n+\t\tk = byte1 | byte2 | byte3 | byte4;\n+\t\tk *= c1;\n+\t\tk = rotate_left(k, r1);\n+\t\tk *= c2;\n+\n+\t\tseed ^= k;\n+\t\tseed = rotate_left(seed, r2) * m + n;\n+\t}\n+\n+\ttail = (data + len4 * sizeof(uint32_t));\n+\n+\tswitch (len & (sizeof(uint32_t) - 1)) {\n+\tcase 3:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n+\t\t/*-fallthrough*/\n+\tcase 2:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n+\t\t/*-fallthrough*/\n+\tcase 1:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n+\t\tk1 *= c1;\n+\t\tk1 = rotate_left(k1, r1);\n+\t\tk1 *= c2;\n+\t\tseed ^= k1;\n+\t\tbreak;\n+\t}\n+\n+\tseed ^= (uint32_t)len;\n+\tseed ^= (seed >> 16);\n+\tseed *= 0x85ebca6b;\n+\tseed ^= (seed >> 13);\n+\tseed *= 0xc2b2ae35;\n+\tseed ^= (seed >> 16);\n+\n+\treturn seed;\n+}\n+\n+static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n {\n \tconst uint32_t c1 = 0xcc9e2d51;\n \tconst uint32_t c2 = 0x1b873593;\n@@ -131,8 +188,14 @@ void fill_bloom_key(const char *data,\n \tint i;\n \tconst uint32_t seed0 = 0x293ae76f;\n \tconst uint32_t seed1 = 0x7e646e2c;\n-\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n-\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n+\tuint32_t hash0, hash1;\n+\tif (settings->hash_version == 2) {\n+\t\thash0 = murmur3_seeded_v2(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v2(seed1, data, len);\n+\t} else {\n+\t\thash0 = murmur3_seeded_v1(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v1(seed1, data, len);\n+\t}\n \n \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n \tfor (i = 0; i < settings->num_hashes; i++)\ndiff --git a/bloom.h b/bloom.h\nindex 1e4f612d2c..138d57a86b 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -8,9 +8,11 @@ struct commit_graph;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * The newest version is 2, which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -80,7 +82,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n \n void fill_bloom_key(const char *data,\n \t\t    size_t len,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1f26c07de4..1a6685f554 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -304,16 +304,25 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct graph_read_bloom_data_context {\n+\tstruct commit_graph *g;\n+\tint *commit_graph_changed_paths_version;\n+};\n+\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t  size_t chunk_size, void *data)\n {\n-\tstruct commit_graph *g = data;\n+\tstruct graph_read_bloom_data_context *c = data;\n+\tstruct commit_graph *g = c->g;\n \tuint32_t hash_version;\n \tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n-\t\treturn 0;\n+\tif (*c->commit_graph_changed_paths_version == -1) {\n+\t\t*c->commit_graph_changed_paths_version = hash_version;\n+\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n+ \t\treturn 0;\n+\t}\n \n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n \tg->bloom_filter_settings->hash_version = hash_version;\n@@ -402,10 +411,14 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t}\n \n \tif (s->commit_graph_changed_paths_version) {\n+\t\tstruct graph_read_bloom_data_context context = {\n+\t\t\t.g = graph,\n+\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n+\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t   graph_read_bloom_data, graph);\n+\t\t\t   graph_read_bloom_data, &context);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -2371,6 +2384,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tif (r->settings.commit_graph_changed_paths_version < -1\n+\t    || r->settings.commit_graph_changed_paths_version > 2) {\n+\t\twarning(_(\"attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported\"),\n+\t\t\tr->settings.commit_graph_changed_paths_version);\n+\t\treturn 0;\n+\t}\n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n+\t\t? 2 : 1;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t      bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2400,7 +2421,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex aabe31d724..3cbc0a5b50 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -50,6 +50,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \"  test-tool bloom get_murmur3 <string>\\n\"\n+\"  test-tool bloom get_murmur3_seven_highbit\\n\"\n \"  test-tool bloom generate_filter <string> [<string>...]\\n\"\n \"  test-tool bloom get_filter_for_commit <commit-hex>\\n\";\n \n@@ -64,7 +65,13 @@ int cmd__bloom(int argc, const char **argv)\n \t\tuint32_t hashed;\n \t\tif (argc < 3)\n \t\t\tusage(bloom_usage);\n-\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n+\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c49528b947..cf4c78ddf9 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -418,7 +418,7 @@ test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n \tgit -C highbit1 commit-graph write --reachable --changed-paths\n '\n \n-test_expect_success 'setup check value of version 1 changed-path' '\n+test_expect_success 'check value of version 1 changed-path' '\n \t(cd highbit1 &&\n \t\techo \"52a9\" >expect &&\n \t\tget_first_changed_path_filter >actual &&\n@@ -443,4 +443,69 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n \t\ttest_bloom_filters_used \"-- $CENT\")\n '\n \n+test_expect_success 'version 1 changed-path not used when version 2 requested' '\n+\t(cd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'version 1 changed-path used when autodetect requested' '\n+\t(cd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n+\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n+\t(cd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(1,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n+\tgit init highbit2 &&\n+\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n+\ttest_commit -C highbit2 c2 \"$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'check value of version 2 changed-path' '\n+\t(cd highbit2 &&\n+\t\techo \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n+test_expect_success 'version 2 changed-path used when version 2 requested' '\n+\t(cd highbit2 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'version 2 changed-path not used when version 1 requested' '\n+\t(cd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'version 2 changed-path used when autodetect requested' '\n+\t(cd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\")\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n+\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n+\t(cd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(2,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual)\n+'\n+\n test_done\n-- \n2.41.0.487.g6d72f3e995-goog\n\n"},{"id":"479703","messageId":"ZLnhihfaRDUmlsi3@nand.local","threadId":"59781","inReplyTo":"20230720202006.3815802-1-jonathantanmy@google.com","subject":"Re: [PATCH v5 1/4] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-21T01:38:18Z","receivedAt":"2023-07-21T01:38:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 20, 2023 at 01:20:06PM -0700, Jonathan Tan wrote:\n> Having said that, I am inclined to not change this, so that the offset\n> calculations are the same for both versions (e.g. in the test tool\n> too), and as far as I know, we haven't had problems with this. But I can\n> change it if people want.\n\nYeah, I definitely get your hesitation. I wonder if the test helper\npatches that I sent change your thinking on this at all, since then\nwe're not relying on those scripts at all to compute the offset of the\nBDAT chunk and dump its contents.\n\nTo be clear, I do think that this is probably outside the immediate\nscope of this series. But since we're changing the on-disk format and\nbumping the version count forward, I think that we want to make sure\nthat we're not missing any other format changes that we'd like to make\nalong the way.\n\nI suppose that a theoretical v3 of the BDAT chunk's format would use the\nsame encoding for Bloom filters. But storing an extra 4-bytes of\ninformation in the header of the Bloom data chunk feels like we could\nsquash that in.\n\nThanks,\nTaylor\n"},{"id":"479853","messageId":"xmqqjzunk91y.fsf@gitster.g","threadId":"59781","inReplyTo":"cover.1689889382.git.jonathantanmy@google.com","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-25T20:52:57Z","receivedAt":"2023-07-25T20:53:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Thanks, Junio and Taylor, for your reviews. I have included Taylor's\n> patches in this patch set.\n>\n> There seemed to be some merge conflicts when I tried to apply the\n> patches Taylor provided on the base that I built my patches on (that is,\n> the base of jt/path-filter-fix, namely, maint-2.40), so I have rebased\n> all my patches onto latest master.\n>\n> Jonathan Tan (4):\n>   gitformat-commit-graph: describe version 2 of BDAT\n>   t4216: test changed path filters with high bit paths\n>   repo-settings: introduce commitgraph.changedPathsVersion\n>   commit-graph: new filter ver. that fixes murmur3\n>\n> Taylor Blau (3):\n>   t/helper/test-read-graph.c: extract `dump_graph_info()`\n>   bloom.h: make `load_bloom_filter_from_graph()` public\n>   t/helper/test-read-graph: implement `bloom-filters` mode\n\nThanks, I seem to have missed this one.  Let's queue this version\nand merge it down to 'next', unless there is no other blocking\ncomments in a few days.\n\nThanks.\n"},{"id":"479919","messageId":"xmqq8rb2csr1.fsf@gitster.g","threadId":"59781","inReplyTo":"cover.1689889382.git.jonathantanmy@google.com","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-26T20:39:14Z","receivedAt":"2023-07-26T20:39:42Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Jonathan Tan (4):\n>   gitformat-commit-graph: describe version 2 of BDAT\n>   t4216: test changed path filters with high bit paths\n>   repo-settings: introduce commitgraph.changedPathsVersion\n>   commit-graph: new filter ver. that fixes murmur3\n>\n> Taylor Blau (3):\n>   t/helper/test-read-graph.c: extract `dump_graph_info()`\n>   bloom.h: make `load_bloom_filter_from_graph()` public\n>   t/helper/test-read-graph: implement `bloom-filters` mode\n\nAfter a week, nobody seems to have found anything worth saying about\nthese patches.  Does the design, especially the migration story, now\nlook sensible to everybody?  I was contemplating to mark the topic\nfor 'next' after reading them myself once more.\n\nThanks.\n"},{"id":"479925","messageId":"ZMGruenDbAo22aWV@nand.local","threadId":"59781","inReplyTo":"1955734d1f7385ccc1bd3bb189b72e36d8a9b6ec.1689889382.git.jonathantanmy@google.com","subject":"Re: [PATCH v6 2/7] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-26T23:26:49Z","receivedAt":"2023-07-26T23:26:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 20, 2023 at 02:46:35PM -0700, Jonathan Tan wrote:\n> From: Taylor Blau <me@ttaylorr.com>\n>\n> Prepare for the 'read-graph' test helper to perform other tasks besides\n> dumping high-level information about the commit-graph by extracting its\n> main routine into a separate function.\n>\n> Signed-off-by: Taylor Blau <me@ttaylorr.com>\n\nMissing sign-off from Jonathan?\n\nThanks,\nTaylor\n"},{"id":"479926","messageId":"ZMGsJTxBXZ94lhMU@nand.local","threadId":"59781","inReplyTo":"5276e6a90ecb84cbca3b14a27dc055cf81d27ac3.1689889382.git.jonathantanmy@google.com","subject":"Re: [PATCH v6 5/7] t4216: test changed path filters with high bit paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-26T23:28:37Z","receivedAt":"2023-07-26T23:28:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 20, 2023 at 02:46:38PM -0700, Jonathan Tan wrote:\n> +test_expect_success 'setup check value of version 1 changed-path' '\n> +\t(cd highbit1 &&\n> +\t\techo \"52a9\" >expect &&\n> +\t\tget_first_changed_path_filter >actual &&\n> +\t\ttest_cmp expect actual)\n> +'\n\nThis is a little bit of funky indentation, I probably would have\nexpected something more along the lines of:\n\n    (\n      cd highbit1 &&\n      echo \"52a9\" >expect &&\n      get_first_changed_path_filter >actual &&\n      test_cmp expect actual\n    )\n\nbut this obviously doesn't merit a reroll on its own.\n\n> +test_expect_success 'version 1 changed-path used when version 1 requested' '\n> +\t(cd highbit1 &&\n> +\t\ttest_bloom_filters_used \"-- $CENT\")\n> +'\n\nSame here, but neither of these is incorrect.\n\nThanks,\nTaylor\n"},{"id":"479927","messageId":"ZMGtEdBmgKu5pWHP@nand.local","threadId":"59781","inReplyTo":"20230720212753.2008505-1-jonathantanmy@google.com","subject":"Re: [PATCH v5 4/4] commit-graph: new filter ver. that fixes murmur3","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-26T23:32:33Z","receivedAt":"2023-07-26T23:32:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 20, 2023 at 02:27:53PM -0700, Jonathan Tan wrote:\n> > I think the early checks would be more expensive, since in the worst\n> > case you have to walk the entire tree, only to realize that you actually\n> > wanted to compute a first-parent tree diff, meaning you have to\n> > essentially repeat the whole walk over again. But for repositories that\n> > have few or no commits whose Bloom filters need computing, I think it\n> > would be significantly faster, since many of the sub-trees wouldn't need\n> > to be visited again.\n>\n> So for repositories that need little-to-no recomputation of Bloom\n> filters, your idea likely means that each tree needs to be read once,\n> as compared to recomputing everything in which, I think, each tree needs\n> to be read roughly twice (once when computing the Bloom filter for the\n> commit that introduces it, and once for the commit that substitutes a\n> different tree in place).\n>\n> I could change the text of the commit message to discuss this (instead\n> of the blanket statement that it would be too hard), although I think\n> that an implementation of this can be done after this patchset. What do\n> you think?\n\nRight, I think that a sizeable portion of repositories will need to\ncompute relatively few Bloom filters overall. If you feel strongly that\nit shouldn't be included in this series, I could live with that since\nthis is all behind a configuration variable anyway.\n\nI think at minimum we should call it out in the documentation, at least\nuntil such functionality is implemented, since unsuspecting users/forge\noperators may bump the filter version forward and be surprised when they\nsuddenly have to recompute every existing Bloom filter.\n\nThanks,\nTaylor\n"},{"id":"479928","messageId":"ZMG3kEO0BzKPaG75@nand.local","threadId":"59781","inReplyTo":"xmqq8rb2csr1.fsf@gitster.g","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-27T00:17:20Z","receivedAt":"2023-07-27T00:17:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Jul 26, 2023 at 01:39:14PM -0700, Junio C Hamano wrote:\n> Jonathan Tan <jonathantanmy@google.com> writes:\n>\n> > Jonathan Tan (4):\n> >   gitformat-commit-graph: describe version 2 of BDAT\n> >   t4216: test changed path filters with high bit paths\n> >   repo-settings: introduce commitgraph.changedPathsVersion\n> >   commit-graph: new filter ver. that fixes murmur3\n> >\n> > Taylor Blau (3):\n> >   t/helper/test-read-graph.c: extract `dump_graph_info()`\n> >   bloom.h: make `load_bloom_filter_from_graph()` public\n> >   t/helper/test-read-graph: implement `bloom-filters` mode\n>\n> After a week, nobody seems to have found anything worth saying about\n> these patches.  Does the design, especially the migration story, now\n> look sensible to everybody?  I was contemplating to mark the topic\n> for 'next' after reading them myself once more.\n\nSorry for not getting to this sooner. I didn't notice anything during my\nreview, but I think there may be a bug here.\n\nSuppose that we have an existing commit-graph with v1 Bloom filters. If\nwe then try to rewrite that commit-graph using v2 Bloom filters, we\n*should* attempt to recompute the filter from scratch. But AFAICT, that\nisn't what happens.\n\nHere's my test setup:\n\n    test_expect_success 'test' '\n      test_commit base &&\n      git repack -d &&\n\n      git -c commitGraph.changedPathVersion=1 commit-graph write --changed-paths &&\n      debug git -c commitGraph.changedPathVersion=2 commit-graph write --changed-paths\n    '\n\nif you attach a debugger to the second process, and break inside of\nget_or_compute_bloom_filter() when compute_if_not_present is set, you'll\nsee that Git will pass along the existing *v1* Bloom filter, and then\nwrite its contents to the new commit-graph:\n\n    (gdb) b get_or_compute_bloom_filter if compute_if_not_present\n    Breakpoint 1 at 0x14340f: file bloom.c, line 260.\n    (gdb) r\n    Starting program: /home/ttaylorr/src/git/git -c commitGraph.changedPathVersion=2 commit-graph write --changed-paths\n    [Thread debugging using libthread_db enabled]\n    Using host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\n\n    Breakpoint 1, get_or_compute_bloom_filter (r=0x5555559bdc80 <the_repo>, c=0x5555559c8ef0,\n        compute_if_not_present=1, settings=0x5555559c6950, computed=0x7fffffffd854) at bloom.c:260\n    260\t\tif (computed)\n    (gdb) until 271\n    get_or_compute_bloom_filter (r=0x5555559bdc80 <the_repo>, c=0x5555559c8ef0, compute_if_not_present=1,\n        settings=0x5555559c6950, computed=0x7fffffffd854) at bloom.c:271\n    271\t\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph,\n    (gdb) p *filter\n    $2 = {data = 0x0, len = 0}\n    (gdb) n\n    275\t\tif (filter->data && filter->len)\n    (gdb) p *filter\n    $3 = {data = 0x7ffff7fc24a8 \"\\210\\210\\322a\\267\\234\\214s}\\004J\\265\\313\\201\\241\\032e\\312\\034\", len = 2}\n\nIf I'm parsing this all correctly, Git used the v1 filter corresponding\nto that commit, and did not recompute a new one.\n\nI think that this could lead to incorrect results if you use this to\nmasquerade a v1 Bloom filter as a v2 one. Since they use different\nimplementations (one correct, one not) of murmur3, that opens us up to\nfalse negatives, at which point all bets are off.\n\nSo I think we want to be more careful about when we load the existing\nBloom data or not. I think we probably want to load it in all cases, but\nonly read it when we have compatible Bloom settings. That should stop us\nfrom exhibiting this kind of bug, but it also gives us a handle on\nexisting Bloom data if we wanted to copy forward existing results where\nwe can.\n\nIf all of this tracks, I think that there is a gap in our test coverage\nthat didn't catch this earlier.\n\nThanks,\nTaylor\n"},{"id":"479929","messageId":"xmqqzg3ib2ma.fsf@gitster.g","threadId":"59781","inReplyTo":"ZMG3kEO0BzKPaG75@nand.local","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-27T00:49:01Z","receivedAt":"2023-07-27T00:49:11Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> Sorry for not getting to this sooner. I didn't notice anything during my\n> review, but I think there may be a bug here.\n>\n> Suppose that we have an existing commit-graph with v1 Bloom filters. If\n> we then try to rewrite that commit-graph using v2 Bloom filters, we\n> *should* attempt to recompute the filter from scratch. But AFAICT, that\n> isn't what happens.\n> ...\n> If I'm parsing this all correctly, Git used the v1 filter corresponding\n> to that commit, and did not recompute a new one.\n>\n> I think that this could lead to incorrect results if you use this to\n> masquerade a v1 Bloom filter as a v2 one.\n\nThat indeed is very bad.  How hard it would be to construct a test\ncase that fails if filter computed as v1 is marketed as v2?  A test\nmay be far easier to construct if it does not have to be end-to-end\n(e.g. instrument the codepath you followed through with the debugger\nwith trace2 annotations and see the control takes the right branches\nby reading the log).\n\n> So I think we want to be more careful about when we load the existing\n> Bloom data or not. I think we probably want to load it in all cases, but\n> only read it when we have compatible Bloom settings. That should stop us\n> from exhibiting this kind of bug, but it also gives us a handle on\n> existing Bloom data if we wanted to copy forward existing results where\n> we can.\n>\n> If all of this tracks, I think that there is a gap in our test coverage\n> that didn't catch this earlier.\n\nYeah, thanks for raising a concern.\n\nJonathan?\n"},{"id":"479931","messageId":"20230727173905.356292-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"xmqqzg3ib2ma.fsf@gitster.g","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-27T17:39:05Z","receivedAt":"2023-07-27T17:39:11Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Junio C Hamano <gitster@pobox.com> writes:\n> Taylor Blau <me@ttaylorr.com> writes:\n> > Suppose that we have an existing commit-graph with v1 Bloom filters. If\n> > we then try to rewrite that commit-graph using v2 Bloom filters, we\n> > *should* attempt to recompute the filter from scratch. But AFAICT, that\n> > isn't what happens.\n> > ...\n> > If I'm parsing this all correctly, Git used the v1 filter corresponding\n> > to that commit, and did not recompute a new one.\n> >\n> > I think that this could lead to incorrect results if you use this to\n> > masquerade a v1 Bloom filter as a v2 one.\n> \n> That indeed is very bad.  How hard it would be to construct a test\n> case that fails if filter computed as v1 is marketed as v2?  A test\n> may be far easier to construct if it does not have to be end-to-end\n> (e.g. instrument the codepath you followed through with the debugger\n> with trace2 annotations and see the control takes the right branches\n> by reading the log).\n\nAh, thanks, Taylor, for so meticulously investigating this. I'll take a look.\n\nA test should be doable - we already have tests (the ones that use\n\"get_first_changed_path_filter\") that check the bytes of the filter\ngenerated, so we should be able to write a test that writes one version,\nwrites the other, then checks the bytes.\n\n> > So I think we want to be more careful about when we load the existing\n> > Bloom data or not. I think we probably want to load it in all cases, but\n> > only read it when we have compatible Bloom settings. That should stop us\n> > from exhibiting this kind of bug, but it also gives us a handle on\n> > existing Bloom data if we wanted to copy forward existing results where\n> > we can.\n\nThe intention in the current patch set was to not load it at all when we\nhave incompatible Bloom settings because it appeared quite troublesome\nto notate which Bloom filter in memory is of which version. If we want\nto copy forward existing results, we can change that, but I don't know\nwhether it's worth doing that (and if we think it's worth doing, this\nshould probably go in another patch set).\n\n> > If all of this tracks, I think that there is a gap in our test coverage\n> > that didn't catch this earlier.\n> \n> Yeah, thanks for raising a concern.\n> \n> Jonathan?\n\nI'll take a look. Yes this does seem like a gap in test coverage -\nI thought the existing test that checks that Bloom filters are not\nused when a different version is requested would be sufficient, but\napparently that's not the case.\n"},{"id":"479933","messageId":"ZMKvsObx+uaKA8zF@nand.local","threadId":"59781","inReplyTo":"20230727173905.356292-1-jonathantanmy@google.com","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-07-27T17:56:00Z","receivedAt":"2023-07-27T17:56:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 27, 2023 at 10:39:05AM -0700, Jonathan Tan wrote:\n> A test should be doable - we already have tests (the ones that use\n> \"get_first_changed_path_filter\") that check the bytes of the filter\n> generated, so we should be able to write a test that writes one version,\n> writes the other, then checks the bytes.\n\nThanks for looking into it!\n\n> > > So I think we want to be more careful about when we load the existing\n> > > Bloom data or not. I think we probably want to load it in all cases, but\n> > > only read it when we have compatible Bloom settings. That should stop us\n> > > from exhibiting this kind of bug, but it also gives us a handle on\n> > > existing Bloom data if we wanted to copy forward existing results where\n> > > we can.\n>\n> The intention in the current patch set was to not load it at all when we\n> have incompatible Bloom settings because it appeared quite troublesome\n> to notate which Bloom filter in memory is of which version. If we want\n> to copy forward existing results, we can change that, but I don't know\n> whether it's worth doing that (and if we think it's worth doing, this\n> should probably go in another patch set).\n\nYeah, I think having Bloom filters accessible from a commit-graph\nregardless of whether or not it matches our Bloom filter version is\nprerequisite to being able to implement something like this.\n\nI feel like this is important enough to do in the same patch set, or the\nsame release to avoid surprising operators when their commit-graph write\nsuddenly recomputes all of its Bloom filters.\n\nSince we already store the Bloom version that we're using in the\ncommit-graph file itself, shouldn't it be something along the lines of\nsticking that value onto the bloom_filter when we read its contents?\n\nAlthough I don't think that you'd even need to annotate each individual\nfilter, since you know that every pre-existing Bloom filter you are able\nto find has the version given by:\n\n    the_repository->objects->commit_graph->bloom_filter_settings->hash_version\n\nright?\n\nThanks,\nTaylor\n"},{"id":"479935","messageId":"xmqqpm4db3dv.fsf@gitster.g","threadId":"59781","inReplyTo":"xmqqzg3ib2ma.fsf@gitster.g","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-27T18:44:44Z","receivedAt":"2023-07-27T18:44:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano <gitster@pobox.com> writes:\n\n> ...  How hard it would be to construct a test\n> case that fails if filter computed as v1 is marketed as v2?\n\nThere may be a more effective way for future-proofing, besides\nensuring the test coverage.\n\nAlthough this series added a way to see which version the on-disk\ndata is using using the version field, I do not think it touched the\n\"struct bloom_filter\" and \"struct bloom_key\" that represent the\nin-core data.  If we had a member in these structures that\nget_or_compute_bloom_filter() can fill in from the on-disk structure\nor the version it used when it computed the filter anew, would it\nbecome easier to catch the case where we try to add a version 2\ncomputed key to a filter that was read from version 1 on-disk\nstructure, presumably at add_key_to_filter()?\n\nThanks.\n\n"},{"id":"479941","messageId":"20230727205308.401364-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"ZMKvsObx+uaKA8zF@nand.local","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-07-27T20:53:08Z","receivedAt":"2023-07-27T20:53:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau <me@ttaylorr.com> writes:\n> > The intention in the current patch set was to not load it at all when we\n> > have incompatible Bloom settings because it appeared quite troublesome\n> > to notate which Bloom filter in memory is of which version. If we want\n> > to copy forward existing results, we can change that, but I don't know\n> > whether it's worth doing that (and if we think it's worth doing, this\n> > should probably go in another patch set).\n> \n> Yeah, I think having Bloom filters accessible from a commit-graph\n> regardless of whether or not it matches our Bloom filter version is\n> prerequisite to being able to implement something like this.\n> \n> I feel like this is important enough to do in the same patch set, or the\n> same release to avoid surprising operators when their commit-graph write\n> suddenly recomputes all of its Bloom filters.\n\nSuddenly reading many (or most) of the repo's trees would be a similar\nsurprise, right?\n\nAlso this would happen only if the server operator explicitly sets a\nchanged path filter version. If they leave it as-is, commit graphs will\nstill be written with the same version as the one on disk.\n\n> Since we already store the Bloom version that we're using in the\n> commit-graph file itself, shouldn't it be something along the lines of\n> sticking that value onto the bloom_filter when we read its contents?\n> \n> Although I don't think that you'd even need to annotate each individual\n> filter, since you know that every pre-existing Bloom filter you are able\n> to find has the version given by:\n> \n>     the_repository->objects->commit_graph->bloom_filter_settings->hash_version\n> \n> right?\n> \n> Thanks,\n> Taylor\n\nRegarding consulting commit_graph->bloom_filter_settings->hash_version,\nthe issue I ran into was that firstly, I didn't know what to do about\ncommit_graph->base_graph (which also has its own bloom_filter_settings)\nand what to do if it had a contradictory hash_version. And even if\nwe found a way to unify those, it is not true that every Bloom filter\nin memory is of that version, since we may have generated some that\ncorrespond to the version we're writing (not the version on disk).\nIn particular, the Bloom filters we write come from a commit slab\n(bloom_filters in bloom.c) and in that slab, both Bloom filters from\ndisk and Bloom filters that were generated in-process coexist.\n\nI also thought of your other proposal of augmenting struct bloom_filter\nto also include the version. The issue I ran into there is if, for a\ngiven commit, there already exists a Bloom filter read from disk with\nthe wrong version, what should we do? Overwrite it, or store both\nversions in memory? (We can't just immediately output the Bloom filter\nto disk and forget about the new version, only storing its size so that\nwe can generate the BIDX, because in the current code, generation and\nwriting to disk are separate. We could try to refactor it, but I didn't\nwant to make such a large change to reduce the possibility of bugs.)\nBoth storing the version number and storing an additional pointer for a\nsecond version would increase memory consumption too, even when\nsupporting two versions isn't needed, but maybe this isn't a big deal.\n"},{"id":"480060","messageId":"ZMlKMmAs3wKULAOd@nand.local","threadId":"59781","inReplyTo":"20230727205308.401364-1-jonathantanmy@google.com","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-01T18:08:50Z","receivedAt":"2023-08-01T18:09:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Jul 27, 2023 at 01:53:08PM -0700, Jonathan Tan wrote:\n> Taylor Blau <me@ttaylorr.com> writes:\n> > > The intention in the current patch set was to not load it at all when we\n> > > have incompatible Bloom settings because it appeared quite troublesome\n> > > to notate which Bloom filter in memory is of which version. If we want\n> > > to copy forward existing results, we can change that, but I don't know\n> > > whether it's worth doing that (and if we think it's worth doing, this\n> > > should probably go in another patch set).\n> >\n> > Yeah, I think having Bloom filters accessible from a commit-graph\n> > regardless of whether or not it matches our Bloom filter version is\n> > prerequisite to being able to implement something like this.\n> >\n> > I feel like this is important enough to do in the same patch set, or the\n> > same release to avoid surprising operators when their commit-graph write\n> > suddenly recomputes all of its Bloom filters.\n>\n> Suddenly reading many (or most) of the repo's trees would be a similar\n> surprise, right?\n\nThat's a good point. I think in general I'd expect Git to avoid\nrecomputing Bloom filters where that work can be avoided, if the work in\norder to detect whether or not we need to recompute a filter is cheap\nenough to carry out.\n\n> Also this would happen only if the server operator explicitly sets a\n> changed path filter version. If they leave it as-is, commit graphs will\n> still be written with the same version as the one on disk.\n\nI think that I could live with that if the default is to leave things\nas-is.\n\nI still think that it's worth it to have this functionality to propagate\nBloom filters forward should ship in Git, but we can work on that\noutside of this series.\n\n> Regarding consulting commit_graph->bloom_filter_settings->hash_version,\n> the issue I ran into was that firstly, I didn't know what to do about\n> commit_graph->base_graph (which also has its own bloom_filter_settings)\n> and what to do if it had a contradictory hash_version. And even if\n> we found a way to unify those, it is not true that every Bloom filter\n> in memory is of that version, since we may have generated some that\n> correspond to the version we're writing (not the version on disk).\n> In particular, the Bloom filters we write come from a commit slab\n> (bloom_filters in bloom.c) and in that slab, both Bloom filters from\n> disk and Bloom filters that were generated in-process coexist.\n\nWould we ever want to have a commit-graph chain with mixed Bloom filter\nversions?\n\nWe avoid mixing generation number schemes across multiple layers of a\ncommit-graph chain. But I don't see any reason that we should or need to\nhave a similar restriction in place for the Bloom filter version. Both\nare readable, as long as the user-provided configuration allows them to\nbe.\n\nWe just have to interpret them differently depending on what layer of\nthe graph (and therefore, what Bloom filter version they are) they come\nfrom.\n\nSorry for thinking aloud a little there. I think that this means that we\nat minimum have to keep in context the commit-graph layer we found the\nBloom filter in so that we can tie that back to its Bloom filter\nversion. That might just mean that we have to tag each Bloom filter with\nthe version it was computed under, or maybe we already have the\ncommit-graph layer in context, in which case we shouldn't need an\nadditional field.\n\nMy gut is telling me that we probably *do* need such a field, since we\ndon't often have a reference to the particular layer that we found a\nBloom filter in, just the tip of the commit-graph chain that it came\nfrom.\n\n> I also thought of your other proposal of augmenting struct bloom_filter\n> to also include the version. The issue I ran into there is if, for a\n> given commit, there already exists a Bloom filter read from disk with\n> the wrong version, what should we do? Overwrite it, or store both\n> versions in memory? (We can't just immediately output the Bloom filter\n> to disk and forget about the new version, only storing its size so that\n> we can generate the BIDX, because in the current code, generation and\n> writing to disk are separate. We could try to refactor it, but I didn't\n> want to make such a large change to reduce the possibility of bugs.)\n> Both storing the version number and storing an additional pointer for a\n> second version would increase memory consumption too, even when\n> supporting two versions isn't needed, but maybe this isn't a big deal.\n\nIt's likely that I'm missing something here, but what is stopping us\nfrom discarding the old Bloom filter as soon as we generate the new\none? We shouldn't need to load the old filter again out of the commit\nslab, right?\n\nThanks,\nTaylor\n"},{"id":"480066","messageId":"3ce6090a4d3634b2e947d57f06714182207bb33a.1690912539.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1690912539.git.jonathantanmy@google.com","subject":"[PATCH v7 1/7] gitformat-commit-graph: describe version 2 of BDAT","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-01T18:41:36Z","receivedAt":"2023-08-01T18:42:11Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The code change to Git to support version 2 will be done in subsequent\ncommits.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/gitformat-commit-graph.txt | 9 ++++++---\n 1 file changed, 6 insertions(+), 3 deletions(-)\n\ndiff --git a/Documentation/gitformat-commit-graph.txt b/Documentation/gitformat-commit-graph.txt\nindex 31cad585e2..3e906e8030 100644\n--- a/Documentation/gitformat-commit-graph.txt\n+++ b/Documentation/gitformat-commit-graph.txt\n@@ -142,13 +142,16 @@ All multi-byte numbers are in network byte order.\n \n ==== Bloom Filter Data (ID: {'B', 'D', 'A', 'T'}) [Optional]\n     * It starts with header consisting of three unsigned 32-bit integers:\n-      - Version of the hash algorithm being used. We currently only support\n-\tvalue 1 which corresponds to the 32-bit version of the murmur3 hash\n+      - Version of the hash algorithm being used. We currently support\n+\tvalue 2 which corresponds to the 32-bit version of the murmur3 hash\n \timplemented exactly as described in\n \thttps://en.wikipedia.org/wiki/MurmurHash#Algorithm and the double\n \thashing technique using seed values 0x293ae76f and 0x7e646e2 as\n \tdescribed in https://doi.org/10.1007/978-3-540-30494-4_26 \"Bloom Filters\n-\tin Probabilistic Verification\"\n+\tin Probabilistic Verification\". Version 1 Bloom filters have a bug that appears\n+\twhen char is signed and the repository has path names that have characters >=\n+\t0x80; Git supports reading and writing them, but this ability will be removed\n+\tin a future version of Git.\n       - The number of times a path is hashed and hence the number of bit positions\n \t      that cumulatively determine whether a file is present in the commit.\n       - The minimum number of bits 'b' per entry in the Bloom filter. If the filter\n-- \n2.41.0.585.gd2178a4bd4-goog\n\n"},{"id":"480069","messageId":"cover.1690912539.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1684790529.git.jonathantanmy@google.com","subject":"[PATCH v7 0/7] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-01T18:41:35Z","receivedAt":"2023-08-01T18:42:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I've fixed the bug that Taylor described. It was an issue where the\npresence of Bloom filters can be indicated both by the presence of the\nchunk and the presence of a bloom_filter_settings, and I've fixed it by\navoiding setting the chunk_bloom_data if we're not using Bloom filters\ndue to an incompatible version. In the future, we might want to refactor\nthe code so that there is only one way to indicate whether the Bloom\nfilters are present.\n\nI've also added sign-offs and changed the indentation of the tests, as\nremarked by Taylor [1] [2].\n\n[1] https://lore.kernel.org/git/ZMGruenDbAo22aWV@nand.local/\n[2] https://lore.kernel.org/git/ZMGsJTxBXZ94lhMU@nand.local/\n\nTaylor also suggested copying forward Bloom filters whenever possible\nin this patch set [3], but also that we could work on this outside this\nseries [4]. I did not implement this in this series.\n\n[3] https://lore.kernel.org/git/ZMKvsObx+uaKA8zF@nand.local/\n[4] https://lore.kernel.org/git/ZMlKMmAs3wKULAOd@nand.local/\n\nJonathan Tan (4):\n  gitformat-commit-graph: describe version 2 of BDAT\n  t4216: test changed path filters with high bit paths\n  repo-settings: introduce commitgraph.changedPathsVersion\n  commit-graph: new filter ver. that fixes murmur3\n\nTaylor Blau (3):\n  t/helper/test-read-graph.c: extract `dump_graph_info()`\n  bloom.h: make `load_bloom_filter_from_graph()` public\n  t/helper/test-read-graph: implement `bloom-filters` mode\n\n Documentation/config/commitgraph.txt     |  26 ++++-\n Documentation/gitformat-commit-graph.txt |   9 +-\n bloom.c                                  |  75 +++++++++++-\n bloom.h                                  |  13 ++-\n commit-graph.c                           |  35 ++++--\n oss-fuzz/fuzz-commit-graph.c             |   2 +-\n repo-settings.c                          |   6 +-\n repository.h                             |   2 +-\n t/helper/test-bloom.c                    |   9 +-\n t/helper/test-read-graph.c               |  67 ++++++++---\n t/t0095-bloom.sh                         |   8 ++\n t/t4216-log-bloom.sh                     | 139 +++++++++++++++++++++++\n 12 files changed, 351 insertions(+), 40 deletions(-)\n\nRange-diff against v6:\n1:  3ce6090a4d = 1:  3ce6090a4d gitformat-commit-graph: describe version 2 of BDAT\n2:  1955734d1f ! 2:  fc6346c039 t/helper/test-read-graph.c: extract `dump_graph_info()`\n    @@ Commit message\n         main routine into a separate function.\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n    +    Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n     \n      ## t/helper/test-read-graph.c ##\n     @@\n3:  4cf7c2f634 ! 3:  f144dc4b15 bloom.h: make `load_bloom_filter_from_graph()` public\n    @@ Commit message\n         for manual inspection (to be used during tests).\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n    +    Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n     \n      ## bloom.c ##\n     @@ bloom.c: static inline unsigned char get_bitmask(uint32_t pos)\n4:  47b55758e6 ! 4:  2ade832a23 t/helper/test-read-graph: implement `bloom-filters` mode\n    @@ Commit message\n         hexadecimal contents of the Bloom filter(s) contained in a commit-graph.\n     \n         Signed-off-by: Taylor Blau <me@ttaylorr.com>\n    +    Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n     \n      ## t/helper/test-read-graph.c ##\n     @@ t/helper/test-read-graph.c: static void dump_graph_info(struct commit_graph *graph)\n5:  5276e6a90e ! 5:  74863c11e5 t4216: test changed path filters with high bit paths\n    @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm\n     +'\n     +\n     +test_expect_success 'setup check value of version 1 changed-path' '\n    -+\t(cd highbit1 &&\n    ++\t(\n    ++\t\tcd highbit1 &&\n     +\t\techo \"52a9\" >expect &&\n     +\t\tget_first_changed_path_filter >actual &&\n    -+\t\ttest_cmp expect actual)\n    ++\t\ttest_cmp expect actual\n    ++\t)\n     +'\n     +\n     +# expect will not match actual if char is unsigned by default. Write the test\n    @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills empty comm\n     +'\n     +\n     +test_expect_success 'version 1 changed-path used when version 1 requested' '\n    -+\t(cd highbit1 &&\n    -+\t\ttest_bloom_filters_used \"-- $CENT\")\n    ++\t(\n    ++\t\tcd highbit1 &&\n    ++\t\ttest_bloom_filters_used \"-- $CENT\"\n    ++\t)\n     +'\n     +\n      test_done\n6:  dc3f6d2d4f = 6:  60f4faeff9 repo-settings: introduce commitgraph.changedPathsVersion\n7:  6e2d797406 ! 7:  68258cfd04 commit-graph: new filter ver. that fixes murmur3\n    @@ commit-graph.c: static int graph_read_oid_lookup(const unsigned char *chunk_star\n     +\tstruct graph_read_bloom_data_context *c = data;\n     +\tstruct commit_graph *g = c->g;\n      \tuint32_t hash_version;\n    - \tg->chunk_bloom_data = chunk_start;\n    +-\tg->chunk_bloom_data = chunk_start;\n      \thash_version = get_be32(chunk_start);\n      \n     -\tif (hash_version != 1)\n    @@ commit-graph.c: static int graph_read_oid_lookup(const unsigned char *chunk_star\n     + \t\treturn 0;\n     +\t}\n      \n    ++\tg->chunk_bloom_data = chunk_start;\n      \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n      \tg->bloom_filter_settings->hash_version = hash_version;\n    + \tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n     @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repo_settings *s,\n      \t}\n      \n    @@ t/t0095-bloom.sh: test_expect_success 'compute unseeded murmur3 hash for test st\n      \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\n     \n      ## t/t4216-log-bloom.sh ##\n    -@@ t/t4216-log-bloom.sh: test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n    - \tgit -C highbit1 commit-graph write --reachable --changed-paths\n    - '\n    - \n    --test_expect_success 'setup check value of version 1 changed-path' '\n    -+test_expect_success 'check value of version 1 changed-path' '\n    - \t(cd highbit1 &&\n    - \t\techo \"52a9\" >expect &&\n    - \t\tget_first_changed_path_filter >actual &&\n     @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when version 1 requested' '\n    - \t\ttest_bloom_filters_used \"-- $CENT\")\n    + \t)\n      '\n      \n     +test_expect_success 'version 1 changed-path not used when version 2 requested' '\n    -+\t(cd highbit1 &&\n    ++\t(\n    ++\t\tcd highbit1 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n    -+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n    ++\t\ttest_bloom_filters_not_used \"-- $CENT\"\n    ++\t)\n     +'\n     +\n     +test_expect_success 'version 1 changed-path used when autodetect requested' '\n    -+\t(cd highbit1 &&\n    ++\t(\n    ++\t\tcd highbit1 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n    -+\t\ttest_bloom_filters_used \"-- $CENT\")\n    ++\t\ttest_bloom_filters_used \"-- $CENT\"\n    ++\t)\n     +'\n     +\n     +test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n     +\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n     +\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n    -+\t(cd highbit1 &&\n    ++\t(\n    ++\t\tcd highbit1 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n     +\t\techo \"options: bloom(1,10,7) read_generation_data\" >expect &&\n     +\t\ttest-tool read-graph >full &&\n     +\t\tgrep options full >actual &&\n    -+\t\ttest_cmp expect actual)\n    ++\t\ttest_cmp expect actual\n    ++\t)\n     +'\n     +\n     +test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n    @@ t/t4216-log-bloom.sh: test_expect_success 'version 1 changed-path used when vers\n     +'\n     +\n     +test_expect_success 'check value of version 2 changed-path' '\n    -+\t(cd highbit2 &&\n    ++\t(\n    ++\t\tcd highbit2 &&\n     +\t\techo \"c01f\" >expect &&\n     +\t\tget_first_changed_path_filter >actual &&\n    -+\t\ttest_cmp expect actual)\n    ++\t\ttest_cmp expect actual\n    ++\t)\n     +'\n     +\n     +test_expect_success 'version 2 changed-path used when version 2 requested' '\n    -+\t(cd highbit2 &&\n    -+\t\ttest_bloom_filters_used \"-- $CENT\")\n    ++\t(\n    ++\t\tcd highbit2 &&\n    ++\t\ttest_bloom_filters_used \"-- $CENT\"\n    ++\t)\n     +'\n     +\n     +test_expect_success 'version 2 changed-path not used when version 1 requested' '\n    -+\t(cd highbit2 &&\n    ++\t(\n    ++\t\tcd highbit2 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n    -+\t\ttest_bloom_filters_not_used \"-- $CENT\")\n    ++\t\ttest_bloom_filters_not_used \"-- $CENT\"\n    ++\t)\n     +'\n     +\n     +test_expect_success 'version 2 changed-path used when autodetect requested' '\n    -+\t(cd highbit2 &&\n    ++\t(\n    ++\t\tcd highbit2 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n    -+\t\ttest_bloom_filters_used \"-- $CENT\")\n    ++\t\ttest_bloom_filters_used \"-- $CENT\"\n    ++\t)\n     +'\n     +\n     +test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n     +\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n     +\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n    -+\t(cd highbit2 &&\n    ++\t(\n    ++\t\tcd highbit2 &&\n     +\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n     +\t\techo \"options: bloom(2,10,7) read_generation_data\" >expect &&\n     +\t\ttest-tool read-graph >full &&\n     +\t\tgrep options full >actual &&\n    -+\t\ttest_cmp expect actual)\n    ++\t\ttest_cmp expect actual\n    ++\t)\n    ++'\n    ++\n    ++test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n    ++\tgit init doublewrite &&\n    ++\ttest_commit -C doublewrite c \"$CENT\" &&\n    ++\tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n    ++\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n    ++\tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n    ++\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n    ++\t(\n    ++\t\tcd doublewrite &&\n    ++\t\techo \"c01f\" >expect &&\n    ++\t\tget_first_changed_path_filter >actual &&\n    ++\t\ttest_cmp expect actual\n    ++\t)\n     +'\n     +\n      test_done\n-- \n2.41.0.585.gd2178a4bd4-goog\n\n"},{"id":"480067","messageId":"fc6346c039db46ad0c35c8464ae6e7c9fe976a0d.1690912539.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1690912539.git.jonathantanmy@google.com","subject":"[PATCH v7 2/7] t/helper/test-read-graph.c: extract `dump_graph_info()`","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-01T18:41:37Z","receivedAt":"2023-08-01T18:42:15Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau <me@ttaylorr.com>\n\nPrepare for the 'read-graph' test helper to perform other tasks besides\ndumping high-level information about the commit-graph by extracting its\nmain routine into a separate function.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n t/helper/test-read-graph.c | 33 ++++++++++++++++++++-------------\n 1 file changed, 20 insertions(+), 13 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 8c7a83f578..c664928412 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -5,20 +5,8 @@\n #include \"bloom.h\"\n #include \"setup.h\"\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_info(struct commit_graph *graph)\n {\n-\tstruct commit_graph *graph = NULL;\n-\tstruct object_directory *odb;\n-\n-\tsetup_git_directory();\n-\todb = the_repository->objects->odb;\n-\n-\tprepare_repo_settings(the_repository);\n-\n-\tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n@@ -57,8 +45,27 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tif (graph->topo_levels)\n \t\tprintf(\" topo_levels\");\n \tprintf(\"\\n\");\n+}\n+\n+int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+{\n+\tstruct commit_graph *graph = NULL;\n+\tstruct object_directory *odb;\n+\n+\tsetup_git_directory();\n+\todb = the_repository->objects->odb;\n+\n+\tprepare_repo_settings(the_repository);\n+\n+\tgraph = read_commit_graph_one(the_repository, odb);\n+\tif (!graph)\n+\t\treturn 1;\n+\n+\tdump_graph_info(graph);\n \n \tUNLEAK(graph);\n \n \treturn 0;\n }\n+\n+\n-- \n2.41.0.585.gd2178a4bd4-goog\n\n"},{"id":"480068","messageId":"f144dc4b15ea97cb6214a4bfbf2d88d8f16f05be.1690912539.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1690912539.git.jonathantanmy@google.com","subject":"[PATCH v7 3/7] bloom.h: make `load_bloom_filter_from_graph()` public","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-01T18:41:38Z","receivedAt":"2023-08-01T18:42:16Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau <me@ttaylorr.com>\n\nPrepare for a future commit to use the load_bloom_filter_from_graph()\nfunction directly to load specific Bloom filters out of the commit-graph\nfor manual inspection (to be used during tests).\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n bloom.c | 6 +++---\n bloom.h | 5 +++++\n 2 files changed, 8 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex aef6b5fea2..3e78cfe79d 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -29,9 +29,9 @@ static inline unsigned char get_bitmask(uint32_t pos)\n \treturn ((unsigned char)1) << (pos & (BITS_PER_WORD - 1));\n }\n \n-static int load_bloom_filter_from_graph(struct commit_graph *g,\n-\t\t\t\t\tstruct bloom_filter *filter,\n-\t\t\t\t\tuint32_t graph_pos)\n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos)\n {\n \tuint32_t lex_pos, start_index, end_index;\n \ndiff --git a/bloom.h b/bloom.h\nindex adde6dfe21..1e4f612d2c 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -3,6 +3,7 @@\n \n struct commit;\n struct repository;\n+struct commit_graph;\n \n struct bloom_filter_settings {\n \t/*\n@@ -68,6 +69,10 @@ struct bloom_key {\n \tuint32_t *hashes;\n };\n \n+int load_bloom_filter_from_graph(struct commit_graph *g,\n+\t\t\t\t struct bloom_filter *filter,\n+\t\t\t\t uint32_t graph_pos);\n+\n /*\n  * Calculate the murmur3 32-bit hash value for the given data\n  * using the given seed.\n-- \n2.41.0.585.gd2178a4bd4-goog\n\n"},{"id":"480070","messageId":"2ade832a230af797ed844947e5e7e90cdbc77c26.1690912539.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1690912539.git.jonathantanmy@google.com","subject":"[PATCH v7 4/7] t/helper/test-read-graph: implement `bloom-filters` mode","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-01T18:41:39Z","receivedAt":"2023-08-01T18:42:19Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"From: Taylor Blau <me@ttaylorr.com>\n\nImplement a mode of the \"read-graph\" test helper to dump out the\nhexadecimal contents of the Bloom filter(s) contained in a commit-graph.\n\nSigned-off-by: Taylor Blau <me@ttaylorr.com>\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n t/helper/test-read-graph.c | 42 +++++++++++++++++++++++++++++++++-----\n 1 file changed, 37 insertions(+), 5 deletions(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex c664928412..da9ac8584d 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -47,10 +47,32 @@ static void dump_graph_info(struct commit_graph *graph)\n \tprintf(\"\\n\");\n }\n \n-int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n+static void dump_graph_bloom_filters(struct commit_graph *graph)\n+{\n+\tuint32_t i;\n+\n+\tfor (i = 0; i < graph->num_commits + graph->num_commits_in_base; i++) {\n+\t\tstruct bloom_filter filter = { 0 };\n+\t\tsize_t j;\n+\n+\t\tif (load_bloom_filter_from_graph(graph, &filter, i) < 0) {\n+\t\t\tfprintf(stderr, \"missing Bloom filter for graph \"\n+\t\t\t\t\"position %\"PRIu32\"\\n\", i);\n+\t\t\tcontinue;\n+\t\t}\n+\n+\t\tfor (j = 0; j < filter.len; j++)\n+\t\t\tprintf(\"%02x\", filter.data[j]);\n+\t\tif (filter.len)\n+\t\t\tprintf(\"\\n\");\n+\t}\n+}\n+\n+int cmd__read_graph(int argc, const char **argv)\n {\n \tstruct commit_graph *graph = NULL;\n \tstruct object_directory *odb;\n+\tint ret = 0;\n \n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n@@ -58,14 +80,24 @@ int cmd__read_graph(int argc UNUSED, const char **argv UNUSED)\n \tprepare_repo_settings(the_repository);\n \n \tgraph = read_commit_graph_one(the_repository, odb);\n-\tif (!graph)\n-\t\treturn 1;\n+\tif (!graph) {\n+\t\tret = 1;\n+\t\tgoto done;\n+\t}\n \n-\tdump_graph_info(graph);\n+\tif (argc <= 1)\n+\t\tdump_graph_info(graph);\n+\telse if (!strcmp(argv[1], \"bloom-filters\"))\n+\t\tdump_graph_bloom_filters(graph);\n+\telse {\n+\t\tfprintf(stderr, \"unknown sub-command: '%s'\\n\", argv[1]);\n+\t\tret = 1;\n+\t}\n \n+done:\n \tUNLEAK(graph);\n \n-\treturn 0;\n+\treturn ret;\n }\n \n \n-- \n2.41.0.585.gd2178a4bd4-goog\n\n"},{"id":"480071","messageId":"74863c11e59c6dbf3673507b58e7c841162d3d18.1690912539.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1690912539.git.jonathantanmy@google.com","subject":"[PATCH v7 5/7] t4216: test changed path filters with high bit paths","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-01T18:41:40Z","receivedAt":"2023-08-01T18:42:21Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Subsequent commits will teach Git another version of changed path\nfilter that has different behavior with paths that contain at least\none character with its high bit set, so test the existing behavior as\na baseline.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n t/t4216-log-bloom.sh | 43 +++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 43 insertions(+)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fa9d32facf..2d4a3fefee 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -404,4 +404,47 @@ test_expect_success 'Bloom generation backfills empty commits' '\n \t)\n '\n \n+get_first_changed_path_filter () {\n+\ttest-tool read-graph bloom-filters >filters.dat &&\n+\thead -n 1 filters.dat\n+}\n+\n+# chosen to be the same under all Unicode normalization forms\n+CENT=$(printf \"\\302\\242\")\n+\n+test_expect_success 'set up repo with high bit path, version 1 changed-path' '\n+\tgit init highbit1 &&\n+\ttest_commit -C highbit1 c1 \"$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'setup check value of version 1 changed-path' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\techo \"52a9\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+# expect will not match actual if char is unsigned by default. Write the test\n+# in this way, so that a user running this test script can still see if the two\n+# files match. (It will appear as an ordinary success if they match, and a skip\n+# if not.)\n+if test_cmp highbit1/expect highbit1/actual\n+then\n+\ttest_set_prereq SIGNED_CHAR_BY_DEFAULT\n+fi\n+test_expect_success SIGNED_CHAR_BY_DEFAULT 'check value of version 1 changed-path' '\n+\t# Only the prereq matters for this test.\n+\ttrue\n+'\n+\n+test_expect_success 'version 1 changed-path used when version 1 requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\"\n+\t)\n+'\n+\n test_done\n-- \n2.41.0.585.gd2178a4bd4-goog\n\n"},{"id":"480072","messageId":"60f4faeff9e3c53c69acc85f033c33c1be632400.1690912539.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1690912539.git.jonathantanmy@google.com","subject":"[PATCH v7 6/7] repo-settings: introduce commitgraph.changedPathsVersion","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-01T18:41:41Z","receivedAt":"2023-08-01T18:42:23Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"A subsequent commit will introduce another version of the changed-path\nfilter in the commit graph file. In order to control which version to\nwrite (and read), a config variable is needed.\n\nTherefore, introduce this config variable. For forwards compatibility,\nteach Git to not read commit graphs when the config variable\nis set to an unsupported version. Because we teach Git this,\ncommitgraph.readChangedPaths is now redundant, so deprecate it and\ndefine its behavior in terms of the config variable we introduce.\n\nThis commit does not change the behavior of writing (Git writes changed\npath filters when explicitly instructed regardless of any config\nvariable), but a subsequent commit will restrict Git such that it will\nonly write when commitgraph.changedPathsVersion is a recognized value.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt | 23 ++++++++++++++++++++---\n commit-graph.c                       |  2 +-\n oss-fuzz/fuzz-commit-graph.c         |  2 +-\n repo-settings.c                      |  6 +++++-\n repository.h                         |  2 +-\n 5 files changed, 28 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 30604e4a4c..2dc9170622 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -9,6 +9,23 @@ commitGraph.maxNewFilters::\n \tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n \n commitGraph.readChangedPaths::\n-\tIf true, then git will use the changed-path Bloom filters in the\n-\tcommit-graph file (if it exists, and they are present). Defaults to\n-\ttrue. See linkgit:git-commit-graph[1] for more information.\n+\tDeprecated. Equivalent to commitGraph.changedPathsVersion=-1 if true, and\n+\tcommitGraph.changedPathsVersion=0 if false. (If commitGraph.changedPathVersion\n+\tis also set, commitGraph.changedPathsVersion takes precedence.)\n+\n+commitGraph.changedPathsVersion::\n+\tSpecifies the version of the changed-path Bloom filters that Git will read and\n+\twrite. May be -1, 0 or 1.\n++\n+Defaults to -1.\n++\n+If -1, Git will use the version of the changed-path Bloom filters in the\n+repository, defaulting to 1 if there are none.\n++\n+If 0, Git will not read any Bloom filters, and will write version 1 Bloom\n+filters when instructed to write.\n++\n+If 1, Git will only read version 1 Bloom filters, and will write version 1\n+Bloom filters.\n++\n+See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex efc697e437..1f26c07de4 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -401,7 +401,7 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t\t\tgraph->read_generation_data = 1;\n \t}\n \n-\tif (s->commit_graph_read_changed_paths) {\n+\tif (s->commit_graph_changed_paths_version) {\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\ndiff --git a/oss-fuzz/fuzz-commit-graph.c b/oss-fuzz/fuzz-commit-graph.c\nindex 2992079dd9..325c0b991a 100644\n--- a/oss-fuzz/fuzz-commit-graph.c\n+++ b/oss-fuzz/fuzz-commit-graph.c\n@@ -19,7 +19,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \t * possible.\n \t */\n \tthe_repository->settings.commit_graph_generation_version = 2;\n-\tthe_repository->settings.commit_graph_read_changed_paths = 1;\n+\tthe_repository->settings.commit_graph_changed_paths_version = 1;\n \tg = parse_commit_graph(&the_repository->settings, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 525f69c0c7..db8fe817f3 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -24,6 +24,7 @@ void prepare_repo_settings(struct repository *r)\n \tint value;\n \tconst char *strval;\n \tint manyfiles;\n+\tint read_changed_paths;\n \n \tif (!r->gitdir)\n \t\tBUG(\"Cannot add settings for uninitialized repository\");\n@@ -54,7 +55,10 @@ void prepare_repo_settings(struct repository *r)\n \t/* Commit graph config or default, does not cascade (simple) */\n \trepo_cfg_bool(r, \"core.commitgraph\", &r->settings.core_commit_graph, 1);\n \trepo_cfg_int(r, \"commitgraph.generationversion\", &r->settings.commit_graph_generation_version, 2);\n-\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &r->settings.commit_graph_read_changed_paths, 1);\n+\trepo_cfg_bool(r, \"commitgraph.readchangedpaths\", &read_changed_paths, 1);\n+\trepo_cfg_int(r, \"commitgraph.changedpathsversion\",\n+\t\t     &r->settings.commit_graph_changed_paths_version,\n+\t\t     read_changed_paths ? -1 : 0);\n \trepo_cfg_bool(r, \"gc.writecommitgraph\", &r->settings.gc_write_commit_graph, 1);\n \trepo_cfg_bool(r, \"fetch.writecommitgraph\", &r->settings.fetch_write_commit_graph, 0);\n \ndiff --git a/repository.h b/repository.h\nindex 5f18486f64..f71154e12c 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -29,7 +29,7 @@ struct repo_settings {\n \n \tint core_commit_graph;\n \tint commit_graph_generation_version;\n-\tint commit_graph_read_changed_paths;\n+\tint commit_graph_changed_paths_version;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \tint command_requires_full_index;\n-- \n2.41.0.585.gd2178a4bd4-goog\n\n"},{"id":"480073","messageId":"68258cfd04cee93d1a1a5bfce674f299dd070b84.1690912539.git.jonathantanmy@google.com","threadId":"59781","inReplyTo":"cover.1690912539.git.jonathantanmy@google.com","subject":"[PATCH v7 7/7] commit-graph: new filter ver. that fixes murmur3","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-01T18:41:42Z","receivedAt":"2023-08-01T18:42:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The murmur3 implementation in bloom.c has a bug when converting series\nof 4 bytes into network-order integers when char is signed (which is\ncontrollable by a compiler option, and the default signedness of char is\nplatform-specific). When a string contains characters with the high bit\nset, this bug causes results that, although internally consistent within\nGit, does not accord with other implementations of murmur3 (thus,\nthe changed path filters wouldn't be readable by other off-the-shelf\nimplementatios of murmur3) and even with Git binaries that were compiled\nwith different signedness of char. This bug affects both how Git writes\nchanged path filters to disk and how Git interprets changed path filters\non disk.\n\nTherefore, introduce a new version (2) of changed path filters that\ncorrects this problem. The existing version (1) is still supported and\nis still the default, but users should migrate away from it as soon\nas possible.\n\nBecause this bug only manifests with characters that have the high bit\nset, it may be possible that some (or all) commits in a given repo would\nhave the same changed path filter both before and after this fix is\napplied. However, in order to determine whether this is the case, the\nchanged paths would first have to be computed, at which point it is not\nmuch more expensive to just compute a new changed path filter.\n\nSo this patch does not include any mechanism to \"salvage\" changed path\nfilters from repositories. There is also no \"mixed\" mode - for each\ninvocation of Git, reading and writing changed path filters are done\nwith the same version number; this version number may be explicitly\nstated (typically if the user knows which version they need) or\nautomatically determined from the version of the existing changed path\nfilters in the repository.\n\nThere is a change in write_commit_graph(). graph_read_bloom_data()\nmakes it possible for chunk_bloom_data to be non-NULL but\nbloom_filter_settings to be NULL, which causes a segfault later on. I\nproduced such a segfault while developing this patch, but couldn't find\na way to reproduce it neither after this complete patch (or before),\nbut in any case it seemed like a good thing to include that might help\nfuture patch authors.\n\nThe value in t0095 was obtained from another murmur3 implementation\nusing the following Go source code:\n\n  package main\n\n  import \"fmt\"\n  import \"github.com/spaolacci/murmur3\"\n\n  func main() {\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte(\"Hello world!\")))\n          fmt.Printf(\"%x\\n\", murmur3.Sum32([]byte{0x99, 0xaa, 0xbb, 0xcc, 0xdd, 0xee, 0xff}))\n  }\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Junio C Hamano <gitster@pobox.com>\n---\n Documentation/config/commitgraph.txt |  5 +-\n bloom.c                              | 69 +++++++++++++++++++-\n bloom.h                              |  8 ++-\n commit-graph.c                       | 33 ++++++++--\n t/helper/test-bloom.c                |  9 ++-\n t/t0095-bloom.sh                     |  8 +++\n t/t4216-log-bloom.sh                 | 96 ++++++++++++++++++++++++++++\n 7 files changed, 214 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex 2dc9170622..acc74a2f27 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -15,7 +15,7 @@ commitGraph.readChangedPaths::\n \n commitGraph.changedPathsVersion::\n \tSpecifies the version of the changed-path Bloom filters that Git will read and\n-\twrite. May be -1, 0 or 1.\n+\twrite. May be -1, 0, 1, or 2.\n +\n Defaults to -1.\n +\n@@ -28,4 +28,7 @@ filters when instructed to write.\n If 1, Git will only read version 1 Bloom filters, and will write version 1\n Bloom filters.\n +\n+If 2, Git will only read version 2 Bloom filters, and will write version 2\n+Bloom filters.\n++\n See linkgit:git-commit-graph[1] for more information.\ndiff --git a/bloom.c b/bloom.c\nindex 3e78cfe79d..ebef5cfd2f 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -66,7 +66,64 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len)\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len)\n+{\n+\tconst uint32_t c1 = 0xcc9e2d51;\n+\tconst uint32_t c2 = 0x1b873593;\n+\tconst uint32_t r1 = 15;\n+\tconst uint32_t r2 = 13;\n+\tconst uint32_t m = 5;\n+\tconst uint32_t n = 0xe6546b64;\n+\tint i;\n+\tuint32_t k1 = 0;\n+\tconst char *tail;\n+\n+\tint len4 = len / sizeof(uint32_t);\n+\n+\tuint32_t k;\n+\tfor (i = 0; i < len4; i++) {\n+\t\tuint32_t byte1 = (uint32_t)(unsigned char)data[4*i];\n+\t\tuint32_t byte2 = ((uint32_t)(unsigned char)data[4*i + 1]) << 8;\n+\t\tuint32_t byte3 = ((uint32_t)(unsigned char)data[4*i + 2]) << 16;\n+\t\tuint32_t byte4 = ((uint32_t)(unsigned char)data[4*i + 3]) << 24;\n+\t\tk = byte1 | byte2 | byte3 | byte4;\n+\t\tk *= c1;\n+\t\tk = rotate_left(k, r1);\n+\t\tk *= c2;\n+\n+\t\tseed ^= k;\n+\t\tseed = rotate_left(seed, r2) * m + n;\n+\t}\n+\n+\ttail = (data + len4 * sizeof(uint32_t));\n+\n+\tswitch (len & (sizeof(uint32_t) - 1)) {\n+\tcase 3:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[2]) << 16;\n+\t\t/*-fallthrough*/\n+\tcase 2:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[1]) << 8;\n+\t\t/*-fallthrough*/\n+\tcase 1:\n+\t\tk1 ^= ((uint32_t)(unsigned char)tail[0]) << 0;\n+\t\tk1 *= c1;\n+\t\tk1 = rotate_left(k1, r1);\n+\t\tk1 *= c2;\n+\t\tseed ^= k1;\n+\t\tbreak;\n+\t}\n+\n+\tseed ^= (uint32_t)len;\n+\tseed ^= (seed >> 16);\n+\tseed *= 0x85ebca6b;\n+\tseed ^= (seed >> 13);\n+\tseed *= 0xc2b2ae35;\n+\tseed ^= (seed >> 16);\n+\n+\treturn seed;\n+}\n+\n+static uint32_t murmur3_seeded_v1(uint32_t seed, const char *data, size_t len)\n {\n \tconst uint32_t c1 = 0xcc9e2d51;\n \tconst uint32_t c2 = 0x1b873593;\n@@ -131,8 +188,14 @@ void fill_bloom_key(const char *data,\n \tint i;\n \tconst uint32_t seed0 = 0x293ae76f;\n \tconst uint32_t seed1 = 0x7e646e2c;\n-\tconst uint32_t hash0 = murmur3_seeded(seed0, data, len);\n-\tconst uint32_t hash1 = murmur3_seeded(seed1, data, len);\n+\tuint32_t hash0, hash1;\n+\tif (settings->hash_version == 2) {\n+\t\thash0 = murmur3_seeded_v2(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v2(seed1, data, len);\n+\t} else {\n+\t\thash0 = murmur3_seeded_v1(seed0, data, len);\n+\t\thash1 = murmur3_seeded_v1(seed1, data, len);\n+\t}\n \n \tkey->hashes = (uint32_t *)xcalloc(settings->num_hashes, sizeof(uint32_t));\n \tfor (i = 0; i < settings->num_hashes; i++)\ndiff --git a/bloom.h b/bloom.h\nindex 1e4f612d2c..138d57a86b 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -8,9 +8,11 @@ struct commit_graph;\n struct bloom_filter_settings {\n \t/*\n \t * The version of the hashing technique being used.\n-\t * We currently only support version = 1 which is\n+\t * The newest version is 2, which is\n \t * the seeded murmur3 hashing technique implemented\n-\t * in bloom.c.\n+\t * in bloom.c. Bloom filters of version 1 were created\n+\t * with prior versions of Git, which had a bug in the\n+\t * implementation of the hash function.\n \t */\n \tuint32_t hash_version;\n \n@@ -80,7 +82,7 @@ int load_bloom_filter_from_graph(struct commit_graph *g,\n  * Not considered to be cryptographically secure.\n  * Implemented as described in https://en.wikipedia.org/wiki/MurmurHash#Algorithm\n  */\n-uint32_t murmur3_seeded(uint32_t seed, const char *data, size_t len);\n+uint32_t murmur3_seeded_v2(uint32_t seed, const char *data, size_t len);\n \n void fill_bloom_key(const char *data,\n \t\t    size_t len,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1f26c07de4..ffbc86151e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -304,17 +304,26 @@ static int graph_read_oid_lookup(const unsigned char *chunk_start,\n \treturn 0;\n }\n \n+struct graph_read_bloom_data_context {\n+\tstruct commit_graph *g;\n+\tint *commit_graph_changed_paths_version;\n+};\n+\n static int graph_read_bloom_data(const unsigned char *chunk_start,\n \t\t\t\t  size_t chunk_size, void *data)\n {\n-\tstruct commit_graph *g = data;\n+\tstruct graph_read_bloom_data_context *c = data;\n+\tstruct commit_graph *g = c->g;\n \tuint32_t hash_version;\n-\tg->chunk_bloom_data = chunk_start;\n \thash_version = get_be32(chunk_start);\n \n-\tif (hash_version != 1)\n-\t\treturn 0;\n+\tif (*c->commit_graph_changed_paths_version == -1) {\n+\t\t*c->commit_graph_changed_paths_version = hash_version;\n+\t} else if (hash_version != *c->commit_graph_changed_paths_version) {\n+ \t\treturn 0;\n+\t}\n \n+\tg->chunk_bloom_data = chunk_start;\n \tg->bloom_filter_settings = xmalloc(sizeof(struct bloom_filter_settings));\n \tg->bloom_filter_settings->hash_version = hash_version;\n \tg->bloom_filter_settings->num_hashes = get_be32(chunk_start + 4);\n@@ -402,10 +411,14 @@ struct commit_graph *parse_commit_graph(struct repo_settings *s,\n \t}\n \n \tif (s->commit_graph_changed_paths_version) {\n+\t\tstruct graph_read_bloom_data_context context = {\n+\t\t\t.g = graph,\n+\t\t\t.commit_graph_changed_paths_version = &s->commit_graph_changed_paths_version\n+\t\t};\n \t\tpair_chunk(cf, GRAPH_CHUNKID_BLOOMINDEXES,\n \t\t\t   &graph->chunk_bloom_indexes);\n \t\tread_chunk(cf, GRAPH_CHUNKID_BLOOMDATA,\n-\t\t\t   graph_read_bloom_data, graph);\n+\t\t\t   graph_read_bloom_data, &context);\n \t}\n \n \tif (graph->chunk_bloom_indexes && graph->chunk_bloom_data) {\n@@ -2371,6 +2384,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->write_generation_data = (get_configured_generation_version(r) == 2);\n \tctx->num_generation_data_overflows = 0;\n \n+\tif (r->settings.commit_graph_changed_paths_version < -1\n+\t    || r->settings.commit_graph_changed_paths_version > 2) {\n+\t\twarning(_(\"attempting to write a commit-graph, but 'commitgraph.changedPathsVersion' (%d) is not supported\"),\n+\t\t\tr->settings.commit_graph_changed_paths_version);\n+\t\treturn 0;\n+\t}\n+\tbloom_settings.hash_version = r->settings.commit_graph_changed_paths_version == 2\n+\t\t? 2 : 1;\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n \t\t\t\t\t\t      bloom_settings.bits_per_entry);\n \tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n@@ -2400,7 +2421,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\tg = ctx->r->objects->commit_graph;\n \n \t\t/* We have changed-paths already. Keep them in the next graph */\n-\t\tif (g && g->chunk_bloom_data) {\n+\t\tif (g && g->bloom_filter_settings) {\n \t\t\tctx->changed_paths = 1;\n \t\t\tctx->bloom_settings = g->bloom_filter_settings;\n \t\t}\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex aabe31d724..3cbc0a5b50 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -50,6 +50,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \n static const char *bloom_usage = \"\\n\"\n \"  test-tool bloom get_murmur3 <string>\\n\"\n+\"  test-tool bloom get_murmur3_seven_highbit\\n\"\n \"  test-tool bloom generate_filter <string> [<string>...]\\n\"\n \"  test-tool bloom get_filter_for_commit <commit-hex>\\n\";\n \n@@ -64,7 +65,13 @@ int cmd__bloom(int argc, const char **argv)\n \t\tuint32_t hashed;\n \t\tif (argc < 3)\n \t\t\tusage(bloom_usage);\n-\t\thashed = murmur3_seeded(0, argv[2], strlen(argv[2]));\n+\t\thashed = murmur3_seeded_v2(0, argv[2], strlen(argv[2]));\n+\t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n+\t}\n+\n+\tif (!strcmp(argv[1], \"get_murmur3_seven_highbit\")) {\n+\t\tuint32_t hashed;\n+\t\thashed = murmur3_seeded_v2(0, \"\\x99\\xaa\\xbb\\xcc\\xdd\\xee\\xff\", 7);\n \t\tprintf(\"Murmur3 Hash with seed=0:0x%08x\\n\", hashed);\n \t}\n \ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex b567383eb8..c8d84ab606 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -29,6 +29,14 @@ test_expect_success 'compute unseeded murmur3 hash for test string 2' '\n \ttest_cmp expect actual\n '\n \n+test_expect_success 'compute unseeded murmur3 hash for test string 3' '\n+\tcat >expect <<-\\EOF &&\n+\tMurmur3 Hash with seed=0:0xa183ccfd\n+\tEOF\n+\ttest-tool bloom get_murmur3_seven_highbit >actual &&\n+\ttest_cmp expect actual\n+'\n+\n test_expect_success 'compute bloom key for empty string' '\n \tcat >expect <<-\\EOF &&\n \tHashes:0x5615800c|0x5b966560|0x61174ab4|0x66983008|0x6c19155c|0x7199fab0|0x771ae004|\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 2d4a3fefee..775e59d864 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -447,4 +447,100 @@ test_expect_success 'version 1 changed-path used when version 1 requested' '\n \t)\n '\n \n+test_expect_success 'version 1 changed-path not used when version 2 requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 2 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 1 changed-path used when autodetect requested' '\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 1 of changed-path' '\n+\ttest_commit -C highbit1 c1double \"$CENT$CENT\" &&\n+\tgit -C highbit1 commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd highbit1 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(1,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'set up repo with high bit path, version 2 changed-path' '\n+\tgit init highbit2 &&\n+\tgit -C highbit2 config --add commitgraph.changedPathsVersion 2 &&\n+\ttest_commit -C highbit2 c2 \"$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths\n+'\n+\n+test_expect_success 'check value of version 2 changed-path' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\techo \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path used when version 2 requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path not used when version 1 requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion 1 &&\n+\t\ttest_bloom_filters_not_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'version 2 changed-path used when autodetect requested' '\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\ttest_bloom_filters_used \"-- $CENT\"\n+\t)\n+'\n+\n+test_expect_success 'when writing another commit graph, preserve existing version 2 of changed-path' '\n+\ttest_commit -C highbit2 c2double \"$CENT$CENT\" &&\n+\tgit -C highbit2 commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd highbit2 &&\n+\t\tgit config --add commitgraph.changedPathsVersion -1 &&\n+\t\techo \"options: bloom(2,10,7) read_generation_data\" >expect &&\n+\t\ttest-tool read-graph >full &&\n+\t\tgrep options full >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n+test_expect_success 'when writing commit graph, do not reuse changed-path of another version' '\n+\tgit init doublewrite &&\n+\ttest_commit -C doublewrite c \"$CENT\" &&\n+\tgit -C doublewrite config --add commitgraph.changedPathsVersion 1 &&\n+\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\tgit -C doublewrite config --add commitgraph.changedPathsVersion 2 &&\n+\tgit -C doublewrite commit-graph write --reachable --changed-paths &&\n+\t(\n+\t\tcd doublewrite &&\n+\t\techo \"c01f\" >expect &&\n+\t\tget_first_changed_path_filter >actual &&\n+\t\ttest_cmp expect actual\n+\t)\n+'\n+\n test_done\n-- \n2.41.0.585.gd2178a4bd4-goog\n\n"},{"id":"480075","messageId":"xmqqbkfqbo21.fsf@gitster.g","threadId":"59781","inReplyTo":"cover.1690912539.git.jonathantanmy@google.com","subject":"Re: [PATCH v7 0/7] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-01T18:44:06Z","receivedAt":"2023-08-01T18:44:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> Taylor also suggested copying forward Bloom filters whenever possible\n> in this patch set [3], but also that we could work on this outside this\n> series [4]. I did not implement this in this series.\n\nI think it is a good place to stop, as it would merely be a quality\nof implementation difference and would not change the transition\nstory very much.\n\nThanks for working well together.  Will queue.\n"},{"id":"480078","messageId":"20230801185232.1457172-1-jonathantanmy@google.com","threadId":"59781","inReplyTo":"ZMlKMmAs3wKULAOd@nand.local","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2023-08-01T18:52:32Z","receivedAt":"2023-08-01T18:52:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Taylor Blau <me@ttaylorr.com> writes:\n> On Thu, Jul 27, 2023 at 01:53:08PM -0700, Jonathan Tan wrote:\n> > Suddenly reading many (or most) of the repo's trees would be a similar\n> > surprise, right?\n> \n> That's a good point. I think in general I'd expect Git to avoid\n> recomputing Bloom filters where that work can be avoided, if the work in\n> order to detect whether or not we need to recompute a filter is cheap\n> enough to carry out.\n\nMakes sense. I just don't think that there is a cheap way to detect if\na filter does not need to be recomputed (the closest way I think we have\nis something that will require reading a lot of trees in a repo).\n\n> > Also this would happen only if the server operator explicitly sets a\n> > changed path filter version. If they leave it as-is, commit graphs will\n> > still be written with the same version as the one on disk.\n> \n> I think that I could live with that if the default is to leave things\n> as-is.\n\nAh, thanks.\n \n> I still think that it's worth it to have this functionality to propagate\n> Bloom filters forward should ship in Git, but we can work on that\n> outside of this series.\n\nMakes sense.\n\n> > Regarding consulting commit_graph->bloom_filter_settings->hash_version,\n> > the issue I ran into was that firstly, I didn't know what to do about\n> > commit_graph->base_graph (which also has its own bloom_filter_settings)\n> > and what to do if it had a contradictory hash_version. And even if\n> > we found a way to unify those, it is not true that every Bloom filter\n> > in memory is of that version, since we may have generated some that\n> > correspond to the version we're writing (not the version on disk).\n> > In particular, the Bloom filters we write come from a commit slab\n> > (bloom_filters in bloom.c) and in that slab, both Bloom filters from\n> > disk and Bloom filters that were generated in-process coexist.\n> \n> Would we ever want to have a commit-graph chain with mixed Bloom filter\n> versions?\n\nProbably not, but I wanted to be robust in case a third-party tool wrote a chain with\nmixed versions.\n\n> We avoid mixing generation number schemes across multiple layers of a\n> commit-graph chain. But I don't see any reason that we should or need to\n> have a similar restriction in place for the Bloom filter version. Both\n> are readable, as long as the user-provided configuration allows them to\n> be.\n\nYes, that's true - there is no inherent reason why we can't mix them,\nunlike with generation numbers.\n\n> We just have to interpret them differently depending on what layer of\n> the graph (and therefore, what Bloom filter version they are) they come\n> from.\n> \n> Sorry for thinking aloud a little there. I think that this means that we\n> at minimum have to keep in context the commit-graph layer we found the\n> Bloom filter in so that we can tie that back to its Bloom filter\n> version. That might just mean that we have to tag each Bloom filter with\n> the version it was computed under, or maybe we already have the\n> commit-graph layer in context, in which case we shouldn't need an\n> additional field.\n> \n> My gut is telling me that we probably *do* need such a field, since we\n> don't often have a reference to the particular layer that we found a\n> Bloom filter in, just the tip of the commit-graph chain that it came\n> from.\n\nWe'll need the additional field because we don't know which commit graph\nlayer it comes from. In fact, we don't even know which *repo* the commit\ncomes from, since the commit slab is global. (Moving the slab to being\nunder a repo or under a commit graph layer would fix this.)\n\nBut I think there still remains the question of whether we really need\nto support multiple versions in one Git invocation.\n\n> > I also thought of your other proposal of augmenting struct bloom_filter\n> > to also include the version. The issue I ran into there is if, for a\n> > given commit, there already exists a Bloom filter read from disk with\n> > the wrong version, what should we do? Overwrite it, or store both\n> > versions in memory? (We can't just immediately output the Bloom filter\n> > to disk and forget about the new version, only storing its size so that\n> > we can generate the BIDX, because in the current code, generation and\n> > writing to disk are separate. We could try to refactor it, but I didn't\n> > want to make such a large change to reduce the possibility of bugs.)\n> > Both storing the version number and storing an additional pointer for a\n> > second version would increase memory consumption too, even when\n> > supporting two versions isn't needed, but maybe this isn't a big deal.\n> \n> It's likely that I'm missing something here, but what is stopping us\n> from discarding the old Bloom filter as soon as we generate the new\n> one? We shouldn't need to load the old filter again out of the commit\n> slab, right?\n> \n> Thanks,\n> Taylor\n\nI did not look at the code closely enough except to see that there was\na gap between generating the new-version Bloom filters and writing them\nto disk, and I was concerned that now, or in the future, there would\nbe some code in that gap that reads the Bloom filter for a commit and\nexpects the old-version Bloom filter there.\n"},{"id":"480088","messageId":"ZMlyEpRCPPNG7jQP@nand.local","threadId":"59781","inReplyTo":"xmqqbkfqbo21.fsf@gitster.g","subject":"Re: [PATCH v7 0/7] Changed path filter hash fix and version bump","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-01T20:58:58Z","receivedAt":"2023-08-01T20:59:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 01, 2023 at 11:44:06AM -0700, Junio C Hamano wrote:\n> Jonathan Tan <jonathantanmy@google.com> writes:\n>\n> > Taylor also suggested copying forward Bloom filters whenever possible\n> > in this patch set [3], but also that we could work on this outside this\n> > series [4]. I did not implement this in this series.\n>\n> I think it is a good place to stop, as it would merely be a quality\n> of implementation difference and would not change the transition\n> story very much.\n>\n> Thanks for working well together.  Will queue.\n\nThanks. I read through this version and feel good about the results.\nI agree that queuing this one down makes sense.\n\nThanks,\nTaylor\n"},{"id":"480089","messageId":"xmqqpm46a2v2.fsf@gitster.g","threadId":"59781","inReplyTo":"ZMlyEpRCPPNG7jQP@nand.local","subject":"Re: [PATCH v7 0/7] Changed path filter hash fix and version bump","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-08-01T21:07:13Z","receivedAt":"2023-08-01T21:07:18Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau <me@ttaylorr.com> writes:\n\n> On Tue, Aug 01, 2023 at 11:44:06AM -0700, Junio C Hamano wrote:\n>> Jonathan Tan <jonathantanmy@google.com> writes:\n>>\n>> > Taylor also suggested copying forward Bloom filters whenever possible\n>> > in this patch set [3], but also that we could work on this outside this\n>> > series [4]. I did not implement this in this series.\n>>\n>> I think it is a good place to stop, as it would merely be a quality\n>> of implementation difference and would not change the transition\n>> story very much.\n>>\n>> Thanks for working well together.  Will queue.\n>\n> Thanks. I read through this version and feel good about the results.\n> I agree that queuing this one down makes sense.\n\nThanks.\n\nps. no more comms from me for the rest of the day as I am feeling\nill.  I've pushed out today's integration result already, merging\nthree or so topics down in 'next' and also absorbing a few new\ntopics in 'seen'.\n\n"},{"id":"480109","messageId":"ZMruSSAGQWS8crvi@nand.local","threadId":"59781","inReplyTo":"ZMlKMmAs3wKULAOd@nand.local","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-03T00:01:13Z","receivedAt":"2023-08-03T00:01:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 01, 2023 at 02:08:50PM -0400, Taylor Blau wrote:\n> On Thu, Jul 27, 2023 at 01:53:08PM -0700, Jonathan Tan wrote:\n> > Taylor Blau <me@ttaylorr.com> writes:\n> > > > The intention in the current patch set was to not load it at all when we\n> > > > have incompatible Bloom settings because it appeared quite troublesome\n> > > > to notate which Bloom filter in memory is of which version. If we want\n> > > > to copy forward existing results, we can change that, but I don't know\n> > > > whether it's worth doing that (and if we think it's worth doing, this\n> > > > should probably go in another patch set).\n> > >\n> > > Yeah, I think having Bloom filters accessible from a commit-graph\n> > > regardless of whether or not it matches our Bloom filter version is\n> > > prerequisite to being able to implement something like this.\n> > >\n> > > I feel like this is important enough to do in the same patch set, or the\n> > > same release to avoid surprising operators when their commit-graph write\n> > > suddenly recomputes all of its Bloom filters.\n> >\n> > Suddenly reading many (or most) of the repo's trees would be a similar\n> > surprise, right?\n>\n> That's a good point. I think in general I'd expect Git to avoid\n> recomputing Bloom filters where that work can be avoided, if the work in\n> order to detect whether or not we need to recompute a filter is cheap\n> enough to carry out.\n\nI spent some time implementing this (patches are available in the branch\n'tb/path-filter-fix-upgrade' from my fork). Handling incompatible Bloom\nfilter versions is kind of tricky, but do-able without having to\nimplement too much on top of what's already there.\n\nBut I don't think that it's enough to say that we can reuse a commit's\nBloom filter iff that commit's tree has no paths with characters >=\n0x80. Suppose we have such a tree, whose Bloom filter we believe to be\nreusable. If its first parent *does* have such a path, then that path\nwould appear as a deletion relative to its first parent. So that path\n*would* be in the filter, meaning that it isn't reusable.\n\nSo I think the revised condition is something like: a commit's Bloom\nfilter is reusable when there are no paths with characters >= 0x80 in\na tree-diff against its first parent. I think that ensuring that there\nare no such paths in both a commit's root tree, as well as its first\nparent's root tree is equivalent, since that removes the possibility of\nsuch a path showing up in its tree-diff.\n\nAs long as we aren't generating a commit-graph with --stdin-packs, then\nwe process commits in generation order, so we will see parents before\ntheir children. I think we could reuse existing filters in that case,\nbut the condition is slightly more complex than I originally thought.\n\nThanks,\nTaylor\n"},{"id":"480122","messageId":"491c237b-de23-52b0-c034-d4d358372864@github.com","threadId":"59781","inReplyTo":"ZMruSSAGQWS8crvi@nand.local","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Derrick Stolee","fromEmail":"derrickstolee@github.com","sentAt":"2023-08-03T13:18:11Z","receivedAt":"2023-08-03T13:18:32Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/2/2023 8:01 PM, Taylor Blau wrote:\n> On Tue, Aug 01, 2023 at 02:08:50PM -0400, Taylor Blau wrote:\n\n>> That's a good point. I think in general I'd expect Git to avoid\n>> recomputing Bloom filters where that work can be avoided, if the work in\n>> order to detect whether or not we need to recompute a filter is cheap\n>> enough to carry out.\n> \n> I spent some time implementing this (patches are available in the branch\n> 'tb/path-filter-fix-upgrade' from my fork). Handling incompatible Bloom\n> filter versions is kind of tricky, but do-able without having to\n> implement too much on top of what's already there.\n> \n> But I don't think that it's enough to say that we can reuse a commit's\n> Bloom filter iff that commit's tree has no paths with characters >=\n> 0x80. Suppose we have such a tree, whose Bloom filter we believe to be\n> reusable. If its first parent *does* have such a path, then that path\n> would appear as a deletion relative to its first parent. So that path\n> *would* be in the filter, meaning that it isn't reusable.\n> \n> So I think the revised condition is something like: a commit's Bloom\n> filter is reusable when there are no paths with characters >= 0x80 in\n> a tree-diff against its first parent. I think that ensuring that there\n> are no such paths in both a commit's root tree, as well as its first\n> parent's root tree is equivalent, since that removes the possibility of\n> such a path showing up in its tree-diff.\n\nThis condition is exactly \"we computed the diff to know which paths were\ninput to the filter\" which is as difficult as recomputing the Bloom filter\nfrom scratch. I don't think there is much room to gain a performance\nimprovement here.\n \nThanks,\n-Stolee\n"},{"id":"480130","messageId":"ZMv14grkM7x7Sf8m@nand.local","threadId":"59781","inReplyTo":"491c237b-de23-52b0-c034-d4d358372864@github.com","subject":"Re: [PATCH v6 0/7] Changed path filter hash fix and version bump","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2023-08-03T18:45:54Z","receivedAt":"2023-08-03T18:47:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Aug 03, 2023 at 09:18:11AM -0400, Derrick Stolee wrote:\n> > So I think the revised condition is something like: a commit's Bloom\n> > filter is reusable when there are no paths with characters >= 0x80 in\n> > a tree-diff against its first parent. I think that ensuring that there\n> > are no such paths in both a commit's root tree, as well as its first\n> > parent's root tree is equivalent, since that removes the possibility of\n> > such a path showing up in its tree-diff.\n>\n> This condition is exactly \"we computed the diff to know which paths were\n> input to the filter\" which is as difficult as recomputing the Bloom filter\n> from scratch. I don't think there is much room to gain a performance\n> improvement here.\n\nI think that's true in the worst case, and certainly for repositories\nwith many tree entries which have characters >= 0x80.\n\nBut note that it's a heuristic in one direction only. If we know that a\ncommit's root tree (and that of it's first parent, if it has one) is\nfree of any such paths, then it's impossible for the first parent\ntree-diff to contain such an entry, and therefore we can reuse any\nexisting filter.\n\nOf course, a commit's root tree (and its parent) may both have a path\nwhose characters >= 0x80 while still not seeing a corresponding entry\nshow up in the tree-diff if that path is unchanged between a commit and\nits first parent.\n\nI think if we were looking at every tree every time only to realize that\nwe have to go back and compute its changed-path Bloom filter, this would\nbe a non-starter. But since we \"cache\" the results of our walk via the\ntree object's flags bits, we can skip looking at many trees.\n\nIn my testing, this showed a significant improvement on linux.git and\ngit.git. My setup for testing is something like:\n\n    $ git clone git@github.com:torvalds/linux.git\n    $ cd linux\n    $ git commit-graph write --reachable --changed-paths\n    $ graph=\".git/objects/info/commit-graph\"\n    $ mv $graph{,.bak}\n\nso that .git/objects/info/commit-graph.bak is a commit-graph with v1\nchanged-path Bloom filters for all commits in generation order.\n\nWith that in place, I can do:\n\n    $ git config commitGraph.changedPathsVersion 2\n    $ hyperfine -p 'cp -f $graph.bak $graph' -L v 0,1 \\\n        'GIT_TEST_UPGRADE_BLOOM_FILTERS={v} git.compile commit-graph write --reachable --changed-paths'\n\n, producing the following results on linux.git:\n\n    Benchmark 1: GIT_TEST_UPGRADE_BLOOM_FILTERS=0 git.compile commit-graph write --reachable --changed-paths\n      Time (mean ± σ):     124.873 s ±  0.316 s    [User: 124.081 s, System: 0.643 s]\n      Range (min … max):   124.621 s … 125.227 s    3 runs\n\n    Benchmark 2: GIT_TEST_UPGRADE_BLOOM_FILTERS=1 git.compile commit-graph write --reachable --changed-paths\n      Time (mean ± σ):     79.271 s ±  0.163 s    [User: 74.611 s, System: 4.521 s]\n      Range (min … max):   79.112 s … 79.437 s    3 runs\n\n    Summary\n      'GIT_TEST_UPGRADE_BLOOM_FILTERS=1 git.compile commit-graph write --reachable --changed-paths' ran\n        1.58 ± 0.01 times faster than 'GIT_TEST_UPGRADE_BLOOM_FILTERS=0 git.compile commit-graph write --reachable --changed-paths'\n\nOn git.git, writing a new commit-graph after upgrading from v1 to v2\nfilters went from taking 4.163 seconds to 3.348 seconds, for a more\nmodest 1.24x speed-up.\n\nOf course, all of this depends on how much of the tree meets the above\nconditions, so we'd expect worse results on repositories with paths that\ncontain characters >= 0x80. I think we'd want some kind of mechanism\n(probably via config, not a GIT_TEST environment variable) to control\nwhether or not to upgrade existing filters.\n\nThanks,\nTaylor\n"}]}