{"thread":{"id":"53681","subject":"[PATCH 0/8] More commit-graph/Bloom filter improvements","startedAt":"2020-06-15T20:15:00Z","lastAt":"2020-10-16T13:53:03Z","messageCount":76,"participants":["Derrick Stolee via GitGitGadget","SZEDER Gábor via GitGitGadget","Junio C Hamano","Derrick Stolee","René Scharfe","Taylor Blau","Taylor Blau via GitGitGadget","SZEDER Gábor"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"399809","messageId":"pull.659.git.1592252093.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":null,"subject":"[PATCH 0/8] More commit-graph/Bloom filter improvements","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-15T20:14:45Z","receivedAt":"2020-06-15T20:15:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This builds on sg/commit-graph-cleanups, which took several patches from\nSzeder's series [1] and applied them almost directly to a more-recent\nversion of Git [2].\n\n[1] https://lore.kernel.org/git/20200529085038.26008-1-szeder.dev@gmail.com/\n[2] \nhttps://lore.kernel.org/git/pull.650.git.1591362032.gitgitgadget@gmail.com/\n\nThis series adds a few extra improvements, several of which are rooted in\nSzeder's original series. I maintained his authorship and sign-off, even\nthough the patches did not apply or cherry-pick at all.\n\n 1. commit-graph: place bloom_settings in context\n 2. commit-graph: unify the signatures of all write_graph_chunk_*()\n functions\n 3. commit-graph: simplify chunk writes into loop\n 4. commit-graph: check chunk sizes after writing\n 5. commit-graph: check all leading directories in changed path Bloom\n filters\n\nPatch 1 is a new preparation patch to then apply Szeder's ideas in the next\nfour. Some are refactoring or defensive programming, but Patch 5 presents a\nmeaningful performance improvement. By creating bloom_keys for each leading\ndirectory in a path, we can greatly improve the false-positive rate.\n\n 6. bloom: enforce a minimum size of 8 bytes\n\nPatch 6 is based on a comment of Szeder's that since we are using 1-byte\nalignment in the filters, that some small filters do not fit the theoretical\nanalysis that calculated the expected false-positive rate. By increasing the\nminimum (non-zero) filter size, we can gain significant performance benefits\nwhile increasing the file size a small amount.\n\n 7. commit-graph: change test to die on parse, not load\n 8. commit-graph: persist existence of changed-paths\n\nThe final two patches handle the unresolved usability issue: if a user\nwrites a commit-graph with --changed-paths, the next write will probably\nclear them out. Think about gc.writeCommitGraph or fetch.writeCommitGraph,\nwhich do not allow for the --changed-paths option directly. Another idea is\nto add a config option, but I will leave that to others [3].\n\n[3] https://github.com/gitgitgadget/git/pull/633\n\nHere is an analysis of the range-diff between this series and Szeder's PoC\nsubmission.\n\nThese patches either are part of sg/commit-graph-cleanups or were discarded\nas unnecessary.\n\n 1: 7a8dbfba53a < -: ----------- tree-walk.c: don't match submodule entries for 'submod/anything'\n 2: df25e984c58 < -: ----------- commit-graph: fix parsing the Chunk Lookup table\n 3: 598f7f9a978 < -: ----------- commit-graph-format.txt: all multi-byte numbers are in network byte order\n 4: b29e5d39ed6 < -: ----------- commit-slab: add a function to deep free entries on the slab\n 5: 18f4db7bfb9 < -: ----------- diff.h: drop diff_tree_oid() & friends' return value\n 6: bf336f109e6 < -: ----------- commit-graph: clean up #includes\n 7: b7f0f831bcf < -: ----------- commit-graph: simplify parse_commit_graph() #1\n 8: f2752000052 < -: ----------- commit-graph: simplify parse_commit_graph() #2\n 9: 4e184b8743c < -: ----------- commit-graph: simplify write_commit_graph_file() #1\n10: 344dd337da5 < -: ----------- commit-graph: simplify write_commit_graph_file() #2\n11: 56e3c4f57b3 < -: ----------- commit-graph: allocate the 'struct chunk_info' array dinamically\n\nThis first patch enables the next refactoring patch.\n\n -: ----------- > 1: c966969071b commit-graph: place bloom_settings in context\n\nThis patch is recognized as similar, but all differences are due to\nwhitespace corrections and the new write_graph_chunk_*() methods.\n\n12: 28fb1b5bdfe ! 2: 65eb15221c8 commit-graph: unify the signatures of all write_graph_chunk_*() functions\n @@ Commit message\n This opens up the possibility for further cleanups and foolproofing in\n the following two patches.\n\n Signed-off-by: SZEDER Gábor \n + Signed-off-by: Derrick Stolee \n\n ## commit-graph.c ##\n @@ commit-graph.c: struct write_commit_graph_context {\n - const struct split_commit_graph_opts *split_opts;\n + struct bloom_filter_settings bloom_settings;\n };\n\n -static void write_graph_chunk_fanout(struct hashfile *f,\n +- struct write_commit_graph_context *ctx)\n +static int write_graph_chunk_fanout(struct hashfile *f,\n - struct write_commit_graph_context *ctx)\n ++ struct write_commit_graph_context *ctx)\n {\n int i, count = 0;\n + struct commit **list = ctx->commits.list;\n @@ commit-graph.c: static void write_graph_chunk_fanout(struct hashfile *f,\n\n hashwrite_be32(f, count);\n }\n ++\n + return 0;\n }\n\n -static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n +- struct write_commit_graph_context *ctx)\n +static int write_graph_chunk_oids(struct hashfile *f,\n - struct write_commit_graph_context *ctx)\n ++ struct write_commit_graph_context *ctx)\n {\n struct commit **list = ctx->commits.list;\n int count;\n for (count = 0; count < ctx->commits.nr; count++, list++) {\n display_progress(ctx->progress, ++ctx->progress_cnt);\n - hashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n -+ hashwrite(f, (*list)->object.oid.hash, the_hash_algo->rawsz);\n ++ hashwrite(f, (*list)->object.oid.hash, (int)the_hash_algo->rawsz);\n }\n ++\n + return 0;\n }\n\n @@ commit-graph.c: static const unsigned char *commit_to_sha1(size_t index, void *t\n }\n\n -static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n +- struct write_commit_graph_context *ctx)\n +static int write_graph_chunk_data(struct hashfile *f,\n - struct write_commit_graph_context *ctx)\n ++ struct write_commit_graph_context *ctx)\n {\n struct commit **list = ctx->commits.list;\n + struct commit **last = ctx->commits.list + ctx->commits.nr;\n @@ commit-graph.c: static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n die(_(\"unable to parse commit %s\"),\n oid_to_hex(&(*list)->object.oid));\n @@ commit-graph.c: static void write_graph_chunk_data(struct hashfile *f, int hash_\n\n list++;\n }\n ++\n + return 0;\n }\n\n -static void write_graph_chunk_extra_edges(struct hashfile *f,\n +- struct write_commit_graph_context *ctx)\n +static int write_graph_chunk_extra_edges(struct hashfile *f,\n - struct write_commit_graph_context *ctx)\n ++ struct write_commit_graph_context *ctx)\n {\n struct commit **list = ctx->commits.list;\n + struct commit **last = ctx->commits.list + ctx->commits.nr;\n @@ commit-graph.c: static void write_graph_chunk_extra_edges(struct hashfile *f,\n\n list++;\n }\n ++\n ++ return 0;\n + }\n + \n +-static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n +- struct write_commit_graph_context *ctx)\n ++static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n ++ struct write_commit_graph_context *ctx)\n + {\n + struct commit **list = ctx->commits.list;\n + struct commit **last = ctx->commits.list + ctx->commits.nr;\n +@@ commit-graph.c: static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n + }\n + \n + stop_progress(&progress);\n ++ return 0;\n + }\n + \n +-static void write_graph_chunk_bloom_data(struct hashfile *f,\n +- struct write_commit_graph_context *ctx)\n ++static int write_graph_chunk_bloom_data(struct hashfile *f,\n ++ struct write_commit_graph_context *ctx)\n + {\n + struct commit **list = ctx->commits.list;\n + struct commit **last = ctx->commits.list + ctx->commits.nr;\n +@@ commit-graph.c: static void write_graph_chunk_bloom_data(struct hashfile *f,\n + }\n + \n + stop_progress(&progress);\n + return 0;\n }\n\n static int oid_compare(const void *_a, const void *_b)\n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n - chunks_nr * ctx->commits.nr);\n + num_chunks * ctx->commits.nr);\n }\n write_graph_chunk_fanout(f, ctx);\n - write_graph_chunk_oids(f, hashsz, ctx);\n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con\n + write_graph_chunk_data(f, ctx);\n if (ctx->num_extra_edges)\n write_graph_chunk_extra_edges(f, ctx);\n - if (ctx->num_commit_graphs_after > 1 &&\n + if (ctx->changed_paths) {\n\nThese patches follow the same intent, but are significantly different\nbecause they are updated with split commit-graphs and the existing\nchanged-path Bloom filters.\n\n13: 1e1e59e2592 < -: ----------- commit-graph: simplify write_commit_graph_file() #3\n -: ----------- > 3: 3d24b9802df commit-graph: simplify chunk writes into loop\n14: 6f0d912e4b8 < -: ----------- commit-graph: check chunk sizes after writing\n -: ----------- > 4: bdca834e6da commit-graph: check chunk sizes after writing\n24: dc96f0d9822 < -: ----------- commit-graph: check all leading directories in modified path Bloom filters\n -: ----------- > 5: 9975fc96f12 commit-graph: check all leading directories in changed path Bloom filters\n\nThese three patches are a few valuable improvements of my own design:\n\n -: ----------- > 6: 2a5f1e17528 bloom: enforce a minimum size of 8 bytes\n -: ----------- > 7: 60bbc15d24a commit-graph: change test to die on parse, not load\n -: ----------- > 8: db5b8fe8439 commit-graph: persist existence of changed-paths\n\nAt this point, we have updated the existing changed-path Bloom filter\nimplementation to be on even terms with Szeder's modified-path Bloom filter\nimplementation.\n\nThe next batch of patches contain Szeder's implementation. These implement a\ncompletely different file format, so they are not intended as ways to move\nforward. If there is a significant improvement to be found by using this\nfile format instead of the established one (comparing the old implementation\nwith these patches), then we could consider swapping the optional chunks for\nthose that he proposes.\n\nWhile I had the motivation and energy to defend the current implementation\nby applying Szeder's (excellent) ideas to the existing format, I do not have\nintent to go through the effort to compare the file formats explicitly at\nthis point. I would be interested to read a performance analysis, if someone\nwere to provide one now.\n\n15: 0ab955aac32 < -: ----------- commit-graph-format.txt: document the modified path Bloom filter chunks\n16: 4c128d51dfe < -: ----------- Add a generic and minimal Bloom filter implementation\n17: 41f02bc38f7 < -: ----------- Import a streaming-capable Murmur3 hash function implementation\n18: e5fd1da48d4 < -: ----------- commit-graph: write \"empty\" Modified Path Bloom Filter Index chunk\n19: 2dd882ec601 < -: ----------- commit-graph: add commit slab for modified path Bloom filters\n20: f30e495c2b0 < -: ----------- commit-graph: fill the Modified Path Bloom Filter Index chunk\n21: e904cb58301 < -: ----------- commit-graph: load and use the Modified Path Bloom Filter Index chunk\n22: c71647ca374 < -: ----------- commit-graph: write the Modified Path Bloom Filters chunk\n23: 50898d42291 < -: ----------- commit-graph: load and use the Modified Path Bloom Filters chunk\n25: 7cbf1bc6b66 < -: ----------- commit-graph: check embedded modified path Bloom filters with a mask\n26: 3951fdedf6a < -: ----------- commit-graph: deduplicate modified path Bloom filters\n27: 5aba19a2766 < -: ----------- commit-graph: load modified path Bloom filters for merge commits\n28: 93fc6af1d2f < -: ----------- commit-graph: write Modified Path Bloom Filter Merge Index chunk\n29: f87b37bf08e < -: ----------- commit-graph: extract init and free write_commit_graph_context\n30: 943b0d9554c < -: ----------- commit-graph: move write_commit_graph_reachable below write_commit_graph\n31: 47b26ea61aa < -: ----------- t7007-show: make the first test compatible with the next patch\n32: 9201b71071c < -: ----------- PoC commit-graph: use revision walk machinery for '--reachable'\n33: 5c72d97e5e9 < -: ----------- commit-graph: write modified path Bloom filters in \"history order\"\n\nThis patch is likely worth investigating again:\n\n34: 8b40ec4cd30 < -: ----------- commit-graph: use modified path Bloom filters with wildcards, if possible\n\nThanks, -Stolee\n\nDerrick Stolee (4):\n commit-graph: place bloom_settings in context\n bloom: enforce a minimum size of 8 bytes\n commit-graph: change test to die on parse, not load\n commit-graph: persist existence of changed-paths\n\nSZEDER Gábor (4):\n commit-graph: unify the signatures of all write_graph_chunk_*()\n functions\n commit-graph: simplify chunk writes into loop\n commit-graph: check chunk sizes after writing\n commit-graph: check all leading directories in changed path Bloom\n filters\n\n Documentation/git-commit-graph.txt | 5 +-\n bloom.c | 4 ++\n builtin/commit-graph.c | 5 +-\n commit-graph.c | 112 ++++++++++++++++++++---------\n commit-graph.h | 3 +-\n revision.c | 35 ++++++---\n revision.h | 6 +-\n t/t4216-log-bloom.sh | 4 +-\n t/t5318-commit-graph.sh | 2 +-\n 9 files changed, 124 insertions(+), 52 deletions(-)\n\n\nbase-commit: 7fbfe07ab4d4e58c0971dac73001b89f180a0af3\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-659%2Fderrickstolee%2Fbloom-2-v1\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-659/derrickstolee/bloom-2-v1\nPull-Request: https://github.com/gitgitgadget/git/pull/659\n-- \ngitgitgadget\n"},{"id":"399812","messageId":"65eb15221c8d3b631d6d7105d0e901433bc7f23b.1592252093.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"[PATCH 2/8] commit-graph: unify the signatures of all write_graph_chunk_*() functions","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-15T20:14:47Z","receivedAt":"2020-06-15T20:15:02Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nUpdate the write_graph_chunk_*() helper functions to have the same\nsignature:\n\n - Return an int error code from all these functions.\n write_graph_chunk_base() already has an int error code, now the\n others will have one, too, but since they don't indicate any\n error, they will always return 0.\n\n - Drop the hash size parameter of write_graph_chunk_oids() and\n write_graph_chunk_data(); its value can be read directly from\n 'the_hash_algo' inside these functions as well.\n\nThis opens up the possibility for further cleanups and foolproofing in\nthe following two patches.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 42 ++++++++++++++++++++++++++----------------\n 1 file changed, 26 insertions(+), 16 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 05b7035d8d5..3bae1e52ed0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -885,8 +885,8 @@ struct write_commit_graph_context {\n \tstruct bloom_filter_settings bloom_settings;\n };\n \n-static void write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_fanout(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tint i, count = 0;\n \tstruct commit **list = ctx->commits.list;\n@@ -907,17 +907,21 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \n \t\thashwrite_be32(f, count);\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_oids(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tint count;\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n-\t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t\thashwrite(f, (*list)->object.oid.hash, (int)the_hash_algo->rawsz);\n \t}\n+\n+\treturn 0;\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -926,8 +930,8 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n \treturn commits[index]->object.oid.hash;\n }\n \n-static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_data(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -944,7 +948,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tdie(_(\"unable to parse commit %s\"),\n \t\t\t\toid_to_hex(&(*list)->object.oid));\n \t\ttree = get_commit_tree_oid(*list);\n-\t\thashwrite(f, tree->hash, hash_len);\n+\t\thashwrite(f, tree->hash, the_hash_algo->rawsz);\n \n \t\tparent = (*list)->parents;\n \n@@ -1024,10 +1028,12 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\tlist++;\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_extra_edges(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1076,10 +1082,12 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \n \t\tlist++;\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1101,10 +1109,11 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n \t}\n \n \tstop_progress(&progress);\n+\treturn 0;\n }\n \n-static void write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_bloom_data(struct hashfile *f,\n+\t\t\t\t\tstruct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1128,6 +1137,7 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t}\n \n \tstop_progress(&progress);\n+\treturn 0;\n }\n \n static int oid_compare(const void *_a, const void *_b)\n@@ -1638,8 +1648,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, hashsz, ctx);\n-\twrite_graph_chunk_data(f, hashsz, ctx);\n+\twrite_graph_chunk_oids(f, ctx);\n+\twrite_graph_chunk_data(f, ctx);\n \tif (ctx->num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n \tif (ctx->changed_paths) {\n-- \ngitgitgadget\n\n"},{"id":"399813","messageId":"c966969071bf0ba135d304bbc0d5c16fbcbedc1e.1592252093.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"[PATCH 1/8] commit-graph: place bloom_settings in context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-15T20:14:46Z","receivedAt":"2020-06-15T20:15:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPlace an instance of struct bloom_settings into the struct\nwrite_commit_graph_context. This allows simplifying the function\nprototype of write_graph_chunk_bloom_data(). This will allow us\nto combine the function prototypes and use function pointers to\nsimplify write_commit_graph_file().\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 887837e8826..05b7035d8d5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -882,6 +882,7 @@ struct write_commit_graph_context {\n \n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n+\tstruct bloom_filter_settings bloom_settings;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -1103,8 +1104,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n }\n \n static void write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx,\n-\t\t\t\t\t const struct bloom_filter_settings *settings)\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1116,9 +1116,9 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t\t\t_(\"Writing changed paths Bloom filters data\"),\n \t\t\tctx->commits.nr);\n \n-\thashwrite_be32(f, settings->hash_version);\n-\thashwrite_be32(f, settings->num_hashes);\n-\thashwrite_be32(f, settings->bits_per_entry);\n+\thashwrite_be32(f, ctx->bloom_settings.hash_version);\n+\thashwrite_be32(f, ctx->bloom_settings.num_hashes);\n+\thashwrite_be32(f, ctx->bloom_settings.bits_per_entry);\n \n \twhile (list < last) {\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n@@ -1541,6 +1541,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tstruct object_id file_hash;\n \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n+\tctx->bloom_settings = bloom_settings;\n+\n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n \n@@ -1642,7 +1644,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n \tif (ctx->changed_paths) {\n \t\twrite_graph_chunk_bloom_indexes(f, ctx);\n-\t\twrite_graph_chunk_bloom_data(f, ctx, &bloom_settings);\n+\t\twrite_graph_chunk_bloom_data(f, ctx);\n \t}\n \tif (ctx->num_commit_graphs_after > 1 &&\n \t write_graph_chunk_base(f, ctx)) {\n-- \ngitgitgadget\n\n"},{"id":"399811","messageId":"3d24b9802df3d1fe8c670ba602890c95df96f2f5.1592252093.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"[PATCH 3/8] commit-graph: simplify chunk writes into loop","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-15T20:14:48Z","receivedAt":"2020-06-15T20:15:03Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn write_commit_graph_file() we now have one block of code filling the\narray of 'struct chunk_info' with the IDs and sizes of chunks to be\nwritten, and an other block of code calling the functions responsible\nfor writing individual chunks. In case of optional chunks like Extra\nEdge List an Base Graphs List there is also a condition checking\nwhether that chunk is necessary/desired, and that same condition is\nrepeated in both blocks of code. Other, newer chunks have similar\noptional conditions.\n\nEliminate these repeated conditions by storing the function pointers\nresponsible for writing individual chunks in the 'struct chunk_info'\narray as well, and calling them in a loop to write the commit-graph\nfile. This will open up the possibility for a bit of foolproofing in\nthe following patch.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 31 +++++++++++++++++++------------\n 1 file changed, 19 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3bae1e52ed0..78e023be664 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1532,9 +1532,13 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n+typedef int (*chunk_write_fn)(struct hashfile *f,\n+\t\t\t struct write_commit_graph_context *ctx);\n+\n struct chunk_info {\n \tuint32_t id;\n \tuint64_t size;\n+\tchunk_write_fn write_fn;\n };\n \n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n@@ -1591,27 +1595,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n \tchunks[0].size = GRAPH_FANOUT_SIZE;\n+\tchunks[0].write_fn = write_graph_chunk_fanout;\n \tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunks[1].size = hashsz * ctx->commits.nr;\n+\tchunks[1].write_fn = write_graph_chunk_oids;\n \tchunks[2].id = GRAPH_CHUNKID_DATA;\n \tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n+\tchunks[2].write_fn = write_graph_chunk_data;\n \tif (ctx->num_extra_edges) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n \t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n \t\tnum_chunks++;\n \t}\n \tif (ctx->changed_paths) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n \t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n \t\tnum_chunks++;\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n \t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n \t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n \t\tnum_chunks++;\n \t}\n \n@@ -1647,19 +1658,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tprogress_title.buf,\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n-\twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, ctx);\n-\twrite_graph_chunk_data(f, ctx);\n-\tif (ctx->num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, ctx);\n-\tif (ctx->changed_paths) {\n-\t\twrite_graph_chunk_bloom_indexes(f, ctx);\n-\t\twrite_graph_chunk_bloom_data(f, ctx);\n-\t}\n-\tif (ctx->num_commit_graphs_after > 1 &&\n-\t write_graph_chunk_base(f, ctx)) {\n-\t\treturn -1;\n+\n+\tfor (i = 0; i < num_chunks; i++) {\n+\t\tif (chunks[i].write_fn(f, ctx)) {\n+\t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n+\t\t\t chunks[i].id);\n+\t\t\treturn -1;\n+\t\t}\n \t}\n+\n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n-- \ngitgitgadget\n\n"},{"id":"399810","messageId":"2a5f1e1752869e54203d7db609b4510a75d8de74.1592252093.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"[PATCH 6/8] bloom: enforce a minimum size of 8 bytes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-15T20:14:51Z","receivedAt":"2020-06-15T20:15:04Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe original design of changed-path Bloom filters included an 8-byte\nblock size for filter lengths. This was changed mid-way through the\nsubmission process, and now the length stored in the commit-graph has\none-byte granularity.\n\nThis can cause some issues for very small filters. The analysis for\nfalse positive rates assume large filters, so rounding errors become\nless important at that scale. When there are only a few paths changed,\na filter that has size only a few bytes could have very different\nbehavior. In fact, this is evidenced in the Git repository due to the\ncode organization and careful patch creation that leads to many commits\nwith very small filters. These small filters frequently have\nfalse-positive rates in the 8-10% range or higher.\n\nThe previous change improved the false-positive rate using multiple\nBloom keys when the path has multiple directory components. However,\nthat does not help at all for files at root. It is typical to have\nseveral commits that change only the README at root, and those commits\nwould be likely to have these artificially high false-positive rates.\n\nCorrect this issue by creating a minimum filters size of 8 bytes. This\nrequires the very small commits (with fewer than six changes, including\nnon-root directories) to have a larger filter. In principle, this\nviolates the bits_per_entry value of struct bloom_filter_settings.\nHowever, it does not actually create a functional problem.\n\nAs for compatibility, this only affects new versions writing filters for\ncommits that do not yet have a filter. Old version will write the\nsmaller filters and this version will persist and properly read that\ndata. Now, the new files will be generated slightly larger.\n\n Bytes before Bytes after Difference\n --------------------------------------------------\n git 4,021,078 4,275,311 +6.32%\n linux 72,212,101 73,909,286 +2.35%\n tensorflow 7,596,359 7,691,646 +1.25%\n\nThis has a measurable improvement in the false-positive rate and the\nend-to-end run time for these repos. The table below compares the average\nfalse-positive rate and runtime of\n\n git rev-list HEAD -- \"$path\"\n\nbefore and after this change for 5000+ randomly* selected paths from\neach repository:\n\n Average false Average Average\n positive rate runtime runtime\n before after before after difference\n ------------------------------------------------------------------\n git 0.786% 0.227% 0.0387s 0.0289s -25.5%\n linux 0.0296% 0.0174% 0.0766s 0.0706s -7.8%\n tensorflow 0.6977% 0.0268% 0.0420s 0.0384s -8.5%\n\n*Path selection was done with the following pipeline:\n\n git ls-tree -r --name-only HEAD | sort -R | head -n 5000\n\nThese relatively-small increases in file size appear to be a fair price\nto pay for these performance improvements.\n\nSigned-off-by: Derrick Stolee \n---\n bloom.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/bloom.c b/bloom.c\nindex c38d1cff0c6..875e3853c2c 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -258,6 +258,10 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \t\t}\n \n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings.bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\n+\t\tif (filter->len && filter->len < 8)\n+\t\t\tfilter->len = 8;\n+\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n-- \ngitgitgadget\n\n"},{"id":"399814","messageId":"60bbc15d24a8de17757d1cb287f5bb2d77c41a72.1592252093.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"[PATCH 7/8] commit-graph: change test to die on parse, not load","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-15T20:14:52Z","receivedAt":"2020-06-15T20:15:05Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n43d3561 (commit-graph write: don't die if the existing graph is corrupt,\n2019-03-25) introduced the GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD environment\nvariable. This was created to verify that commit-graph was not loaded\nwhen writing a new non-incremental commit-graph.\n\nAn upcoming change wants to load a commit-graph in some valuable cases,\nbut we want to maintain that we don't trust the commit-graph data when\nwriting our new file. Instead of dying on load, instead die if we ever\ntry to parse a commit from the commit-graph. This functionally verifies\nthe same intended behavior, but allows a more advanced feature in the\nnext change.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 ++++++++----\n commit-graph.h | 2 +-\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 5c8f210cada..3a64e3b382d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -564,10 +564,6 @@ static int prepare_commit_graph(struct repository *r)\n \t\treturn !!r->objects->commit_graph;\n \tr->objects->commit_graph_attempted = 1;\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD, 0))\n-\t\tdie(\"dying as requested by the '%s' variable on commit-graph load!\",\n-\t\t GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD);\n-\n \tprepare_repo_settings(r);\n \n \tif (!git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n@@ -790,6 +786,14 @@ static int parse_commit_in_graph_one(struct repository *r,\n \n int parse_commit_in_graph(struct repository *r, struct commit *item)\n {\n+\tstatic int checked_env = 0;\n+\n+\tif (!checked_env &&\n+\t git_env_bool(GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE, 0))\n+\t\tdie(\"dying as requested by the '%s' variable on commit-graph parse!\",\n+\t\t GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE);\n+\tchecked_env = 1;\n+\n \tif (!prepare_commit_graph(r))\n \t\treturn 0;\n \treturn parse_commit_in_graph_one(r, r->objects->commit_graph, item);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 881c9b46e57..f0fb13e3f28 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -5,7 +5,7 @@\n #include \"object-store.h\"\n \n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n-#define GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD \"GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD\"\n+#define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n #define GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS \"GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS\"\n \n /*\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 1073f9e3cf2..5ec01abdaa9 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -436,7 +436,7 @@ corrupt_graph_verify() {\n \t\tcp $objdir/info/commit-graph commit-graph-pre-write-test\n \tfi &&\n \tgit status --short &&\n-\tGIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD=true git commit-graph write &&\n+\tGIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE=true git commit-graph write &&\n \tgit commit-graph verify\n }\n \n-- \ngitgitgadget\n\n"},{"id":"399817","messageId":"db5b8fe843951a13b791841772f480f839bd1db5.1592252093.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"[PATCH 8/8] commit-graph: persist existence of changed-paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-15T20:14:53Z","receivedAt":"2020-06-15T20:15:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe changed-path Bloom filters were released in v2.27.0, but have a\nsignificant drawback. A user can opt-in to writing the changed-path\nfilters using the \"--changed-paths\" option to \"git commit-graph write\"\nbut the next write will drop the filters unless that option is\nspecified.\n\nThis becomes even more important when considering the interaction with\ngc.writeCommitGraph (on by default) or fetch.writeCommitGraph (part of\nfeatures.experimental). These config options trigger commit-graph writes\nthat the user did not signal, and hence there is no --changed-paths\noption available.\n\nAllow a user that opts-in to the changed-path filters to persist the\nproperty of \"my commit-graph has changed-path filters\" automatically. A\nuser can drop filters using the --no-changed-paths option.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 5 ++++-\n builtin/commit-graph.c | 5 ++++-\n commit-graph.c | 12 +++++++++++-\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 2 +-\n 5 files changed, 21 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex f4b13c005b8..369b222b08b 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -60,7 +60,10 @@ existing commit-graph file.\n With the `--changed-paths` option, compute and write information about the\n paths changed between a commit and it's first parent. This operation can\n take a while on large repositories. It provides significant performance gains\n-for getting history of a directory or a file with `git log -- `.\n+for getting history of a directory or a file with `git log -- `. If\n+this option is given, future commit-graph writes will automatically assume\n+that this option was intended. Use `--no-changed-paths` to stop storing this\n+data.\n +\n With the `--split` option, write the commit-graph as a chain of multiple\n commit-graph files stored in `/info/commit-graphs`. The new commits\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 59009837dc9..ff7b177c337 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -151,6 +151,7 @@ static int graph_write(int argc, const char **argv)\n \t};\n \n \topts.progress = isatty(2);\n+\topts.enable_changed_paths = -1;\n \tsplit_opts.size_multiple = 2;\n \tsplit_opts.max_commits = 0;\n \tsplit_opts.expire_time = 0;\n@@ -171,7 +172,9 @@ static int graph_write(int argc, const char **argv)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n \tif (opts.progress)\n \t\tflags |= COMMIT_GRAPH_WRITE_PROGRESS;\n-\tif (opts.enable_changed_paths ||\n+\tif (!opts.enable_changed_paths)\n+\t\tflags |= COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS;\n+\tif (opts.enable_changed_paths == 1 ||\n \t git_env_bool(GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS, 0))\n \t\tflags |= COMMIT_GRAPH_WRITE_BLOOM_FILTERS;\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 3a64e3b382d..04eea725232 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1996,9 +1996,19 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n \tctx->check_oids = flags & COMMIT_GRAPH_WRITE_CHECK_OIDS ? 1 : 0;\n \tctx->split_opts = split_opts;\n-\tctx->changed_paths = flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS ? 1 : 0;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n+\t\tctx->changed_paths = 1;\n+\telse if (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\n+\t\tprepare_commit_graph_one(ctx->r, ctx->odb);\n+\n+\t\t/* We have changed-paths already. Keep them in the next graph */\n+\t\tif (ctx->r->objects->commit_graph &&\n+\t\t ctx->r->objects->commit_graph->chunk_bloom_data)\n+\t\t\tctx->changed_paths = 1;\n+\t}\n+\n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n \t\tprepare_commit_graph(ctx->r);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex f0fb13e3f28..45b1e5bca39 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -96,6 +96,7 @@ enum commit_graph_write_flags {\n \t/* Make sure that each OID in the input is a valid commit OID. */\n \tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3),\n \tCOMMIT_GRAPH_WRITE_BLOOM_FILTERS = (1 << 4),\n+\tCOMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS = (1 << 5),\n };\n \n struct split_commit_graph_opts {\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c13b97d3bda..30c8d9562e8 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -126,7 +126,7 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_commit c14 A/anotherFile2 &&\n \ttest_commit c15 A/B/anotherFile2 &&\n \ttest_commit c16 A/B/C/anotherFile2 &&\n-\tGIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0 git commit-graph write --reachable --split &&\n+\tgit commit-graph write --reachable --split --no-changed-paths &&\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n-- \ngitgitgadget\n"},{"id":"399815","messageId":"9975fc96f1269b049ebdd54835a63480c3dcbe4e.1592252093.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"[PATCH 5/8] commit-graph: check all leading directories in changed path Bloom filters","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-15T20:14:50Z","receivedAt":"2020-06-15T20:15:09Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nThe file 'dir/subdir/file' can only be modified if its leading\ndirectories 'dir' and 'dir/subdir' are modified as well.\n\nSo when checking modified path Bloom filters looking for commits\nmodifying a path with multiple path components, then check not only\nthe full path in the Bloom filters, but all its leading directories as\nwell. Take care to check these paths in \"deepest first\" order,\nbecause it's the full path that is least likely to be modified, and\nthe Bloom filter queries can short circuit sooner.\n\nThis can significantly reduce the average false positive rate, by\nabout an order of magnitude or three(!), and can further speed up\npathspec-limited revision walks. The table below compares the average\nfalse positive rate and runtime of\n\n git rev-list HEAD -- \"$path\"\n\nbefore and after this change for 5000+ randomly* selected paths from\neach repository:\n\n Average false Average Average\n positive rate runtime runtime\n before after before after difference\n ------------------------------------------------------------------\n git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n\n*Path selection was done with the following pipeline:\n\n\tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n\nThe improvements in runtime are much smaller than the improvements in\naverage false positive rate, as we are clearly reaching diminishing\nreturns here. However, all these timings depend on that accessing\ntree objects is reasonably fast (warm caches). If we had a partial\nclone and the tree objects had to be fetched from a promisor remote,\ne.g.:\n\n $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n commit-graph write --reachable\n $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n rev-list HEAD -- \"$path\"\n\nthen checking all leading path component can reduce the runtime from\nover an hour to a few seconds (and this is with the clone and the\npromisor on the same machine).\n\nThis adjusts the tracing values in t4216-log-bloom.sh, which provides a\nconcrete way to notice the improvement.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n revision.c | 35 ++++++++++++++++++++++++++---------\n revision.h | 6 ++++--\n t/t4216-log-bloom.sh | 2 +-\n 3 files changed, 31 insertions(+), 12 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex c644c660917..027ae3982b4 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n {\n \tstruct pathspec_item *pi;\n \tchar *path_alloc = NULL;\n-\tconst char *path;\n+\tconst char *path, *p;\n \tint last_index;\n-\tint len;\n+\tsize_t len;\n+\tint path_component_nr = 0, j;\n \n \tif (!revs->commits)\n \t\treturn;\n@@ -705,8 +706,22 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \tlen = strlen(path);\n \n-\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n-\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n+\tp = path;\n+\tdo {\n+\t\tp = strchrnul(p + 1, '/');\n+\t\tpath_component_nr++;\n+\t} while (p - path < len);\n+\n+\trevs->bloom_keys_nr = path_component_nr;\n+\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n+\n+\tp = path;\n+\tfor (j = 0; j < revs->bloom_keys_nr; j++) {\n+\t\tp = strchrnul(p + 1, '/');\n+\n+\t\tfill_bloom_key(path, p - path, &revs->bloom_keys[j],\n+\t\t\t revs->bloom_filter_settings);\n+\t}\n \n \tif (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n \t\tatexit(trace2_bloom_filter_statistics_atexit);\n@@ -720,7 +735,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\t\t\t\t\t struct commit *commit)\n {\n \tstruct bloom_filter *filter;\n-\tint result;\n+\tint result = 1, j;\n \n \tif (!revs->repo->objects->commit_graph)\n \t\treturn -1;\n@@ -740,9 +755,11 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\treturn -1;\n \t}\n \n-\tresult = bloom_filter_contains(filter,\n-\t\t\t\t revs->bloom_key,\n-\t\t\t\t revs->bloom_filter_settings);\n+\tfor (j = 0; result && j < revs->bloom_keys_nr; j++) {\n+\t\tresult = bloom_filter_contains(filter,\n+\t\t\t\t\t &revs->bloom_keys[j],\n+\t\t\t\t\t revs->bloom_filter_settings);\n+\t}\n \n \tif (result)\n \t\tcount_bloom_filter_maybe++;\n@@ -782,7 +799,7 @@ static int rev_compare_tree(struct rev_info *revs,\n \t\t\treturn REV_TREE_SAME;\n \t}\n \n-\tif (revs->bloom_key && !nth_parent) {\n+\tif (revs->bloom_keys_nr && !nth_parent) {\n \t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, commit);\n \n \t\tif (bloom_ret == 0)\ndiff --git a/revision.h b/revision.h\nindex 7c026fe41fc..abbfb4ab59a 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -295,8 +295,10 @@ struct rev_info {\n \tstruct topo_walk_info *topo_walk_info;\n \n \t/* Commit graph bloom filter fields */\n-\t/* The bloom filter key for the pathspec */\n-\tstruct bloom_key *bloom_key;\n+\t/* The bloom filter key(s) for the pathspec */\n+\tstruct bloom_key *bloom_keys;\n+\tint bloom_keys_nr;\n+\n \t/*\n \t * The bloom filter settings used to generate the key.\n \t * This is loaded from the commit-graph being used.\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c7011f33e2c..c13b97d3bda 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -142,7 +142,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n-- \ngitgitgadget\n\n"},{"id":"399816","messageId":"bdca834e6da21e75fa96bd1967d36b3a018d9545.1592252093.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"[PATCH 4/8] commit-graph: check chunk sizes after writing","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-15T20:14:49Z","receivedAt":"2020-06-15T20:15:09Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn my experience while experimenting with new commit-graph chunks,\nearly versions of the corresponding new write_commit_graph_my_chunk()\nfunctions are, sadly but not surprisingly, often buggy, and write more\nor less data than they are supposed to, especially if the chunk size\nis not directly proportional to the number of commits. This then\ncauses all kinds of issues when reading such a bogus commit-graph\nfile, raising the question of whether the writing or the reading part\nhappens to be buggy this time.\n\nLet's catch such issues early, already when writing the commit-graph\nfile, and check that each write_graph_chunk_*() function wrote the\namount of data that it was expected to, and what has been encoded in\nthe Chunk Lookup table. Now that all commit-graph chunks are written\nin a loop we can do this check in a single place for all chunks, and\nany chunks added in the future will get checked as well.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 78e023be664..5c8f210cada 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1659,12 +1659,21 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \n+\tchunk_offset = f->total + f->offset;\n \tfor (i = 0; i < num_chunks; i++) {\n+\t\tuint64_t end_offset;\n+\n \t\tif (chunks[i].write_fn(f, ctx)) {\n \t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n \t\t\t chunks[i].id);\n \t\t\treturn -1;\n \t\t}\n+\n+\t\tend_offset = f->total + f->offset;\n+\t\tif (end_offset - chunk_offset != chunks[i].size)\n+\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n+\t\t\t chunks[i].size, chunks[i].id, end_offset - chunk_offset);\n+\t\tchunk_offset = end_offset;\n \t}\n \n \tstop_progress(&ctx->progress);\n-- \ngitgitgadget\n\n"},{"id":"400008","messageId":"xmqqimfps81d.fsf@gitster.c.googlers.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"Re: [PATCH 0/8] More commit-graph/Bloom filter improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-06-17T21:21:18Z","receivedAt":"2020-06-17T21:21:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"Derrick Stolee via GitGitGadget\" writes:\n\n> This builds on sg/commit-graph-cleanups,...\n\nHow ready is that topic, do you think? I'd rather not to see too\nmany patches piled on top of what is not even in 'next', but I do\nnot remember it reviewed seriously (I did take a look or two at it\nmyself before queuing the series, but that does not quite count).\n\nWill queue to extend the topic for now.\n\nThanks.\n"},{"id":"400018","messageId":"c2b40d09-ca17-6b0e-c7da-722d92a2a669@gmail.com","threadId":"53681","inReplyTo":"xmqqimfps81d.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 0/8] More commit-graph/Bloom filter improvements","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-18T01:46:51Z","receivedAt":"2020-06-18T01:46:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/17/2020 5:21 PM, Junio C Hamano wrote:\n> \"Derrick Stolee via GitGitGadget\" writes:\n> \n>> This builds on sg/commit-graph-cleanups,...\n> \n> How ready is that topic, do you think? I'd rather not to see too\n> many patches piled on top of what is not even in 'next', but I do\n> not remember it reviewed seriously (I did take a look or two at it\n> myself before queuing the series, but that does not quite count).\n\nThat topic was my attempt to apply the \"easy and obvious\" changes\nfrom Szeder's proof-of-concept. In some sense, you could consider\nthem authored by Szeder and reviewed by me. But also, I did need to\ntweak some things, so some review from others would be helpful.\n\n> Will queue to extend the topic for now.\n\nI'll refrain from pushing any more in this direction until more\nreview comes along. I found myself with a need to do something\nproductive and familiar, so I started doing commit-graph\nperformance stuff.\n\nThanks,\n-Stolee\n"},{"id":"400080","messageId":"4a7fbd99-c581-fb30-90b3-e78c30708ff2@web.de","threadId":"53681","inReplyTo":"c966969071bf0ba135d304bbc0d5c16fbcbedc1e.1592252093.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 1/8] commit-graph: place bloom_settings in context","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-18T20:30:38Z","receivedAt":"2020-06-18T20:30:49Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 15.06.20 um 22:14 schrieb Derrick Stolee via GitGitGadget:\n> From: Derrick Stolee \n>\n> Place an instance of struct bloom_settings into the struct\n> write_commit_graph_context. This allows simplifying the function\n> prototype of write_graph_chunk_bloom_data(). This will allow us\n> to combine the function prototypes and use function pointers to\n> simplify write_commit_graph_file().\n>\n> Reported-by: SZEDER Gábor \n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 14 ++++++++------\n> 1 file changed, 8 insertions(+), 6 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 887837e8826..05b7035d8d5 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -882,6 +882,7 @@ struct write_commit_graph_context {\n>\n> \tconst struct split_commit_graph_opts *split_opts;\n> \tsize_t total_bloom_filter_data_size;\n> +\tstruct bloom_filter_settings bloom_settings;\n\nThat structure is quite busy already, so adding one more member wouldn't\nmatter much.\n\nPassing so many things to lots of functions makes it harder to argue\nabout them, though, as all of them effectively become part of their\nsignature, and you have to look at their implementation to see which\npseudo-parameters they actually use. It's like a God object.\n\n> };\n>\n> static void write_graph_chunk_fanout(struct hashfile *f,\n> @@ -1103,8 +1104,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n> }\n>\n> static void write_graph_chunk_bloom_data(struct hashfile *f,\n> -\t\t\t\t\t struct write_commit_graph_context *ctx,\n> -\t\t\t\t\t const struct bloom_filter_settings *settings)\n> +\t\t\t\t\t struct write_commit_graph_context *ctx)\n> {\n> \tstruct commit **list = ctx->commits.list;\n> \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n> @@ -1116,9 +1116,9 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n> \t\t\t_(\"Writing changed paths Bloom filters data\"),\n> \t\t\tctx->commits.nr);\n>\n> -\thashwrite_be32(f, settings->hash_version);\n> -\thashwrite_be32(f, settings->num_hashes);\n> -\thashwrite_be32(f, settings->bits_per_entry);\n> +\thashwrite_be32(f, ctx->bloom_settings.hash_version);\n> +\thashwrite_be32(f, ctx->bloom_settings.num_hashes);\n> +\thashwrite_be32(f, ctx->bloom_settings.bits_per_entry);\n>\n> \twhile (list < last) {\n> \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n> @@ -1541,6 +1541,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> \tstruct object_id file_hash;\n> \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n>\n> +\tctx->bloom_settings = bloom_settings;\n\nSo we use the defaults, no customization? Then you could simply move\nthe declaration of bloom_settings from write_commit_graph_file() to\nwrite_graph_chunk_bloom_data(). Glancing at pu I don't see additional\nuses there, so no need to put it into the context (yet?).\n\nRené\n"},{"id":"400081","messageId":"1e0c501e-3d2d-55b2-431a-f6760a184a80@web.de","threadId":"53681","inReplyTo":"65eb15221c8d3b631d6d7105d0e901433bc7f23b.1592252093.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 2/8] commit-graph: unify the signatures of all write_graph_chunk_*() functions","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-18T20:30:46Z","receivedAt":"2020-06-18T20:30:56Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 15.06.20 um 22:14 schrieb SZEDER Gábor via GitGitGadget:\n> -static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n> -\t\t\t\t struct write_commit_graph_context *ctx)\n> +static int write_graph_chunk_oids(struct hashfile *f,\n> +\t\t\t\t struct write_commit_graph_context *ctx)\n> {\n> \tstruct commit **list = ctx->commits.list;\n> \tint count;\n> \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n> \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n> -\t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n> +\t\thashwrite(f, (*list)->object.oid.hash, (int)the_hash_algo->rawsz);\n\nBefore the cast was forcing an int into an int (huh?), now it forces a\nsize_t into an int, but hashwrite() expects an unsigned int. Do we\nreally need that cast?\n\n> \t}\n> +\n> +\treturn 0;\n> }\n>\n> static const unsigned char *commit_to_sha1(size_t index, void *table)\n> @@ -926,8 +930,8 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n> \treturn commits[index]->object.oid.hash;\n> }\n>\n> -static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> -\t\t\t\t struct write_commit_graph_context *ctx)\n> +static int write_graph_chunk_data(struct hashfile *f,\n> +\t\t\t\t struct write_commit_graph_context *ctx)\n> {\n> \tstruct commit **list = ctx->commits.list;\n> \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n> @@ -944,7 +948,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \t\t\tdie(_(\"unable to parse commit %s\"),\n> \t\t\t\toid_to_hex(&(*list)->object.oid));\n> \t\ttree = get_commit_tree_oid(*list);\n> -\t\thashwrite(f, tree->hash, hash_len);\n> +\t\thashwrite(f, tree->hash, the_hash_algo->rawsz);\n\n... and here's the answer: No, we don't need to cast.\n\nRené\n"},{"id":"400082","messageId":"6779f98a-8ba3-089b-afd6-889c2caba2ff@web.de","threadId":"53681","inReplyTo":"3d24b9802df3d1fe8c670ba602890c95df96f2f5.1592252093.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 3/8] commit-graph: simplify chunk writes into loop","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-18T20:30:49Z","receivedAt":"2020-06-18T20:31:00Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 15.06.20 um 22:14 schrieb SZEDER Gábor via GitGitGadget:\n> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>\n> In write_commit_graph_file() we now have one block of code filling the\n> array of 'struct chunk_info' with the IDs and sizes of chunks to be\n> written, and an other block of code calling the functions responsible\n> for writing individual chunks. In case of optional chunks like Extra\n> Edge List an Base Graphs List there is also a condition checking\n> whether that chunk is necessary/desired, and that same condition is\n> repeated in both blocks of code. Other, newer chunks have similar\n> optional conditions.\n>\n> Eliminate these repeated conditions by storing the function pointers\n> responsible for writing individual chunks in the 'struct chunk_info'\n> array as well, and calling them in a loop to write the commit-graph\n> file. This will open up the possibility for a bit of foolproofing in\n> the following patch.\n\nOK. An alternative would be a switch in the loop that calls the right\nfunction based on the chunk id. That would not require uniform\ninterfaces for all write functions; patch 2 would not be necessary.\n\n>\n> Signed-off-by: SZEDER Gábor \n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 31 +++++++++++++++++++------------\n> 1 file changed, 19 insertions(+), 12 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 3bae1e52ed0..78e023be664 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1532,9 +1532,13 @@ static int write_graph_chunk_base(struct hashfile *f,\n> \treturn 0;\n> }\n>\n> +typedef int (*chunk_write_fn)(struct hashfile *f,\n> +\t\t\t struct write_commit_graph_context *ctx);\n> +\n> struct chunk_info {\n> \tuint32_t id;\n> \tuint64_t size;\n> +\tchunk_write_fn write_fn;\n> };\n>\n> static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> @@ -1591,27 +1595,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n>\n> \tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n> \tchunks[0].size = GRAPH_FANOUT_SIZE;\n> +\tchunks[0].write_fn = write_graph_chunk_fanout;\n> \tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n> \tchunks[1].size = hashsz * ctx->commits.nr;\n> +\tchunks[1].write_fn = write_graph_chunk_oids;\n> \tchunks[2].id = GRAPH_CHUNKID_DATA;\n> \tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n> +\tchunks[2].write_fn = write_graph_chunk_data;\n> \tif (ctx->num_extra_edges) {\n> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n> \t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n> \t\tnum_chunks++;\n> \t}\n> \tif (ctx->changed_paths) {\n> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n> \t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n> \t\tnum_chunks++;\n> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n> \t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n> \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n> \t\tnum_chunks++;\n> \t}\n> \tif (ctx->num_commit_graphs_after > 1) {\n> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n> \t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n> \t\tnum_chunks++;\n> \t}\n>\n> @@ -1647,19 +1658,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> \t\t\tprogress_title.buf,\n> \t\t\tnum_chunks * ctx->commits.nr);\n> \t}\n> -\twrite_graph_chunk_fanout(f, ctx);\n> -\twrite_graph_chunk_oids(f, ctx);\n> -\twrite_graph_chunk_data(f, ctx);\n> -\tif (ctx->num_extra_edges)\n> -\t\twrite_graph_chunk_extra_edges(f, ctx);\n> -\tif (ctx->changed_paths) {\n> -\t\twrite_graph_chunk_bloom_indexes(f, ctx);\n> -\t\twrite_graph_chunk_bloom_data(f, ctx);\n> -\t}\n> -\tif (ctx->num_commit_graphs_after > 1 &&\n> -\t write_graph_chunk_base(f, ctx)) {\n> -\t\treturn -1;\n> +\n> +\tfor (i = 0; i < num_chunks; i++) {\n> +\t\tif (chunks[i].write_fn(f, ctx)) {\n> +\t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n> +\t\t\t chunks[i].id);\n\nThis error message is new and not mentioned in the commit message.\nwrite_graph_chunk_base() seems to be the only write function that can\nreturn something else than 0, and it already reports an error in that\ncase. So do we really want the one here as well?\n\nRené\n"},{"id":"400083","messageId":"acfd97a7-9a53-0ad1-41f5-48524ceb99db@web.de","threadId":"53681","inReplyTo":"9975fc96f1269b049ebdd54835a63480c3dcbe4e.1592252093.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/8] commit-graph: check all leading directories in changed path Bloom filters","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-18T20:31:01Z","receivedAt":"2020-06-18T20:31:12Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 15.06.20 um 22:14 schrieb SZEDER Gábor via GitGitGadget:\n> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>\n> The file 'dir/subdir/file' can only be modified if its leading\n> directories 'dir' and 'dir/subdir' are modified as well.\n>\n> So when checking modified path Bloom filters looking for commits\n> modifying a path with multiple path components, then check not only\n> the full path in the Bloom filters, but all its leading directories as\n> well. Take care to check these paths in \"deepest first\" order,\n> because it's the full path that is least likely to be modified, and\n> the Bloom filter queries can short circuit sooner.\n>\n> This can significantly reduce the average false positive rate, by\n> about an order of magnitude or three(!), and can further speed up\n> pathspec-limited revision walks. The table below compares the average\n> false positive rate and runtime of\n>\n> git rev-list HEAD -- \"$path\"\n>\n> before and after this change for 5000+ randomly* selected paths from\n> each repository:\n>\n> Average false Average Average\n> positive rate runtime runtime\n> before after before after difference\n> ------------------------------------------------------------------\n> git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n> linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n> tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n\nNice!\n\n>\n> *Path selection was done with the following pipeline:\n>\n> \tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n>\n> The improvements in runtime are much smaller than the improvements in\n> average false positive rate, as we are clearly reaching diminishing\n> returns here. However, all these timings depend on that accessing\n> tree objects is reasonably fast (warm caches). If we had a partial\n> clone and the tree objects had to be fetched from a promisor remote,\n> e.g.:\n>\n> $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n> $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n> commit-graph write --reachable\n> $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n> $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n> rev-list HEAD -- \"$path\"\n>\n> then checking all leading path component can reduce the runtime from\n> over an hour to a few seconds (and this is with the clone and the\n> promisor on the same machine).\n>\n> This adjusts the tracing values in t4216-log-bloom.sh, which provides a\n> concrete way to notice the improvement.\n>\n> Signed-off-by: SZEDER Gábor \n> Signed-off-by: Derrick Stolee \n> ---\n> revision.c | 35 ++++++++++++++++++++++++++---------\n> revision.h | 6 ++++--\n> t/t4216-log-bloom.sh | 2 +-\n> 3 files changed, 31 insertions(+), 12 deletions(-)\n>\n> diff --git a/revision.c b/revision.c\n> index c644c660917..027ae3982b4 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> {\n> \tstruct pathspec_item *pi;\n> \tchar *path_alloc = NULL;\n> -\tconst char *path;\n> +\tconst char *path, *p;\n> \tint last_index;\n> -\tint len;\n> +\tsize_t len;\n> +\tint path_component_nr = 0, j;\n>\n> \tif (!revs->commits)\n> \t\treturn;\n> @@ -705,8 +706,22 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n>\n> \tlen = strlen(path);\n>\n> -\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n> -\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n> +\tp = path;\n> +\tdo {\n> +\t\tp = strchrnul(p + 1, '/');\n> +\t\tpath_component_nr++;\n> +\t} while (p - path < len);\n\nHmm, that \"+ 1\" makes me a bit nervous. Can we be sure that path is not\nan empty string?\n\nAnd shouldn't we use is_dir_sep() or find_last_dir_sep() instead of\nhard-coding a slash?\n\n> +\n> +\trevs->bloom_keys_nr = path_component_nr;\n> +\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n> +\n> +\tp = path;\n> +\tfor (j = 0; j < revs->bloom_keys_nr; j++) {\n> +\t\tp = strchrnul(p + 1, '/');\n\nSame here, of course.\n\nAlso note that this puts shorter sub-strings first.\n\n\n> +\n> +\t\tfill_bloom_key(path, p - path, &revs->bloom_keys[j],\n> +\t\t\t revs->bloom_filter_settings);\n> +\t}\n>\n> \tif (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n> \t\tatexit(trace2_bloom_filter_statistics_atexit);\n> @@ -720,7 +735,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n> \t\t\t\t\t\t struct commit *commit)\n> {\n> \tstruct bloom_filter *filter;\n> -\tint result;\n> +\tint result = 1, j;\n>\n> \tif (!revs->repo->objects->commit_graph)\n> \t\treturn -1;\n> @@ -740,9 +755,11 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n> \t\treturn -1;\n> \t}\n>\n> -\tresult = bloom_filter_contains(filter,\n> -\t\t\t\t revs->bloom_key,\n> -\t\t\t\t revs->bloom_filter_settings);\n> +\tfor (j = 0; result && j < revs->bloom_keys_nr; j++) {\n> +\t\tresult = bloom_filter_contains(filter,\n> +\t\t\t\t\t &revs->bloom_keys[j],\n> +\t\t\t\t\t revs->bloom_filter_settings);\n> +\t}\n\nThis checks shorter sub-strings first, contradicting the \"deepest first\"\nstrategy mentioned in the commit message.\n\nThis can easily be fixed by inverting the traversal of one of the loops,\nof course. Or perhaps do something like this?\n\n\trevs->bloom_keys = NULL;\n\trevs->bloom_keys_nr = 0;\n\tstrbuf_add(&path, pi->match, pi->len);\n\tstrbuf_trim_trailing_dir_sep(&path);\n\tfor (;;) {\n\t\tconst char *sep;\n\t\tALLOC_GROW(revs->bloom_keys, revs->bloom_keys_nr + 1, alloc);\n\t\tfill_bloom_key(path.buf, path.len,\n\t\t\t &revs->bloom_keys[revs->bloom_keys_nr++],\n\t\t\t revs->bloom_filter_settings);\n\t\tsep = find_last_dir_sep(path.buf);\n\t\tif (!sep)\n\t\t\tbreak;\n\t\tstrbuf_setlen(&path, sep - path.buf);\n\t}\n\tstrbuf_release(&path);\n\nThe find_last_dir_sep() calls scan the first part of the string over and\nover, which is a bit silly. A strbuf_trim_trailing_path_component()\ncould start at the end and scan backwards if that turns out to be an\nactual problem.\n\nALLOC_GROW wastes memory on revs->bloom_keys, and reallocating instead\nof allocating the right size from the start has a cost as well, but I'd\nexpect this to be dwarfed by the actual revision walk.\n\n>\n> \tif (result)\n> \t\tcount_bloom_filter_maybe++;\n> @@ -782,7 +799,7 @@ static int rev_compare_tree(struct rev_info *revs,\n> \t\t\treturn REV_TREE_SAME;\n> \t}\n>\n> -\tif (revs->bloom_key && !nth_parent) {\n> +\tif (revs->bloom_keys_nr && !nth_parent) {\n> \t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, commit);\n>\n> \t\tif (bloom_ret == 0)\n> diff --git a/revision.h b/revision.h\n> index 7c026fe41fc..abbfb4ab59a 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -295,8 +295,10 @@ struct rev_info {\n> \tstruct topo_walk_info *topo_walk_info;\n>\n> \t/* Commit graph bloom filter fields */\n> -\t/* The bloom filter key for the pathspec */\n> -\tstruct bloom_key *bloom_key;\n> +\t/* The bloom filter key(s) for the pathspec */\n> +\tstruct bloom_key *bloom_keys;\n> +\tint bloom_keys_nr;\n> +\n> \t/*\n> \t * The bloom filter settings used to generate the key.\n> \t * This is loaded from the commit-graph being used.\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index c7011f33e2c..c13b97d3bda 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -142,7 +142,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n>\n> test_bloom_filters_used_when_some_filters_are_missing () {\n> \tlog_args=$1\n> -\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n> +\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n> \tsetup \"$log_args\" &&\n> \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n> \ttest_cmp log_wo_bloom log_w_bloom\n>\n\n"},{"id":"400102","messageId":"0b70fca2-5c3f-5d9b-d56c-ca731450a212@web.de","threadId":"53681","inReplyTo":"acfd97a7-9a53-0ad1-41f5-48524ceb99db@web.de","subject":"Re: [PATCH 5/8] commit-graph: check all leading directories in changed path Bloom filters","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-19T09:14:20Z","receivedAt":"2020-06-19T09:14:48Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 18.06.20 um 22:31 schrieb René Scharfe:\n> Am 15.06.20 um 22:14 schrieb SZEDER Gábor via GitGitGadget:\n>> --- a/revision.c\n>> +++ b/revision.c\n>> @@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n>> {\n>> \tstruct pathspec_item *pi;\n>> \tchar *path_alloc = NULL;\n>> -\tconst char *path;\n>> +\tconst char *path, *p;\n>> \tint last_index;\n>> -\tint len;\n>> +\tsize_t len;\n>> +\tint path_component_nr = 0, j;\n>>\n>> \tif (!revs->commits)\n>> \t\treturn;\n>> @@ -705,8 +706,22 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n>>\n>> \tlen = strlen(path);\n>>\n>> -\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n>> -\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n>> +\tp = path;\n>> +\tdo {\n>> +\t\tp = strchrnul(p + 1, '/');\n>> +\t\tpath_component_nr++;\n>> +\t} while (p - path < len);\n\n> And shouldn't we use is_dir_sep() or find_last_dir_sep() instead of\n> hard-coding a slash?\n\nNot necessarily. Paths should be normalized to use one specific\nseparator, probably slash, both when building and querying the Bloom\nfilter. Otherwise a filter that knows e.g. \"foo/bar\" could confidently\nclaim that \"foo\\bar\" does not match. If this is done in a previous\nstep then using a literal slash here would be correct.\n\nRené\n"},{"id":"400110","messageId":"38a8ba15-7fed-8eec-41bf-ccae068335af@gmail.com","threadId":"53681","inReplyTo":"4a7fbd99-c581-fb30-90b3-e78c30708ff2@web.de","subject":"Re: [PATCH 1/8] commit-graph: place bloom_settings in context","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-19T12:58:54Z","receivedAt":"2020-06-19T12:59:02Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/18/2020 4:30 PM, René Scharfe wrote:\n> Am 15.06.20 um 22:14 schrieb Derrick Stolee via GitGitGadget:\n>> From: Derrick Stolee \n>>\n>> Place an instance of struct bloom_settings into the struct\n>> write_commit_graph_context. This allows simplifying the function\n>> prototype of write_graph_chunk_bloom_data(). This will allow us\n>> to combine the function prototypes and use function pointers to\n>> simplify write_commit_graph_file().\n>>\n>> Reported-by: SZEDER Gábor \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 14 ++++++++------\n>> 1 file changed, 8 insertions(+), 6 deletions(-)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 887837e8826..05b7035d8d5 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -882,6 +882,7 @@ struct write_commit_graph_context {\n>>\n>> \tconst struct split_commit_graph_opts *split_opts;\n>> \tsize_t total_bloom_filter_data_size;\n>> +\tstruct bloom_filter_settings bloom_settings;\n> \n> That structure is quite busy already, so adding one more member wouldn't\n> matter much.\n> \n> Passing so many things to lots of functions makes it harder to argue\n> about them, though, as all of them effectively become part of their\n> signature, and you have to look at their implementation to see which\n> pseudo-parameters they actually use. It's like a God object.\n\nCorrect. The write_commit_graph_context _is_ a God object for the\ncommit-graph write. The good news is that it is limited only to\ncommit-graph.c and the write operations therein. Hopefully, the\ncode organization benefits enough from this structure to justify\nthe massive struct.\n\nIn contrast, it's still smaller and more contained than\n\"struct rev_info\"!\n\n>> };\n>>\n>> static void write_graph_chunk_fanout(struct hashfile *f,\n>> @@ -1103,8 +1104,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n>> }\n>>\n>> static void write_graph_chunk_bloom_data(struct hashfile *f,\n>> -\t\t\t\t\t struct write_commit_graph_context *ctx,\n>> -\t\t\t\t\t const struct bloom_filter_settings *settings)\n>> +\t\t\t\t\t struct write_commit_graph_context *ctx)\n>> {\n>> \tstruct commit **list = ctx->commits.list;\n>> \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n>> @@ -1116,9 +1116,9 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n>> \t\t\t_(\"Writing changed paths Bloom filters data\"),\n>> \t\t\tctx->commits.nr);\n>>\n>> -\thashwrite_be32(f, settings->hash_version);\n>> -\thashwrite_be32(f, settings->num_hashes);\n>> -\thashwrite_be32(f, settings->bits_per_entry);\n>> +\thashwrite_be32(f, ctx->bloom_settings.hash_version);\n>> +\thashwrite_be32(f, ctx->bloom_settings.num_hashes);\n>> +\thashwrite_be32(f, ctx->bloom_settings.bits_per_entry);\n>>\n>> \twhile (list < last) {\n>> \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n>> @@ -1541,6 +1541,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n>> \tstruct object_id file_hash;\n>> \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n>>\n>> +\tctx->bloom_settings = bloom_settings;\n> \n> So we use the defaults, no customization? Then you could simply move\n> the declaration of bloom_settings from write_commit_graph_file() to\n> write_graph_chunk_bloom_data(). Glancing at pu I don't see additional\n> uses there, so no need to put it into the context (yet?).\n\nIt certainly is not customized by a user (yet). However, you do make an\nexcellent point that I need to be more careful here! Patch 8\n(commit-graph: persist existence of changed-paths) needs to load the\nbloom_filter_settings from the existing commit-graph so we can be\nfuture-proof from a future version customizing the settings inside the\ncommit-graph file!\n\nThis means that in v2 I'll move patches 7 & 8 to be after patch 1 and\nadd a test to verify the filter settings are preserved (after manually\nchanging the data in the file).\n\nThanks!\n-Stolee\n"},{"id":"400150","messageId":"20200619171717.GA55911@syl.local","threadId":"53681","inReplyTo":"9975fc96f1269b049ebdd54835a63480c3dcbe4e.1592252093.git.gitgitgadget@gmail.com","subject":"Re: [PATCH 5/8] commit-graph: check all leading directories in changed path Bloom filters","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-06-19T17:17:17Z","receivedAt":"2020-06-19T17:17:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi Stolee,\n\nOn Mon, Jun 15, 2020 at 08:14:50PM +0000, SZEDER Gábor via GitGitGadget wrote:\n> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>\n> The file 'dir/subdir/file' can only be modified if its leading\n> directories 'dir' and 'dir/subdir' are modified as well.\n>\n> So when checking modified path Bloom filters looking for commits\n> modifying a path with multiple path components, then check not only\n> the full path in the Bloom filters, but all its leading directories as\n> well. Take care to check these paths in \"deepest first\" order,\n> because it's the full path that is least likely to be modified, and\n> the Bloom filter queries can short circuit sooner.\n>\n> This can significantly reduce the average false positive rate, by\n> about an order of magnitude or three(!), and can further speed up\n> pathspec-limited revision walks. The table below compares the average\n> false positive rate and runtime of\n>\n> git rev-list HEAD -- \"$path\"\n>\n> before and after this change for 5000+ randomly* selected paths from\n> each repository:\n>\n> Average false Average Average\n> positive rate runtime runtime\n> before after before after difference\n> ------------------------------------------------------------------\n> git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n> linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n> tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n>\n> *Path selection was done with the following pipeline:\n>\n> \tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n>\n> The improvements in runtime are much smaller than the improvements in\n> average false positive rate, as we are clearly reaching diminishing\n> returns here. However, all these timings depend on that accessing\n> tree objects is reasonably fast (warm caches). If we had a partial\n> clone and the tree objects had to be fetched from a promisor remote,\n> e.g.:\n>\n> $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n> $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n> commit-graph write --reachable\n> $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n> $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n> rev-list HEAD -- \"$path\"\n>\n> then checking all leading path component can reduce the runtime from\n> over an hour to a few seconds (and this is with the clone and the\n> promisor on the same machine).\n>\n> This adjusts the tracing values in t4216-log-bloom.sh, which provides a\n> concrete way to notice the improvement.\n>\n> Signed-off-by: SZEDER Gábor \n> Signed-off-by: Derrick Stolee \n> ---\n> revision.c | 35 ++++++++++++++++++++++++++---------\n> revision.h | 6 ++++--\n> t/t4216-log-bloom.sh | 2 +-\n> 3 files changed, 31 insertions(+), 12 deletions(-)\n>\n> diff --git a/revision.c b/revision.c\n> index c644c660917..027ae3982b4 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> {\n> \tstruct pathspec_item *pi;\n> \tchar *path_alloc = NULL;\n> -\tconst char *path;\n> +\tconst char *path, *p;\n> \tint last_index;\n> -\tint len;\n> +\tsize_t len;\n> +\tint path_component_nr = 0, j;\n>\n> \tif (!revs->commits)\n> \t\treturn;\n> @@ -705,8 +706,22 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n>\n> \tlen = strlen(path);\n>\n> -\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n> -\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n> +\tp = path;\n> +\tdo {\n> +\t\tp = strchrnul(p + 1, '/');\n> +\t\tpath_component_nr++;\n> +\t} while (p - path < len);\n> +\n> +\trevs->bloom_keys_nr = path_component_nr;\n> +\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n> +\n> +\tp = path;\n> +\tfor (j = 0; j < revs->bloom_keys_nr; j++) {\n> +\t\tp = strchrnul(p + 1, '/');\n> +\n> +\t\tfill_bloom_key(path, p - path, &revs->bloom_keys[j],\n> +\t\t\t revs->bloom_filter_settings);\n> +\t}\n>\n\nSomewhat related to our off-list discussion yesterday, there is a bug in\nboth 2.27 and this patch which produces incorrect results when (1)\nBloom filters are enabled, and (2) we are doing a revision walk from\nroot with the pathspec '.'.\n\nWhat appears to be going on is that our normalization takes '.' -> '',\nand then we form a Bloom key based on the empty string, which will\nreturn 'definitely not' when querying the Bloom filter some of the time,\nwhich should never happen. This is a consequence of never inserting the\nempty key into the Bloom filter upon generation.\n\nAs a result, I have patched this in GitHub's fork (which is currently\nbased on 2.27 and doesn't have these patches yet) by doing an early\nreturn when 'strlen(path) == 0'. Since it looks like these patches are\ngoing to land, here is some clean-up and a fix for the bug that you\nshould feel free to test with and apply on top:\n\n--- >8 ---\n\ndiff --git a/revision.c b/revision.c\nindex 8bd383b1dd..123e72698d 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -670,10 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n {\n struct pathspec_item *pi;\n char *path_alloc = NULL;\n- const char *path, *p;\n+ char *path, *p;\n int last_index;\n size_t len;\n- int path_component_nr = 0, j;\n+ int path_component_nr = 1, j;\n\n if (!revs->commits)\n return;\n@@ -698,29 +698,33 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n\n /* remove single trailing slash from path, if needed */\n if (pi->match[last_index] == '/') {\n- path_alloc = xstrdup(pi->match);\n- path_alloc[last_index] = '\\0';\n- path = path_alloc;\n- } else\n- path = pi->match;\n+ path_alloc = xstrdup(pi->match);\n+ path_alloc[last_index] = '\\0';\n+ path = path_alloc;\n+ } else {\n+ path = pi->match;\n+ len = pi->len;\n+ }\n\n- len = strlen(path);\n+ if (!len)\n+ return;\n\n- p = path;\n do {\n- p = strchrnul(p + 1, '/');\n- path_component_nr++;\n- } while (p - path < len);\n+ if (is_dir_sep(*p)) {\n+ *p = '\\0';\n+ path_component_nr++;\n+ }\n+ } while (*p++);\n\n revs->bloom_keys_nr = path_component_nr;\n ALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n\n p = path;\n for (j = 0; j < revs->bloom_keys_nr; j++) {\n- p = strchrnul(p + 1, '/');\n-\n- fill_bloom_key(path, p - path, &revs->bloom_keys[j],\n+ size_t plen = strlen(p);\n+ fill_bloom_key(p, plen, &revs->bloom_keys[j],\n revs->bloom_filter_settings);\n+ p += plen;\n }\n\n if (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n\n> \tif (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n> \t\tatexit(trace2_bloom_filter_statistics_atexit);\n> @@ -720,7 +735,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n> \t\t\t\t\t\t struct commit *commit)\n> {\n> \tstruct bloom_filter *filter;\n> -\tint result;\n> +\tint result = 1, j;\n>\n> \tif (!revs->repo->objects->commit_graph)\n> \t\treturn -1;\n> @@ -740,9 +755,11 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n> \t\treturn -1;\n> \t}\n>\n> -\tresult = bloom_filter_contains(filter,\n> -\t\t\t\t revs->bloom_key,\n> -\t\t\t\t revs->bloom_filter_settings);\n> +\tfor (j = 0; result && j < revs->bloom_keys_nr; j++) {\n> +\t\tresult = bloom_filter_contains(filter,\n> +\t\t\t\t\t &revs->bloom_keys[j],\n> +\t\t\t\t\t revs->bloom_filter_settings);\n> +\t}\n>\n> \tif (result)\n> \t\tcount_bloom_filter_maybe++;\n> @@ -782,7 +799,7 @@ static int rev_compare_tree(struct rev_info *revs,\n> \t\t\treturn REV_TREE_SAME;\n> \t}\n>\n> -\tif (revs->bloom_key && !nth_parent) {\n> +\tif (revs->bloom_keys_nr && !nth_parent) {\n> \t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, commit);\n>\n> \t\tif (bloom_ret == 0)\n> diff --git a/revision.h b/revision.h\n> index 7c026fe41fc..abbfb4ab59a 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -295,8 +295,10 @@ struct rev_info {\n> \tstruct topo_walk_info *topo_walk_info;\n>\n> \t/* Commit graph bloom filter fields */\n> -\t/* The bloom filter key for the pathspec */\n> -\tstruct bloom_key *bloom_key;\n> +\t/* The bloom filter key(s) for the pathspec */\n> +\tstruct bloom_key *bloom_keys;\n> +\tint bloom_keys_nr;\n> +\n> \t/*\n> \t * The bloom filter settings used to generate the key.\n> \t * This is loaded from the commit-graph being used.\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index c7011f33e2c..c13b97d3bda 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -142,7 +142,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n>\n> test_bloom_filters_used_when_some_filters_are_missing () {\n> \tlog_args=$1\n> -\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n> +\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n> \tsetup \"$log_args\" &&\n> \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n> \ttest_cmp log_wo_bloom log_w_bloom\n> --\n> gitgitgadget\n>\nThanks,\nTaylor\n"},{"id":"400153","messageId":"20200619171922.GA57283@syl.local","threadId":"53681","inReplyTo":"20200619171717.GA55911@syl.local","subject":"Re: [PATCH 5/8] commit-graph: check all leading directories in changed path Bloom filters","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-06-19T17:19:22Z","receivedAt":"2020-06-19T17:19:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Jun 19, 2020 at 11:17:17AM -0600, Taylor Blau wrote:\n> Hi Stolee,\n>\n> On Mon, Jun 15, 2020 at 08:14:50PM +0000, SZEDER Gábor via GitGitGadget wrote:\n> > From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n> >\n> > The file 'dir/subdir/file' can only be modified if its leading\n> > directories 'dir' and 'dir/subdir' are modified as well.\n> >\n> > So when checking modified path Bloom filters looking for commits\n> > modifying a path with multiple path components, then check not only\n> > the full path in the Bloom filters, but all its leading directories as\n> > well. Take care to check these paths in \"deepest first\" order,\n> > because it's the full path that is least likely to be modified, and\n> > the Bloom filter queries can short circuit sooner.\n> >\n> > This can significantly reduce the average false positive rate, by\n> > about an order of magnitude or three(!), and can further speed up\n> > pathspec-limited revision walks. The table below compares the average\n> > false positive rate and runtime of\n> >\n> > git rev-list HEAD -- \"$path\"\n> >\n> > before and after this change for 5000+ randomly* selected paths from\n> > each repository:\n> >\n> > Average false Average Average\n> > positive rate runtime runtime\n> > before after before after difference\n> > ------------------------------------------------------------------\n> > git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n> > linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n> > tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n> >\n> > *Path selection was done with the following pipeline:\n> >\n> > \tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n> >\n> > The improvements in runtime are much smaller than the improvements in\n> > average false positive rate, as we are clearly reaching diminishing\n> > returns here. However, all these timings depend on that accessing\n> > tree objects is reasonably fast (warm caches). If we had a partial\n> > clone and the tree objects had to be fetched from a promisor remote,\n> > e.g.:\n> >\n> > $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n> > $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n> > commit-graph write --reachable\n> > $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n> > $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n> > rev-list HEAD -- \"$path\"\n> >\n> > then checking all leading path component can reduce the runtime from\n> > over an hour to a few seconds (and this is with the clone and the\n> > promisor on the same machine).\n> >\n> > This adjusts the tracing values in t4216-log-bloom.sh, which provides a\n> > concrete way to notice the improvement.\n> >\n> > Signed-off-by: SZEDER Gábor \n> > Signed-off-by: Derrick Stolee \n> > ---\n> > revision.c | 35 ++++++++++++++++++++++++++---------\n> > revision.h | 6 ++++--\n> > t/t4216-log-bloom.sh | 2 +-\n> > 3 files changed, 31 insertions(+), 12 deletions(-)\n> >\n> > diff --git a/revision.c b/revision.c\n> > index c644c660917..027ae3982b4 100644\n> > --- a/revision.c\n> > +++ b/revision.c\n> > @@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> > {\n> > \tstruct pathspec_item *pi;\n> > \tchar *path_alloc = NULL;\n> > -\tconst char *path;\n> > +\tconst char *path, *p;\n> > \tint last_index;\n> > -\tint len;\n> > +\tsize_t len;\n> > +\tint path_component_nr = 0, j;\n> >\n> > \tif (!revs->commits)\n> > \t\treturn;\n> > @@ -705,8 +706,22 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> >\n> > \tlen = strlen(path);\n> >\n> > -\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n> > -\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n> > +\tp = path;\n> > +\tdo {\n> > +\t\tp = strchrnul(p + 1, '/');\n> > +\t\tpath_component_nr++;\n> > +\t} while (p - path < len);\n> > +\n> > +\trevs->bloom_keys_nr = path_component_nr;\n> > +\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n> > +\n> > +\tp = path;\n> > +\tfor (j = 0; j < revs->bloom_keys_nr; j++) {\n> > +\t\tp = strchrnul(p + 1, '/');\n> > +\n> > +\t\tfill_bloom_key(path, p - path, &revs->bloom_keys[j],\n> > +\t\t\t revs->bloom_filter_settings);\n> > +\t}\n> >\n>\n> Somewhat related to our off-list discussion yesterday, there is a bug in\n> both 2.27 and this patch which produces incorrect results when (1)\n> Bloom filters are enabled, and (2) we are doing a revision walk from\n> root with the pathspec '.'.\n>\n> What appears to be going on is that our normalization takes '.' -> '',\n> and then we form a Bloom key based on the empty string, which will\n> return 'definitely not' when querying the Bloom filter some of the time,\n> which should never happen. This is a consequence of never inserting the\n> empty key into the Bloom filter upon generation.\n>\n> As a result, I have patched this in GitHub's fork (which is currently\n> based on 2.27 and doesn't have these patches yet) by doing an early\n> return when 'strlen(path) == 0'. Since it looks like these patches are\n> going to land, here is some clean-up and a fix for the bug that you\n> should feel free to test with and apply on top:\n>\n> --- >8 ---\n>\n> diff --git a/revision.c b/revision.c\n> index 8bd383b1dd..123e72698d 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -670,10 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> {\n> struct pathspec_item *pi;\n> char *path_alloc = NULL;\n> - const char *path, *p;\n> + char *path, *p;\n> int last_index;\n> size_t len;\n> - int path_component_nr = 0, j;\n> + int path_component_nr = 1, j;\n>\n> if (!revs->commits)\n> return;\n> @@ -698,29 +698,33 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n>\n> /* remove single trailing slash from path, if needed */\n> if (pi->match[last_index] == '/') {\n> - path_alloc = xstrdup(pi->match);\n> - path_alloc[last_index] = '\\0';\n> - path = path_alloc;\n> - } else\n> - path = pi->match;\n> + path_alloc = xstrdup(pi->match);\n> + path_alloc[last_index] = '\\0';\n> + path = path_alloc;\n> + } else {\n> + path = pi->match;\n> + len = pi->len;\n> + }\n>\n> - len = strlen(path);\n> + if (!len)\n> + return;\n\nI should note that _this_ is the critical fix, and it should fix the bug\nif you only applied just this hunk.\n\nEverything else is purely style clean-ups on top (ranging from the four\nspaces used instead of a tab, to some string processing niceties that I\n_think_ should address Rene's concern, although I'm not sure if an\nactual bug is lurking there or not...)\n\n> - p = path;\n> do {\n> - p = strchrnul(p + 1, '/');\n> - path_component_nr++;\n> - } while (p - path < len);\n> + if (is_dir_sep(*p)) {\n> + *p = '\\0';\n> + path_component_nr++;\n> + }\n> + } while (*p++);\n>\n> revs->bloom_keys_nr = path_component_nr;\n> ALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n>\n> p = path;\n> for (j = 0; j < revs->bloom_keys_nr; j++) {\n> - p = strchrnul(p + 1, '/');\n> -\n> - fill_bloom_key(path, p - path, &revs->bloom_keys[j],\n> + size_t plen = strlen(p);\n> + fill_bloom_key(p, plen, &revs->bloom_keys[j],\n> revs->bloom_filter_settings);\n> + p += plen;\n> }\n>\n> if (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n>\n> > \tif (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n> > \t\tatexit(trace2_bloom_filter_statistics_atexit);\n> > @@ -720,7 +735,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n> > \t\t\t\t\t\t struct commit *commit)\n> > {\n> > \tstruct bloom_filter *filter;\n> > -\tint result;\n> > +\tint result = 1, j;\n> >\n> > \tif (!revs->repo->objects->commit_graph)\n> > \t\treturn -1;\n> > @@ -740,9 +755,11 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n> > \t\treturn -1;\n> > \t}\n> >\n> > -\tresult = bloom_filter_contains(filter,\n> > -\t\t\t\t revs->bloom_key,\n> > -\t\t\t\t revs->bloom_filter_settings);\n> > +\tfor (j = 0; result && j < revs->bloom_keys_nr; j++) {\n> > +\t\tresult = bloom_filter_contains(filter,\n> > +\t\t\t\t\t &revs->bloom_keys[j],\n> > +\t\t\t\t\t revs->bloom_filter_settings);\n> > +\t}\n> >\n> > \tif (result)\n> > \t\tcount_bloom_filter_maybe++;\n> > @@ -782,7 +799,7 @@ static int rev_compare_tree(struct rev_info *revs,\n> > \t\t\treturn REV_TREE_SAME;\n> > \t}\n> >\n> > -\tif (revs->bloom_key && !nth_parent) {\n> > +\tif (revs->bloom_keys_nr && !nth_parent) {\n> > \t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, commit);\n> >\n> > \t\tif (bloom_ret == 0)\n> > diff --git a/revision.h b/revision.h\n> > index 7c026fe41fc..abbfb4ab59a 100644\n> > --- a/revision.h\n> > +++ b/revision.h\n> > @@ -295,8 +295,10 @@ struct rev_info {\n> > \tstruct topo_walk_info *topo_walk_info;\n> >\n> > \t/* Commit graph bloom filter fields */\n> > -\t/* The bloom filter key for the pathspec */\n> > -\tstruct bloom_key *bloom_key;\n> > +\t/* The bloom filter key(s) for the pathspec */\n> > +\tstruct bloom_key *bloom_keys;\n> > +\tint bloom_keys_nr;\n> > +\n> > \t/*\n> > \t * The bloom filter settings used to generate the key.\n> > \t * This is loaded from the commit-graph being used.\n> > diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> > index c7011f33e2c..c13b97d3bda 100755\n> > --- a/t/t4216-log-bloom.sh\n> > +++ b/t/t4216-log-bloom.sh\n> > @@ -142,7 +142,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n> >\n> > test_bloom_filters_used_when_some_filters_are_missing () {\n> > \tlog_args=$1\n> > -\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n> > +\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n> > \tsetup \"$log_args\" &&\n> > \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n> > \ttest_cmp log_wo_bloom log_w_bloom\n> > --\n> > gitgitgadget\n> >\n> Thanks,\n> Taylor\nThanks,\nTaylor\n"},{"id":"400368","messageId":"b6a748d0-0fd8-5d43-012d-fabeff75ff31@gmail.com","threadId":"53681","inReplyTo":"20200619171717.GA55911@syl.local","subject":"Re: [PATCH 5/8] commit-graph: check all leading directories in changed path Bloom filters","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-23T13:47:40Z","receivedAt":"2020-06-23T13:47:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/19/2020 1:17 PM, Taylor Blau wrote:\n> Hi Stolee,\n> \n> On Mon, Jun 15, 2020 at 08:14:50PM +0000, SZEDER Gábor via GitGitGadget wrote:\n>> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>>\n>> The file 'dir/subdir/file' can only be modified if its leading\n>> directories 'dir' and 'dir/subdir' are modified as well.\n>>\n>> So when checking modified path Bloom filters looking for commits\n>> modifying a path with multiple path components, then check not only\n>> the full path in the Bloom filters, but all its leading directories as\n>> well. Take care to check these paths in \"deepest first\" order,\n>> because it's the full path that is least likely to be modified, and\n>> the Bloom filter queries can short circuit sooner.\n>>\n>> This can significantly reduce the average false positive rate, by\n>> about an order of magnitude or three(!), and can further speed up\n>> pathspec-limited revision walks. The table below compares the average\n>> false positive rate and runtime of\n>>\n>> git rev-list HEAD -- \"$path\"\n>>\n>> before and after this change for 5000+ randomly* selected paths from\n>> each repository:\n>>\n>> Average false Average Average\n>> positive rate runtime runtime\n>> before after before after difference\n>> ------------------------------------------------------------------\n>> git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n>> linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n>> tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n>>\n>> *Path selection was done with the following pipeline:\n>>\n>> \tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n>>\n>> The improvements in runtime are much smaller than the improvements in\n>> average false positive rate, as we are clearly reaching diminishing\n>> returns here. However, all these timings depend on that accessing\n>> tree objects is reasonably fast (warm caches). If we had a partial\n>> clone and the tree objects had to be fetched from a promisor remote,\n>> e.g.:\n>>\n>> $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n>> $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n>> commit-graph write --reachable\n>> $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n>> $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n>> rev-list HEAD -- \"$path\"\n>>\n>> then checking all leading path component can reduce the runtime from\n>> over an hour to a few seconds (and this is with the clone and the\n>> promisor on the same machine).\n>>\n>> This adjusts the tracing values in t4216-log-bloom.sh, which provides a\n>> concrete way to notice the improvement.\n>>\n>> Signed-off-by: SZEDER Gábor \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> revision.c | 35 ++++++++++++++++++++++++++---------\n>> revision.h | 6 ++++--\n>> t/t4216-log-bloom.sh | 2 +-\n>> 3 files changed, 31 insertions(+), 12 deletions(-)\n>>\n>> diff --git a/revision.c b/revision.c\n>> index c644c660917..027ae3982b4 100644\n>> --- a/revision.c\n>> +++ b/revision.c\n>> @@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n>> {\n>> \tstruct pathspec_item *pi;\n>> \tchar *path_alloc = NULL;\n>> -\tconst char *path;\n>> +\tconst char *path, *p;\n>> \tint last_index;\n>> -\tint len;\n>> +\tsize_t len;\n>> +\tint path_component_nr = 0, j;\n>>\n>> \tif (!revs->commits)\n>> \t\treturn;\n>> @@ -705,8 +706,22 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n>>\n>> \tlen = strlen(path);\n>>\n>> -\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n>> -\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n>> +\tp = path;\n>> +\tdo {\n>> +\t\tp = strchrnul(p + 1, '/');\n>> +\t\tpath_component_nr++;\n>> +\t} while (p - path < len);\n>> +\n>> +\trevs->bloom_keys_nr = path_component_nr;\n>> +\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n>> +\n>> +\tp = path;\n>> +\tfor (j = 0; j < revs->bloom_keys_nr; j++) {\n>> +\t\tp = strchrnul(p + 1, '/');\n>> +\n>> +\t\tfill_bloom_key(path, p - path, &revs->bloom_keys[j],\n>> +\t\t\t revs->bloom_filter_settings);\n>> +\t}\n>>\n> \n> Somewhat related to our off-list discussion yesterday, there is a bug in\n> both 2.27 and this patch which produces incorrect results when (1)\n> Bloom filters are enabled, and (2) we are doing a revision walk from\n> root with the pathspec '.'.\n> \n> What appears to be going on is that our normalization takes '.' -> '',\n> and then we form a Bloom key based on the empty string, which will\n> return 'definitely not' when querying the Bloom filter some of the time,\n> which should never happen. This is a consequence of never inserting the\n> empty key into the Bloom filter upon generation.\n> \n> As a result, I have patched this in GitHub's fork (which is currently\n> based on 2.27 and doesn't have these patches yet) by doing an early\n> return when 'strlen(path) == 0'. Since it looks like these patches are\n> going to land, here is some clean-up and a fix for the bug that you\n> should feel free to test with and apply on top:\n> \n> --- >8 ---\n> \n> diff --git a/revision.c b/revision.c\n> index 8bd383b1dd..123e72698d 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -670,10 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> {\n> struct pathspec_item *pi;\n> char *path_alloc = NULL;\n> - const char *path, *p;\n> + char *path, *p;\n> int last_index;\n> size_t len;\n> - int path_component_nr = 0, j;\n> + int path_component_nr = 1, j;\n> \n> if (!revs->commits)\n> return;\n> @@ -698,29 +698,33 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> \n> /* remove single trailing slash from path, if needed */\n> if (pi->match[last_index] == '/') {\n> - path_alloc = xstrdup(pi->match);\n> - path_alloc[last_index] = '\\0';\n> - path = path_alloc;\n> - } else\n> - path = pi->match;\n> + path_alloc = xstrdup(pi->match);\n> + path_alloc[last_index] = '\\0';\n> + path = path_alloc;\n> + } else {\n> + path = pi->match;\n> + len = pi->len;\n> + }\n> \n> - len = strlen(path);\n> + if (!len)\n> + return;\n> \n> - p = path;\n> do {\n> - p = strchrnul(p + 1, '/');\n> - path_component_nr++;\n> - } while (p - path < len);\n> + if (is_dir_sep(*p)) {\n> + *p = '\\0';\n> + path_component_nr++;\n> + }\n> + } while (*p++);\n> \n> revs->bloom_keys_nr = path_component_nr;\n> ALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n> \n> p = path;\n> for (j = 0; j < revs->bloom_keys_nr; j++) {\n> - p = strchrnul(p + 1, '/');\n> -\n> - fill_bloom_key(path, p - path, &revs->bloom_keys[j],\n> + size_t plen = strlen(p);\n> + fill_bloom_key(p, plen, &revs->bloom_keys[j],\n> revs->bloom_filter_settings);\n> + p += plen;\n\nI don't think this is correct at all. Looking at it, it seems\nthat it would take a path \"A/B/C\" and add keys for \"A\", \"B\", and\n\"C\" instead of \"A\", \"A/B\", and \"A/B/C\".\n\nLooking more closely, there are a few issues that makes it clear\nwhy you didn't see a failing test:\n\n1. You use \"while (*p++)\" instead of \"while (*++p)\" so the scan\n terminates after the first directory split. (So only \"A\" is\n added, which won't fail, but will be slower than intended.)\n\nChanging that, we see the next problem:\n\n2. You use \"p += plen\" instead of \"p += plen + 1\". This causes\n the filters to add \"A\", \"\", and \"\" (because we don't skip the\n terminating character). This _is_ incorrect and would result\n in test failures.\n\nChanging that, we then see \"A\", \"B\", and \"C\" are added as keys.\n\nI'm going to take the style issues that you presented, and\nchange them in one commit (reported-by you) and then the\nif (!len) fix in a separate patch.\n\nI'll update the scan loops to use is_dir_sep() accordingly\nin this patch.\n\nThanks,\n-Stolee\n"},{"id":"400398","messageId":"57002040bc83668d4998da6a1e34a6efe287ae22.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 01/11] commit-graph: place bloom_settings in context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:00Z","receivedAt":"2020-06-23T17:47:15Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPlace an instance of struct bloom_settings into the struct\nwrite_commit_graph_context. This allows simplifying the function\nprototype of write_graph_chunk_bloom_data(). This will allow us\nto combine the function prototypes and use function pointers to\nsimplify write_commit_graph_file().\n\nBy using a pointer, we can later replace the settings to match those\nthat exist in the current commit-graph, in case a future Git version\nallows customization of these parameters.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 887837e882..d0fedcd9b1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -882,6 +882,7 @@ struct write_commit_graph_context {\n \n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n+\tconst struct bloom_filter_settings *bloom_settings;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -1103,8 +1104,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n }\n \n static void write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx,\n-\t\t\t\t\t const struct bloom_filter_settings *settings)\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1116,9 +1116,9 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t\t\t_(\"Writing changed paths Bloom filters data\"),\n \t\t\tctx->commits.nr);\n \n-\thashwrite_be32(f, settings->hash_version);\n-\thashwrite_be32(f, settings->num_hashes);\n-\thashwrite_be32(f, settings->bits_per_entry);\n+\thashwrite_be32(f, ctx->bloom_settings->hash_version);\n+\thashwrite_be32(f, ctx->bloom_settings->num_hashes);\n+\thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n@@ -1541,6 +1541,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tstruct object_id file_hash;\n \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n+\tctx->bloom_settings = &bloom_settings;\n+\n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n \n@@ -1642,7 +1644,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n \tif (ctx->changed_paths) {\n \t\twrite_graph_chunk_bloom_indexes(f, ctx);\n-\t\twrite_graph_chunk_bloom_data(f, ctx, &bloom_settings);\n+\t\twrite_graph_chunk_bloom_data(f, ctx);\n \t}\n \tif (ctx->num_commit_graphs_after > 1 &&\n \t write_graph_chunk_base(f, ctx)) {\n-- \ngitgitgadget\n\n"},{"id":"400399","messageId":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.git.1592252093.gitgitgadget@gmail.com","subject":"[PATCH v2 00/11] More commit-graph/Bloom filter improvements","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:46:59Z","receivedAt":"2020-06-23T17:47:18Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This builds on sg/commit-graph-cleanups, which took several patches from\nSzeder's series [1] and applied them almost directly to a more-recent\nversion of Git [2].\n\n[1] https://lore.kernel.org/git/20200529085038.26008-1-szeder.dev@gmail.com/\n[2] \nhttps://lore.kernel.org/git/pull.650.git.1591362032.gitgitgadget@gmail.com/\n\nThis series adds a few extra improvements, several of which are rooted in\nSzeder's original series. I maintained his authorship and sign-off, even\nthough the patches did not apply or cherry-pick at all.\n\n(In v2, I have removed the range-diff comparison to Szeder's series, so look\nat the v1 cover letter for that.)\n\nThe patches have been significantly reordered. René pointed out (and Szeder\ndiscovered in the old thread) that we are not re-using the\nbloom_filter_settings from the existing commit-graph when writing a new one.\n\n 1. commit-graph: place bloom_settings in context\n 2. commit-graph: change test to die on parse, not load\n\nThese are mostly the same, except we now use a pointer to the settings in\nthe commit-graph write context.\n\n 3. bloom: get_bloom_filter() cleanups\n\nThis new patch is a subtle change in behavior that will become relevant in\nthe very next patch. In fact, if we swap patch 3 and 4, then\nt4216-log-bloom.sh fails with a segfault due to a NULL filter.\n\n 4. commit-graph: persist existence of changed-paths\n\nThis patch is now updated to use the existing changed-path filter settings.\n\n 5. commit-graph: unify the signatures of all write_graph_chunk_*()\n functions\n 6. commit-graph: simplify chunk writes into loop\n 7. commit-graph: check chunk sizes after writing\n\nThese are all the same as before.\n\n 8. revision.c: fix whitespace\n\nThis patch is the cleanup part of Taylor's patch.\n\n 9. revision: empty pathspecs should not use Bloom filters\n\nHere is Taylor's fix for empty pathspecs.\n\n 10. commit-graph: check all leading directories in changed path Bloom\n filters\n 11. bloom: enforce a minimum size of 8 bytes\n\nFinally, we get these performance patches. Patch 10 is updated to have the\nbetter logic around directory separators and empty paths. Also, the list of\nBloom keys is ordered with the deepest path first. That has some tiny\nperformance benefits for deep paths since we can short-circuit the multi-key\nchecks more often. That code path is much faster than the tree parsing, so\nit is hard to measure any change.\n\nThanks, -Stolee\n\nDerrick Stolee (6):\n commit-graph: place bloom_settings in context\n commit-graph: change test to die on parse, not load\n bloom: get_bloom_filter() cleanups\n commit-graph: persist existence of changed-paths\n revision.c: fix whitespace\n bloom: enforce a minimum size of 8 bytes\n\nSZEDER Gábor (4):\n commit-graph: unify the signatures of all write_graph_chunk_*()\n functions\n commit-graph: simplify chunk writes into loop\n commit-graph: check chunk sizes after writing\n commit-graph: check all leading directories in changed path Bloom\n filters\n\nTaylor Blau (1):\n revision: empty pathspecs should not use Bloom filters\n\n Documentation/git-commit-graph.txt | 5 +-\n bloom.c | 19 ++--\n builtin/commit-graph.c | 5 +-\n commit-graph.c | 136 +++++++++++++++++++++--------\n commit-graph.h | 3 +-\n revision.c | 53 ++++++++---\n revision.h | 6 +-\n t/t4216-log-bloom.sh | 35 +++++++-\n t/t5318-commit-graph.sh | 2 +-\n 9 files changed, 200 insertions(+), 64 deletions(-)\n\n\nbase-commit: 7fbfe07ab4d4e58c0971dac73001b89f180a0af3\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-659%2Fderrickstolee%2Fbloom-2-v2\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-659/derrickstolee/bloom-2-v2\nPull-Request: https://github.com/gitgitgadget/git/pull/659\n\nRange-diff vs v1:\n\n 1: c966969071 ! 1: 57002040bc commit-graph: place bloom_settings in context\n @@ Commit message\n to combine the function prototypes and use function pointers to\n simplify write_commit_graph_file().\n \n + By using a pointer, we can later replace the settings to match those\n + that exist in the current commit-graph, in case a future Git version\n + allows customization of these parameters.\n +\n Reported-by: SZEDER Gábor \n Signed-off-by: Derrick Stolee \n \n @@ commit-graph.c: struct write_commit_graph_context {\n \n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n -+\tstruct bloom_filter_settings bloom_settings;\n ++\tconst struct bloom_filter_settings *bloom_settings;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n @@ commit-graph.c: static void write_graph_chunk_bloom_data(struct hashfile *f,\n -\thashwrite_be32(f, settings->hash_version);\n -\thashwrite_be32(f, settings->num_hashes);\n -\thashwrite_be32(f, settings->bits_per_entry);\n -+\thashwrite_be32(f, ctx->bloom_settings.hash_version);\n -+\thashwrite_be32(f, ctx->bloom_settings.num_hashes);\n -+\thashwrite_be32(f, ctx->bloom_settings.bits_per_entry);\n ++\thashwrite_be32(f, ctx->bloom_settings->hash_version);\n ++\thashwrite_be32(f, ctx->bloom_settings->num_hashes);\n ++\thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con\n \tstruct object_id file_hash;\n \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n -+\tctx->bloom_settings = bloom_settings;\n ++\tctx->bloom_settings = &bloom_settings;\n +\n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n 7: 60bbc15d24 = 2: 6b63f9bd8a commit-graph: change test to die on parse, not load\n -: ---------- > 3: 492deaf916 bloom: get_bloom_filter() cleanups\n 8: db5b8fe843 ! 4: 8727b25468 commit-graph: persist existence of changed-paths\n @@ Commit message\n property of \"my commit-graph has changed-path filters\" automatically. A\n user can drop filters using the --no-changed-paths option.\n \n + In the process, we need to be extremely careful to match the Bloom\n + filter settings as specified by the commit-graph. This will allow future\n + versions of Git to customize these settings, and the version with this\n + change will persist those settings as commit-graphs are rewritten on\n + top.\n +\n + Use the trace2 API to signal the settings used during the write, and\n + check that output in a test after manually adjusting the correct bytes\n + in the commit-graph file.\n +\n Signed-off-by: Derrick Stolee \n \n ## Documentation/git-commit-graph.txt ##\n @@ builtin/commit-graph.c: static int graph_write(int argc, const char **argv)\n \n \n ## commit-graph.c ##\n +@@\n + #include \"progress.h\"\n + #include \"bloom.h\"\n + #include \"commit-slab.h\"\n ++#include \"json-writer.h\"\n ++#include \"trace2.h\"\n + \n + void git_test_write_commit_graph_or_die(void)\n + {\n +@@ commit-graph.c: static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n + \tstop_progress(&progress);\n + }\n + \n ++static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n ++{\n ++\tstruct json_writer jw = JSON_WRITER_INIT;\n ++\n ++\tjw_object_begin(&jw, 0);\n ++\tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n ++\tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n ++\tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n ++\tjw_end(&jw);\n ++\n ++\ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n ++\n ++\tjw_release(&jw);\n ++}\n ++\n + static void write_graph_chunk_bloom_data(struct hashfile *f,\n + \t\t\t\t\t struct write_commit_graph_context *ctx)\n + {\n +@@ commit-graph.c: static void write_graph_chunk_bloom_data(struct hashfile *f,\n + \tstruct progress *progress = NULL;\n + \tint i = 0;\n + \n ++\ttrace2_bloom_filter_settings(ctx);\n ++\n + \tif (ctx->report_progress)\n + \t\tprogress = start_delayed_progress(\n + \t\t\t_(\"Writing changed paths Bloom filters data\"),\n +@@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n + \tstruct object_id file_hash;\n + \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n + \n +-\tctx->bloom_settings = &bloom_settings;\n ++\tif (!ctx->bloom_settings)\n ++\t\tctx->bloom_settings = &bloom_settings;\n + \n + \tif (ctx->split) {\n + \t\tstruct strbuf tmp_file = STRBUF_INIT;\n @@ commit-graph.c: int write_commit_graph(struct object_directory *odb,\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n \tctx->check_oids = flags & COMMIT_GRAPH_WRITE_CHECK_OIDS ? 1 : 0;\n @@ commit-graph.c: int write_commit_graph(struct object_directory *odb,\n \n +\tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n +\t\tctx->changed_paths = 1;\n -+\telse if (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\n ++\tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\n ++\t\tstruct commit_graph *g;\n +\t\tprepare_commit_graph_one(ctx->r, ctx->odb);\n +\n ++\t\tg = ctx->r->objects->commit_graph;\n ++\n +\t\t/* We have changed-paths already. Keep them in the next graph */\n -+\t\tif (ctx->r->objects->commit_graph &&\n -+\t\t ctx->r->objects->commit_graph->chunk_bloom_data)\n ++\t\tif (g && g->chunk_bloom_data) {\n +\t\t\tctx->changed_paths = 1;\n ++\t\t\tctx->bloom_settings = g->bloom_filter_settings;\n ++\t\t}\n +\t}\n +\n \tif (ctx->split) {\n @@ t/t4216-log-bloom.sh: test_expect_success 'setup - add commit-graph to the chain\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n +@@ t/t4216-log-bloom.sh: test_expect_success 'Use Bloom filters if they exist in the latest but not all c\n + \ttest_bloom_filters_used_when_some_filters_are_missing \"-- A/B\"\n + '\n + \n ++BASE_BDAT_OFFSET=2240\n ++BASE_K_BYTE_OFFSET=$((BASE_BDAT_OFFSET + 10))\n ++BASE_LEN_BYTE_OFFSET=$((BASE_BDAT_OFFSET + 14))\n ++\n ++corrupt_graph() {\n ++\tpos=$1\n ++\tdata=\"${2:-\\0}\"\n ++\tgrepstr=$3\n ++\torig_size=$(wc -c < .git/objects/info/commit-graph) &&\n ++\tzero_pos=${4:-${orig_size}} &&\n ++\tprintf \"$data\" | dd of=\".git/objects/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n ++\tdd of=\".git/objects/info/commit-graph\" bs=1 seek=\"$zero_pos\" if=/dev/null\n ++}\n ++\n ++test_expect_success 'persist filter settings' '\n ++\ttest_when_finished rm -rf .git/objects/info/commit-graph* &&\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n ++\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":7,\\\"bits_per_entry\\\":10}\" trace2.txt &&\n ++\tcp .git/objects/info/commit-graph commit-graph-before &&\n ++\tcorrupt_graph $BASE_K_BYTE_OFFSET \"\\09\" &&\n ++\tcorrupt_graph $BASE_LEN_BYTE_OFFSET \"\\0F\" &&\n ++\tcp .git/objects/info/commit-graph commit-graph-after &&\n ++\ttest_commit c18 A/corrupt &&\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n ++\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":57,\\\"bits_per_entry\\\":70}\" trace2.txt\n ++'\n ++\n + test_done\n + \\ No newline at end of file\n 2: 65eb15221c ! 5: 244668fec4 commit-graph: unify the signatures of all write_graph_chunk_*() functions\n @@ Commit message\n \n ## commit-graph.c ##\n @@ commit-graph.c: struct write_commit_graph_context {\n - \tstruct bloom_filter_settings bloom_settings;\n + \tconst struct bloom_filter_settings *bloom_settings;\n };\n \n -static void write_graph_chunk_fanout(struct hashfile *f,\n @@ commit-graph.c: static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n +\treturn 0;\n }\n \n + static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n +@@ commit-graph.c: static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n + \tjw_release(&jw);\n + }\n + \n -static void write_graph_chunk_bloom_data(struct hashfile *f,\n -\t\t\t\t\t struct write_commit_graph_context *ctx)\n +static int write_graph_chunk_bloom_data(struct hashfile *f,\n 3: 3d24b9802d = 6: 8b959f2f37 commit-graph: simplify chunk writes into loop\n 4: bdca834e6d = 7: 3eb10933dc commit-graph: check chunk sizes after writing\n -: ---------- > 8: 0bcfc1f051 revision.c: fix whitespace\n -: ---------- > 9: 719c7091a7 revision: empty pathspecs should not use Bloom filters\n 5: 9975fc96f1 ! 10: 9c2076b4ce commit-graph: check all leading directories in changed path Bloom filters\n @@ Commit message\n This adjusts the tracing values in t4216-log-bloom.sh, which provides a\n concrete way to notice the improvement.\n \n + Helped-by: Taylor Blau \n + Helped-by: René Scharfe \n Signed-off-by: SZEDER Gábor \n Signed-off-by: Derrick Stolee \n \n @@ revision.c: static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \tint last_index;\n -\tint len;\n +\tsize_t len;\n -+\tint path_component_nr = 0, j;\n ++\tint path_component_nr = 1;\n \n \tif (!revs->commits)\n \t\treturn;\n @@ revision.c: static void prepare_to_use_bloom_filter(struct rev_info *revs)\n - \n - \tlen = strlen(path);\n + \t\treturn;\n + \t}\n \n -\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n -\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n +\tp = path;\n -+\tdo {\n -+\t\tp = strchrnul(p + 1, '/');\n -+\t\tpath_component_nr++;\n -+\t} while (p - path < len);\n ++\twhile (*p) {\n ++\t\tif (is_dir_sep(*p))\n ++\t\t\tpath_component_nr++;\n ++\t\tp++;\n ++\t}\n +\n +\trevs->bloom_keys_nr = path_component_nr;\n +\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n +\n -+\tp = path;\n -+\tfor (j = 0; j < revs->bloom_keys_nr; j++) {\n -+\t\tp = strchrnul(p + 1, '/');\n ++\tfill_bloom_key(path, len, &revs->bloom_keys[0],\n ++\t\t revs->bloom_filter_settings);\n ++\tpath_component_nr = 1;\n +\n -+\t\tfill_bloom_key(path, p - path, &revs->bloom_keys[j],\n -+\t\t\t revs->bloom_filter_settings);\n ++\tp = path + len - 1;\n ++\twhile (p > path) {\n ++\t\tif (is_dir_sep(*p))\n ++\t\t\tfill_bloom_key(path, p - path,\n ++\t\t\t\t &revs->bloom_keys[path_component_nr++],\n ++\t\t\t\t revs->bloom_filter_settings);\n ++\t\tp--;\n +\t}\n \n \tif (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n 6: 2a5f1e1752 = 11: 1022c0ad21 bloom: enforce a minimum size of 8 bytes\n\n-- \ngitgitgadget\n"},{"id":"400401","messageId":"492deaf916464abedc7fc2d3de41fe690a558d9b.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 03/11] bloom: get_bloom_filter() cleanups","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:02Z","receivedAt":"2020-06-23T17:47:19Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe get_bloom_filter() method is a bit complicated in some parts where\nit does not need to be. In particular, it needs to return a NULL filter\nonly when compute_if_not_present is zero AND the filter data cannot be\nloaded from a commit-graph file. This currently happens by accident\nbecause the commit-graph does not load changed-path Bloom filters from\nan existing commit-graph when writing a new one. This will change in a\nlater patch.\n\nAlso clean up some style issues while we are here.\n\nSigned-off-by: Derrick Stolee \n---\n bloom.c | 15 +++++++--------\n 1 file changed, 7 insertions(+), 8 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex c38d1cff0c..7291506564 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -186,7 +186,7 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n-\tif (bloom_filters.slab_size == 0)\n+\tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n \tfilter = bloom_filter_slab_at(&bloom_filters, c);\n@@ -194,16 +194,15 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (c->graph_pos != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes) {\n-\t\t\tif (load_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n-\t\t\telse\n-\t\t\t\treturn NULL;\n-\t\t}\n+\t\t r->objects->commit_graph->chunk_bloom_indexes &&\n+\t\t load_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n+\t\t\treturn filter;\n \t}\n \n-\tif (filter->data || !compute_if_not_present)\n+\tif (filter->data)\n \t\treturn filter;\n+\tif (!filter->data && !compute_if_not_present)\n+\t\treturn NULL;\n \n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n-- \ngitgitgadget\n\n"},{"id":"400400","messageId":"6b63f9bd8a2a7e18d7ac1be7066d4bcd1df2a729.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 02/11] commit-graph: change test to die on parse, not load","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:01Z","receivedAt":"2020-06-23T17:47:20Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n43d3561 (commit-graph write: don't die if the existing graph is corrupt,\n2019-03-25) introduced the GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD environment\nvariable. This was created to verify that commit-graph was not loaded\nwhen writing a new non-incremental commit-graph.\n\nAn upcoming change wants to load a commit-graph in some valuable cases,\nbut we want to maintain that we don't trust the commit-graph data when\nwriting our new file. Instead of dying on load, instead die if we ever\ntry to parse a commit from the commit-graph. This functionally verifies\nthe same intended behavior, but allows a more advanced feature in the\nnext change.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 ++++++++----\n commit-graph.h | 2 +-\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d0fedcd9b1..6a28d4a5a6 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -564,10 +564,6 @@ static int prepare_commit_graph(struct repository *r)\n \t\treturn !!r->objects->commit_graph;\n \tr->objects->commit_graph_attempted = 1;\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD, 0))\n-\t\tdie(\"dying as requested by the '%s' variable on commit-graph load!\",\n-\t\t GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD);\n-\n \tprepare_repo_settings(r);\n \n \tif (!git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n@@ -790,6 +786,14 @@ static int parse_commit_in_graph_one(struct repository *r,\n \n int parse_commit_in_graph(struct repository *r, struct commit *item)\n {\n+\tstatic int checked_env = 0;\n+\n+\tif (!checked_env &&\n+\t git_env_bool(GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE, 0))\n+\t\tdie(\"dying as requested by the '%s' variable on commit-graph parse!\",\n+\t\t GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE);\n+\tchecked_env = 1;\n+\n \tif (!prepare_commit_graph(r))\n \t\treturn 0;\n \treturn parse_commit_in_graph_one(r, r->objects->commit_graph, item);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 881c9b46e5..f0fb13e3f2 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -5,7 +5,7 @@\n #include \"object-store.h\"\n \n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n-#define GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD \"GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD\"\n+#define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n #define GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS \"GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS\"\n \n /*\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 1073f9e3cf..5ec01abdaa 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -436,7 +436,7 @@ corrupt_graph_verify() {\n \t\tcp $objdir/info/commit-graph commit-graph-pre-write-test\n \tfi &&\n \tgit status --short &&\n-\tGIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD=true git commit-graph write &&\n+\tGIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE=true git commit-graph write &&\n \tgit commit-graph verify\n }\n \n-- \ngitgitgadget\n\n"},{"id":"400402","messageId":"8727b254680fdbc4f42f8b46c6ec686e7fa33fbe.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 04/11] commit-graph: persist existence of changed-paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:03Z","receivedAt":"2020-06-23T17:47:22Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe changed-path Bloom filters were released in v2.27.0, but have a\nsignificant drawback. A user can opt-in to writing the changed-path\nfilters using the \"--changed-paths\" option to \"git commit-graph write\"\nbut the next write will drop the filters unless that option is\nspecified.\n\nThis becomes even more important when considering the interaction with\ngc.writeCommitGraph (on by default) or fetch.writeCommitGraph (part of\nfeatures.experimental). These config options trigger commit-graph writes\nthat the user did not signal, and hence there is no --changed-paths\noption available.\n\nAllow a user that opts-in to the changed-path filters to persist the\nproperty of \"my commit-graph has changed-path filters\" automatically. A\nuser can drop filters using the --no-changed-paths option.\n\nIn the process, we need to be extremely careful to match the Bloom\nfilter settings as specified by the commit-graph. This will allow future\nversions of Git to customize these settings, and the version with this\nchange will persist those settings as commit-graphs are rewritten on\ntop.\n\nUse the trace2 API to signal the settings used during the write, and\ncheck that output in a test after manually adjusting the correct bytes\nin the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 5 +++-\n builtin/commit-graph.c | 5 +++-\n commit-graph.c | 38 ++++++++++++++++++++++++++++--\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 29 ++++++++++++++++++++++-\n 5 files changed, 73 insertions(+), 5 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex f4b13c005b..369b222b08 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -60,7 +60,10 @@ existing commit-graph file.\n With the `--changed-paths` option, compute and write information about the\n paths changed between a commit and it's first parent. This operation can\n take a while on large repositories. It provides significant performance gains\n-for getting history of a directory or a file with `git log -- `.\n+for getting history of a directory or a file with `git log -- `. If\n+this option is given, future commit-graph writes will automatically assume\n+that this option was intended. Use `--no-changed-paths` to stop storing this\n+data.\n +\n With the `--split` option, write the commit-graph as a chain of multiple\n commit-graph files stored in `/info/commit-graphs`. The new commits\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 59009837dc..ff7b177c33 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -151,6 +151,7 @@ static int graph_write(int argc, const char **argv)\n \t};\n \n \topts.progress = isatty(2);\n+\topts.enable_changed_paths = -1;\n \tsplit_opts.size_multiple = 2;\n \tsplit_opts.max_commits = 0;\n \tsplit_opts.expire_time = 0;\n@@ -171,7 +172,9 @@ static int graph_write(int argc, const char **argv)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n \tif (opts.progress)\n \t\tflags |= COMMIT_GRAPH_WRITE_PROGRESS;\n-\tif (opts.enable_changed_paths ||\n+\tif (!opts.enable_changed_paths)\n+\t\tflags |= COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS;\n+\tif (opts.enable_changed_paths == 1 ||\n \t git_env_bool(GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS, 0))\n \t\tflags |= COMMIT_GRAPH_WRITE_BLOOM_FILTERS;\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6a28d4a5a6..908f094271 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -16,6 +16,8 @@\n #include \"progress.h\"\n #include \"bloom.h\"\n #include \"commit-slab.h\"\n+#include \"json-writer.h\"\n+#include \"trace2.h\"\n \n void git_test_write_commit_graph_or_die(void)\n {\n@@ -1107,6 +1109,21 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tstop_progress(&progress);\n }\n \n+static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n+{\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n+\tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n+\tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_end(&jw);\n+\n+\ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n+\n+\tjw_release(&jw);\n+}\n+\n static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n@@ -1115,6 +1132,8 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \tstruct progress *progress = NULL;\n \tint i = 0;\n \n+\ttrace2_bloom_filter_settings(ctx);\n+\n \tif (ctx->report_progress)\n \t\tprogress = start_delayed_progress(\n \t\t\t_(\"Writing changed paths Bloom filters data\"),\n@@ -1545,7 +1564,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tstruct object_id file_hash;\n \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n-\tctx->bloom_settings = &bloom_settings;\n+\tif (!ctx->bloom_settings)\n+\t\tctx->bloom_settings = &bloom_settings;\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -1970,9 +1990,23 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n \tctx->check_oids = flags & COMMIT_GRAPH_WRITE_CHECK_OIDS ? 1 : 0;\n \tctx->split_opts = split_opts;\n-\tctx->changed_paths = flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS ? 1 : 0;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n+\t\tctx->changed_paths = 1;\n+\tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\n+\t\tstruct commit_graph *g;\n+\t\tprepare_commit_graph_one(ctx->r, ctx->odb);\n+\n+\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\t/* We have changed-paths already. Keep them in the next graph */\n+\t\tif (g && g->chunk_bloom_data) {\n+\t\t\tctx->changed_paths = 1;\n+\t\t\tctx->bloom_settings = g->bloom_filter_settings;\n+\t\t}\n+\t}\n+\n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n \t\tprepare_commit_graph(ctx->r);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex f0fb13e3f2..45b1e5bca3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -96,6 +96,7 @@ enum commit_graph_write_flags {\n \t/* Make sure that each OID in the input is a valid commit OID. */\n \tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3),\n \tCOMMIT_GRAPH_WRITE_BLOOM_FILTERS = (1 << 4),\n+\tCOMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS = (1 << 5),\n };\n \n struct split_commit_graph_opts {\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c7011f33e2..426de10041 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -126,7 +126,7 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_commit c14 A/anotherFile2 &&\n \ttest_commit c15 A/B/anotherFile2 &&\n \ttest_commit c16 A/B/C/anotherFile2 &&\n-\tGIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0 git commit-graph write --reachable --split &&\n+\tgit commit-graph write --reachable --split --no-changed-paths &&\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n@@ -152,4 +152,31 @@ test_expect_success 'Use Bloom filters if they exist in the latest but not all c\n \ttest_bloom_filters_used_when_some_filters_are_missing \"-- A/B\"\n '\n \n+BASE_BDAT_OFFSET=2240\n+BASE_K_BYTE_OFFSET=$((BASE_BDAT_OFFSET + 10))\n+BASE_LEN_BYTE_OFFSET=$((BASE_BDAT_OFFSET + 14))\n+\n+corrupt_graph() {\n+\tpos=$1\n+\tdata=\"${2:-\\0}\"\n+\tgrepstr=$3\n+\torig_size=$(wc -c < .git/objects/info/commit-graph) &&\n+\tzero_pos=${4:-${orig_size}} &&\n+\tprintf \"$data\" | dd of=\".git/objects/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n+\tdd of=\".git/objects/info/commit-graph\" bs=1 seek=\"$zero_pos\" if=/dev/null\n+}\n+\n+test_expect_success 'persist filter settings' '\n+\ttest_when_finished rm -rf .git/objects/info/commit-graph* &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":7,\\\"bits_per_entry\\\":10}\" trace2.txt &&\n+\tcp .git/objects/info/commit-graph commit-graph-before &&\n+\tcorrupt_graph $BASE_K_BYTE_OFFSET \"\\09\" &&\n+\tcorrupt_graph $BASE_LEN_BYTE_OFFSET \"\\0F\" &&\n+\tcp .git/objects/info/commit-graph commit-graph-after &&\n+\ttest_commit c18 A/corrupt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":57,\\\"bits_per_entry\\\":70}\" trace2.txt\n+'\n+\n test_done\n\\ No newline at end of file\n-- \ngitgitgadget\n\n"},{"id":"400404","messageId":"244668fec4428872f441d515e570e923808f0db1.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 05/11] commit-graph: unify the signatures of all write_graph_chunk_*() functions","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:04Z","receivedAt":"2020-06-23T17:47:22Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nUpdate the write_graph_chunk_*() helper functions to have the same\nsignature:\n\n - Return an int error code from all these functions.\n write_graph_chunk_base() already has an int error code, now the\n others will have one, too, but since they don't indicate any\n error, they will always return 0.\n\n - Drop the hash size parameter of write_graph_chunk_oids() and\n write_graph_chunk_data(); its value can be read directly from\n 'the_hash_algo' inside these functions as well.\n\nThis opens up the possibility for further cleanups and foolproofing in\nthe following two patches.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 42 ++++++++++++++++++++++++++----------------\n 1 file changed, 26 insertions(+), 16 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 908f094271..f33bfe49b3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -891,8 +891,8 @@ struct write_commit_graph_context {\n \tconst struct bloom_filter_settings *bloom_settings;\n };\n \n-static void write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_fanout(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tint i, count = 0;\n \tstruct commit **list = ctx->commits.list;\n@@ -913,17 +913,21 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \n \t\thashwrite_be32(f, count);\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_oids(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tint count;\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n-\t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t\thashwrite(f, (*list)->object.oid.hash, (int)the_hash_algo->rawsz);\n \t}\n+\n+\treturn 0;\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -932,8 +936,8 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n \treturn commits[index]->object.oid.hash;\n }\n \n-static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_data(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -950,7 +954,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tdie(_(\"unable to parse commit %s\"),\n \t\t\t\toid_to_hex(&(*list)->object.oid));\n \t\ttree = get_commit_tree_oid(*list);\n-\t\thashwrite(f, tree->hash, hash_len);\n+\t\thashwrite(f, tree->hash, the_hash_algo->rawsz);\n \n \t\tparent = (*list)->parents;\n \n@@ -1030,10 +1034,12 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\tlist++;\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_extra_edges(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1082,10 +1088,12 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \n \t\tlist++;\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1107,6 +1115,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n \t}\n \n \tstop_progress(&progress);\n+\treturn 0;\n }\n \n static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n@@ -1124,8 +1133,8 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_release(&jw);\n }\n \n-static void write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_bloom_data(struct hashfile *f,\n+\t\t\t\t\tstruct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1151,6 +1160,7 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t}\n \n \tstop_progress(&progress);\n+\treturn 0;\n }\n \n static int oid_compare(const void *_a, const void *_b)\n@@ -1662,8 +1672,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, hashsz, ctx);\n-\twrite_graph_chunk_data(f, hashsz, ctx);\n+\twrite_graph_chunk_oids(f, ctx);\n+\twrite_graph_chunk_data(f, ctx);\n \tif (ctx->num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n \tif (ctx->changed_paths) {\n-- \ngitgitgadget\n\n"},{"id":"400403","messageId":"8b959f2f374654aeb87b847560761890c2f9aa2c.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 06/11] commit-graph: simplify chunk writes into loop","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:05Z","receivedAt":"2020-06-23T17:47:23Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn write_commit_graph_file() we now have one block of code filling the\narray of 'struct chunk_info' with the IDs and sizes of chunks to be\nwritten, and an other block of code calling the functions responsible\nfor writing individual chunks. In case of optional chunks like Extra\nEdge List an Base Graphs List there is also a condition checking\nwhether that chunk is necessary/desired, and that same condition is\nrepeated in both blocks of code. Other, newer chunks have similar\noptional conditions.\n\nEliminate these repeated conditions by storing the function pointers\nresponsible for writing individual chunks in the 'struct chunk_info'\narray as well, and calling them in a loop to write the commit-graph\nfile. This will open up the possibility for a bit of foolproofing in\nthe following patch.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 31 +++++++++++++++++++------------\n 1 file changed, 19 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex f33bfe49b3..086fc2d070 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1555,9 +1555,13 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n+typedef int (*chunk_write_fn)(struct hashfile *f,\n+\t\t\t struct write_commit_graph_context *ctx);\n+\n struct chunk_info {\n \tuint32_t id;\n \tuint64_t size;\n+\tchunk_write_fn write_fn;\n };\n \n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n@@ -1615,27 +1619,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n \tchunks[0].size = GRAPH_FANOUT_SIZE;\n+\tchunks[0].write_fn = write_graph_chunk_fanout;\n \tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunks[1].size = hashsz * ctx->commits.nr;\n+\tchunks[1].write_fn = write_graph_chunk_oids;\n \tchunks[2].id = GRAPH_CHUNKID_DATA;\n \tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n+\tchunks[2].write_fn = write_graph_chunk_data;\n \tif (ctx->num_extra_edges) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n \t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n \t\tnum_chunks++;\n \t}\n \tif (ctx->changed_paths) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n \t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n \t\tnum_chunks++;\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n \t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n \t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n \t\tnum_chunks++;\n \t}\n \n@@ -1671,19 +1682,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tprogress_title.buf,\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n-\twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, ctx);\n-\twrite_graph_chunk_data(f, ctx);\n-\tif (ctx->num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, ctx);\n-\tif (ctx->changed_paths) {\n-\t\twrite_graph_chunk_bloom_indexes(f, ctx);\n-\t\twrite_graph_chunk_bloom_data(f, ctx);\n-\t}\n-\tif (ctx->num_commit_graphs_after > 1 &&\n-\t write_graph_chunk_base(f, ctx)) {\n-\t\treturn -1;\n+\n+\tfor (i = 0; i < num_chunks; i++) {\n+\t\tif (chunks[i].write_fn(f, ctx)) {\n+\t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n+\t\t\t chunks[i].id);\n+\t\t\treturn -1;\n+\t\t}\n \t}\n+\n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n-- \ngitgitgadget\n\n"},{"id":"400406","messageId":"0bcfc1f05126767ab6d6125a2b04d6d7392eb519.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 08/11] revision.c: fix whitespace","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:07Z","receivedAt":"2020-06-23T17:47:25Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nHere, four spaces were used instead of tab characters.\n\nReported-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n revision.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex c644c66091..ed59084f50 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -697,11 +697,11 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \t/* remove single trailing slash from path, if needed */\n \tif (pi->match[last_index] == '/') {\n-\t path_alloc = xstrdup(pi->match);\n-\t path_alloc[last_index] = '\\0';\n-\t path = path_alloc;\n+\t\tpath_alloc = xstrdup(pi->match);\n+\t\tpath_alloc[last_index] = '\\0';\n+\t\tpath = path_alloc;\n \t} else\n-\t path = pi->match;\n+\t\tpath = pi->match;\n \n \tlen = strlen(path);\n \n-- \ngitgitgadget\n\n"},{"id":"400405","messageId":"719c7091a7a48434ce7534c5b617f4238d96e22d.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 09/11] revision: empty pathspecs should not use Bloom filters","fromName":"Taylor Blau via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:08Z","receivedAt":"2020-06-23T17:47:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Taylor Blau \n\nThe prepare_to_use_bloom_filter() method was not intended to be called\non an empty pathspec. However, 'git log -- .' and 'git log' are subtly\ndifferent: the latter reports all commits while the former will simplify\ncommits that do not change the root tree.\n\nThis means that the path used to construct the bloom_key might be empty,\nand that value is not added to the Bloom filter during construction.\nThat means that the results are likely incorrect!\n\nTo resolve the issue, be careful about the length of the path and stop\nfilling Bloom filters. To be completely sure we do not use them, drop\nthe pointer to the bloom_filter_settings from the commit-graph. That\nallows our test to look at the trace2 logs to verify no Bloom filter\nstatistics are reported.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n revision.c | 4 ++++\n t/t4216-log-bloom.sh | 4 ++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex ed59084f50..b53377cd52 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -704,6 +704,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \t\tpath = pi->match;\n \n \tlen = strlen(path);\n+\tif (!len) {\n+\t\trevs->bloom_filter_settings = NULL;\n+\t\treturn;\n+\t}\n \n \trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n \tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 426de10041..f890cc4737 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -112,6 +112,10 @@ test_expect_success 'git log -- multiple path specs does not use Bloom filters'\n \ttest_bloom_filters_not_used \"-- file4 A/file1\"\n '\n \n+test_expect_success 'git log -- \".\" pathspec at root does not use Bloom filters' '\n+\ttest_bloom_filters_not_used \"-- .\"\n+'\n+\n test_expect_success 'git log with wildcard that resolves to a single path uses Bloom filters' '\n \ttest_bloom_filters_used \"-- *4\" &&\n \ttest_bloom_filters_used \"-- *renamed\"\n-- \ngitgitgadget\n\n"},{"id":"400408","messageId":"9c2076b4ce46918fce8f05e609b057611ec56e13.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 10/11] commit-graph: check all leading directories in changed path Bloom filters","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:09Z","receivedAt":"2020-06-23T17:47:28Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nThe file 'dir/subdir/file' can only be modified if its leading\ndirectories 'dir' and 'dir/subdir' are modified as well.\n\nSo when checking modified path Bloom filters looking for commits\nmodifying a path with multiple path components, then check not only\nthe full path in the Bloom filters, but all its leading directories as\nwell. Take care to check these paths in \"deepest first\" order,\nbecause it's the full path that is least likely to be modified, and\nthe Bloom filter queries can short circuit sooner.\n\nThis can significantly reduce the average false positive rate, by\nabout an order of magnitude or three(!), and can further speed up\npathspec-limited revision walks. The table below compares the average\nfalse positive rate and runtime of\n\n git rev-list HEAD -- \"$path\"\n\nbefore and after this change for 5000+ randomly* selected paths from\neach repository:\n\n Average false Average Average\n positive rate runtime runtime\n before after before after difference\n ------------------------------------------------------------------\n git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n\n*Path selection was done with the following pipeline:\n\n\tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n\nThe improvements in runtime are much smaller than the improvements in\naverage false positive rate, as we are clearly reaching diminishing\nreturns here. However, all these timings depend on that accessing\ntree objects is reasonably fast (warm caches). If we had a partial\nclone and the tree objects had to be fetched from a promisor remote,\ne.g.:\n\n $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n commit-graph write --reachable\n $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n rev-list HEAD -- \"$path\"\n\nthen checking all leading path component can reduce the runtime from\nover an hour to a few seconds (and this is with the clone and the\npromisor on the same machine).\n\nThis adjusts the tracing values in t4216-log-bloom.sh, which provides a\nconcrete way to notice the improvement.\n\nHelped-by: Taylor Blau \nHelped-by: René Scharfe \nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n revision.c | 41 ++++++++++++++++++++++++++++++++---------\n revision.h | 6 ++++--\n t/t4216-log-bloom.sh | 2 +-\n 3 files changed, 37 insertions(+), 12 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex b53377cd52..077888ee51 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n {\n \tstruct pathspec_item *pi;\n \tchar *path_alloc = NULL;\n-\tconst char *path;\n+\tconst char *path, *p;\n \tint last_index;\n-\tint len;\n+\tsize_t len;\n+\tint path_component_nr = 1;\n \n \tif (!revs->commits)\n \t\treturn;\n@@ -709,8 +710,28 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \t\treturn;\n \t}\n \n-\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n-\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n+\tp = path;\n+\twhile (*p) {\n+\t\tif (is_dir_sep(*p))\n+\t\t\tpath_component_nr++;\n+\t\tp++;\n+\t}\n+\n+\trevs->bloom_keys_nr = path_component_nr;\n+\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n+\n+\tfill_bloom_key(path, len, &revs->bloom_keys[0],\n+\t\t revs->bloom_filter_settings);\n+\tpath_component_nr = 1;\n+\n+\tp = path + len - 1;\n+\twhile (p > path) {\n+\t\tif (is_dir_sep(*p))\n+\t\t\tfill_bloom_key(path, p - path,\n+\t\t\t\t &revs->bloom_keys[path_component_nr++],\n+\t\t\t\t revs->bloom_filter_settings);\n+\t\tp--;\n+\t}\n \n \tif (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n \t\tatexit(trace2_bloom_filter_statistics_atexit);\n@@ -724,7 +745,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\t\t\t\t\t struct commit *commit)\n {\n \tstruct bloom_filter *filter;\n-\tint result;\n+\tint result = 1, j;\n \n \tif (!revs->repo->objects->commit_graph)\n \t\treturn -1;\n@@ -744,9 +765,11 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\treturn -1;\n \t}\n \n-\tresult = bloom_filter_contains(filter,\n-\t\t\t\t revs->bloom_key,\n-\t\t\t\t revs->bloom_filter_settings);\n+\tfor (j = 0; result && j < revs->bloom_keys_nr; j++) {\n+\t\tresult = bloom_filter_contains(filter,\n+\t\t\t\t\t &revs->bloom_keys[j],\n+\t\t\t\t\t revs->bloom_filter_settings);\n+\t}\n \n \tif (result)\n \t\tcount_bloom_filter_maybe++;\n@@ -786,7 +809,7 @@ static int rev_compare_tree(struct rev_info *revs,\n \t\t\treturn REV_TREE_SAME;\n \t}\n \n-\tif (revs->bloom_key && !nth_parent) {\n+\tif (revs->bloom_keys_nr && !nth_parent) {\n \t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, commit);\n \n \t\tif (bloom_ret == 0)\ndiff --git a/revision.h b/revision.h\nindex 7c026fe41f..abbfb4ab59 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -295,8 +295,10 @@ struct rev_info {\n \tstruct topo_walk_info *topo_walk_info;\n \n \t/* Commit graph bloom filter fields */\n-\t/* The bloom filter key for the pathspec */\n-\tstruct bloom_key *bloom_key;\n+\t/* The bloom filter key(s) for the pathspec */\n+\tstruct bloom_key *bloom_keys;\n+\tint bloom_keys_nr;\n+\n \t/*\n \t * The bloom filter settings used to generate the key.\n \t * This is loaded from the commit-graph being used.\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex f890cc4737..84f95972ca 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -146,7 +146,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n-- \ngitgitgadget\n\n"},{"id":"400407","messageId":"1022c0ad21379dae825ca8e47b3da8b62c0b59dc.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 11/11] bloom: enforce a minimum size of 8 bytes","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:10Z","receivedAt":"2020-06-23T17:47:31Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe original design of changed-path Bloom filters included an 8-byte\nblock size for filter lengths. This was changed mid-way through the\nsubmission process, and now the length stored in the commit-graph has\none-byte granularity.\n\nThis can cause some issues for very small filters. The analysis for\nfalse positive rates assume large filters, so rounding errors become\nless important at that scale. When there are only a few paths changed,\na filter that has size only a few bytes could have very different\nbehavior. In fact, this is evidenced in the Git repository due to the\ncode organization and careful patch creation that leads to many commits\nwith very small filters. These small filters frequently have\nfalse-positive rates in the 8-10% range or higher.\n\nThe previous change improved the false-positive rate using multiple\nBloom keys when the path has multiple directory components. However,\nthat does not help at all for files at root. It is typical to have\nseveral commits that change only the README at root, and those commits\nwould be likely to have these artificially high false-positive rates.\n\nCorrect this issue by creating a minimum filters size of 8 bytes. This\nrequires the very small commits (with fewer than six changes, including\nnon-root directories) to have a larger filter. In principle, this\nviolates the bits_per_entry value of struct bloom_filter_settings.\nHowever, it does not actually create a functional problem.\n\nAs for compatibility, this only affects new versions writing filters for\ncommits that do not yet have a filter. Old version will write the\nsmaller filters and this version will persist and properly read that\ndata. Now, the new files will be generated slightly larger.\n\n Bytes before Bytes after Difference\n --------------------------------------------------\n git 4,021,078 4,275,311 +6.32%\n linux 72,212,101 73,909,286 +2.35%\n tensorflow 7,596,359 7,691,646 +1.25%\n\nThis has a measurable improvement in the false-positive rate and the\nend-to-end run time for these repos. The table below compares the average\nfalse-positive rate and runtime of\n\n git rev-list HEAD -- \"$path\"\n\nbefore and after this change for 5000+ randomly* selected paths from\neach repository:\n\n Average false Average Average\n positive rate runtime runtime\n before after before after difference\n ------------------------------------------------------------------\n git 0.786% 0.227% 0.0387s 0.0289s -25.5%\n linux 0.0296% 0.0174% 0.0766s 0.0706s -7.8%\n tensorflow 0.6977% 0.0268% 0.0420s 0.0384s -8.5%\n\n*Path selection was done with the following pipeline:\n\n git ls-tree -r --name-only HEAD | sort -R | head -n 5000\n\nThese relatively-small increases in file size appear to be a fair price\nto pay for these performance improvements.\n\nSigned-off-by: Derrick Stolee \n---\n bloom.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/bloom.c b/bloom.c\nindex 7291506564..e9dc15976c 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -257,6 +257,10 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \t\t}\n \n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings.bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\n+\t\tif (filter->len && filter->len < 8)\n+\t\t\tfilter->len = 8;\n+\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n-- \ngitgitgadget\n"},{"id":"400409","messageId":"3eb10933dc8517ab01eff17a179e484068d29724.1592934430.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v2 07/11] commit-graph: check chunk sizes after writing","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-23T17:47:06Z","receivedAt":"2020-06-23T17:47:32Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn my experience while experimenting with new commit-graph chunks,\nearly versions of the corresponding new write_commit_graph_my_chunk()\nfunctions are, sadly but not surprisingly, often buggy, and write more\nor less data than they are supposed to, especially if the chunk size\nis not directly proportional to the number of commits. This then\ncauses all kinds of issues when reading such a bogus commit-graph\nfile, raising the question of whether the writing or the reading part\nhappens to be buggy this time.\n\nLet's catch such issues early, already when writing the commit-graph\nfile, and check that each write_graph_chunk_*() function wrote the\namount of data that it was expected to, and what has been encoded in\nthe Chunk Lookup table. Now that all commit-graph chunks are written\nin a loop we can do this check in a single place for all chunks, and\nany chunks added in the future will get checked as well.\n\nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 9 +++++++++\n 1 file changed, 9 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 086fc2d070..1de6800d74 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1683,12 +1683,21 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \n+\tchunk_offset = f->total + f->offset;\n \tfor (i = 0; i < num_chunks; i++) {\n+\t\tuint64_t end_offset;\n+\n \t\tif (chunks[i].write_fn(f, ctx)) {\n \t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n \t\t\t chunks[i].id);\n \t\t\treturn -1;\n \t\t}\n+\n+\t\tend_offset = f->total + f->offset;\n+\t\tif (end_offset - chunk_offset != chunks[i].size)\n+\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n+\t\t\t chunks[i].size, chunks[i].id, end_offset - chunk_offset);\n+\t\tchunk_offset = end_offset;\n \t}\n \n \tstop_progress(&ctx->progress);\n-- \ngitgitgadget\n\n"},{"id":"400564","messageId":"xmqqftak5aa0.fsf@gitster.c.googlers.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 00/11] More commit-graph/Bloom filter improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-06-24T23:11:51Z","receivedAt":"2020-06-24T23:12:06Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"This does not seem to play well with what is in flight. Tests seem\nto pass with topics up to es/config-hooks merged but not with this\ntopic merged on top.\n\n 1b5d3d8260 Merge branch 'ds/commit-graph-bloom-updates' into seen\n 32169c595c Merge branch 'es/config-hooks' into seen\n ...\n\n$ sh t4216-log-bloom.sh -i -v\n\nends like so:\n\nok 133 - Use Bloom filters if they exist in the latest but not all commit graphs in the chain.\n\nexpecting success of 4216.134 'persist filter settings':\n test_when_finished rm -rf .git/objects/info/commit-graph* &&\n GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n grep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":7,\\\"bits_per_entry\\\":10}\" trace2.txt &&\n cp .git/objects/info/commit-graph commit-graph-before &&\n corrupt_graph $BASE_K_BYTE_OFFSET \"\\09\" &&\n corrupt_graph $BASE_LEN_BYTE_OFFSET \"\\0F\" &&\n cp .git/objects/info/commit-graph commit-graph-after &&\n test_commit c18 A/corrupt &&\n GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n grep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":57,\\\"bits_per_entry\\\":70}\" trace2.txt\n\nnot ok 134 - persist filter settings\n# ...\n\nThanks.\n"},{"id":"400565","messageId":"429f38a2-9346-f6f7-bd69-2fd63828ae7c@gmail.com","threadId":"53681","inReplyTo":"xmqqftak5aa0.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 00/11] More commit-graph/Bloom filter improvements","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-24T23:32:56Z","receivedAt":"2020-06-24T23:33:01Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/24/2020 7:11 PM, Junio C Hamano wrote:\n> This does not seem to play well with what is in flight. Tests seem\n> to pass with topics up to es/config-hooks merged but not with this\n> topic merged on top.\n> \n> 1b5d3d8260 Merge branch 'ds/commit-graph-bloom-updates' into seen\n> 32169c595c Merge branch 'es/config-hooks' into seen\n> ...\n> \n> $ sh t4216-log-bloom.sh -i -v\n> \n> ends like so:\n> \n> ok 133 - Use Bloom filters if they exist in the latest but not all commit graphs in the chain.\n> \n> expecting success of 4216.134 'persist filter settings':\n> test_when_finished rm -rf .git/objects/info/commit-graph* &&\n> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n> grep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":7,\\\"bits_per_entry\\\":10}\" trace2.txt &&\n> cp .git/objects/info/commit-graph commit-graph-before &&\n> corrupt_graph $BASE_K_BYTE_OFFSET \"\\09\" &&\n> corrupt_graph $BASE_LEN_BYTE_OFFSET \"\\0F\" &&\n> cp .git/objects/info/commit-graph commit-graph-after &&\n> test_commit c18 A/corrupt &&\n> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n> grep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":57,\\\"bits_per_entry\\\":70}\" trace2.txt\n> \n> not ok 134 - persist filter settings\n> # ...\n> \n> Thanks.\n\nThanks for letting me know. I'll investigate carefully with the\nrest of the 'seen' branch. This test is a bit fragile due to\ncomputed values for which bytes to replace, so anything that\ncould have changed the length or order of chunks would lead to\na failure here.\n\nSorry for the disruption.\n\n-Stolee\n\n"},{"id":"400568","messageId":"xmqq7dvw5699.fsf@gitster.c.googlers.com","threadId":"53681","inReplyTo":"429f38a2-9346-f6f7-bd69-2fd63828ae7c@gmail.com","subject":"Re: [PATCH v2 00/11] More commit-graph/Bloom filter improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-06-25T00:38:42Z","receivedAt":"2020-06-25T00:38:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 6/24/2020 7:11 PM, Junio C Hamano wrote:\n>> This does not seem to play well with what is in flight. Tests seem\n>> to pass with topics up to es/config-hooks merged but not with this\n>> topic merged on top.\n>> \n>> 1b5d3d8260 Merge branch 'ds/commit-graph-bloom-updates' into seen\n>> 32169c595c Merge branch 'es/config-hooks' into seen\n>> ...\n>> \n>> $ sh t4216-log-bloom.sh -i -v\n>> \n>> ends like so:\n>> \n>> ok 133 - Use Bloom filters if they exist in the latest but not all commit graphs in the chain.\n>> \n>> expecting success of 4216.134 'persist filter settings':\n>> test_when_finished rm -rf .git/objects/info/commit-graph* &&\n>> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n>> grep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":7,\\\"bits_per_entry\\\":10}\" trace2.txt &&\n>> cp .git/objects/info/commit-graph commit-graph-before &&\n>> corrupt_graph $BASE_K_BYTE_OFFSET \"\\09\" &&\n>> corrupt_graph $BASE_LEN_BYTE_OFFSET \"\\0F\" &&\n>> cp .git/objects/info/commit-graph commit-graph-after &&\n>> test_commit c18 A/corrupt &&\n>> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n>> grep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":57,\\\"bits_per_entry\\\":70}\" trace2.txt\n>> \n>> not ok 134 - persist filter settings\n>> # ...\n>> \n>> Thanks.\n>\n> Thanks for letting me know. I'll investigate carefully with the\n> rest of the 'seen' branch. This test is a bit fragile due to\n> computed values for which bytes to replace, so anything that\n> could have changed the length or order of chunks would lead to\n> a failure here.\n>\n> Sorry for the disruption.\n\nOh, not at all. Thanks for helping.\n"},{"id":"400571","messageId":"2d64707c-a538-118c-d7c3-7a38bf08fa79@web.de","threadId":"53681","inReplyTo":"492deaf916464abedc7fc2d3de41fe690a558d9b.1592934430.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 03/11] bloom: get_bloom_filter() cleanups","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-25T07:24:51Z","receivedAt":"2020-06-25T07:25:23Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 23.06.20 um 19:47 schrieb Derrick Stolee via GitGitGadget:\n> From: Derrick Stolee \n>\n> The get_bloom_filter() method is a bit complicated in some parts where\n> it does not need to be. In particular, it needs to return a NULL filter\n> only when compute_if_not_present is zero AND the filter data cannot be\n> loaded from a commit-graph file. This currently happens by accident\n> because the commit-graph does not load changed-path Bloom filters from\n> an existing commit-graph when writing a new one. This will change in a\n> later patch.\n\nSo this is actually a logic fix, not just a cleanup as the subject says?\n\n>\n> Also clean up some style issues while we are here.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> bloom.c | 15 +++++++--------\n> 1 file changed, 7 insertions(+), 8 deletions(-)\n>\n> diff --git a/bloom.c b/bloom.c\n> index c38d1cff0c..7291506564 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -186,7 +186,7 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n> \tstruct diff_options diffopt;\n> \tint max_changes = 512;\n>\n> -\tif (bloom_filters.slab_size == 0)\n> +\tif (!bloom_filters.slab_size)\n> \t\treturn NULL;\n>\n> \tfilter = bloom_filter_slab_at(&bloom_filters, c);\n> @@ -194,16 +194,15 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n> \tif (!filter->data) {\n> \t\tload_commit_graph_info(r, c);\n> \t\tif (c->graph_pos != COMMIT_NOT_FROM_GRAPH &&\n> -\t\t\tr->objects->commit_graph->chunk_bloom_indexes) {\n> -\t\t\tif (load_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n> -\t\t\t\treturn filter;\n> -\t\t\telse\n> -\t\t\t\treturn NULL;\n\n... and the fix is that this else branch should not be taken if\ncompute_if_not_present is set.\n\n> -\t\t}\n> +\t\t r->objects->commit_graph->chunk_bloom_indexes &&\n> +\t\t load_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n> +\t\t\treturn filter;\n\nYou could even drop this return as well and have the check below handle the\nsuccessful load case.\n\n> \t}\n>\n> -\tif (filter->data || !compute_if_not_present)\n> +\tif (filter->data)\n> \t\treturn filter;\n> +\tif (!filter->data && !compute_if_not_present)\n ^^^^^^^^^^^^^\nThe first condition is always true, as the check two lines above makes sure.\nRemoving it would be cleaner IMHO.\n\n> +\t\treturn NULL;\n>\n> \trepo_diff_setup(r, &diffopt);\n> \tdiffopt.flags.recursive = 1;\n>\n"},{"id":"400572","messageId":"562263d0-9376-bf90-254c-880e611ffd2b@web.de","threadId":"53681","inReplyTo":"244668fec4428872f441d515e570e923808f0db1.1592934430.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 05/11] commit-graph: unify the signatures of all write_graph_chunk_*() functions","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-25T07:25:09Z","receivedAt":"2020-06-25T07:25:23Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 23.06.20 um 19:47 schrieb SZEDER Gábor via GitGitGadget:\n> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>\n> Update the write_graph_chunk_*() helper functions to have the same\n> signature:\n>\n> - Return an int error code from all these functions.\n> write_graph_chunk_base() already has an int error code, now the\n> others will have one, too, but since they don't indicate any\n> error, they will always return 0.\n>\n> - Drop the hash size parameter of write_graph_chunk_oids() and\n> write_graph_chunk_data(); its value can be read directly from\n> 'the_hash_algo' inside these functions as well.\n>\n> This opens up the possibility for further cleanups and foolproofing in\n> the following two patches.\n>\n> Signed-off-by: SZEDER Gábor \n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 42 ++++++++++++++++++++++++++----------------\n> 1 file changed, 26 insertions(+), 16 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 908f094271..f33bfe49b3 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -891,8 +891,8 @@ struct write_commit_graph_context {\n> \tconst struct bloom_filter_settings *bloom_settings;\n> };\n>\n> -static void write_graph_chunk_fanout(struct hashfile *f,\n> -\t\t\t\t struct write_commit_graph_context *ctx)\n> +static int write_graph_chunk_fanout(struct hashfile *f,\n> +\t\t\t\t struct write_commit_graph_context *ctx)\n> {\n> \tint i, count = 0;\n> \tstruct commit **list = ctx->commits.list;\n> @@ -913,17 +913,21 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n>\n> \t\thashwrite_be32(f, count);\n> \t}\n> +\n> +\treturn 0;\n> }\n>\n> -static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n> -\t\t\t\t struct write_commit_graph_context *ctx)\n> +static int write_graph_chunk_oids(struct hashfile *f,\n> +\t\t\t\t struct write_commit_graph_context *ctx)\n> {\n> \tstruct commit **list = ctx->commits.list;\n> \tint count;\n> \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n> \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n> -\t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n> +\t\thashwrite(f, (*list)->object.oid.hash, (int)the_hash_algo->rawsz);\n ^^^^^\nThis cast is confusing...\n\n> \t}\n> +\n> +\treturn 0;\n> }\n>\n> static const unsigned char *commit_to_sha1(size_t index, void *table)\n> @@ -932,8 +936,8 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n> \treturn commits[index]->object.oid.hash;\n> }\n>\n> -static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> -\t\t\t\t struct write_commit_graph_context *ctx)\n> +static int write_graph_chunk_data(struct hashfile *f,\n> +\t\t\t\t struct write_commit_graph_context *ctx)\n> {\n> \tstruct commit **list = ctx->commits.list;\n> \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n> @@ -950,7 +954,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n> \t\t\tdie(_(\"unable to parse commit %s\"),\n> \t\t\t\toid_to_hex(&(*list)->object.oid));\n> \t\ttree = get_commit_tree_oid(*list);\n> -\t\thashwrite(f, tree->hash, hash_len);\n> +\t\thashwrite(f, tree->hash, the_hash_algo->rawsz);\n\n... and obviously not needed, as this example shows.\n\n>\n> \t\tparent = (*list)->parents;\n>\n> @@ -1030,10 +1034,12 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n>\n> \t\tlist++;\n> \t}\n> +\n> +\treturn 0;\n> }\n>\n> -static void write_graph_chunk_extra_edges(struct hashfile *f,\n> -\t\t\t\t\t struct write_commit_graph_context *ctx)\n> +static int write_graph_chunk_extra_edges(struct hashfile *f,\n> +\t\t\t\t\t struct write_commit_graph_context *ctx)\n> {\n> \tstruct commit **list = ctx->commits.list;\n> \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n> @@ -1082,10 +1088,12 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n>\n> \t\tlist++;\n> \t}\n> +\n> +\treturn 0;\n> }\n>\n> -static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n> -\t\t\t\t\t struct write_commit_graph_context *ctx)\n> +static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n> +\t\t\t\t\t struct write_commit_graph_context *ctx)\n> {\n> \tstruct commit **list = ctx->commits.list;\n> \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n> @@ -1107,6 +1115,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n> \t}\n>\n> \tstop_progress(&progress);\n> +\treturn 0;\n> }\n>\n> static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n> @@ -1124,8 +1133,8 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n> \tjw_release(&jw);\n> }\n>\n> -static void write_graph_chunk_bloom_data(struct hashfile *f,\n> -\t\t\t\t\t struct write_commit_graph_context *ctx)\n> +static int write_graph_chunk_bloom_data(struct hashfile *f,\n> +\t\t\t\t\tstruct write_commit_graph_context *ctx)\n> {\n> \tstruct commit **list = ctx->commits.list;\n> \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n> @@ -1151,6 +1160,7 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n> \t}\n>\n> \tstop_progress(&progress);\n> +\treturn 0;\n> }\n>\n> static int oid_compare(const void *_a, const void *_b)\n> @@ -1662,8 +1672,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> \t\t\tnum_chunks * ctx->commits.nr);\n> \t}\n> \twrite_graph_chunk_fanout(f, ctx);\n> -\twrite_graph_chunk_oids(f, hashsz, ctx);\n> -\twrite_graph_chunk_data(f, hashsz, ctx);\n> +\twrite_graph_chunk_oids(f, ctx);\n> +\twrite_graph_chunk_data(f, ctx);\n> \tif (ctx->num_extra_edges)\n> \t\twrite_graph_chunk_extra_edges(f, ctx);\n> \tif (ctx->changed_paths) {\n>\n"},{"id":"400573","messageId":"de5453ed-6db9-ebd5-28f9-8db4f9ed8447@web.de","threadId":"53681","inReplyTo":"3eb10933dc8517ab01eff17a179e484068d29724.1592934430.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 07/11] commit-graph: check chunk sizes after writing","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-25T07:25:12Z","receivedAt":"2020-06-25T07:25:23Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 23.06.20 um 19:47 schrieb SZEDER Gábor via GitGitGadget:\n> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>\n> In my experience while experimenting with new commit-graph chunks,\n> early versions of the corresponding new write_commit_graph_my_chunk()\n> functions are, sadly but not surprisingly, often buggy, and write more\n> or less data than they are supposed to, especially if the chunk size\n> is not directly proportional to the number of commits. This then\n> causes all kinds of issues when reading such a bogus commit-graph\n> file, raising the question of whether the writing or the reading part\n> happens to be buggy this time.\n>\n> Let's catch such issues early, already when writing the commit-graph\n> file, and check that each write_graph_chunk_*() function wrote the\n> amount of data that it was expected to, and what has been encoded in\n> the Chunk Lookup table. Now that all commit-graph chunks are written\n> in a loop we can do this check in a single place for all chunks, and\n> any chunks added in the future will get checked as well.\n>\n> Signed-off-by: SZEDER Gábor \n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 9 +++++++++\n> 1 file changed, 9 insertions(+)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 086fc2d070..1de6800d74 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1683,12 +1683,21 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> \t\t\tnum_chunks * ctx->commits.nr);\n> \t}\n>\n> +\tchunk_offset = f->total + f->offset;\n> \tfor (i = 0; i < num_chunks; i++) {\n> +\t\tuint64_t end_offset;\n> +\n\nHmm, the added code looks complicated because it keeps state outside the\nloop, but it could be replace by this:\n\n\t\tuint64_t start_offset = f->total + f->offset;\n\n> \t\tif (chunks[i].write_fn(f, ctx)) {\n> \t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n> \t\t\t chunks[i].id);\n> \t\t\treturn -1;\n> \t\t}\n> +\n> +\t\tend_offset = f->total + f->offset;\n> +\t\tif (end_offset - chunk_offset != chunks[i].size)\n> +\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n> +\t\t\t chunks[i].size, chunks[i].id, end_offset - chunk_offset);\n> +\t\tchunk_offset = end_offset;\n\n... and that:\n\n\t\tif (f->total + f->offset != start_offset + chunks[i].size)\n\t\t\tBUG(...);\n\n> \t}\n>\n> \tstop_progress(&ctx->progress);\n>\n"},{"id":"400575","messageId":"675f46df-c57c-bcd6-3f6a-fdd40b2f3066@web.de","threadId":"53681","inReplyTo":"8b959f2f374654aeb87b847560761890c2f9aa2c.1592934430.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 06/11] commit-graph: simplify chunk writes into loop","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-25T07:25:14Z","receivedAt":"2020-06-25T07:25:23Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 23.06.20 um 19:47 schrieb SZEDER Gábor via GitGitGadget:\n> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>\n> In write_commit_graph_file() we now have one block of code filling the\n> array of 'struct chunk_info' with the IDs and sizes of chunks to be\n> written, and an other block of code calling the functions responsible\n> for writing individual chunks. In case of optional chunks like Extra\n> Edge List an Base Graphs List there is also a condition checking\n> whether that chunk is necessary/desired, and that same condition is\n> repeated in both blocks of code. Other, newer chunks have similar\n> optional conditions.\n>\n> Eliminate these repeated conditions by storing the function pointers\n> responsible for writing individual chunks in the 'struct chunk_info'\n> array as well, and calling them in a loop to write the commit-graph\n> file. This will open up the possibility for a bit of foolproofing in\n> the following patch.\n\nYou can do that without storing function pointers by selecting the\nfunction to use based on the chunk ID -- like parse_commit_graph() does\non the read side. Advantage: You don't need to press all write\nfunctions into the same mold and can keep their individual signatures.\n\n>\n> Signed-off-by: SZEDER Gábor \n> Signed-off-by: Derrick Stolee \n> ---\n> commit-graph.c | 31 +++++++++++++++++++------------\n> 1 file changed, 19 insertions(+), 12 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index f33bfe49b3..086fc2d070 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1555,9 +1555,13 @@ static int write_graph_chunk_base(struct hashfile *f,\n> \treturn 0;\n> }\n>\n> +typedef int (*chunk_write_fn)(struct hashfile *f,\n> +\t\t\t struct write_commit_graph_context *ctx);\n> +\n> struct chunk_info {\n> \tuint32_t id;\n> \tuint64_t size;\n> +\tchunk_write_fn write_fn;\n> };\n>\n> static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> @@ -1615,27 +1619,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n>\n> \tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n> \tchunks[0].size = GRAPH_FANOUT_SIZE;\n> +\tchunks[0].write_fn = write_graph_chunk_fanout;\n> \tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n> \tchunks[1].size = hashsz * ctx->commits.nr;\n> +\tchunks[1].write_fn = write_graph_chunk_oids;\n> \tchunks[2].id = GRAPH_CHUNKID_DATA;\n> \tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n> +\tchunks[2].write_fn = write_graph_chunk_data;\n> \tif (ctx->num_extra_edges) {\n> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n> \t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n> \t\tnum_chunks++;\n> \t}\n> \tif (ctx->changed_paths) {\n> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n> \t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n> \t\tnum_chunks++;\n> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n> \t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n> \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n> \t\tnum_chunks++;\n> \t}\n> \tif (ctx->num_commit_graphs_after > 1) {\n> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n> \t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n> \t\tnum_chunks++;\n> \t}\n>\n> @@ -1671,19 +1682,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> \t\t\tprogress_title.buf,\n> \t\t\tnum_chunks * ctx->commits.nr);\n> \t}\n> -\twrite_graph_chunk_fanout(f, ctx);\n> -\twrite_graph_chunk_oids(f, ctx);\n> -\twrite_graph_chunk_data(f, ctx);\n> -\tif (ctx->num_extra_edges)\n> -\t\twrite_graph_chunk_extra_edges(f, ctx);\n> -\tif (ctx->changed_paths) {\n> -\t\twrite_graph_chunk_bloom_indexes(f, ctx);\n> -\t\twrite_graph_chunk_bloom_data(f, ctx);\n> -\t}\n> -\tif (ctx->num_commit_graphs_after > 1 &&\n> -\t write_graph_chunk_base(f, ctx)) {\n> -\t\treturn -1;\n> +\n> +\tfor (i = 0; i < num_chunks; i++) {\n> +\t\tif (chunks[i].write_fn(f, ctx)) {\n> +\t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n> +\t\t\t chunks[i].id);\n\nOf all the write functions only write_graph_chunk_base() can return\nnon-zero and it already prints an error message in that case (\"failed to\nwrite correct number of base graph ids\"). Why add this one?\n\n> +\t\t\treturn -1;\n> +\t\t}\n> \t}\n> +\n> \tstop_progress(&ctx->progress);\n> \tstrbuf_release(&progress_title);\n>\n>\n"},{"id":"400574","messageId":"8d8b1763-bbf3-682e-0649-f5a0885399e6@web.de","threadId":"53681","inReplyTo":"9c2076b4ce46918fce8f05e609b057611ec56e13.1592934430.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v2 10/11] commit-graph: check all leading directories in changed path Bloom filters","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-06-25T07:25:18Z","receivedAt":"2020-06-25T07:25:25Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 23.06.20 um 19:47 schrieb SZEDER Gábor via GitGitGadget:\n> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>\n> The file 'dir/subdir/file' can only be modified if its leading\n> directories 'dir' and 'dir/subdir' are modified as well.\n>\n> So when checking modified path Bloom filters looking for commits\n> modifying a path with multiple path components, then check not only\n> the full path in the Bloom filters, but all its leading directories as\n> well. Take care to check these paths in \"deepest first\" order,\n> because it's the full path that is least likely to be modified, and\n> the Bloom filter queries can short circuit sooner.\n>\n> This can significantly reduce the average false positive rate, by\n> about an order of magnitude or three(!), and can further speed up\n> pathspec-limited revision walks. The table below compares the average\n> false positive rate and runtime of\n>\n> git rev-list HEAD -- \"$path\"\n>\n> before and after this change for 5000+ randomly* selected paths from\n> each repository:\n>\n> Average false Average Average\n> positive rate runtime runtime\n> before after before after difference\n> ------------------------------------------------------------------\n> git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n> linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n> tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n>\n> *Path selection was done with the following pipeline:\n>\n> \tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n>\n> The improvements in runtime are much smaller than the improvements in\n> average false positive rate, as we are clearly reaching diminishing\n> returns here. However, all these timings depend on that accessing\n> tree objects is reasonably fast (warm caches). If we had a partial\n> clone and the tree objects had to be fetched from a promisor remote,\n> e.g.:\n>\n> $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n> $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n> commit-graph write --reachable\n> $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n> $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n> rev-list HEAD -- \"$path\"\n>\n> then checking all leading path component can reduce the runtime from\n> over an hour to a few seconds (and this is with the clone and the\n> promisor on the same machine).\n>\n> This adjusts the tracing values in t4216-log-bloom.sh, which provides a\n> concrete way to notice the improvement.\n>\n> Helped-by: Taylor Blau \n> Helped-by: René Scharfe \n> Signed-off-by: SZEDER Gábor \n> Signed-off-by: Derrick Stolee \n> ---\n> revision.c | 41 ++++++++++++++++++++++++++++++++---------\n> revision.h | 6 ++++--\n> t/t4216-log-bloom.sh | 2 +-\n> 3 files changed, 37 insertions(+), 12 deletions(-)\n>\n> diff --git a/revision.c b/revision.c\n> index b53377cd52..077888ee51 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> {\n> \tstruct pathspec_item *pi;\n> \tchar *path_alloc = NULL;\n> -\tconst char *path;\n> +\tconst char *path, *p;\n> \tint last_index;\n> -\tint len;\n> +\tsize_t len;\n> +\tint path_component_nr = 1;\n>\n> \tif (!revs->commits)\n> \t\treturn;\n> @@ -709,8 +710,28 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> \t\treturn;\n> \t}\n>\n> -\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n> -\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n> +\tp = path;\n> +\twhile (*p) {\n> +\t\tif (is_dir_sep(*p))\n> +\t\t\tpath_component_nr++;\n> +\t\tp++;\n> +\t}\n> +\n> +\trevs->bloom_keys_nr = path_component_nr;\n> +\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n> +\n> +\tfill_bloom_key(path, len, &revs->bloom_keys[0],\n> +\t\t revs->bloom_filter_settings);\n> +\tpath_component_nr = 1;\n> +\n> +\tp = path + len - 1;\n\nlen cannot be 0 at this point, as patch 9 made sure, so this is safe.\nGood.\n\n> +\twhile (p > path) {\n> +\t\tif (is_dir_sep(*p))\n> +\t\t\tfill_bloom_key(path, p - path,\n> +\t\t\t\t &revs->bloom_keys[path_component_nr++],\n> +\t\t\t\t revs->bloom_filter_settings);\n> +\t\tp--;\n> +\t}\n\nThis walks the directory hierarchy upwards and adds bloom filters for\nshorter and shorter paths, (\"deepest first\"). Good.\n\nAnd it supports all directory separators. On Windows that would be\nslash (/) and backslash (\\). I assume paths are normalized to use\nonly slashes when bloom filters are written, correct? Then the lookup\nside needs to normalize a given path to only use slashes as well,\notherwise paths with backslashes cannot be found. This part seems to\nbe missing.\n\n>\n> \tif (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n> \t\tatexit(trace2_bloom_filter_statistics_atexit);\n> @@ -724,7 +745,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n> \t\t\t\t\t\t struct commit *commit)\n> {\n> \tstruct bloom_filter *filter;\n> -\tint result;\n> +\tint result = 1, j;\n>\n> \tif (!revs->repo->objects->commit_graph)\n> \t\treturn -1;\n> @@ -744,9 +765,11 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n> \t\treturn -1;\n> \t}\n>\n> -\tresult = bloom_filter_contains(filter,\n> -\t\t\t\t revs->bloom_key,\n> -\t\t\t\t revs->bloom_filter_settings);\n> +\tfor (j = 0; result && j < revs->bloom_keys_nr; j++) {\n> +\t\tresult = bloom_filter_contains(filter,\n> +\t\t\t\t\t &revs->bloom_keys[j],\n> +\t\t\t\t\t revs->bloom_filter_settings);\n> +\t}\n>\n> \tif (result)\n> \t\tcount_bloom_filter_maybe++;\n> @@ -786,7 +809,7 @@ static int rev_compare_tree(struct rev_info *revs,\n> \t\t\treturn REV_TREE_SAME;\n> \t}\n>\n> -\tif (revs->bloom_key && !nth_parent) {\n> +\tif (revs->bloom_keys_nr && !nth_parent) {\n> \t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, commit);\n>\n> \t\tif (bloom_ret == 0)\n> diff --git a/revision.h b/revision.h\n> index 7c026fe41f..abbfb4ab59 100644\n> --- a/revision.h\n> +++ b/revision.h\n> @@ -295,8 +295,10 @@ struct rev_info {\n> \tstruct topo_walk_info *topo_walk_info;\n>\n> \t/* Commit graph bloom filter fields */\n> -\t/* The bloom filter key for the pathspec */\n> -\tstruct bloom_key *bloom_key;\n> +\t/* The bloom filter key(s) for the pathspec */\n> +\tstruct bloom_key *bloom_keys;\n> +\tint bloom_keys_nr;\n> +\n> \t/*\n> \t * The bloom filter settings used to generate the key.\n> \t * This is loaded from the commit-graph being used.\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index f890cc4737..84f95972ca 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -146,7 +146,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n>\n> test_bloom_filters_used_when_some_filters_are_missing () {\n> \tlog_args=$1\n> -\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n> +\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n> \tsetup \"$log_args\" &&\n> \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n> \ttest_cmp log_wo_bloom log_w_bloom\n>\n"},{"id":"400605","messageId":"4b08360d-04ab-b614-196b-76124f1d54ef@gmail.com","threadId":"53681","inReplyTo":"xmqq7dvw5699.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 00/11] More commit-graph/Bloom filter improvements","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-25T13:38:42Z","receivedAt":"2020-06-25T13:38:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/24/2020 8:38 PM, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n>> On 6/24/2020 7:11 PM, Junio C Hamano wrote:\n>>> This does not seem to play well with what is in flight. Tests seem\n>>> to pass with topics up to es/config-hooks merged but not with this\n>>> topic merged on top.\n>>>\n>>> 1b5d3d8260 Merge branch 'ds/commit-graph-bloom-updates' into seen\n>>> 32169c595c Merge branch 'es/config-hooks' into seen\n>>> ...\n>>>\n>>> $ sh t4216-log-bloom.sh -i -v\n>>>\n>>> ends like so:\n>>>\n>>> ok 133 - Use Bloom filters if they exist in the latest but not all commit graphs in the chain.\n>>>\n>>> expecting success of 4216.134 'persist filter settings':\n>>> test_when_finished rm -rf .git/objects/info/commit-graph* &&\n>>> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n>>> grep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":7,\\\"bits_per_entry\\\":10}\" trace2.txt &&\n>>> cp .git/objects/info/commit-graph commit-graph-before &&\n>>> corrupt_graph $BASE_K_BYTE_OFFSET \"\\09\" &&\n>>> corrupt_graph $BASE_LEN_BYTE_OFFSET \"\\0F\" &&\n>>> cp .git/objects/info/commit-graph commit-graph-after &&\n>>> test_commit c18 A/corrupt &&\n>>> GIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n>>> grep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":57,\\\"bits_per_entry\\\":70}\" trace2.txt\n>>>\n>>> not ok 134 - persist filter settings\n>>> # ...\n>>>\n>>> Thanks.\n>>\n>> Thanks for letting me know. I'll investigate carefully with the\n>> rest of the 'seen' branch. This test is a bit fragile due to\n>> computed values for which bytes to replace, so anything that\n>> could have changed the length or order of chunks would lead to\n>> a failure here.\n>>\n>> Sorry for the disruption.\n> \n> Oh, not at all. Thanks for helping\nI'll squash the patch into my v3, but here it is now to make 'seen'\npass tests again.\n\nThe _real_ reason for the failure was that some changes in trace2\npushed the events out of the nesting limits. I also think it is a\ngood idea to make the test less brittle. Adding GIT_TEST_* variables\nwill also help anyone who wants to adjust the Bloom filter settings\nfor testing.\n\nQuestion: Should these be GIT_BLOOM_SETTINGS_* instead of GIT_TEST_...?\nI ask because this _could_ be a way to allow user customization,\nwithout making it as public as a config option. Or, should I just do\nthe work and add config settings in this series?\n\nThanks,\n-Stolee\n\n-- >8 --\nFrom 9245d31f0431eceec60f0b7a90900d2825787530 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Thu, 25 Jun 2020 09:19:13 -0400\nSubject: [PATCH] fixup! commit-graph: persist existence of changed-paths\n\nThe previous version of this test was too fragile to subtle changes\nin the commit-graph file size. This version now uses two environment\nvariables to customize the Bloom filter settings before rewriting\nwithout those environment variables. This demonstrates that we\npersist the settings correctly.\n\nThe issue with the 'seen' branch is due to es/trace-log-progress\nadding trace2 regions in the progress indicators. This pushed the\ntrace2 data that the test was expecting outside the nesting limit.\nSet GIT_TRACE2_EVENT_NESTING to ensure we still record those items.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 9 +++++++--\n t/t4216-log-bloom.sh | 34 +++++++++++-----------------------\n 2 files changed, 18 insertions(+), 25 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1de6800d74..026ec63d38 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1576,10 +1576,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tint num_chunks = 3;\n \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n-\tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n+\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n-\tif (!ctx->bloom_settings)\n+\tif (!ctx->bloom_settings) {\n+\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n+\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n+\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n+\t\t\t\t\t\t\t bloom_settings.num_hashes);\n \t\tctx->bloom_settings = &bloom_settings;\n+\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 84f95972ca..d7dd717347 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -156,31 +156,19 @@ test_expect_success 'Use Bloom filters if they exist in the latest but not all c\n \ttest_bloom_filters_used_when_some_filters_are_missing \"-- A/B\"\n '\n \n-BASE_BDAT_OFFSET=2240\n-BASE_K_BYTE_OFFSET=$((BASE_BDAT_OFFSET + 10))\n-BASE_LEN_BYTE_OFFSET=$((BASE_BDAT_OFFSET + 14))\n-\n-corrupt_graph() {\n-\tpos=$1\n-\tdata=\"${2:-\\0}\"\n-\tgrepstr=$3\n-\torig_size=$(wc -c < .git/objects/info/commit-graph) &&\n-\tzero_pos=${4:-${orig_size}} &&\n-\tprintf \"$data\" | dd of=\".git/objects/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n-\tdd of=\".git/objects/info/commit-graph\" bs=1 seek=\"$zero_pos\" if=/dev/null\n-}\n-\n test_expect_success 'persist filter settings' '\n \ttest_when_finished rm -rf .git/objects/info/commit-graph* &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":7,\\\"bits_per_entry\\\":10}\" trace2.txt &&\n-\tcp .git/objects/info/commit-graph commit-graph-before &&\n-\tcorrupt_graph $BASE_K_BYTE_OFFSET \"\\09\" &&\n-\tcorrupt_graph $BASE_LEN_BYTE_OFFSET \"\\0F\" &&\n-\tcp .git/objects/info/commit-graph commit-graph-after &&\n-\ttest_commit c18 A/corrupt &&\n-\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":57,\\\"bits_per_entry\\\":70}\" trace2.txt\n+\trm -rf .git/objects/info/commit-graph* &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tGIT_TRACE2_EVENT_NESTING=5 \\\n+\t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n+\t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n+\t\tgit commit-graph write --reachable --changed-paths &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n+\t\tGIT_TRACE2_EVENT_NESTING=5 \\\n+\t\tgit commit-graph write --reachable --changed-paths &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n '\n \n test_done\n\\ No newline at end of file\n-- \n2.27.0\n\n\n\n"},{"id":"400609","messageId":"467ecf3b-2635-5d92-c8f9-9e89dd6580ee@gmail.com","threadId":"53681","inReplyTo":"675f46df-c57c-bcd6-3f6a-fdd40b2f3066@web.de","subject":"Re: [PATCH v2 06/11] commit-graph: simplify chunk writes into loop","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-25T14:59:22Z","receivedAt":"2020-06-25T14:59:28Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/25/2020 3:25 AM, René Scharfe wrote:\n> Am 23.06.20 um 19:47 schrieb SZEDER Gábor via GitGitGadget:\n>> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>>\n>> In write_commit_graph_file() we now have one block of code filling the\n>> array of 'struct chunk_info' with the IDs and sizes of chunks to be\n>> written, and an other block of code calling the functions responsible\n>> for writing individual chunks. In case of optional chunks like Extra\n>> Edge List an Base Graphs List there is also a condition checking\n>> whether that chunk is necessary/desired, and that same condition is\n>> repeated in both blocks of code. Other, newer chunks have similar\n>> optional conditions.\n>>\n>> Eliminate these repeated conditions by storing the function pointers\n>> responsible for writing individual chunks in the 'struct chunk_info'\n>> array as well, and calling them in a loop to write the commit-graph\n>> file. This will open up the possibility for a bit of foolproofing in\n>> the following patch.\n> \n> You can do that without storing function pointers by selecting the\n> function to use based on the chunk ID -- like parse_commit_graph() does\n> on the read side. Advantage: You don't need to press all write\n> functions into the same mold and can keep their individual signatures.\n\nI do think that the loop without a switch statement is valuable.\nIt focuses the updates for new chunks to be localized to the\nsection that calculates the offset values.\n\n>>\n>> Signed-off-by: SZEDER Gábor \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 31 +++++++++++++++++++------------\n>> 1 file changed, 19 insertions(+), 12 deletions(-)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index f33bfe49b3..086fc2d070 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -1555,9 +1555,13 @@ static int write_graph_chunk_base(struct hashfile *f,\n>> \treturn 0;\n>> }\n>>\n>> +typedef int (*chunk_write_fn)(struct hashfile *f,\n>> +\t\t\t struct write_commit_graph_context *ctx);\n>> +\n>> struct chunk_info {\n>> \tuint32_t id;\n>> \tuint64_t size;\n>> +\tchunk_write_fn write_fn;\n>> };\n>>\n>> static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n>> @@ -1615,27 +1619,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n>>\n>> \tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n>> \tchunks[0].size = GRAPH_FANOUT_SIZE;\n>> +\tchunks[0].write_fn = write_graph_chunk_fanout;\n>> \tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n>> \tchunks[1].size = hashsz * ctx->commits.nr;\n>> +\tchunks[1].write_fn = write_graph_chunk_oids;\n>> \tchunks[2].id = GRAPH_CHUNKID_DATA;\n>> \tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n>> +\tchunks[2].write_fn = write_graph_chunk_data;\n>> \tif (ctx->num_extra_edges) {\n>> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n>> \t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n>> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n>> \t\tnum_chunks++;\n>> \t}\n>> \tif (ctx->changed_paths) {\n>> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n>> \t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n>> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n>> \t\tnum_chunks++;\n>> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n>> \t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n>> \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n>> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n>> \t\tnum_chunks++;\n>> \t}\n>> \tif (ctx->num_commit_graphs_after > 1) {\n>> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n>> \t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n>> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n>> \t\tnum_chunks++;\n>> \t}\n>>\n>> @@ -1671,19 +1682,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n>> \t\t\tprogress_title.buf,\n>> \t\t\tnum_chunks * ctx->commits.nr);\n>> \t}\n>> -\twrite_graph_chunk_fanout(f, ctx);\n>> -\twrite_graph_chunk_oids(f, ctx);\n>> -\twrite_graph_chunk_data(f, ctx);\n>> -\tif (ctx->num_extra_edges)\n>> -\t\twrite_graph_chunk_extra_edges(f, ctx);\n>> -\tif (ctx->changed_paths) {\n>> -\t\twrite_graph_chunk_bloom_indexes(f, ctx);\n>> -\t\twrite_graph_chunk_bloom_data(f, ctx);\n>> -\t}\n>> -\tif (ctx->num_commit_graphs_after > 1 &&\n>> -\t write_graph_chunk_base(f, ctx)) {\n>> -\t\treturn -1;\n>> +\n>> +\tfor (i = 0; i < num_chunks; i++) {\n>> +\t\tif (chunks[i].write_fn(f, ctx)) {\n>> +\t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n>> +\t\t\t chunks[i].id);\n> \n> Of all the write functions only write_graph_chunk_base() can return\n> non-zero and it already prints an error message in that case (\"failed to\n> write correct number of base graph ids\"). Why add this one?\n\nOk, we can require the chunk methods to write an error() message with\nappropriate context and simply return -1 here.\n\nThanks,\n-Stolee\n"},{"id":"400610","messageId":"5cb45264-1788-a563-a0e1-40ec7e64644a@gmail.com","threadId":"53681","inReplyTo":"de5453ed-6db9-ebd5-28f9-8db4f9ed8447@web.de","subject":"Re: [PATCH v2 07/11] commit-graph: check chunk sizes after writing","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-25T15:02:49Z","receivedAt":"2020-06-25T15:02:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/25/2020 3:25 AM, René Scharfe wrote:\n> Am 23.06.20 um 19:47 schrieb SZEDER Gábor via GitGitGadget:\n>> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>>\n>> In my experience while experimenting with new commit-graph chunks,\n>> early versions of the corresponding new write_commit_graph_my_chunk()\n>> functions are, sadly but not surprisingly, often buggy, and write more\n>> or less data than they are supposed to, especially if the chunk size\n>> is not directly proportional to the number of commits. This then\n>> causes all kinds of issues when reading such a bogus commit-graph\n>> file, raising the question of whether the writing or the reading part\n>> happens to be buggy this time.\n>>\n>> Let's catch such issues early, already when writing the commit-graph\n>> file, and check that each write_graph_chunk_*() function wrote the\n>> amount of data that it was expected to, and what has been encoded in\n>> the Chunk Lookup table. Now that all commit-graph chunks are written\n>> in a loop we can do this check in a single place for all chunks, and\n>> any chunks added in the future will get checked as well.\n>>\n>> Signed-off-by: SZEDER Gábor \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> commit-graph.c | 9 +++++++++\n>> 1 file changed, 9 insertions(+)\n>>\n>> diff --git a/commit-graph.c b/commit-graph.c\n>> index 086fc2d070..1de6800d74 100644\n>> --- a/commit-graph.c\n>> +++ b/commit-graph.c\n>> @@ -1683,12 +1683,21 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n>> \t\t\tnum_chunks * ctx->commits.nr);\n>> \t}\n>>\n>> +\tchunk_offset = f->total + f->offset;\n>> \tfor (i = 0; i < num_chunks; i++) {\n>> +\t\tuint64_t end_offset;\n>> +\n> \n> Hmm, the added code looks complicated because it keeps state outside the\n> loop, but it could be replace by this:\n> \n> \t\tuint64_t start_offset = f->total + f->offset;\n> \n>> \t\tif (chunks[i].write_fn(f, ctx)) {\n>> \t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n>> \t\t\t chunks[i].id);\n>> \t\t\treturn -1;\n>> \t\t}\n>> +\n>> +\t\tend_offset = f->total + f->offset;\n>> +\t\tif (end_offset - chunk_offset != chunks[i].size)\n>> +\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n>> +\t\t\t chunks[i].size, chunks[i].id, end_offset - chunk_offset);\n>> +\t\tchunk_offset = end_offset;\n> \n> ... and that:\n> \n> \t\tif (f->total + f->offset != start_offset + chunks[i].size)\n> \t\t\tBUG(...);\n\nThanks! I agree this approach is simpler and less prone to\nbugs since we are using the local state.\n\n-Stolee\n"},{"id":"400611","messageId":"0fe96c75-2946-8160-2ced-3d9781dca8c0@gmail.com","threadId":"53681","inReplyTo":"8d8b1763-bbf3-682e-0649-f5a0885399e6@web.de","subject":"Re: [PATCH v2 10/11] commit-graph: check all leading directories in changed path Bloom filters","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-25T15:05:04Z","receivedAt":"2020-06-25T15:05:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/25/2020 3:25 AM, René Scharfe wrote:\n> Am 23.06.20 um 19:47 schrieb SZEDER Gábor via GitGitGadget:\n>> From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n>>\n>> The file 'dir/subdir/file' can only be modified if its leading\n>> directories 'dir' and 'dir/subdir' are modified as well.\n>>\n>> So when checking modified path Bloom filters looking for commits\n>> modifying a path with multiple path components, then check not only\n>> the full path in the Bloom filters, but all its leading directories as\n>> well. Take care to check these paths in \"deepest first\" order,\n>> because it's the full path that is least likely to be modified, and\n>> the Bloom filter queries can short circuit sooner.\n>>\n>> This can significantly reduce the average false positive rate, by\n>> about an order of magnitude or three(!), and can further speed up\n>> pathspec-limited revision walks. The table below compares the average\n>> false positive rate and runtime of\n>>\n>> git rev-list HEAD -- \"$path\"\n>>\n>> before and after this change for 5000+ randomly* selected paths from\n>> each repository:\n>>\n>> Average false Average Average\n>> positive rate runtime runtime\n>> before after before after difference\n>> ------------------------------------------------------------------\n>> git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n>> linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n>> tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n>>\n>> *Path selection was done with the following pipeline:\n>>\n>> \tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n>>\n>> The improvements in runtime are much smaller than the improvements in\n>> average false positive rate, as we are clearly reaching diminishing\n>> returns here. However, all these timings depend on that accessing\n>> tree objects is reasonably fast (warm caches). If we had a partial\n>> clone and the tree objects had to be fetched from a promisor remote,\n>> e.g.:\n>>\n>> $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n>> $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n>> commit-graph write --reachable\n>> $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n>> $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n>> rev-list HEAD -- \"$path\"\n>>\n>> then checking all leading path component can reduce the runtime from\n>> over an hour to a few seconds (and this is with the clone and the\n>> promisor on the same machine).\n>>\n>> This adjusts the tracing values in t4216-log-bloom.sh, which provides a\n>> concrete way to notice the improvement.\n>>\n>> Helped-by: Taylor Blau \n>> Helped-by: René Scharfe \n>> Signed-off-by: SZEDER Gábor \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> revision.c | 41 ++++++++++++++++++++++++++++++++---------\n>> revision.h | 6 ++++--\n>> t/t4216-log-bloom.sh | 2 +-\n>> 3 files changed, 37 insertions(+), 12 deletions(-)\n>>\n>> diff --git a/revision.c b/revision.c\n>> index b53377cd52..077888ee51 100644\n>> --- a/revision.c\n>> +++ b/revision.c\n>> @@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n>> {\n>> \tstruct pathspec_item *pi;\n>> \tchar *path_alloc = NULL;\n>> -\tconst char *path;\n>> +\tconst char *path, *p;\n>> \tint last_index;\n>> -\tint len;\n>> +\tsize_t len;\n>> +\tint path_component_nr = 1;\n>>\n>> \tif (!revs->commits)\n>> \t\treturn;\n>> @@ -709,8 +710,28 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n>> \t\treturn;\n>> \t}\n>>\n>> -\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n>> -\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n>> +\tp = path;\n>> +\twhile (*p) {\n>> +\t\tif (is_dir_sep(*p))\n>> +\t\t\tpath_component_nr++;\n>> +\t\tp++;\n>> +\t}\n>> +\n>> +\trevs->bloom_keys_nr = path_component_nr;\n>> +\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n>> +\n>> +\tfill_bloom_key(path, len, &revs->bloom_keys[0],\n>> +\t\t revs->bloom_filter_settings);\n>> +\tpath_component_nr = 1;\n>> +\n>> +\tp = path + len - 1;\n> \n> len cannot be 0 at this point, as patch 9 made sure, so this is safe.\n> Good.\n> \n>> +\twhile (p > path) {\n>> +\t\tif (is_dir_sep(*p))\n>> +\t\t\tfill_bloom_key(path, p - path,\n>> +\t\t\t\t &revs->bloom_keys[path_component_nr++],\n>> +\t\t\t\t revs->bloom_filter_settings);\n>> +\t\tp--;\n>> +\t}\n> \n> This walks the directory hierarchy upwards and adds bloom filters for\n> shorter and shorter paths, (\"deepest first\"). Good.\n> \n> And it supports all directory separators. On Windows that would be\n> slash (/) and backslash (\\). I assume paths are normalized to use\n> only slashes when bloom filters are written, correct? Then the lookup\n> side needs to normalize a given path to only use slashes as well,\n> otherwise paths with backslashes cannot be found. This part seems to\n> be missing.\n\nYes, that's a good point. We _require_ the paths to be normalized\nhere to be Unix-style paths or else the Bloom filter keys are\nincorrect. Thankfully, they are. Let's make that clear in-code by\nusing '/' instead of is_dir_sep().\n\nThanks,\n-Stolee\n"},{"id":"400616","messageId":"xmqqy2ob3xzq.fsf@gitster.c.googlers.com","threadId":"53681","inReplyTo":"4b08360d-04ab-b614-196b-76124f1d54ef@gmail.com","subject":"Re: [PATCH v2 00/11] More commit-graph/Bloom filter improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-06-25T16:34:49Z","receivedAt":"2020-06-25T16:34:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Question: Should these be GIT_BLOOM_SETTINGS_* instead of GIT_TEST_...?\n> I ask because this _could_ be a way to allow user customization,\n> without making it as public as a config option. Or, should I just do\n> the work and add config settings in this series?\n\nOther than when testing and/or debugging, what are the expected\nreasons and situations an individual would want to use customized\nsettings? Once a decision is made to use one customized setting for\na repository, does it make sense for a setting other than that one\nsetting for the same repository, or is it something very handy if we\ncan use different settings on a whim? \n\nMy gut feeling is that it should be added as per-repo configuration\nbut only after a use case is found, and GIT_TEST_* would be the way\nto go.\n\nThanks.\n"},{"id":"400654","messageId":"20200626063436.GA11341@szeder.dev","threadId":"53681","inReplyTo":"0fe96c75-2946-8160-2ced-3d9781dca8c0@gmail.com","subject":"Re: [PATCH v2 10/11] commit-graph: check all leading directories in changed path Bloom filters","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-06-26T06:34:36Z","receivedAt":"2020-06-26T06:34:47Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Jun 25, 2020 at 11:05:04AM -0400, Derrick Stolee wrote:\n\n> >> +\twhile (p > path) {\n> >> +\t\tif (is_dir_sep(*p))\n> >> +\t\t\tfill_bloom_key(path, p - path,\n> >> +\t\t\t\t &revs->bloom_keys[path_component_nr++],\n> >> +\t\t\t\t revs->bloom_filter_settings);\n> >> +\t\tp--;\n> >> +\t}\n> > \n> > This walks the directory hierarchy upwards and adds bloom filters for\n> > shorter and shorter paths, (\"deepest first\"). Good.\n> > \n> > And it supports all directory separators. On Windows that would be\n> > slash (/) and backslash (\\). I assume paths are normalized to use\n> > only slashes when bloom filters are written, correct? Then the lookup\n> > side needs to normalize a given path to only use slashes as well,\n> > otherwise paths with backslashes cannot be found. This part seems to\n> > be missing.\n> \n> Yes, that's a good point. We _require_ the paths to be normalized\n> here to be Unix-style paths or else the Bloom filter keys are\n> incorrect. Thankfully, they are.\n\nUnfortunately, they aren't always...\n\nPath normalization is done in normalize_path_copy_len(), whose\ndescription says, among other things:\n\n * Performs the following normalizations on src, storing the result in dst:\n * - Ensures that components are separated by '/' (Windows only)\n\nand the code indeed does:\n\n if (is_dir_sep(c)) {\n *dst++ = '/';\n\nNow, while parsing pathspecs this function is called via:\n\n parse_pathspec()\n init_pathspec_item()\n prefix_path_gently()\n normalize_path_copy_len()\n\nUnfortunately, init_pathspec_item() has this chain of conditions:\n\n /* Create match string which will be used for pathspec matching */\n if (pathspec_prefix >= 0) {\n match = xstrdup(copyfrom);\n prefixlen = pathspec_prefix;\n } else if (magic & PATHSPEC_FROMTOP) {\n match = xstrdup(copyfrom);\n prefixlen = 0;\n } else {\n match = prefix_path_gently(prefix, prefixlen,\n &prefixlen, copyfrom);\n if (!match) {\n const char *hint_path = get_git_work_tree();\n if (!hint_path)\n hint_path = get_git_dir();\n die(_(\"%s: '%s' is outside repository at '%s'\"), elt,\n copyfrom, absolute_path(hint_path));\n }\n }\n\nwhich means that it doesn't always calls prefix_path_gently(), which,\nin turn, means that 'pathspec_item->match' might remain un-normalized\nin case of some unusual pathspecs.\n\nThe first condition is supposed to handle the case when one Git\nprocess passes pathspecs to another, and is supposed to be \"internal\nuse only\"; see 233c3e6c59 (parse_pathspec: preserve prefix length via\nPATHSPEC_PREFIX_ORIGIN, 2013-07-14), I haven't even tried to grok what\nthat might entail.\n\nThe second condition handles pathspecs explicitly relative to the root\nof the work tree, i.e. ':/path'. Adding a printf() to show the\noriginal path and the resulting 'pathspec_item->match' does confirm\nthat no normalization is performed:\n\n expecting success of 9999.1 'test': \n mkdir -p dir &&\n >dir/file &&\n git add \":/dir/file\" &&\n git add \":(top)dir/file\" &&\n test_might_fail git add \":/dir//file\" &&\n git add \":(top)dir//file\"\n \n orig: ':/dir/file'\n match: 'dir/file'\n orig: ':(top)dir/file'\n match: 'dir/file'\n orig: ':/dir//file'\n match: 'dir//file'\n fatal: oops in prep_exclude\n orig: ':(top)dir//file'\n match: 'dir//file'\n fatal: oops in prep_exclude\n not ok 1 - test\n\nThis is, of course, bad for Bloom filters, because the repeated\nslashes are hashed as well and commits will be omitted from the output\nof pathspec-limited revision walks, but apparently it also affects\nother parts of Git.\n\nAnd the else branch handles the rest, which, I believe, is by far the\nmost common case.\n\n> Let's make that clear in-code by\n> using '/' instead of is_dir_sep().\n> \n> Thanks,\n> -Stolee\n"},{"id":"400670","messageId":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v2.git.1592934430.gitgitgadget@gmail.com","subject":"[PATCH v3 00/10] More commit-graph/Bloom filter improvements","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:26Z","receivedAt":"2020-06-26T12:30:43Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This builds on sg/commit-graph-cleanups, which took several patches from\nSzeder's series [1] and applied them almost directly to a more-recent\nversion of Git [2].\n\n[1] https://lore.kernel.org/git/20200529085038.26008-1-szeder.dev@gmail.com/\n[2] \nhttps://lore.kernel.org/git/pull.650.git.1591362032.gitgitgadget@gmail.com/\n\nThis series adds a few extra improvements, several of which are rooted in\nSzeder's original series. I maintained his authorship and sign-off, even\nthough the patches did not apply or cherry-pick at all.\n\n(In v2, I have removed the range-diff comparison to Szeder's series, so look\nat the v1 cover letter for that.)\n\nThe patches have been significantly reordered. René pointed out (and Szeder\ndiscovered in the old thread) that we are not re-using the\nbloom_filter_settings from the existing commit-graph when writing a new one.\n\n 1. commit-graph: place bloom_settings in context\n 2. commit-graph: change test to die on parse, not load\n\nThese are mostly the same, except we now use a pointer to the settings in\nthe commit-graph write context.\n\n 3. bloom: get_bloom_filter() cleanups\n\nThis new patch is a subtle change in behavior that will become relevant in\nthe very next patch. In fact, if we swap patch 3 and 4, then\nt4216-log-bloom.sh fails with a segfault due to a NULL filter.\n\n 4. commit-graph: persist existence of changed-paths\n\nThis patch is now updated to use the existing changed-path filter settings.\n\n 5. commit-graph: unify the signatures of all write_graph_chunk_*()\n functions\n 6. commit-graph: simplify chunk writes into loop\n 7. commit-graph: check chunk sizes after writing\n\nThese are all the same as before.\n\n 8. revision.c: fix whitespace\n\nThis patch is the cleanup part of Taylor's patch.\n\n 9. revision: empty pathspecs should not use Bloom filters\n\nHere is Taylor's fix for empty pathspecs.\n\n 10. commit-graph: check all leading directories in changed path Bloom\n filters\n 11. bloom: enforce a minimum size of 8 bytes\n\nFinally, we get these performance patches. Patch 10 is updated to have the\nbetter logic around directory separators and empty paths. Also, the list of\nBloom keys is ordered with the deepest path first. That has some tiny\nperformance benefits for deep paths since we can short-circuit the multi-key\nchecks more often. That code path is much faster than the tree parsing, so\nit is hard to measure any change.\n\nUpdates in V3:\n\n * Responded to René's feedback.\n * Fixed the test in Patch 4 to use GIT_TEST_ variables and extend the\n GIT_TRACE2 depth to work with 'seen' branch.\n\nThanks, -Stolee\n\nDerrick Stolee (5):\n commit-graph: place bloom_settings in context\n commit-graph: change test to die on parse, not load\n bloom: fix logic in get_bloom_filter()\n commit-graph: persist existence of changed-paths\n revision.c: fix whitespace\n\nSZEDER Gábor (4):\n commit-graph: unify the signatures of all write_graph_chunk_*()\n functions\n commit-graph: simplify chunk writes into loop\n commit-graph: check chunk sizes after writing\n commit-graph: check all leading directories in changed path Bloom\n filters\n\nTaylor Blau (1):\n revision: empty pathspecs should not use Bloom filters\n\n Documentation/git-commit-graph.txt | 5 +-\n bloom.c | 14 ++-\n builtin/commit-graph.c | 5 +-\n commit-graph.c | 138 +++++++++++++++++++++--------\n commit-graph.h | 3 +-\n revision.c | 58 +++++++++---\n revision.h | 6 +-\n t/t4216-log-bloom.sh | 23 ++++-\n t/t5318-commit-graph.sh | 2 +-\n 9 files changed, 189 insertions(+), 65 deletions(-)\n\n\nbase-commit: 7fbfe07ab4d4e58c0971dac73001b89f180a0af3\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-659%2Fderrickstolee%2Fbloom-2-v3\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-659/derrickstolee/bloom-2-v3\nPull-Request: https://github.com/gitgitgadget/git/pull/659\n\nRange-diff vs v2:\n\n 1: 57002040bc = 1: 57002040bc commit-graph: place bloom_settings in context\n 2: 6b63f9bd8a = 2: 6b63f9bd8a commit-graph: change test to die on parse, not load\n 3: 492deaf916 ! 3: 2f809499ab bloom: get_bloom_filter() cleanups\n @@ Metadata\n Author: Derrick Stolee \n \n ## Commit message ##\n - bloom: get_bloom_filter() cleanups\n + bloom: fix logic in get_bloom_filter()\n \n The get_bloom_filter() method is a bit complicated in some parts where\n it does not need to be. In particular, it needs to return a NULL filter\n @@ Commit message\n \n Also clean up some style issues while we are here.\n \n + Helped-by: René Scharfe \n Signed-off-by: Derrick Stolee \n \n ## bloom.c ##\n @@ bloom.c: struct bloom_filter *get_bloom_filter(struct repository *r,\n -\t\t\telse\n -\t\t\t\treturn NULL;\n -\t\t}\n -+\t\t r->objects->commit_graph->chunk_bloom_indexes &&\n -+\t\t load_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n -+\t\t\treturn filter;\n ++\t\t r->objects->commit_graph->chunk_bloom_indexes)\n ++\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n \n -\tif (filter->data || !compute_if_not_present)\n +\tif (filter->data)\n \t\treturn filter;\n -+\tif (!filter->data && !compute_if_not_present)\n ++\tif (!compute_if_not_present)\n +\t\treturn NULL;\n \n \trepo_diff_setup(r, &diffopt);\n 4: 8727b25468 ! 4: 33e22d05cb commit-graph: persist existence of changed-paths\n @@ commit-graph.c: static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t\tprogress = start_delayed_progress(\n \t\t\t_(\"Writing changed paths Bloom filters data\"),\n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n + \tint num_chunks = 3;\n + \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n - \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n +-\tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n ++\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n -\tctx->bloom_settings = &bloom_settings;\n -+\tif (!ctx->bloom_settings)\n ++\tif (!ctx->bloom_settings) {\n ++\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n ++\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n ++\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n ++\t\t\t\t\t\t\t bloom_settings.num_hashes);\n +\t\tctx->bloom_settings = &bloom_settings;\n ++\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n @@ t/t4216-log-bloom.sh: test_expect_success 'Use Bloom filters if they exist in th\n \ttest_bloom_filters_used_when_some_filters_are_missing \"-- A/B\"\n '\n \n -+BASE_BDAT_OFFSET=2240\n -+BASE_K_BYTE_OFFSET=$((BASE_BDAT_OFFSET + 10))\n -+BASE_LEN_BYTE_OFFSET=$((BASE_BDAT_OFFSET + 14))\n -+\n -+corrupt_graph() {\n -+\tpos=$1\n -+\tdata=\"${2:-\\0}\"\n -+\tgrepstr=$3\n -+\torig_size=$(wc -c < .git/objects/info/commit-graph) &&\n -+\tzero_pos=${4:-${orig_size}} &&\n -+\tprintf \"$data\" | dd of=\".git/objects/info/commit-graph\" bs=1 seek=\"$pos\" conv=notrunc &&\n -+\tdd of=\".git/objects/info/commit-graph\" bs=1 seek=\"$zero_pos\" if=/dev/null\n -+}\n -+\n +test_expect_success 'persist filter settings' '\n +\ttest_when_finished rm -rf .git/objects/info/commit-graph* &&\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n -+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":7,\\\"bits_per_entry\\\":10}\" trace2.txt &&\n -+\tcp .git/objects/info/commit-graph commit-graph-before &&\n -+\tcorrupt_graph $BASE_K_BYTE_OFFSET \"\\09\" &&\n -+\tcorrupt_graph $BASE_LEN_BYTE_OFFSET \"\\0F\" &&\n -+\tcp .git/objects/info/commit-graph commit-graph-after &&\n -+\ttest_commit c18 A/corrupt &&\n -+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" git commit-graph write --reachable --changed-paths &&\n -+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":57,\\\"bits_per_entry\\\":70}\" trace2.txt\n ++\trm -rf .git/objects/info/commit-graph* &&\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n ++\t\tGIT_TRACE2_EVENT_NESTING=5 \\\n ++\t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n ++\t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n ++\t\tgit commit-graph write --reachable --changed-paths &&\n ++\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n ++\tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n ++\t\tGIT_TRACE2_EVENT_NESTING=5 \\\n ++\t\tgit commit-graph write --reachable --changed-paths &&\n ++\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n +'\n +\n test_done\n 5: 244668fec4 ! 5: 81c45d5260 commit-graph: unify the signatures of all write_graph_chunk_*() functions\n @@ Commit message\n This opens up the possibility for further cleanups and foolproofing in\n the following two patches.\n \n + Helped-by: René Scharfe \n Signed-off-by: SZEDER Gábor \n Signed-off-by: Derrick Stolee \n \n @@ commit-graph.c: static void write_graph_chunk_fanout(struct hashfile *f,\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n -\t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n -+\t\thashwrite(f, (*list)->object.oid.hash, (int)the_hash_algo->rawsz);\n ++\t\thashwrite(f, (*list)->object.oid.hash, the_hash_algo->rawsz);\n \t}\n +\n +\treturn 0;\n 6: 8b959f2f37 ! 6: 8828dcd906 commit-graph: simplify chunk writes into loop\n @@ Commit message\n file. This will open up the possibility for a bit of foolproofing in\n the following patch.\n \n + Helped-by: René Scharfe \n Signed-off-by: SZEDER Gábor \n Signed-off-by: Derrick Stolee \n \n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_con\n -\t\treturn -1;\n +\n +\tfor (i = 0; i < num_chunks; i++) {\n -+\t\tif (chunks[i].write_fn(f, ctx)) {\n -+\t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n -+\t\t\t chunks[i].id);\n ++\t\tif (chunks[i].write_fn(f, ctx))\n +\t\t\treturn -1;\n -+\t\t}\n \t}\n +\n \tstop_progress(&ctx->progress);\n 7: 3eb10933dc ! 7: ddbf297755 commit-graph: check chunk sizes after writing\n @@ Commit message\n in a loop we can do this check in a single place for all chunks, and\n any chunks added in the future will get checked as well.\n \n + Helped-by: René Scharfe \n Signed-off-by: SZEDER Gábor \n Signed-off-by: Derrick Stolee \n \n ## commit-graph.c ##\n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n - \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \n -+\tchunk_offset = f->total + f->offset;\n \tfor (i = 0; i < num_chunks; i++) {\n -+\t\tuint64_t end_offset;\n ++\t\tuint64_t start_offset = f->total + f->offset;\n +\n - \t\tif (chunks[i].write_fn(f, ctx)) {\n - \t\t\terror(_(\"failed writing chunk with id %\"PRIx32\"\"),\n - \t\t\t chunks[i].id);\n + \t\tif (chunks[i].write_fn(f, ctx))\n \t\t\treturn -1;\n - \t\t}\n +\n -+\t\tend_offset = f->total + f->offset;\n -+\t\tif (end_offset - chunk_offset != chunks[i].size)\n ++\t\tif (f->total + f->offset != start_offset + chunks[i].size)\n +\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n -+\t\t\t chunks[i].size, chunks[i].id, end_offset - chunk_offset);\n -+\t\tchunk_offset = end_offset;\n ++\t\t\t chunks[i].size, chunks[i].id,\n ++\t\t\t f->total + f->offset - start_offset);\n \t}\n \n \tstop_progress(&ctx->progress);\n 8: 0bcfc1f051 = 8: 8b63706141 revision.c: fix whitespace\n 9: 719c7091a7 = 9: 7d6163305a revision: empty pathspecs should not use Bloom filters\n 10: 9c2076b4ce ! 10: 40061233ca commit-graph: check all leading directories in changed path Bloom filters\n @@ revision.c: static void prepare_to_use_bloom_filter(struct rev_info *revs)\n -\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n +\tp = path;\n +\twhile (*p) {\n -+\t\tif (is_dir_sep(*p))\n ++\t\t/*\n ++\t\t * At this point, the path is normalized to use Unix-style\n ++\t\t * path separators. This is required due to how the\n ++\t\t * changed-path Bloom filters store the paths.\n ++\t\t */\n ++\t\tif (*p == '/')\n +\t\t\tpath_component_nr++;\n +\t\tp++;\n +\t}\n @@ revision.c: static void prepare_to_use_bloom_filter(struct rev_info *revs)\n +\n +\tp = path + len - 1;\n +\twhile (p > path) {\n -+\t\tif (is_dir_sep(*p))\n ++\t\tif (*p == '/')\n +\t\t\tfill_bloom_key(path, p - path,\n +\t\t\t\t &revs->bloom_keys[path_component_nr++],\n +\t\t\t\t revs->bloom_filter_settings);\n 11: 1022c0ad21 < -: ---------- bloom: enforce a minimum size of 8 bytes\n\n-- \ngitgitgadget\n"},{"id":"400672","messageId":"6b63f9bd8a2a7e18d7ac1be7066d4bcd1df2a729.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 02/10] commit-graph: change test to die on parse, not load","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:28Z","receivedAt":"2020-06-26T12:30:44Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n43d3561 (commit-graph write: don't die if the existing graph is corrupt,\n2019-03-25) introduced the GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD environment\nvariable. This was created to verify that commit-graph was not loaded\nwhen writing a new non-incremental commit-graph.\n\nAn upcoming change wants to load a commit-graph in some valuable cases,\nbut we want to maintain that we don't trust the commit-graph data when\nwriting our new file. Instead of dying on load, instead die if we ever\ntry to parse a commit from the commit-graph. This functionally verifies\nthe same intended behavior, but allows a more advanced feature in the\nnext change.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 ++++++++----\n commit-graph.h | 2 +-\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d0fedcd9b1..6a28d4a5a6 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -564,10 +564,6 @@ static int prepare_commit_graph(struct repository *r)\n \t\treturn !!r->objects->commit_graph;\n \tr->objects->commit_graph_attempted = 1;\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD, 0))\n-\t\tdie(\"dying as requested by the '%s' variable on commit-graph load!\",\n-\t\t GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD);\n-\n \tprepare_repo_settings(r);\n \n \tif (!git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n@@ -790,6 +786,14 @@ static int parse_commit_in_graph_one(struct repository *r,\n \n int parse_commit_in_graph(struct repository *r, struct commit *item)\n {\n+\tstatic int checked_env = 0;\n+\n+\tif (!checked_env &&\n+\t git_env_bool(GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE, 0))\n+\t\tdie(\"dying as requested by the '%s' variable on commit-graph parse!\",\n+\t\t GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE);\n+\tchecked_env = 1;\n+\n \tif (!prepare_commit_graph(r))\n \t\treturn 0;\n \treturn parse_commit_in_graph_one(r, r->objects->commit_graph, item);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 881c9b46e5..f0fb13e3f2 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -5,7 +5,7 @@\n #include \"object-store.h\"\n \n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n-#define GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD \"GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD\"\n+#define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n #define GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS \"GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS\"\n \n /*\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 1073f9e3cf..5ec01abdaa 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -436,7 +436,7 @@ corrupt_graph_verify() {\n \t\tcp $objdir/info/commit-graph commit-graph-pre-write-test\n \tfi &&\n \tgit status --short &&\n-\tGIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD=true git commit-graph write &&\n+\tGIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE=true git commit-graph write &&\n \tgit commit-graph verify\n }\n \n-- \ngitgitgadget\n\n"},{"id":"400671","messageId":"57002040bc83668d4998da6a1e34a6efe287ae22.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 01/10] commit-graph: place bloom_settings in context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:27Z","receivedAt":"2020-06-26T12:30:46Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPlace an instance of struct bloom_settings into the struct\nwrite_commit_graph_context. This allows simplifying the function\nprototype of write_graph_chunk_bloom_data(). This will allow us\nto combine the function prototypes and use function pointers to\nsimplify write_commit_graph_file().\n\nBy using a pointer, we can later replace the settings to match those\nthat exist in the current commit-graph, in case a future Git version\nallows customization of these parameters.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 887837e882..d0fedcd9b1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -882,6 +882,7 @@ struct write_commit_graph_context {\n \n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n+\tconst struct bloom_filter_settings *bloom_settings;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -1103,8 +1104,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n }\n \n static void write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx,\n-\t\t\t\t\t const struct bloom_filter_settings *settings)\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1116,9 +1116,9 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t\t\t_(\"Writing changed paths Bloom filters data\"),\n \t\t\tctx->commits.nr);\n \n-\thashwrite_be32(f, settings->hash_version);\n-\thashwrite_be32(f, settings->num_hashes);\n-\thashwrite_be32(f, settings->bits_per_entry);\n+\thashwrite_be32(f, ctx->bloom_settings->hash_version);\n+\thashwrite_be32(f, ctx->bloom_settings->num_hashes);\n+\thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n@@ -1541,6 +1541,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tstruct object_id file_hash;\n \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n+\tctx->bloom_settings = &bloom_settings;\n+\n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n \n@@ -1642,7 +1644,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n \tif (ctx->changed_paths) {\n \t\twrite_graph_chunk_bloom_indexes(f, ctx);\n-\t\twrite_graph_chunk_bloom_data(f, ctx, &bloom_settings);\n+\t\twrite_graph_chunk_bloom_data(f, ctx);\n \t}\n \tif (ctx->num_commit_graphs_after > 1 &&\n \t write_graph_chunk_base(f, ctx)) {\n-- \ngitgitgadget\n\n"},{"id":"400673","messageId":"2f809499abadd83b81b3d38d0cad9a2fd08b5440.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 03/10] bloom: fix logic in get_bloom_filter()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:29Z","receivedAt":"2020-06-26T12:30:48Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe get_bloom_filter() method is a bit complicated in some parts where\nit does not need to be. In particular, it needs to return a NULL filter\nonly when compute_if_not_present is zero AND the filter data cannot be\nloaded from a commit-graph file. This currently happens by accident\nbecause the commit-graph does not load changed-path Bloom filters from\nan existing commit-graph when writing a new one. This will change in a\nlater patch.\n\nAlso clean up some style issues while we are here.\n\nHelped-by: René Scharfe \nSigned-off-by: Derrick Stolee \n---\n bloom.c | 14 ++++++--------\n 1 file changed, 6 insertions(+), 8 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex c38d1cff0c..2af5389795 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -186,7 +186,7 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n-\tif (bloom_filters.slab_size == 0)\n+\tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n \tfilter = bloom_filter_slab_at(&bloom_filters, c);\n@@ -194,16 +194,14 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (c->graph_pos != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes) {\n-\t\t\tif (load_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n-\t\t\telse\n-\t\t\t\treturn NULL;\n-\t\t}\n+\t\t r->objects->commit_graph->chunk_bloom_indexes)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n \n-\tif (filter->data || !compute_if_not_present)\n+\tif (filter->data)\n \t\treturn filter;\n+\tif (!compute_if_not_present)\n+\t\treturn NULL;\n \n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n-- \ngitgitgadget\n\n"},{"id":"400674","messageId":"33e22d05cbf9d60e5f0164985d3481643220d8fa.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 04/10] commit-graph: persist existence of changed-paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:30Z","receivedAt":"2020-06-26T12:30:50Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe changed-path Bloom filters were released in v2.27.0, but have a\nsignificant drawback. A user can opt-in to writing the changed-path\nfilters using the \"--changed-paths\" option to \"git commit-graph write\"\nbut the next write will drop the filters unless that option is\nspecified.\n\nThis becomes even more important when considering the interaction with\ngc.writeCommitGraph (on by default) or fetch.writeCommitGraph (part of\nfeatures.experimental). These config options trigger commit-graph writes\nthat the user did not signal, and hence there is no --changed-paths\noption available.\n\nAllow a user that opts-in to the changed-path filters to persist the\nproperty of \"my commit-graph has changed-path filters\" automatically. A\nuser can drop filters using the --no-changed-paths option.\n\nIn the process, we need to be extremely careful to match the Bloom\nfilter settings as specified by the commit-graph. This will allow future\nversions of Git to customize these settings, and the version with this\nchange will persist those settings as commit-graphs are rewritten on\ntop.\n\nUse the trace2 API to signal the settings used during the write, and\ncheck that output in a test after manually adjusting the correct bytes\nin the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 5 +++-\n builtin/commit-graph.c | 5 +++-\n commit-graph.c | 45 ++++++++++++++++++++++++++++--\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 17 ++++++++++-\n 5 files changed, 67 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex f4b13c005b..369b222b08 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -60,7 +60,10 @@ existing commit-graph file.\n With the `--changed-paths` option, compute and write information about the\n paths changed between a commit and it's first parent. This operation can\n take a while on large repositories. It provides significant performance gains\n-for getting history of a directory or a file with `git log -- `.\n+for getting history of a directory or a file with `git log -- `. If\n+this option is given, future commit-graph writes will automatically assume\n+that this option was intended. Use `--no-changed-paths` to stop storing this\n+data.\n +\n With the `--split` option, write the commit-graph as a chain of multiple\n commit-graph files stored in `/info/commit-graphs`. The new commits\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 59009837dc..ff7b177c33 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -151,6 +151,7 @@ static int graph_write(int argc, const char **argv)\n \t};\n \n \topts.progress = isatty(2);\n+\topts.enable_changed_paths = -1;\n \tsplit_opts.size_multiple = 2;\n \tsplit_opts.max_commits = 0;\n \tsplit_opts.expire_time = 0;\n@@ -171,7 +172,9 @@ static int graph_write(int argc, const char **argv)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n \tif (opts.progress)\n \t\tflags |= COMMIT_GRAPH_WRITE_PROGRESS;\n-\tif (opts.enable_changed_paths ||\n+\tif (!opts.enable_changed_paths)\n+\t\tflags |= COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS;\n+\tif (opts.enable_changed_paths == 1 ||\n \t git_env_bool(GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS, 0))\n \t\tflags |= COMMIT_GRAPH_WRITE_BLOOM_FILTERS;\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6a28d4a5a6..11088fc11f 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -16,6 +16,8 @@\n #include \"progress.h\"\n #include \"bloom.h\"\n #include \"commit-slab.h\"\n+#include \"json-writer.h\"\n+#include \"trace2.h\"\n \n void git_test_write_commit_graph_or_die(void)\n {\n@@ -1107,6 +1109,21 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tstop_progress(&progress);\n }\n \n+static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n+{\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n+\tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n+\tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_end(&jw);\n+\n+\ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n+\n+\tjw_release(&jw);\n+}\n+\n static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n@@ -1115,6 +1132,8 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \tstruct progress *progress = NULL;\n \tint i = 0;\n \n+\ttrace2_bloom_filter_settings(ctx);\n+\n \tif (ctx->report_progress)\n \t\tprogress = start_delayed_progress(\n \t\t\t_(\"Writing changed paths Bloom filters data\"),\n@@ -1543,9 +1562,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tint num_chunks = 3;\n \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n-\tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n+\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n-\tctx->bloom_settings = &bloom_settings;\n+\tif (!ctx->bloom_settings) {\n+\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n+\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n+\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n+\t\t\t\t\t\t\t bloom_settings.num_hashes);\n+\t\tctx->bloom_settings = &bloom_settings;\n+\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -1970,9 +1995,23 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n \tctx->check_oids = flags & COMMIT_GRAPH_WRITE_CHECK_OIDS ? 1 : 0;\n \tctx->split_opts = split_opts;\n-\tctx->changed_paths = flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS ? 1 : 0;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n+\t\tctx->changed_paths = 1;\n+\tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\n+\t\tstruct commit_graph *g;\n+\t\tprepare_commit_graph_one(ctx->r, ctx->odb);\n+\n+\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\t/* We have changed-paths already. Keep them in the next graph */\n+\t\tif (g && g->chunk_bloom_data) {\n+\t\t\tctx->changed_paths = 1;\n+\t\t\tctx->bloom_settings = g->bloom_filter_settings;\n+\t\t}\n+\t}\n+\n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n \t\tprepare_commit_graph(ctx->r);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex f0fb13e3f2..45b1e5bca3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -96,6 +96,7 @@ enum commit_graph_write_flags {\n \t/* Make sure that each OID in the input is a valid commit OID. */\n \tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3),\n \tCOMMIT_GRAPH_WRITE_BLOOM_FILTERS = (1 << 4),\n+\tCOMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS = (1 << 5),\n };\n \n struct split_commit_graph_opts {\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c7011f33e2..73ed51b595 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -126,7 +126,7 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_commit c14 A/anotherFile2 &&\n \ttest_commit c15 A/B/anotherFile2 &&\n \ttest_commit c16 A/B/C/anotherFile2 &&\n-\tGIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0 git commit-graph write --reachable --split &&\n+\tgit commit-graph write --reachable --split --no-changed-paths &&\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n@@ -152,4 +152,19 @@ test_expect_success 'Use Bloom filters if they exist in the latest but not all c\n \ttest_bloom_filters_used_when_some_filters_are_missing \"-- A/B\"\n '\n \n+test_expect_success 'persist filter settings' '\n+\ttest_when_finished rm -rf .git/objects/info/commit-graph* &&\n+\trm -rf .git/objects/info/commit-graph* &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tGIT_TRACE2_EVENT_NESTING=5 \\\n+\t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n+\t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n+\t\tgit commit-graph write --reachable --changed-paths &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n+\t\tGIT_TRACE2_EVENT_NESTING=5 \\\n+\t\tgit commit-graph write --reachable --changed-paths &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n+'\n+\n test_done\n\\ No newline at end of file\n-- \ngitgitgadget\n\n"},{"id":"400675","messageId":"81c45d5260f4aaf885aa51d19e3ab891aa2988c1.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 05/10] commit-graph: unify the signatures of all write_graph_chunk_*() functions","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:31Z","receivedAt":"2020-06-26T12:30:51Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nUpdate the write_graph_chunk_*() helper functions to have the same\nsignature:\n\n - Return an int error code from all these functions.\n write_graph_chunk_base() already has an int error code, now the\n others will have one, too, but since they don't indicate any\n error, they will always return 0.\n\n - Drop the hash size parameter of write_graph_chunk_oids() and\n write_graph_chunk_data(); its value can be read directly from\n 'the_hash_algo' inside these functions as well.\n\nThis opens up the possibility for further cleanups and foolproofing in\nthe following two patches.\n\nHelped-by: René Scharfe \nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 42 ++++++++++++++++++++++++++----------------\n 1 file changed, 26 insertions(+), 16 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 11088fc11f..d51682998d 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -891,8 +891,8 @@ struct write_commit_graph_context {\n \tconst struct bloom_filter_settings *bloom_settings;\n };\n \n-static void write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_fanout(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tint i, count = 0;\n \tstruct commit **list = ctx->commits.list;\n@@ -913,17 +913,21 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \n \t\thashwrite_be32(f, count);\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_oids(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tint count;\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n-\t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t\thashwrite(f, (*list)->object.oid.hash, the_hash_algo->rawsz);\n \t}\n+\n+\treturn 0;\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -932,8 +936,8 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n \treturn commits[index]->object.oid.hash;\n }\n \n-static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_data(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -950,7 +954,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tdie(_(\"unable to parse commit %s\"),\n \t\t\t\toid_to_hex(&(*list)->object.oid));\n \t\ttree = get_commit_tree_oid(*list);\n-\t\thashwrite(f, tree->hash, hash_len);\n+\t\thashwrite(f, tree->hash, the_hash_algo->rawsz);\n \n \t\tparent = (*list)->parents;\n \n@@ -1030,10 +1034,12 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\tlist++;\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_extra_edges(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1082,10 +1088,12 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \n \t\tlist++;\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1107,6 +1115,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n \t}\n \n \tstop_progress(&progress);\n+\treturn 0;\n }\n \n static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n@@ -1124,8 +1133,8 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_release(&jw);\n }\n \n-static void write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_bloom_data(struct hashfile *f,\n+\t\t\t\t\tstruct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1151,6 +1160,7 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t}\n \n \tstop_progress(&progress);\n+\treturn 0;\n }\n \n static int oid_compare(const void *_a, const void *_b)\n@@ -1667,8 +1677,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, hashsz, ctx);\n-\twrite_graph_chunk_data(f, hashsz, ctx);\n+\twrite_graph_chunk_oids(f, ctx);\n+\twrite_graph_chunk_data(f, ctx);\n \tif (ctx->num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n \tif (ctx->changed_paths) {\n-- \ngitgitgadget\n\n"},{"id":"400677","messageId":"8828dcd9068abdc44d321900280fdb51d2b7f138.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 06/10] commit-graph: simplify chunk writes into loop","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:32Z","receivedAt":"2020-06-26T12:30:52Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn write_commit_graph_file() we now have one block of code filling the\narray of 'struct chunk_info' with the IDs and sizes of chunks to be\nwritten, and an other block of code calling the functions responsible\nfor writing individual chunks. In case of optional chunks like Extra\nEdge List an Base Graphs List there is also a condition checking\nwhether that chunk is necessary/desired, and that same condition is\nrepeated in both blocks of code. Other, newer chunks have similar\noptional conditions.\n\nEliminate these repeated conditions by storing the function pointers\nresponsible for writing individual chunks in the 'struct chunk_info'\narray as well, and calling them in a loop to write the commit-graph\nfile. This will open up the possibility for a bit of foolproofing in\nthe following patch.\n\nHelped-by: René Scharfe \nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 28 ++++++++++++++++------------\n 1 file changed, 16 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d51682998d..e43ee58ea6 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1555,9 +1555,13 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n+typedef int (*chunk_write_fn)(struct hashfile *f,\n+\t\t\t struct write_commit_graph_context *ctx);\n+\n struct chunk_info {\n \tuint32_t id;\n \tuint64_t size;\n+\tchunk_write_fn write_fn;\n };\n \n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n@@ -1620,27 +1624,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n \tchunks[0].size = GRAPH_FANOUT_SIZE;\n+\tchunks[0].write_fn = write_graph_chunk_fanout;\n \tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunks[1].size = hashsz * ctx->commits.nr;\n+\tchunks[1].write_fn = write_graph_chunk_oids;\n \tchunks[2].id = GRAPH_CHUNKID_DATA;\n \tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n+\tchunks[2].write_fn = write_graph_chunk_data;\n \tif (ctx->num_extra_edges) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n \t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n \t\tnum_chunks++;\n \t}\n \tif (ctx->changed_paths) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n \t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n \t\tnum_chunks++;\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n \t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n \t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n \t\tnum_chunks++;\n \t}\n \n@@ -1676,19 +1687,12 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tprogress_title.buf,\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n-\twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, ctx);\n-\twrite_graph_chunk_data(f, ctx);\n-\tif (ctx->num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, ctx);\n-\tif (ctx->changed_paths) {\n-\t\twrite_graph_chunk_bloom_indexes(f, ctx);\n-\t\twrite_graph_chunk_bloom_data(f, ctx);\n-\t}\n-\tif (ctx->num_commit_graphs_after > 1 &&\n-\t write_graph_chunk_base(f, ctx)) {\n-\t\treturn -1;\n+\n+\tfor (i = 0; i < num_chunks; i++) {\n+\t\tif (chunks[i].write_fn(f, ctx))\n+\t\t\treturn -1;\n \t}\n+\n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n-- \ngitgitgadget\n\n"},{"id":"400676","messageId":"ddbf297755cc133fd3111a4accfee306e9f27c9b.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 07/10] commit-graph: check chunk sizes after writing","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:33Z","receivedAt":"2020-06-26T12:30:53Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn my experience while experimenting with new commit-graph chunks,\nearly versions of the corresponding new write_commit_graph_my_chunk()\nfunctions are, sadly but not surprisingly, often buggy, and write more\nor less data than they are supposed to, especially if the chunk size\nis not directly proportional to the number of commits. This then\ncauses all kinds of issues when reading such a bogus commit-graph\nfile, raising the question of whether the writing or the reading part\nhappens to be buggy this time.\n\nLet's catch such issues early, already when writing the commit-graph\nfile, and check that each write_graph_chunk_*() function wrote the\namount of data that it was expected to, and what has been encoded in\nthe Chunk Lookup table. Now that all commit-graph chunks are written\nin a loop we can do this check in a single place for all chunks, and\nany chunks added in the future will get checked as well.\n\nHelped-by: René Scharfe \nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e43ee58ea6..a0766a86f5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1689,8 +1689,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t}\n \n \tfor (i = 0; i < num_chunks; i++) {\n+\t\tuint64_t start_offset = f->total + f->offset;\n+\n \t\tif (chunks[i].write_fn(f, ctx))\n \t\t\treturn -1;\n+\n+\t\tif (f->total + f->offset != start_offset + chunks[i].size)\n+\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n+\t\t\t chunks[i].size, chunks[i].id,\n+\t\t\t f->total + f->offset - start_offset);\n \t}\n \n \tstop_progress(&ctx->progress);\n-- \ngitgitgadget\n\n"},{"id":"400678","messageId":"40061233ca2dee2ecdb9f197fc3e5999793594b3.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 10/10] commit-graph: check all leading directories in changed path Bloom filters","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:36Z","receivedAt":"2020-06-26T12:30:57Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nThe file 'dir/subdir/file' can only be modified if its leading\ndirectories 'dir' and 'dir/subdir' are modified as well.\n\nSo when checking modified path Bloom filters looking for commits\nmodifying a path with multiple path components, then check not only\nthe full path in the Bloom filters, but all its leading directories as\nwell. Take care to check these paths in \"deepest first\" order,\nbecause it's the full path that is least likely to be modified, and\nthe Bloom filter queries can short circuit sooner.\n\nThis can significantly reduce the average false positive rate, by\nabout an order of magnitude or three(!), and can further speed up\npathspec-limited revision walks. The table below compares the average\nfalse positive rate and runtime of\n\n git rev-list HEAD -- \"$path\"\n\nbefore and after this change for 5000+ randomly* selected paths from\neach repository:\n\n Average false Average Average\n positive rate runtime runtime\n before after before after difference\n ------------------------------------------------------------------\n git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n\n*Path selection was done with the following pipeline:\n\n\tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n\nThe improvements in runtime are much smaller than the improvements in\naverage false positive rate, as we are clearly reaching diminishing\nreturns here. However, all these timings depend on that accessing\ntree objects is reasonably fast (warm caches). If we had a partial\nclone and the tree objects had to be fetched from a promisor remote,\ne.g.:\n\n $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n commit-graph write --reachable\n $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n rev-list HEAD -- \"$path\"\n\nthen checking all leading path component can reduce the runtime from\nover an hour to a few seconds (and this is with the clone and the\npromisor on the same machine).\n\nThis adjusts the tracing values in t4216-log-bloom.sh, which provides a\nconcrete way to notice the improvement.\n\nHelped-by: Taylor Blau \nHelped-by: René Scharfe \nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n revision.c | 46 +++++++++++++++++++++++++++++++++++---------\n revision.h | 6 ++++--\n t/t4216-log-bloom.sh | 2 +-\n 3 files changed, 42 insertions(+), 12 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex b53377cd52..b40bc5b51b 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -670,9 +670,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n {\n \tstruct pathspec_item *pi;\n \tchar *path_alloc = NULL;\n-\tconst char *path;\n+\tconst char *path, *p;\n \tint last_index;\n-\tint len;\n+\tsize_t len;\n+\tint path_component_nr = 1;\n \n \tif (!revs->commits)\n \t\treturn;\n@@ -709,8 +710,33 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \t\treturn;\n \t}\n \n-\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n-\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n+\tp = path;\n+\twhile (*p) {\n+\t\t/*\n+\t\t * At this point, the path is normalized to use Unix-style\n+\t\t * path separators. This is required due to how the\n+\t\t * changed-path Bloom filters store the paths.\n+\t\t */\n+\t\tif (*p == '/')\n+\t\t\tpath_component_nr++;\n+\t\tp++;\n+\t}\n+\n+\trevs->bloom_keys_nr = path_component_nr;\n+\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n+\n+\tfill_bloom_key(path, len, &revs->bloom_keys[0],\n+\t\t revs->bloom_filter_settings);\n+\tpath_component_nr = 1;\n+\n+\tp = path + len - 1;\n+\twhile (p > path) {\n+\t\tif (*p == '/')\n+\t\t\tfill_bloom_key(path, p - path,\n+\t\t\t\t &revs->bloom_keys[path_component_nr++],\n+\t\t\t\t revs->bloom_filter_settings);\n+\t\tp--;\n+\t}\n \n \tif (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n \t\tatexit(trace2_bloom_filter_statistics_atexit);\n@@ -724,7 +750,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\t\t\t\t\t struct commit *commit)\n {\n \tstruct bloom_filter *filter;\n-\tint result;\n+\tint result = 1, j;\n \n \tif (!revs->repo->objects->commit_graph)\n \t\treturn -1;\n@@ -744,9 +770,11 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\treturn -1;\n \t}\n \n-\tresult = bloom_filter_contains(filter,\n-\t\t\t\t revs->bloom_key,\n-\t\t\t\t revs->bloom_filter_settings);\n+\tfor (j = 0; result && j < revs->bloom_keys_nr; j++) {\n+\t\tresult = bloom_filter_contains(filter,\n+\t\t\t\t\t &revs->bloom_keys[j],\n+\t\t\t\t\t revs->bloom_filter_settings);\n+\t}\n \n \tif (result)\n \t\tcount_bloom_filter_maybe++;\n@@ -786,7 +814,7 @@ static int rev_compare_tree(struct rev_info *revs,\n \t\t\treturn REV_TREE_SAME;\n \t}\n \n-\tif (revs->bloom_key && !nth_parent) {\n+\tif (revs->bloom_keys_nr && !nth_parent) {\n \t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, commit);\n \n \t\tif (bloom_ret == 0)\ndiff --git a/revision.h b/revision.h\nindex 7c026fe41f..abbfb4ab59 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -295,8 +295,10 @@ struct rev_info {\n \tstruct topo_walk_info *topo_walk_info;\n \n \t/* Commit graph bloom filter fields */\n-\t/* The bloom filter key for the pathspec */\n-\tstruct bloom_key *bloom_key;\n+\t/* The bloom filter key(s) for the pathspec */\n+\tstruct bloom_key *bloom_keys;\n+\tint bloom_keys_nr;\n+\n \t/*\n \t * The bloom filter settings used to generate the key.\n \t * This is loaded from the commit-graph being used.\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex e3e4badd4c..d7dd717347 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -146,7 +146,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n-- \ngitgitgadget\n"},{"id":"400679","messageId":"7d6163305aa6ba295d5bc5ee685b158ae2853d93.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 09/10] revision: empty pathspecs should not use Bloom filters","fromName":"Taylor Blau via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:35Z","receivedAt":"2020-06-26T12:30:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Taylor Blau \n\nThe prepare_to_use_bloom_filter() method was not intended to be called\non an empty pathspec. However, 'git log -- .' and 'git log' are subtly\ndifferent: the latter reports all commits while the former will simplify\ncommits that do not change the root tree.\n\nThis means that the path used to construct the bloom_key might be empty,\nand that value is not added to the Bloom filter during construction.\nThat means that the results are likely incorrect!\n\nTo resolve the issue, be careful about the length of the path and stop\nfilling Bloom filters. To be completely sure we do not use them, drop\nthe pointer to the bloom_filter_settings from the commit-graph. That\nallows our test to look at the trace2 logs to verify no Bloom filter\nstatistics are reported.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n revision.c | 4 ++++\n t/t4216-log-bloom.sh | 4 ++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex ed59084f50..b53377cd52 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -704,6 +704,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \t\tpath = pi->match;\n \n \tlen = strlen(path);\n+\tif (!len) {\n+\t\trevs->bloom_filter_settings = NULL;\n+\t\treturn;\n+\t}\n \n \trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n \tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 73ed51b595..e3e4badd4c 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -112,6 +112,10 @@ test_expect_success 'git log -- multiple path specs does not use Bloom filters'\n \ttest_bloom_filters_not_used \"-- file4 A/file1\"\n '\n \n+test_expect_success 'git log -- \".\" pathspec at root does not use Bloom filters' '\n+\ttest_bloom_filters_not_used \"-- .\"\n+'\n+\n test_expect_success 'git log with wildcard that resolves to a single path uses Bloom filters' '\n \ttest_bloom_filters_used \"-- *4\" &&\n \ttest_bloom_filters_used \"-- *renamed\"\n-- \ngitgitgadget\n\n"},{"id":"400680","messageId":"8b6370614124dba15902db43dd1db496ecc62612.1593174637.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v3 08/10] revision.c: fix whitespace","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-06-26T12:30:34Z","receivedAt":"2020-06-26T12:30:59Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nHere, four spaces were used instead of tab characters.\n\nReported-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n revision.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex c644c66091..ed59084f50 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -697,11 +697,11 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \t/* remove single trailing slash from path, if needed */\n \tif (pi->match[last_index] == '/') {\n-\t path_alloc = xstrdup(pi->match);\n-\t path_alloc[last_index] = '\\0';\n-\t path = path_alloc;\n+\t\tpath_alloc = xstrdup(pi->match);\n+\t\tpath_alloc[last_index] = '\\0';\n+\t\tpath = path_alloc;\n \t} else\n-\t path = pi->match;\n+\t\tpath = pi->match;\n \n \tlen = strlen(path);\n \n-- \ngitgitgadget\n\n"},{"id":"400690","messageId":"452d363a-d3c1-e80f-2354-13d2070afbee@gmail.com","threadId":"53681","inReplyTo":"20200626063436.GA11341@szeder.dev","subject":"Re: [PATCH v2 10/11] commit-graph: check all leading directories in changed path Bloom filters","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-26T14:42:45Z","receivedAt":"2020-06-26T14:42:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/26/2020 2:34 AM, SZEDER Gábor wrote:\n> On Thu, Jun 25, 2020 at 11:05:04AM -0400, Derrick Stolee wrote:\n> \n>>>> +\twhile (p > path) {\n>>>> +\t\tif (is_dir_sep(*p))\n>>>> +\t\t\tfill_bloom_key(path, p - path,\n>>>> +\t\t\t\t &revs->bloom_keys[path_component_nr++],\n>>>> +\t\t\t\t revs->bloom_filter_settings);\n>>>> +\t\tp--;\n>>>> +\t}\n>>>\n>>> This walks the directory hierarchy upwards and adds bloom filters for\n>>> shorter and shorter paths, (\"deepest first\"). Good.\n>>>\n>>> And it supports all directory separators. On Windows that would be\n>>> slash (/) and backslash (\\). I assume paths are normalized to use\n>>> only slashes when bloom filters are written, correct? Then the lookup\n>>> side needs to normalize a given path to only use slashes as well,\n>>> otherwise paths with backslashes cannot be found. This part seems to\n>>> be missing.\n>>\n>> Yes, that's a good point. We _require_ the paths to be normalized\n>> here to be Unix-style paths or else the Bloom filter keys are\n>> incorrect. Thankfully, they are.\n> \n> Unfortunately, they aren't always...\n> \n> Path normalization is done in normalize_path_copy_len(), whose\n> description says, among other things:\n> \n> * Performs the following normalizations on src, storing the result in dst:\n> * - Ensures that components are separated by '/' (Windows only)\n> \n> and the code indeed does:\n> \n> if (is_dir_sep(c)) {\n> *dst++ = '/';\n> \n> Now, while parsing pathspecs this function is called via:\n> \n> parse_pathspec()\n> init_pathspec_item()\n> prefix_path_gently()\n> normalize_path_copy_len()\n> \n> Unfortunately, init_pathspec_item() has this chain of conditions:\n> \n> /* Create match string which will be used for pathspec matching */\n> if (pathspec_prefix >= 0) {\n> match = xstrdup(copyfrom);\n> prefixlen = pathspec_prefix;\n> } else if (magic & PATHSPEC_FROMTOP) {\n> match = xstrdup(copyfrom);\n> prefixlen = 0;\n> } else {\n> match = prefix_path_gently(prefix, prefixlen,\n> &prefixlen, copyfrom);\n> if (!match) {\n> const char *hint_path = get_git_work_tree();\n> if (!hint_path)\n> hint_path = get_git_dir();\n> die(_(\"%s: '%s' is outside repository at '%s'\"), elt,\n> copyfrom, absolute_path(hint_path));\n> }\n> }\n> \n> which means that it doesn't always calls prefix_path_gently(), which,\n> in turn, means that 'pathspec_item->match' might remain un-normalized\n> in case of some unusual pathspecs.\n> \n> The first condition is supposed to handle the case when one Git\n> process passes pathspecs to another, and is supposed to be \"internal\n> use only\"; see 233c3e6c59 (parse_pathspec: preserve prefix length via\n> PATHSPEC_PREFIX_ORIGIN, 2013-07-14), I haven't even tried to grok what\n> that might entail.\n> \n> The second condition handles pathspecs explicitly relative to the root\n> of the work tree, i.e. ':/path'. Adding a printf() to show the\n> original path and the resulting 'pathspec_item->match' does confirm\n> that no normalization is performed:\n> \n> expecting success of 9999.1 'test': \n> mkdir -p dir &&\n> >dir/file &&\n> git add \":/dir/file\" &&\n> git add \":(top)dir/file\" &&\n> test_might_fail git add \":/dir//file\" &&\n> git add \":(top)dir//file\"\n> \n> orig: ':/dir/file'\n> match: 'dir/file'\n> orig: ':(top)dir/file'\n> match: 'dir/file'\n> orig: ':/dir//file'\n> match: 'dir//file'\n> fatal: oops in prep_exclude\n> orig: ':(top)dir//file'\n> match: 'dir//file'\n> fatal: oops in prep_exclude\n> not ok 1 - test\n> \n> This is, of course, bad for Bloom filters, because the repeated\n> slashes are hashed as well and commits will be omitted from the output\n> of pathspec-limited revision walks, but apparently it also affects\n> other parts of Git.\n> \n> And the else branch handles the rest, which, I believe, is by far the\n> most common case.\n\nThanks for this analysis. Clearly, there is already a bug here\nwhen the input data is not pristine. I didn't see this message\nwhen I submitted my v3, but normalizing the path data before\ncomputing filters can (hopefully) be done as a small patch\nbefore or after my v3 PATCH 10 without much conflict.\n\nThanks,\n-Stolee\n"},{"id":"400731","messageId":"20200627163335.GO2898@szeder.dev","threadId":"53681","inReplyTo":"2f809499abadd83b81b3d38d0cad9a2fd08b5440.1593174637.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v3 03/10] bloom: fix logic in get_bloom_filter()","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-06-27T16:33:35Z","receivedAt":"2020-06-27T16:33:42Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Jun 26, 2020 at 12:30:29PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The get_bloom_filter() method is a bit complicated in some parts where\n> it does not need to be. In particular, it needs to return a NULL filter\n> only when compute_if_not_present is zero AND the filter data cannot be\n> loaded from a commit-graph file. This currently happens by accident\n> because the commit-graph does not load changed-path Bloom filters from\n> an existing commit-graph when writing a new one. This will change in a\n> later patch.\n> \n> Also clean up some style issues while we are here.\n> \n> Helped-by: René Scharfe \n> Signed-off-by: Derrick Stolee \n> ---\n> bloom.c | 14 ++++++--------\n> 1 file changed, 6 insertions(+), 8 deletions(-)\n> \n> diff --git a/bloom.c b/bloom.c\n> index c38d1cff0c..2af5389795 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -186,7 +186,7 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n> \tstruct diff_options diffopt;\n> \tint max_changes = 512;\n> \n> -\tif (bloom_filters.slab_size == 0)\n> +\tif (!bloom_filters.slab_size)\n> \t\treturn NULL;\n>\n> \tfilter = bloom_filter_slab_at(&bloom_filters, c);\n> @@ -194,16 +194,14 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n> \tif (!filter->data) {\n> \t\tload_commit_graph_info(r, c);\n> \t\tif (c->graph_pos != COMMIT_NOT_FROM_GRAPH &&\n> -\t\t\tr->objects->commit_graph->chunk_bloom_indexes) {\n> -\t\t\tif (load_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n> -\t\t\t\treturn filter;\n> -\t\t\telse\n> -\t\t\t\treturn NULL;\n> -\t\t}\n> +\t\t r->objects->commit_graph->chunk_bloom_indexes)\n> +\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n> \t}\n> \n> -\tif (filter->data || !compute_if_not_present)\n> +\tif (filter->data)\n> \t\treturn filter;\n> +\tif (!compute_if_not_present)\n> +\t\treturn NULL;\n\nSome callers of get_bloom_filter() invoke it with\ncompute_if_not_present=0, but are not prepared to handle a NULL return\nvalue and dereference it right away:\n\n write_graph_chunk_bloom_indexes():\n\n struct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n cur_pos += filter->len;\n\n write_graph_chunk_bloom_data():\n\n struct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n display_progress(progress, ++i);\n hashwrite(f, filter->data, filter->len * sizeof(unsigned char));\n\nI don't know whether this was an issue before, but I didn't really\ntried. Unfortunately, starting with this patch this causes\nsegmentation faults basically in all real repositories I use for\ntesting.\n\n expecting success of 9999.1 'test': \n for i in $(test_seq 1 513)\n do\n >file-$i || return 1\n done &&\n git add file-* &&\n git commit -q -m one &&\n \n git commit-graph write --reachable --changed-paths\n \n Segmentation fault\n not ok 1 - test\n\n\n Program received signal SIGSEGV, Segmentation fault.\n 0x0000000000515848 in write_graph_chunk_bloom_indexes (f=0x9fe650, \n ctx=0x9d2000) at commit-graph.c:1101\n 1101 cur_pos += filter->len;\n (gdb) print filter\n $1 = (struct bloom_filter *) 0x0\n\n\n\n> \trepo_diff_setup(r, &diffopt);\n> \tdiffopt.flags.recursive = 1;\n> -- \n> gitgitgadget\n> \n"},{"id":"400754","messageId":"e998499b-6c27-e4c8-fa92-d83167ab41b6@gmail.com","threadId":"53681","inReplyTo":"20200627163335.GO2898@szeder.dev","subject":"Re: [PATCH v3 03/10] bloom: fix logic in get_bloom_filter()","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-06-29T13:02:52Z","receivedAt":"2020-06-29T19:03:14Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 6/27/2020 12:33 PM, SZEDER Gábor wrote:\n> On Fri, Jun 26, 2020 at 12:30:29PM +0000, Derrick Stolee via GitGitGadget wrote:\n>> From: Derrick Stolee \n>>\n>> The get_bloom_filter() method is a bit complicated in some parts where\n>> it does not need to be. In particular, it needs to return a NULL filter\n>> only when compute_if_not_present is zero AND the filter data cannot be\n>> loaded from a commit-graph file. This currently happens by accident\n>> because the commit-graph does not load changed-path Bloom filters from\n>> an existing commit-graph when writing a new one. This will change in a\n>> later patch.\n>>\n>> Also clean up some style issues while we are here.\n>>\n>> Helped-by: René Scharfe \n>> Signed-off-by: Derrick Stolee \n>> ---\n>> bloom.c | 14 ++++++--------\n>> 1 file changed, 6 insertions(+), 8 deletions(-)\n>>\n>> diff --git a/bloom.c b/bloom.c\n>> index c38d1cff0c..2af5389795 100644\n>> --- a/bloom.c\n>> +++ b/bloom.c\n>> @@ -186,7 +186,7 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n>> \tstruct diff_options diffopt;\n>> \tint max_changes = 512;\n>> \n>> -\tif (bloom_filters.slab_size == 0)\n>> +\tif (!bloom_filters.slab_size)\n>> \t\treturn NULL;\n>>\n>> \tfilter = bloom_filter_slab_at(&bloom_filters, c);\n>> @@ -194,16 +194,14 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n>> \tif (!filter->data) {\n>> \t\tload_commit_graph_info(r, c);\n>> \t\tif (c->graph_pos != COMMIT_NOT_FROM_GRAPH &&\n>> -\t\t\tr->objects->commit_graph->chunk_bloom_indexes) {\n>> -\t\t\tif (load_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n>> -\t\t\t\treturn filter;\n>> -\t\t\telse\n>> -\t\t\t\treturn NULL;\n>> -\t\t}\n>> +\t\t r->objects->commit_graph->chunk_bloom_indexes)\n>> +\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n>> \t}\n>> \n>> -\tif (filter->data || !compute_if_not_present)\n>> +\tif (filter->data)\n>> \t\treturn filter;\n>> +\tif (!compute_if_not_present)\n>> +\t\treturn NULL;\n> \n> Some callers of get_bloom_filter() invoke it with\n> compute_if_not_present=0, but are not prepared to handle a NULL return\n> value and dereference it right away:\n> \n> write_graph_chunk_bloom_indexes():\n> \n> struct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n> cur_pos += filter->len;\n> \n> write_graph_chunk_bloom_data():\n> \n> struct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n> display_progress(progress, ++i);\n> hashwrite(f, filter->data, filter->len * sizeof(unsigned char));\n\nIn theory, these _should_ be safe, because we already computed\nthe filters in an earlier step, right? We should have generated\nthe filter and populated it in the slab.\n\n> I don't know whether this was an issue before, but I didn't really\n> tried. Unfortunately, starting with this patch this causes\n> segmentation faults basically in all real repositories I use for\n> testing.\n> \n> expecting success of 9999.1 'test': \n> for i in $(test_seq 1 513)\n> do\n> >file-$i || return 1\n> done &&\n> git add file-* &&\n> git commit -q -m one &&\n> \n> git commit-graph write --reachable --changed-paths\n> \n> Segmentation fault\n> not ok 1 - test\n\nHowever, you are demonstrating a failure that doesn't appear\nin our test suite. I was able to reproduce it.\n\nI can confirm that this patch causes a SIGSEGV when writing\nthe commit-graph in the Git repository, too.\n\nSo, what is wrong with my earlier assumption? There are\ntwo problems.\n\nThe thing I notice is that an empty filter (no changes\nwith respect to the first parent) will have NULL\nfilter->data, so we are returning NULL instead of a\ncorrectly-empty filter (with len zero).\n\nBut what you are hitting here is the max number of changes\nlimit. That also returns a NULL filter, because we mark\nthe filter as \"TOO LARGE\" to store. We store that as a\nzero-length filter.\n\nThe following fixup corrects the bug and adds a test\nsimilar to yours, but with extra care around ensuring the\nrevision walk still works appropriately for that large\ncommit.\n\nIn the next version, I will include more in the commit\nmessage about these side-effect changes, especially around\nthe stats for zero-length filters. The trace2 message will\nno longer differentiate between zero-length filters and\nNULL filters.\n\nThanks,\n-Stolee\n\n-- >8 --\n\nFrom f9867adc5de8a072f41b91fd6cd87edfcc92e05e Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Mon, 29 Jun 2020 08:52:33 -0400\nSubject: [PATCH] fixup! bloom: fix logic in get_bloom_filter()\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 8 ++++++--\n revision.c | 7 -------\n t/t4216-log-bloom.sh | 24 ++++++++++++++++++++++--\n 3 files changed, 28 insertions(+), 11 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a0766a86f5..6752916c1a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1108,7 +1108,8 @@ static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \n \twhile (list < last) {\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n-\t\tcur_pos += filter->len;\n+\t\tsize_t len = filter ? filter->len : 0;\n+\t\tcur_pos += len;\n \t\tdisplay_progress(progress, ++i);\n \t\thashwrite_be32(f, cur_pos);\n \t\tlist++;\n@@ -1154,8 +1155,11 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \n \twhile (list < last) {\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tsize_t len = filter ? filter->len : 0;\n \t\tdisplay_progress(progress, ++i);\n-\t\thashwrite(f, filter->data, filter->len * sizeof(unsigned char));\n+\n+\t\tif (len)\n+\t\t\thashwrite(f, filter->data, len * sizeof(unsigned char));\n \t\tlist++;\n \t}\n \ndiff --git a/revision.c b/revision.c\nindex b40bc5b51b..b9118001f9 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -633,7 +633,6 @@ static unsigned int count_bloom_filter_maybe;\n static unsigned int count_bloom_filter_definitely_not;\n static unsigned int count_bloom_filter_false_positive;\n static unsigned int count_bloom_filter_not_present;\n-static unsigned int count_bloom_filter_length_zero;\n \n static void trace2_bloom_filter_statistics_atexit(void)\n {\n@@ -641,7 +640,6 @@ static void trace2_bloom_filter_statistics_atexit(void)\n \n \tjw_object_begin(&jw, 0);\n \tjw_object_intmax(&jw, \"filter_not_present\", count_bloom_filter_not_present);\n-\tjw_object_intmax(&jw, \"zero_length_filter\", count_bloom_filter_length_zero);\n \tjw_object_intmax(&jw, \"maybe\", count_bloom_filter_maybe);\n \tjw_object_intmax(&jw, \"definitely_not\", count_bloom_filter_definitely_not);\n \tjw_object_intmax(&jw, \"false_positive\", count_bloom_filter_false_positive);\n@@ -765,11 +763,6 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\treturn -1;\n \t}\n \n-\tif (!filter->len) {\n-\t\tcount_bloom_filter_length_zero++;\n-\t\treturn -1;\n-\t}\n-\n \tfor (j = 0; result && j < revs->bloom_keys_nr; j++) {\n \t\tresult = bloom_filter_contains(filter,\n \t\t\t\t\t &revs->bloom_keys[j],\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex d7dd717347..4892364e74 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -60,7 +60,7 @@ setup () {\n \n test_bloom_filters_used () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"zero_length_filter\\\":0,\\\"maybe\\\"\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom &&\n@@ -146,7 +146,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n@@ -171,4 +171,24 @@ test_expect_success 'persist filter settings' '\n \tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n '\n \n+test_expect_success 'correctly report changes over limit' '\n+\tgit init 513changes &&\n+\t(\n+\t\tcd 513changes &&\n+\t\tfor i in $(test_seq 1 513)\n+\t\tdo\n+\t\t\techo $i >file$i.txt || return 1\n+\t\tdone &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"files\" &&\n+\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tfor i in $(test_seq 1 513)\n+\t\tdo\n+\t\t\tgit -c core.commitGraph=false log -- file$i.txt >expect &&\n+\t\t\tgit log -- file$i.txt >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone\n+\t)\n+'\n+\n test_done\n\\ No newline at end of file\n-- \n2.27.0.203.gf402ea6816\n\n"},{"id":"400894","messageId":"57002040bc83668d4998da6a1e34a6efe287ae22.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 01/10] commit-graph: place bloom_settings in context","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:21Z","receivedAt":"2020-07-01T13:27:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nPlace an instance of struct bloom_settings into the struct\nwrite_commit_graph_context. This allows simplifying the function\nprototype of write_graph_chunk_bloom_data(). This will allow us\nto combine the function prototypes and use function pointers to\nsimplify write_commit_graph_file().\n\nBy using a pointer, we can later replace the settings to match those\nthat exist in the current commit-graph, in case a future Git version\nallows customization of these parameters.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 887837e882..d0fedcd9b1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -882,6 +882,7 @@ struct write_commit_graph_context {\n \n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n+\tconst struct bloom_filter_settings *bloom_settings;\n };\n \n static void write_graph_chunk_fanout(struct hashfile *f,\n@@ -1103,8 +1104,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n }\n \n static void write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx,\n-\t\t\t\t\t const struct bloom_filter_settings *settings)\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1116,9 +1116,9 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t\t\t_(\"Writing changed paths Bloom filters data\"),\n \t\t\tctx->commits.nr);\n \n-\thashwrite_be32(f, settings->hash_version);\n-\thashwrite_be32(f, settings->num_hashes);\n-\thashwrite_be32(f, settings->bits_per_entry);\n+\thashwrite_be32(f, ctx->bloom_settings->hash_version);\n+\thashwrite_be32(f, ctx->bloom_settings->num_hashes);\n+\thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n@@ -1541,6 +1541,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tstruct object_id file_hash;\n \tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n+\tctx->bloom_settings = &bloom_settings;\n+\n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n \n@@ -1642,7 +1644,7 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n \tif (ctx->changed_paths) {\n \t\twrite_graph_chunk_bloom_indexes(f, ctx);\n-\t\twrite_graph_chunk_bloom_data(f, ctx, &bloom_settings);\n+\t\twrite_graph_chunk_bloom_data(f, ctx);\n \t}\n \tif (ctx->num_commit_graphs_after > 1 &&\n \t write_graph_chunk_base(f, ctx)) {\n-- \ngitgitgadget\n\n"},{"id":"400895","messageId":"6b63f9bd8a2a7e18d7ac1be7066d4bcd1df2a729.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 02/10] commit-graph: change test to die on parse, not load","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:22Z","receivedAt":"2020-07-01T13:27:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\n43d3561 (commit-graph write: don't die if the existing graph is corrupt,\n2019-03-25) introduced the GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD environment\nvariable. This was created to verify that commit-graph was not loaded\nwhen writing a new non-incremental commit-graph.\n\nAn upcoming change wants to load a commit-graph in some valuable cases,\nbut we want to maintain that we don't trust the commit-graph data when\nwriting our new file. Instead of dying on load, instead die if we ever\ntry to parse a commit from the commit-graph. This functionally verifies\nthe same intended behavior, but allows a more advanced feature in the\nnext change.\n\nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 12 ++++++++----\n commit-graph.h | 2 +-\n t/t5318-commit-graph.sh | 2 +-\n 3 files changed, 10 insertions(+), 6 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d0fedcd9b1..6a28d4a5a6 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -564,10 +564,6 @@ static int prepare_commit_graph(struct repository *r)\n \t\treturn !!r->objects->commit_graph;\n \tr->objects->commit_graph_attempted = 1;\n \n-\tif (git_env_bool(GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD, 0))\n-\t\tdie(\"dying as requested by the '%s' variable on commit-graph load!\",\n-\t\t GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD);\n-\n \tprepare_repo_settings(r);\n \n \tif (!git_env_bool(GIT_TEST_COMMIT_GRAPH, 0) &&\n@@ -790,6 +786,14 @@ static int parse_commit_in_graph_one(struct repository *r,\n \n int parse_commit_in_graph(struct repository *r, struct commit *item)\n {\n+\tstatic int checked_env = 0;\n+\n+\tif (!checked_env &&\n+\t git_env_bool(GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE, 0))\n+\t\tdie(\"dying as requested by the '%s' variable on commit-graph parse!\",\n+\t\t GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE);\n+\tchecked_env = 1;\n+\n \tif (!prepare_commit_graph(r))\n \t\treturn 0;\n \treturn parse_commit_in_graph_one(r, r->objects->commit_graph, item);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 881c9b46e5..f0fb13e3f2 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -5,7 +5,7 @@\n #include \"object-store.h\"\n \n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n-#define GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD \"GIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD\"\n+#define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n #define GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS \"GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS\"\n \n /*\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 1073f9e3cf..5ec01abdaa 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -436,7 +436,7 @@ corrupt_graph_verify() {\n \t\tcp $objdir/info/commit-graph commit-graph-pre-write-test\n \tfi &&\n \tgit status --short &&\n-\tGIT_TEST_COMMIT_GRAPH_DIE_ON_LOAD=true git commit-graph write &&\n+\tGIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE=true git commit-graph write &&\n \tgit commit-graph verify\n }\n \n-- \ngitgitgadget\n\n"},{"id":"400896","messageId":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v3.git.1593174636.gitgitgadget@gmail.com","subject":"[PATCH v4 00/10] More commit-graph/Bloom filter improvements","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:20Z","receivedAt":"2020-07-01T13:27:38Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"This builds on sg/commit-graph-cleanups, which took several patches from\nSzeder's series [1] and applied them almost directly to a more-recent\nversion of Git [2].\n\n[1] https://lore.kernel.org/git/20200529085038.26008-1-szeder.dev@gmail.com/\n[2] \nhttps://lore.kernel.org/git/pull.650.git.1591362032.gitgitgadget@gmail.com/\n\nThis series adds a few extra improvements, several of which are rooted in\nSzeder's original series. I maintained his authorship and sign-off, even\nthough the patches did not apply or cherry-pick at all.\n\n(In v2, I have removed the range-diff comparison to Szeder's series, so look\nat the v1 cover letter for that.)\n\nThe patches have been significantly reordered. René pointed out (and Szeder\ndiscovered in the old thread) that we are not re-using the\nbloom_filter_settings from the existing commit-graph when writing a new one.\n\n 1. commit-graph: place bloom_settings in context\n 2. commit-graph: change test to die on parse, not load\n\nThese are mostly the same, except we now use a pointer to the settings in\nthe commit-graph write context.\n\n 3. bloom: get_bloom_filter() cleanups\n\nThis new patch is a subtle change in behavior that will become relevant in\nthe very next patch. In fact, if we swap patch 3 and 4, then\nt4216-log-bloom.sh fails with a segfault due to a NULL filter.\n\n 4. commit-graph: persist existence of changed-paths\n\nThis patch is now updated to use the existing changed-path filter settings.\n\n 5. commit-graph: unify the signatures of all write_graph_chunk_*()\n functions\n 6. commit-graph: simplify chunk writes into loop\n 7. commit-graph: check chunk sizes after writing\n\nThese are all the same as before.\n\n 8. revision.c: fix whitespace\n\nThis patch is the cleanup part of Taylor's patch.\n\n 9. revision: empty pathspecs should not use Bloom filters\n\nHere is Taylor's fix for empty pathspecs.\n\n 10. commit-graph: check all leading directories in changed path Bloom\n filters\n 11. bloom: enforce a minimum size of 8 bytes\n\nFinally, we get these performance patches. Patch 10 is updated to have the\nbetter logic around directory separators and empty paths. Also, the list of\nBloom keys is ordered with the deepest path first. That has some tiny\nperformance benefits for deep paths since we can short-circuit the multi-key\nchecks more often. That code path is much faster than the tree parsing, so\nit is hard to measure any change.\n\nUpdates in V3:\n\n * Responded to René's feedback.\n * Fixed the test in Patch 4 to use GIT_TEST_ variables and extend the\n GIT_TRACE2 depth to work with 'seen' branch.\n\nUpdate in V4;\n\n * Fixed the bug with \"too large\" commits. Test is added. The fixup! I sent\n earlier doesn't actually squash cleanly, so I resolved the conflicts\n during the rebase.\n\nThanks, -Stolee\n\nDerrick Stolee (5):\n commit-graph: place bloom_settings in context\n commit-graph: change test to die on parse, not load\n bloom: fix logic in get_bloom_filter()\n commit-graph: persist existence of changed-paths\n revision.c: fix whitespace\n\nSZEDER Gábor (4):\n commit-graph: unify the signatures of all write_graph_chunk_*()\n functions\n commit-graph: simplify chunk writes into loop\n commit-graph: check chunk sizes after writing\n commit-graph: check all leading directories in changed path Bloom\n filters\n\nTaylor Blau (1):\n revision: empty pathspecs should not use Bloom filters\n\n Documentation/git-commit-graph.txt | 5 +-\n bloom.c | 14 ++-\n builtin/commit-graph.c | 5 +-\n commit-graph.c | 146 +++++++++++++++++++++--------\n commit-graph.h | 3 +-\n revision.c | 63 +++++++++----\n revision.h | 6 +-\n t/t4216-log-bloom.sh | 45 ++++++++-\n t/t5318-commit-graph.sh | 2 +-\n 9 files changed, 215 insertions(+), 74 deletions(-)\n\n\nbase-commit: 7fbfe07ab4d4e58c0971dac73001b89f180a0af3\nPublished-As: https://github.com/gitgitgadget/git/releases/tag/pr-659%2Fderrickstolee%2Fbloom-2-v4\nFetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-659/derrickstolee/bloom-2-v4\nPull-Request: https://github.com/gitgitgadget/git/pull/659\n\nRange-diff vs v3:\n\n 1: 57002040bc = 1: 57002040bc commit-graph: place bloom_settings in context\n 2: 6b63f9bd8a = 2: 6b63f9bd8a commit-graph: change test to die on parse, not load\n 3: 2f809499ab ! 3: 3c532ebabc bloom: fix logic in get_bloom_filter()\n @@ Commit message\n \n Also clean up some style issues while we are here.\n \n + One side-effect of returning a NULL filter is that the filters that are\n + reported as \"too large\" will now be reported as NULL insead of length\n + zero. This case was not properly covered before, so add a test. Further,\n + remote the counting of the zero-length filters from revision.c and the\n + trace2 logs.\n +\n Helped-by: René Scharfe \n + Helped-by: SZEDER Gábor \n Signed-off-by: Derrick Stolee \n \n ## bloom.c ##\n @@ bloom.c: struct bloom_filter *get_bloom_filter(struct repository *r,\n \n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n +\n + ## commit-graph.c ##\n +@@ commit-graph.c: static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n + \n + \twhile (list < last) {\n + \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n +-\t\tcur_pos += filter->len;\n ++\t\tsize_t len = filter ? filter->len : 0;\n ++\t\tcur_pos += len;\n + \t\tdisplay_progress(progress, ++i);\n + \t\thashwrite_be32(f, cur_pos);\n + \t\tlist++;\n +@@ commit-graph.c: static void write_graph_chunk_bloom_data(struct hashfile *f,\n + \n + \twhile (list < last) {\n + \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n ++\t\tsize_t len = filter ? filter->len : 0;\n + \t\tdisplay_progress(progress, ++i);\n +-\t\thashwrite(f, filter->data, filter->len * sizeof(unsigned char));\n ++\n ++\t\tif (len)\n ++\t\t\thashwrite(f, filter->data, len * sizeof(unsigned char));\n + \t\tlist++;\n + \t}\n + \n +\n + ## revision.c ##\n +@@ revision.c: static unsigned int count_bloom_filter_maybe;\n + static unsigned int count_bloom_filter_definitely_not;\n + static unsigned int count_bloom_filter_false_positive;\n + static unsigned int count_bloom_filter_not_present;\n +-static unsigned int count_bloom_filter_length_zero;\n + \n + static void trace2_bloom_filter_statistics_atexit(void)\n + {\n +@@ revision.c: static void trace2_bloom_filter_statistics_atexit(void)\n + \n + \tjw_object_begin(&jw, 0);\n + \tjw_object_intmax(&jw, \"filter_not_present\", count_bloom_filter_not_present);\n +-\tjw_object_intmax(&jw, \"zero_length_filter\", count_bloom_filter_length_zero);\n + \tjw_object_intmax(&jw, \"maybe\", count_bloom_filter_maybe);\n + \tjw_object_intmax(&jw, \"definitely_not\", count_bloom_filter_definitely_not);\n + \tjw_object_intmax(&jw, \"false_positive\", count_bloom_filter_false_positive);\n +@@ revision.c: static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n + \t\treturn -1;\n + \t}\n + \n +-\tif (!filter->len) {\n +-\t\tcount_bloom_filter_length_zero++;\n +-\t\treturn -1;\n +-\t}\n +-\n + \tresult = bloom_filter_contains(filter,\n + \t\t\t\t revs->bloom_key,\n + \t\t\t\t revs->bloom_filter_settings);\n +\n + ## t/t4216-log-bloom.sh ##\n +@@ t/t4216-log-bloom.sh: setup () {\n + \n + test_bloom_filters_used () {\n + \tlog_args=$1\n +-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"zero_length_filter\\\":0,\\\"maybe\\\"\"\n ++\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n + \tsetup \"$log_args\" &&\n + \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n + \ttest_cmp log_wo_bloom log_w_bloom &&\n +@@ t/t4216-log-bloom.sh: test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n + \n + test_bloom_filters_used_when_some_filters_are_missing () {\n + \tlog_args=$1\n +-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n ++\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n + \tsetup \"$log_args\" &&\n + \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n + \ttest_cmp log_wo_bloom log_w_bloom\n +@@ t/t4216-log-bloom.sh: test_expect_success 'Use Bloom filters if they exist in the latest but not all c\n + \ttest_bloom_filters_used_when_some_filters_are_missing \"-- A/B\"\n + '\n + \n ++test_expect_success 'correctly report changes over limit' '\n ++\tgit init 513changes &&\n ++\t(\n ++\t\tcd 513changes &&\n ++\t\tfor i in $(test_seq 1 513)\n ++\t\tdo\n ++\t\t\techo $i >file$i.txt || return 1\n ++\t\tdone &&\n ++\t\tgit add . &&\n ++\t\tgit commit -m \"files\" &&\n ++\t\tgit commit-graph write --reachable --changed-paths &&\n ++\t\tfor i in $(test_seq 1 513)\n ++\t\tdo\n ++\t\t\tgit -c core.commitGraph=false log -- file$i.txt >expect &&\n ++\t\t\tgit log -- file$i.txt >actual &&\n ++\t\t\ttest_cmp expect actual || return 1\n ++\t\tdone\n ++\t)\n ++'\n ++\n + test_done\n + \\ No newline at end of file\n 4: 33e22d05cb ! 4: f1e3a8516e commit-graph: persist existence of changed-paths\n @@ t/t4216-log-bloom.sh: test_expect_success 'Use Bloom filters if they exist in th\n +\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n +'\n +\n - test_done\n - \\ No newline at end of file\n + test_expect_success 'correctly report changes over limit' '\n + \tgit init 513changes &&\n + \t(\n 5: 81c45d5260 = 5: c079921473 commit-graph: unify the signatures of all write_graph_chunk_*() functions\n 6: 8828dcd906 = 6: 5ed0ce20a4 commit-graph: simplify chunk writes into loop\n 7: ddbf297755 = 7: b982c9bf80 commit-graph: check chunk sizes after writing\n 8: 8b63706141 = 8: af750d8887 revision.c: fix whitespace\n 9: 7d6163305a = 9: a95de3cceb revision: empty pathspecs should not use Bloom filters\n 10: 40061233ca ! 10: 9c4a00ab08 commit-graph: check all leading directories in changed path Bloom filters\n @@ t/t4216-log-bloom.sh: test_expect_success 'setup - add commit-graph to the chain\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n --\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n -+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n +-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n ++\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n\n-- \ngitgitgadget\n"},{"id":"400897","messageId":"3c532ebabc17fef9149e477b7263cbf11c4d7cb9.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 03/10] bloom: fix logic in get_bloom_filter()","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:23Z","receivedAt":"2020-07-01T13:27:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe get_bloom_filter() method is a bit complicated in some parts where\nit does not need to be. In particular, it needs to return a NULL filter\nonly when compute_if_not_present is zero AND the filter data cannot be\nloaded from a commit-graph file. This currently happens by accident\nbecause the commit-graph does not load changed-path Bloom filters from\nan existing commit-graph when writing a new one. This will change in a\nlater patch.\n\nAlso clean up some style issues while we are here.\n\nOne side-effect of returning a NULL filter is that the filters that are\nreported as \"too large\" will now be reported as NULL insead of length\nzero. This case was not properly covered before, so add a test. Further,\nremote the counting of the zero-length filters from revision.c and the\ntrace2 logs.\n\nHelped-by: René Scharfe \nHelped-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n bloom.c | 14 ++++++--------\n commit-graph.c | 8 ++++++--\n revision.c | 7 -------\n t/t4216-log-bloom.sh | 24 ++++++++++++++++++++++--\n 4 files changed, 34 insertions(+), 19 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex c38d1cff0c..2af5389795 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -186,7 +186,7 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n-\tif (bloom_filters.slab_size == 0)\n+\tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n \tfilter = bloom_filter_slab_at(&bloom_filters, c);\n@@ -194,16 +194,14 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (c->graph_pos != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes) {\n-\t\t\tif (load_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n-\t\t\telse\n-\t\t\t\treturn NULL;\n-\t\t}\n+\t\t r->objects->commit_graph->chunk_bloom_indexes)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n \n-\tif (filter->data || !compute_if_not_present)\n+\tif (filter->data)\n \t\treturn filter;\n+\tif (!compute_if_not_present)\n+\t\treturn NULL;\n \n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6a28d4a5a6..50ce039a53 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1098,7 +1098,8 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n \n \twhile (list < last) {\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n-\t\tcur_pos += filter->len;\n+\t\tsize_t len = filter ? filter->len : 0;\n+\t\tcur_pos += len;\n \t\tdisplay_progress(progress, ++i);\n \t\thashwrite_be32(f, cur_pos);\n \t\tlist++;\n@@ -1126,8 +1127,11 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \n \twhile (list < last) {\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tsize_t len = filter ? filter->len : 0;\n \t\tdisplay_progress(progress, ++i);\n-\t\thashwrite(f, filter->data, filter->len * sizeof(unsigned char));\n+\n+\t\tif (len)\n+\t\t\thashwrite(f, filter->data, len * sizeof(unsigned char));\n \t\tlist++;\n \t}\n \ndiff --git a/revision.c b/revision.c\nindex c644c66091..7339750af1 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -633,7 +633,6 @@ static unsigned int count_bloom_filter_maybe;\n static unsigned int count_bloom_filter_definitely_not;\n static unsigned int count_bloom_filter_false_positive;\n static unsigned int count_bloom_filter_not_present;\n-static unsigned int count_bloom_filter_length_zero;\n \n static void trace2_bloom_filter_statistics_atexit(void)\n {\n@@ -641,7 +640,6 @@ static void trace2_bloom_filter_statistics_atexit(void)\n \n \tjw_object_begin(&jw, 0);\n \tjw_object_intmax(&jw, \"filter_not_present\", count_bloom_filter_not_present);\n-\tjw_object_intmax(&jw, \"zero_length_filter\", count_bloom_filter_length_zero);\n \tjw_object_intmax(&jw, \"maybe\", count_bloom_filter_maybe);\n \tjw_object_intmax(&jw, \"definitely_not\", count_bloom_filter_definitely_not);\n \tjw_object_intmax(&jw, \"false_positive\", count_bloom_filter_false_positive);\n@@ -735,11 +733,6 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\treturn -1;\n \t}\n \n-\tif (!filter->len) {\n-\t\tcount_bloom_filter_length_zero++;\n-\t\treturn -1;\n-\t}\n-\n \tresult = bloom_filter_contains(filter,\n \t\t\t\t revs->bloom_key,\n \t\t\t\t revs->bloom_filter_settings);\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c7011f33e2..2761208e74 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -60,7 +60,7 @@ setup () {\n \n test_bloom_filters_used () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"zero_length_filter\\\":0,\\\"maybe\\\"\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom &&\n@@ -142,7 +142,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"zero_length_filter\\\":0,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n@@ -152,4 +152,24 @@ test_expect_success 'Use Bloom filters if they exist in the latest but not all c\n \ttest_bloom_filters_used_when_some_filters_are_missing \"-- A/B\"\n '\n \n+test_expect_success 'correctly report changes over limit' '\n+\tgit init 513changes &&\n+\t(\n+\t\tcd 513changes &&\n+\t\tfor i in $(test_seq 1 513)\n+\t\tdo\n+\t\t\techo $i >file$i.txt || return 1\n+\t\tdone &&\n+\t\tgit add . &&\n+\t\tgit commit -m \"files\" &&\n+\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tfor i in $(test_seq 1 513)\n+\t\tdo\n+\t\t\tgit -c core.commitGraph=false log -- file$i.txt >expect &&\n+\t\t\tgit log -- file$i.txt >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone\n+\t)\n+'\n+\n test_done\n\\ No newline at end of file\n-- \ngitgitgadget\n\n"},{"id":"400898","messageId":"c07992147354258c3ea76bba4d85f430ddda5632.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 05/10] commit-graph: unify the signatures of all write_graph_chunk_*() functions","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:25Z","receivedAt":"2020-07-01T13:27:44Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nUpdate the write_graph_chunk_*() helper functions to have the same\nsignature:\n\n - Return an int error code from all these functions.\n write_graph_chunk_base() already has an int error code, now the\n others will have one, too, but since they don't indicate any\n error, they will always return 0.\n\n - Drop the hash size parameter of write_graph_chunk_oids() and\n write_graph_chunk_data(); its value can be read directly from\n 'the_hash_algo' inside these functions as well.\n\nThis opens up the possibility for further cleanups and foolproofing in\nthe following two patches.\n\nHelped-by: René Scharfe \nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 42 ++++++++++++++++++++++++++----------------\n 1 file changed, 26 insertions(+), 16 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6762704324..1a6d26f864 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -891,8 +891,8 @@ struct write_commit_graph_context {\n \tconst struct bloom_filter_settings *bloom_settings;\n };\n \n-static void write_graph_chunk_fanout(struct hashfile *f,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_fanout(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tint i, count = 0;\n \tstruct commit **list = ctx->commits.list;\n@@ -913,17 +913,21 @@ static void write_graph_chunk_fanout(struct hashfile *f,\n \n \t\thashwrite_be32(f, count);\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_oids(struct hashfile *f, int hash_len,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_oids(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tint count;\n \tfor (count = 0; count < ctx->commits.nr; count++, list++) {\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n-\t\thashwrite(f, (*list)->object.oid.hash, (int)hash_len);\n+\t\thashwrite(f, (*list)->object.oid.hash, the_hash_algo->rawsz);\n \t}\n+\n+\treturn 0;\n }\n \n static const unsigned char *commit_to_sha1(size_t index, void *table)\n@@ -932,8 +936,8 @@ static const unsigned char *commit_to_sha1(size_t index, void *table)\n \treturn commits[index]->object.oid.hash;\n }\n \n-static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n-\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_data(struct hashfile *f,\n+\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -950,7 +954,7 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \t\t\tdie(_(\"unable to parse commit %s\"),\n \t\t\t\toid_to_hex(&(*list)->object.oid));\n \t\ttree = get_commit_tree_oid(*list);\n-\t\thashwrite(f, tree->hash, hash_len);\n+\t\thashwrite(f, tree->hash, the_hash_algo->rawsz);\n \n \t\tparent = (*list)->parents;\n \n@@ -1030,10 +1034,12 @@ static void write_graph_chunk_data(struct hashfile *f, int hash_len,\n \n \t\tlist++;\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_extra_edges(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_extra_edges(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1082,10 +1088,12 @@ static void write_graph_chunk_extra_edges(struct hashfile *f,\n \n \t\tlist++;\n \t}\n+\n+\treturn 0;\n }\n \n-static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1108,6 +1116,7 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n \t}\n \n \tstop_progress(&progress);\n+\treturn 0;\n }\n \n static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n@@ -1125,8 +1134,8 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_release(&jw);\n }\n \n-static void write_graph_chunk_bloom_data(struct hashfile *f,\n-\t\t\t\t\t struct write_commit_graph_context *ctx)\n+static int write_graph_chunk_bloom_data(struct hashfile *f,\n+\t\t\t\t\tstruct write_commit_graph_context *ctx)\n {\n \tstruct commit **list = ctx->commits.list;\n \tstruct commit **last = ctx->commits.list + ctx->commits.nr;\n@@ -1155,6 +1164,7 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t}\n \n \tstop_progress(&progress);\n+\treturn 0;\n }\n \n static int oid_compare(const void *_a, const void *_b)\n@@ -1671,8 +1681,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n \twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, hashsz, ctx);\n-\twrite_graph_chunk_data(f, hashsz, ctx);\n+\twrite_graph_chunk_oids(f, ctx);\n+\twrite_graph_chunk_data(f, ctx);\n \tif (ctx->num_extra_edges)\n \t\twrite_graph_chunk_extra_edges(f, ctx);\n \tif (ctx->changed_paths) {\n-- \ngitgitgadget\n\n"},{"id":"400899","messageId":"5ed0ce20a4a82d09ec2e9b82e12b0315dc6fd7d4.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 06/10] commit-graph: simplify chunk writes into loop","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:26Z","receivedAt":"2020-07-01T13:27:44Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn write_commit_graph_file() we now have one block of code filling the\narray of 'struct chunk_info' with the IDs and sizes of chunks to be\nwritten, and an other block of code calling the functions responsible\nfor writing individual chunks. In case of optional chunks like Extra\nEdge List an Base Graphs List there is also a condition checking\nwhether that chunk is necessary/desired, and that same condition is\nrepeated in both blocks of code. Other, newer chunks have similar\noptional conditions.\n\nEliminate these repeated conditions by storing the function pointers\nresponsible for writing individual chunks in the 'struct chunk_info'\narray as well, and calling them in a loop to write the commit-graph\nfile. This will open up the possibility for a bit of foolproofing in\nthe following patch.\n\nHelped-by: René Scharfe \nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 28 ++++++++++++++++------------\n 1 file changed, 16 insertions(+), 12 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1a6d26f864..2b26a9dad3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1559,9 +1559,13 @@ static int write_graph_chunk_base(struct hashfile *f,\n \treturn 0;\n }\n \n+typedef int (*chunk_write_fn)(struct hashfile *f,\n+\t\t\t struct write_commit_graph_context *ctx);\n+\n struct chunk_info {\n \tuint32_t id;\n \tuint64_t size;\n+\tchunk_write_fn write_fn;\n };\n \n static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n@@ -1624,27 +1628,34 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tchunks[0].id = GRAPH_CHUNKID_OIDFANOUT;\n \tchunks[0].size = GRAPH_FANOUT_SIZE;\n+\tchunks[0].write_fn = write_graph_chunk_fanout;\n \tchunks[1].id = GRAPH_CHUNKID_OIDLOOKUP;\n \tchunks[1].size = hashsz * ctx->commits.nr;\n+\tchunks[1].write_fn = write_graph_chunk_oids;\n \tchunks[2].id = GRAPH_CHUNKID_DATA;\n \tchunks[2].size = (hashsz + 16) * ctx->commits.nr;\n+\tchunks[2].write_fn = write_graph_chunk_data;\n \tif (ctx->num_extra_edges) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_EXTRAEDGES;\n \t\tchunks[num_chunks].size = 4 * ctx->num_extra_edges;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_extra_edges;\n \t\tnum_chunks++;\n \t}\n \tif (ctx->changed_paths) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMINDEXES;\n \t\tchunks[num_chunks].size = sizeof(uint32_t) * ctx->commits.nr;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_indexes;\n \t\tnum_chunks++;\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMDATA;\n \t\tchunks[num_chunks].size = sizeof(uint32_t) * 3\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n \t\tchunks[num_chunks].size = hashsz * (ctx->num_commit_graphs_after - 1);\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_base;\n \t\tnum_chunks++;\n \t}\n \n@@ -1680,19 +1691,12 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\tprogress_title.buf,\n \t\t\tnum_chunks * ctx->commits.nr);\n \t}\n-\twrite_graph_chunk_fanout(f, ctx);\n-\twrite_graph_chunk_oids(f, ctx);\n-\twrite_graph_chunk_data(f, ctx);\n-\tif (ctx->num_extra_edges)\n-\t\twrite_graph_chunk_extra_edges(f, ctx);\n-\tif (ctx->changed_paths) {\n-\t\twrite_graph_chunk_bloom_indexes(f, ctx);\n-\t\twrite_graph_chunk_bloom_data(f, ctx);\n-\t}\n-\tif (ctx->num_commit_graphs_after > 1 &&\n-\t write_graph_chunk_base(f, ctx)) {\n-\t\treturn -1;\n+\n+\tfor (i = 0; i < num_chunks; i++) {\n+\t\tif (chunks[i].write_fn(f, ctx))\n+\t\t\treturn -1;\n \t}\n+\n \tstop_progress(&ctx->progress);\n \tstrbuf_release(&progress_title);\n \n-- \ngitgitgadget\n\n"},{"id":"400900","messageId":"b982c9bf80198506e3179badd12c63bb2566bc6f.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 07/10] commit-graph: check chunk sizes after writing","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:27Z","receivedAt":"2020-07-01T13:27:45Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nIn my experience while experimenting with new commit-graph chunks,\nearly versions of the corresponding new write_commit_graph_my_chunk()\nfunctions are, sadly but not surprisingly, often buggy, and write more\nor less data than they are supposed to, especially if the chunk size\nis not directly proportional to the number of commits. This then\ncauses all kinds of issues when reading such a bogus commit-graph\nfile, raising the question of whether the writing or the reading part\nhappens to be buggy this time.\n\nLet's catch such issues early, already when writing the commit-graph\nfile, and check that each write_graph_chunk_*() function wrote the\namount of data that it was expected to, and what has been encoded in\nthe Chunk Lookup table. Now that all commit-graph chunks are written\nin a loop we can do this check in a single place for all chunks, and\nany chunks added in the future will get checked as well.\n\nHelped-by: René Scharfe \nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n commit-graph.c | 7 +++++++\n 1 file changed, 7 insertions(+)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 2b26a9dad3..6752916c1a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1693,8 +1693,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t}\n \n \tfor (i = 0; i < num_chunks; i++) {\n+\t\tuint64_t start_offset = f->total + f->offset;\n+\n \t\tif (chunks[i].write_fn(f, ctx))\n \t\t\treturn -1;\n+\n+\t\tif (f->total + f->offset != start_offset + chunks[i].size)\n+\t\t\tBUG(\"expected to write %\"PRId64\" bytes to chunk %\"PRIx32\", but wrote %\"PRId64\" instead\",\n+\t\t\t chunks[i].size, chunks[i].id,\n+\t\t\t f->total + f->offset - start_offset);\n \t}\n \n \tstop_progress(&ctx->progress);\n-- \ngitgitgadget\n\n"},{"id":"400902","messageId":"9c4a00ab089c48870f0e3ad02ed15c1cc71f8625.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 10/10] commit-graph: check all leading directories in changed path Bloom filters","fromName":"SZEDER Gábor via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:30Z","receivedAt":"2020-07-01T13:27:47Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"From: =?UTF-8?q?SZEDER=20G=C3=A1bor?= \n\nThe file 'dir/subdir/file' can only be modified if its leading\ndirectories 'dir' and 'dir/subdir' are modified as well.\n\nSo when checking modified path Bloom filters looking for commits\nmodifying a path with multiple path components, then check not only\nthe full path in the Bloom filters, but all its leading directories as\nwell. Take care to check these paths in \"deepest first\" order,\nbecause it's the full path that is least likely to be modified, and\nthe Bloom filter queries can short circuit sooner.\n\nThis can significantly reduce the average false positive rate, by\nabout an order of magnitude or three(!), and can further speed up\npathspec-limited revision walks. The table below compares the average\nfalse positive rate and runtime of\n\n git rev-list HEAD -- \"$path\"\n\nbefore and after this change for 5000+ randomly* selected paths from\neach repository:\n\n Average false Average Average\n positive rate runtime runtime\n before after before after difference\n ------------------------------------------------------------------\n git 3.220% 0.7853% 0.0558s 0.0387s -30.6%\n linux 2.453% 0.0296% 0.1046s 0.0766s -26.8%\n tensorflow 2.536% 0.6977% 0.0594s 0.0420s -29.2%\n\n*Path selection was done with the following pipeline:\n\n\tgit ls-tree -r --name-only HEAD | sort -R | head -n 5000\n\nThe improvements in runtime are much smaller than the improvements in\naverage false positive rate, as we are clearly reaching diminishing\nreturns here. However, all these timings depend on that accessing\ntree objects is reasonably fast (warm caches). If we had a partial\nclone and the tree objects had to be fetched from a promisor remote,\ne.g.:\n\n $ git clone --filter=tree:0 --bare file://.../webkit.git webkit.notrees.git\n $ git -C webkit.git -c core.modifiedPathBloomFilters=1 \\\n commit-graph write --reachable\n $ cp webkit.git/objects/info/commit-graph webkit.notrees.git/objects/info/\n $ git -C webkit.notrees.git -c core.modifiedPathBloomFilters=1 \\\n rev-list HEAD -- \"$path\"\n\nthen checking all leading path component can reduce the runtime from\nover an hour to a few seconds (and this is with the clone and the\npromisor on the same machine).\n\nThis adjusts the tracing values in t4216-log-bloom.sh, which provides a\nconcrete way to notice the improvement.\n\nHelped-by: Taylor Blau \nHelped-by: René Scharfe \nSigned-off-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \n---\n revision.c | 46 +++++++++++++++++++++++++++++++++++---------\n revision.h | 6 ++++--\n t/t4216-log-bloom.sh | 2 +-\n 3 files changed, 42 insertions(+), 12 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 667ca36e1c..b9118001f9 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -668,9 +668,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n {\n \tstruct pathspec_item *pi;\n \tchar *path_alloc = NULL;\n-\tconst char *path;\n+\tconst char *path, *p;\n \tint last_index;\n-\tint len;\n+\tsize_t len;\n+\tint path_component_nr = 1;\n \n \tif (!revs->commits)\n \t\treturn;\n@@ -707,8 +708,33 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \t\treturn;\n \t}\n \n-\trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n-\tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\n+\tp = path;\n+\twhile (*p) {\n+\t\t/*\n+\t\t * At this point, the path is normalized to use Unix-style\n+\t\t * path separators. This is required due to how the\n+\t\t * changed-path Bloom filters store the paths.\n+\t\t */\n+\t\tif (*p == '/')\n+\t\t\tpath_component_nr++;\n+\t\tp++;\n+\t}\n+\n+\trevs->bloom_keys_nr = path_component_nr;\n+\tALLOC_ARRAY(revs->bloom_keys, revs->bloom_keys_nr);\n+\n+\tfill_bloom_key(path, len, &revs->bloom_keys[0],\n+\t\t revs->bloom_filter_settings);\n+\tpath_component_nr = 1;\n+\n+\tp = path + len - 1;\n+\twhile (p > path) {\n+\t\tif (*p == '/')\n+\t\t\tfill_bloom_key(path, p - path,\n+\t\t\t\t &revs->bloom_keys[path_component_nr++],\n+\t\t\t\t revs->bloom_filter_settings);\n+\t\tp--;\n+\t}\n \n \tif (trace2_is_enabled() && !bloom_filter_atexit_registered) {\n \t\tatexit(trace2_bloom_filter_statistics_atexit);\n@@ -722,7 +748,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\t\t\t\t\t struct commit *commit)\n {\n \tstruct bloom_filter *filter;\n-\tint result;\n+\tint result = 1, j;\n \n \tif (!revs->repo->objects->commit_graph)\n \t\treturn -1;\n@@ -737,9 +763,11 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \t\treturn -1;\n \t}\n \n-\tresult = bloom_filter_contains(filter,\n-\t\t\t\t revs->bloom_key,\n-\t\t\t\t revs->bloom_filter_settings);\n+\tfor (j = 0; result && j < revs->bloom_keys_nr; j++) {\n+\t\tresult = bloom_filter_contains(filter,\n+\t\t\t\t\t &revs->bloom_keys[j],\n+\t\t\t\t\t revs->bloom_filter_settings);\n+\t}\n \n \tif (result)\n \t\tcount_bloom_filter_maybe++;\n@@ -779,7 +807,7 @@ static int rev_compare_tree(struct rev_info *revs,\n \t\t\treturn REV_TREE_SAME;\n \t}\n \n-\tif (revs->bloom_key && !nth_parent) {\n+\tif (revs->bloom_keys_nr && !nth_parent) {\n \t\tbloom_ret = check_maybe_different_in_bloom_filter(revs, commit);\n \n \t\tif (bloom_ret == 0)\ndiff --git a/revision.h b/revision.h\nindex 7c026fe41f..abbfb4ab59 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -295,8 +295,10 @@ struct rev_info {\n \tstruct topo_walk_info *topo_walk_info;\n \n \t/* Commit graph bloom filter fields */\n-\t/* The bloom filter key for the pathspec */\n-\tstruct bloom_key *bloom_key;\n+\t/* The bloom filter key(s) for the pathspec */\n+\tstruct bloom_key *bloom_keys;\n+\tint bloom_keys_nr;\n+\n \t/*\n \t * The bloom filter settings used to generate the key.\n \t * This is loaded from the commit-graph being used.\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 29338f36bf..4892364e74 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -146,7 +146,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":8,\\\"definitely_not\\\":6\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n-- \ngitgitgadget\n"},{"id":"400901","messageId":"a95de3cceb39ea499d18e20aa0e3de973765cb26.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 09/10] revision: empty pathspecs should not use Bloom filters","fromName":"Taylor Blau via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:29Z","receivedAt":"2020-07-01T13:27:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Taylor Blau \n\nThe prepare_to_use_bloom_filter() method was not intended to be called\non an empty pathspec. However, 'git log -- .' and 'git log' are subtly\ndifferent: the latter reports all commits while the former will simplify\ncommits that do not change the root tree.\n\nThis means that the path used to construct the bloom_key might be empty,\nand that value is not added to the Bloom filter during construction.\nThat means that the results are likely incorrect!\n\nTo resolve the issue, be careful about the length of the path and stop\nfilling Bloom filters. To be completely sure we do not use them, drop\nthe pointer to the bloom_filter_settings from the commit-graph. That\nallows our test to look at the trace2 logs to verify no Bloom filter\nstatistics are reported.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n revision.c | 4 ++++\n t/t4216-log-bloom.sh | 4 ++++\n 2 files changed, 8 insertions(+)\n\ndiff --git a/revision.c b/revision.c\nindex ddf09ab0aa..667ca36e1c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -702,6 +702,10 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \t\tpath = pi->match;\n \n \tlen = strlen(path);\n+\tif (!len) {\n+\t\trevs->bloom_filter_settings = NULL;\n+\t\treturn;\n+\t}\n \n \trevs->bloom_key = xmalloc(sizeof(struct bloom_key));\n \tfill_bloom_key(path, len, revs->bloom_key, revs->bloom_filter_settings);\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 52ad998f9e..29338f36bf 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -112,6 +112,10 @@ test_expect_success 'git log -- multiple path specs does not use Bloom filters'\n \ttest_bloom_filters_not_used \"-- file4 A/file1\"\n '\n \n+test_expect_success 'git log -- \".\" pathspec at root does not use Bloom filters' '\n+\ttest_bloom_filters_not_used \"-- .\"\n+'\n+\n test_expect_success 'git log with wildcard that resolves to a single path uses Bloom filters' '\n \ttest_bloom_filters_used \"-- *4\" &&\n \ttest_bloom_filters_used \"-- *renamed\"\n-- \ngitgitgadget\n\n"},{"id":"400903","messageId":"f1e3a8516ebd58b283166a5374843f5cb3332d08.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 04/10] commit-graph: persist existence of changed-paths","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:24Z","receivedAt":"2020-07-01T13:27:49Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nThe changed-path Bloom filters were released in v2.27.0, but have a\nsignificant drawback. A user can opt-in to writing the changed-path\nfilters using the \"--changed-paths\" option to \"git commit-graph write\"\nbut the next write will drop the filters unless that option is\nspecified.\n\nThis becomes even more important when considering the interaction with\ngc.writeCommitGraph (on by default) or fetch.writeCommitGraph (part of\nfeatures.experimental). These config options trigger commit-graph writes\nthat the user did not signal, and hence there is no --changed-paths\noption available.\n\nAllow a user that opts-in to the changed-path filters to persist the\nproperty of \"my commit-graph has changed-path filters\" automatically. A\nuser can drop filters using the --no-changed-paths option.\n\nIn the process, we need to be extremely careful to match the Bloom\nfilter settings as specified by the commit-graph. This will allow future\nversions of Git to customize these settings, and the version with this\nchange will persist those settings as commit-graphs are rewritten on\ntop.\n\nUse the trace2 API to signal the settings used during the write, and\ncheck that output in a test after manually adjusting the correct bytes\nin the commit-graph file.\n\nSigned-off-by: Derrick Stolee \n---\n Documentation/git-commit-graph.txt | 5 +++-\n builtin/commit-graph.c | 5 +++-\n commit-graph.c | 45 ++++++++++++++++++++++++++++--\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 17 ++++++++++-\n 5 files changed, 67 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex f4b13c005b..369b222b08 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -60,7 +60,10 @@ existing commit-graph file.\n With the `--changed-paths` option, compute and write information about the\n paths changed between a commit and it's first parent. This operation can\n take a while on large repositories. It provides significant performance gains\n-for getting history of a directory or a file with `git log -- `.\n+for getting history of a directory or a file with `git log -- `. If\n+this option is given, future commit-graph writes will automatically assume\n+that this option was intended. Use `--no-changed-paths` to stop storing this\n+data.\n +\n With the `--split` option, write the commit-graph as a chain of multiple\n commit-graph files stored in `/info/commit-graphs`. The new commits\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 59009837dc..ff7b177c33 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -151,6 +151,7 @@ static int graph_write(int argc, const char **argv)\n \t};\n \n \topts.progress = isatty(2);\n+\topts.enable_changed_paths = -1;\n \tsplit_opts.size_multiple = 2;\n \tsplit_opts.max_commits = 0;\n \tsplit_opts.expire_time = 0;\n@@ -171,7 +172,9 @@ static int graph_write(int argc, const char **argv)\n \t\tflags |= COMMIT_GRAPH_WRITE_SPLIT;\n \tif (opts.progress)\n \t\tflags |= COMMIT_GRAPH_WRITE_PROGRESS;\n-\tif (opts.enable_changed_paths ||\n+\tif (!opts.enable_changed_paths)\n+\t\tflags |= COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS;\n+\tif (opts.enable_changed_paths == 1 ||\n \t git_env_bool(GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS, 0))\n \t\tflags |= COMMIT_GRAPH_WRITE_BLOOM_FILTERS;\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 50ce039a53..6762704324 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -16,6 +16,8 @@\n #include \"progress.h\"\n #include \"bloom.h\"\n #include \"commit-slab.h\"\n+#include \"json-writer.h\"\n+#include \"trace2.h\"\n \n void git_test_write_commit_graph_or_die(void)\n {\n@@ -1108,6 +1110,21 @@ static void write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tstop_progress(&progress);\n }\n \n+static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n+{\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n+\tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n+\tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_end(&jw);\n+\n+\ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n+\n+\tjw_release(&jw);\n+}\n+\n static void write_graph_chunk_bloom_data(struct hashfile *f,\n \t\t\t\t\t struct write_commit_graph_context *ctx)\n {\n@@ -1116,6 +1133,8 @@ static void write_graph_chunk_bloom_data(struct hashfile *f,\n \tstruct progress *progress = NULL;\n \tint i = 0;\n \n+\ttrace2_bloom_filter_settings(ctx);\n+\n \tif (ctx->report_progress)\n \t\tprogress = start_delayed_progress(\n \t\t\t_(\"Writing changed paths Bloom filters data\"),\n@@ -1547,9 +1566,15 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tint num_chunks = 3;\n \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n-\tconst struct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n+\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n-\tctx->bloom_settings = &bloom_settings;\n+\tif (!ctx->bloom_settings) {\n+\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n+\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n+\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n+\t\t\t\t\t\t\t bloom_settings.num_hashes);\n+\t\tctx->bloom_settings = &bloom_settings;\n+\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -1974,9 +1999,23 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n \tctx->check_oids = flags & COMMIT_GRAPH_WRITE_CHECK_OIDS ? 1 : 0;\n \tctx->split_opts = split_opts;\n-\tctx->changed_paths = flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS ? 1 : 0;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n+\t\tctx->changed_paths = 1;\n+\tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\n+\t\tstruct commit_graph *g;\n+\t\tprepare_commit_graph_one(ctx->r, ctx->odb);\n+\n+\t\tg = ctx->r->objects->commit_graph;\n+\n+\t\t/* We have changed-paths already. Keep them in the next graph */\n+\t\tif (g && g->chunk_bloom_data) {\n+\t\t\tctx->changed_paths = 1;\n+\t\t\tctx->bloom_settings = g->bloom_filter_settings;\n+\t\t}\n+\t}\n+\n \tif (ctx->split) {\n \t\tstruct commit_graph *g;\n \t\tprepare_commit_graph(ctx->r);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex f0fb13e3f2..45b1e5bca3 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -96,6 +96,7 @@ enum commit_graph_write_flags {\n \t/* Make sure that each OID in the input is a valid commit OID. */\n \tCOMMIT_GRAPH_WRITE_CHECK_OIDS = (1 << 3),\n \tCOMMIT_GRAPH_WRITE_BLOOM_FILTERS = (1 << 4),\n+\tCOMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS = (1 << 5),\n };\n \n struct split_commit_graph_opts {\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 2761208e74..52ad998f9e 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -126,7 +126,7 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_commit c14 A/anotherFile2 &&\n \ttest_commit c15 A/B/anotherFile2 &&\n \ttest_commit c16 A/B/C/anotherFile2 &&\n-\tGIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=0 git commit-graph write --reachable --split &&\n+\tgit commit-graph write --reachable --split --no-changed-paths &&\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n@@ -152,6 +152,21 @@ test_expect_success 'Use Bloom filters if they exist in the latest but not all c\n \ttest_bloom_filters_used_when_some_filters_are_missing \"-- A/B\"\n '\n \n+test_expect_success 'persist filter settings' '\n+\ttest_when_finished rm -rf .git/objects/info/commit-graph* &&\n+\trm -rf .git/objects/info/commit-graph* &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2.txt\" \\\n+\t\tGIT_TRACE2_EVENT_NESTING=5 \\\n+\t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n+\t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n+\t\tgit commit-graph write --reachable --changed-paths &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n+\t\tGIT_TRACE2_EVENT_NESTING=5 \\\n+\t\tgit commit-graph write --reachable --changed-paths &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n+'\n+\n test_expect_success 'correctly report changes over limit' '\n \tgit init 513changes &&\n \t(\n-- \ngitgitgadget\n\n"},{"id":"400904","messageId":"af750d88875903c023c97c519999ed720f27a0cc.1593610050.git.gitgitgadget@gmail.com","threadId":"53681","inReplyTo":"pull.659.v4.git.1593610050.gitgitgadget@gmail.com","subject":"[PATCH v4 08/10] revision.c: fix whitespace","fromName":"Derrick Stolee via GitGitGadget","fromEmail":"gitgitgadget@gmail.com","sentAt":"2020-07-01T13:27:28Z","receivedAt":"2020-07-01T13:27:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"From: Derrick Stolee \n\nHere, four spaces were used instead of tab characters.\n\nReported-by: Taylor Blau \nSigned-off-by: Derrick Stolee \n---\n revision.c | 8 ++++----\n 1 file changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/revision.c b/revision.c\nindex 7339750af1..ddf09ab0aa 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -695,11 +695,11 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \t/* remove single trailing slash from path, if needed */\n \tif (pi->match[last_index] == '/') {\n-\t path_alloc = xstrdup(pi->match);\n-\t path_alloc[last_index] = '\\0';\n-\t path = path_alloc;\n+\t\tpath_alloc = xstrdup(pi->match);\n+\t\tpath_alloc[last_index] = '\\0';\n+\t\tpath = path_alloc;\n \t} else\n-\t path = pi->match;\n+\t\tpath = pi->match;\n \n \tlen = strlen(path);\n \n-- \ngitgitgadget\n\n"},{"id":"407599","messageId":"20201015132147.GB24954@szeder.dev","threadId":"53681","inReplyTo":"f1e3a8516ebd58b283166a5374843f5cb3332d08.1593610050.git.gitgitgadget@gmail.com","subject":"Re: [PATCH v4 04/10] commit-graph: persist existence of changed-paths","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-10-15T13:21:47Z","receivedAt":"2020-10-15T13:21:54Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Jul 01, 2020 at 01:27:24PM +0000, Derrick Stolee via GitGitGadget wrote:\n> From: Derrick Stolee \n> \n> The changed-path Bloom filters were released in v2.27.0, but have a\n> significant drawback. A user can opt-in to writing the changed-path\n> filters using the \"--changed-paths\" option to \"git commit-graph write\"\n> but the next write will drop the filters unless that option is\n> specified.\n> \n> This becomes even more important when considering the interaction with\n> gc.writeCommitGraph (on by default) or fetch.writeCommitGraph (part of\n> features.experimental). These config options trigger commit-graph writes\n> that the user did not signal, and hence there is no --changed-paths\n> option available.\n> \n> Allow a user that opts-in to the changed-path filters to persist the\n> property of \"my commit-graph has changed-path filters\" automatically. A\n> user can drop filters using the --no-changed-paths option.\n\nThe above parts of the commit message and the corresponding changes\nare OK, but ...\n\n> In the process, we need to be extremely careful to match the Bloom\n> filter settings as specified by the commit-graph. This will allow future\n> versions of Git to customize these settings, and the version with this\n> change will persist those settings as commit-graphs are rewritten on\n> top.\n\nAs pointed out in my original bug report [1], modified path Bloom\nfilters are computed with hardcoded settings in\nbloom.c:get_bloom_filter(). Since this patch does not touch bloom.c\nat all, it still computes Bloom filters with those hardcoded settings,\nand, consequently, despite the commit message's claims, it does not\npersist the settings in the existing commit-graph.\n\n[1] https://public-inbox.org/git/20200619140230.GB22200@szeder.dev/\n\n> Use the trace2 API to signal the settings used during the write, and\n> check that output in a test after manually adjusting the correct bytes\n> in the commit-graph file.\n\nThis test is insufficient, as it only checks what settings trace2\nbelieves the Bloom filters are computed with, not what settings they\nare actually computed with; that's why it succeeded while the bug\nwhose absence it was supposed to ensure was still there.\n\nMore robust tests should instead look at what actually gets written to\nthe commit-graph, and how that is interpreted during pathspec-limited\nrevision walks.\n\n> Signed-off-by: Derrick Stolee \n\nA \"Reported-by: me\" trailer would have been appropriate here.\n\n> ---\n> Documentation/git-commit-graph.txt | 5 +++-\n> builtin/commit-graph.c | 5 +++-\n> commit-graph.c | 45 ++++++++++++++++++++++++++++--\n> commit-graph.h | 1 +\n> t/t4216-log-bloom.sh | 17 ++++++++++-\n> 5 files changed, 67 insertions(+), 6 deletions(-)\n\nAnyway, this is now partially fixed in 9a7a9ed10d (bloom: use provided\n'struct bloom_filter_settings', 2020-09-16), though, unfortunately,\nits commit message is not quite clear on this. Alas, that's only a\npartial fix, because we still only look at the top level commit-graph\nfile for existing Bloom filter settings. However, deeper commit-graph\nlayers can contain Bloom filters with non-default settings even when\nthe top level doesn't, and these failing tests below demonstrate:\n\n --- >8 ---\n\n#!/bin/sh\n\ntest_description='test'\n\n. ./test-lib.sh\n\ntest_expect_success 'setup' '\n\tgit commit --allow-empty -m \"Bloom filters are written but ignored for root commits :(\" &&\n\tfor i in 1 2 3\n\tdo\n\t\techo $i >file &&\n\t\tgit add file &&\n\t\tgit commit -m \"$i\" || return 1\n\tdone &&\n\tgit log --oneline --no-decorate -- file >expect\n'\n\ntest_expect_success 'split' '\n\t# Compute Bloom filters with \"unusual\" settings.\n\tgit rev-parse HEAD^^ | GIT_TEST_BLOOM_SETTINGS_NUM_HASHES=3 git commit-graph write --stdin-commits --changed-paths --split &&\n\t# A commit-graph layer without Bloom filters \"hides\" the layers\n\t# below ...\n\tgit rev-parse HEAD^ | git commit-graph write --stdin-commits --no-changed-paths --split=no-merge &&\n\t# ... so this does not look at existing Bloom filters and their\n\t# settings in the bottom commit-graph layer and computes new\n\t# Bloom filters using the default 7 hashes.\n\tgit rev-parse HEAD | git commit-graph write --stdin-commits --changed-paths --split=no-merge &&\n\n\t# Just to make sure that there are as many graph layers as I\n\t# think there should be.\n\ttest_line_count = 3 .git/objects/info/commit-graphs/commit-graph-chain &&\n\n\t# This checks Bloom filters using settings in the top layer,\n\t# thus misses commits modifying file in the bottom commit-graph\n\t# layer.\n\tgit log --oneline --no-decorate -- file >actual &&\n\ttest_cmp expect actual\n'\n\ntest_expect_success 'merged' '\n\t# This merges all existing layers, and computes missing Bloom\n\t# filters with the settings in the top layer, without noticing\n\t# that filters in the bottom layer were computed with different\n\t# settings.\n\tgit commit-graph write --reachable --changed-paths &&\n\n\t# Just to make sure...\n\ttest_path_is_file .git/objects/info/commit-graph &&\n\n\t# This misses commits modifying file that were merged from the\n\t# bottom commit-graph layer.\n\tgit log --oneline --no-decorate -- file >actual &&\n\ttest_cmp expect actual\n'\n\ntest_done\n\n"},{"id":"407685","messageId":"20201015214111.GA2176154@nand.local","threadId":"53681","inReplyTo":"20201015132147.GB24954@szeder.dev","subject":"Re: [PATCH v4 04/10] commit-graph: persist existence of changed-paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-10-15T21:41:11Z","receivedAt":"2020-10-15T21:41:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Oct 15, 2020 at 03:21:47PM +0200, SZEDER Gábor wrote:\n> As pointed out in my original bug report [1], modified path Bloom\n> filters are computed with hardcoded settings in\n> bloom.c:get_bloom_filter(). Since this patch does not touch bloom.c\n> at all, it still computes Bloom filters with those hardcoded settings,\n> and, consequently, despite the commit message's claims, it does not\n> persist the settings in the existing commit-graph.\n>\n> [1] https://public-inbox.org/git/20200619140230.GB22200@szeder.dev/\n\nRight. It is worth mentioning here (as you do below) that this was fixed\nas of 9a7a9ed10d (bloom: use provided 'struct bloom_filter_settings',\n2020-09-16).\n\n> > Use the trace2 API to signal the settings used during the write, and\n> > check that output in a test after manually adjusting the correct bytes\n> > in the commit-graph file.\n>\n> This test is insufficient, as it only checks what settings trace2\n> believes the Bloom filters are computed with, not what settings they\n> are actually computed with; that's why it succeeded while the bug\n> whose absence it was supposed to ensure was still there.\n>\n> More robust tests should instead look at what actually gets written to\n> the commit-graph, and how that is interpreted during pathspec-limited\n> revision walks.\n\nThanks for the test! That saved me a little bit of work trying to set up\nthe scenario you're describing in a reproducible way. I think that the\nthird test can be fixed relatively easily. The crux of the issue there\nis that we are computing Bloom filters for commits, some of which\nalready had Bloom filters computed in an earlier layer, but with\ndifferent settings than the one that we're using to write the current\nlayer.\n\nSo, we need to be more aggressively checking the Bloom filter settings\nin any layer we want to reuse a Bloom filter out of before reusing it\nverbatim in the current layer. The patch below the scissors line is\nsufficient to do that, and it causes the third test to start passing.\n\n...But, teaching the revision machinery how to handle a walk through\ncommits in multiple commit-graph layers with incompatible Bloom filter\nsettings is trickier. Right now we compute all of the Bloom keys up\nfront using the Bloom filter settings in the top layer. I think we'd\nprobably want to change this to lazily compute those keys by:\n\n - Keeping around the contents of the Bloom keys, i.e., the pathspecs\n themselves.\n\n - Keeping a hashmap from Bloom filter settings to computed Bloom keys\n corresponding to those settings.\n\n - Lazily filling in that hashmap as we traverse through different\n commits.\n\nAt least, that's the best way that I can think to do it. It does get\nkind of slow, though; we'd have to scan every commit graph layer at each\ncommit in the worst case to find the actual 'struct commit_graph *' in\norder to get its Bloom filter settings. So, I think that's sort of\nshow-stoppingly slow, and that we should probably think more deeply\nabout it before taking up that direction.\n\nMaybe Stolee has some better thoughts about how we can quickly go from a\ncommit to either (a) the commit-graph struct that that commit is stored\nin, or (b) the Bloom filter settings belonging to that struct.\n\nThanks,\nTaylor\n\n--- >8 ---\n\nSubject: [PATCH] bloom: recompute filters with incompatible settings\n---\n bloom.c | 21 +++++++++++++++++++--\n commit-graph.c | 4 ++--\n 2 files changed, 21 insertions(+), 4 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 68c73200a5..30da128474 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -30,7 +30,8 @@ static inline unsigned char get_bitmask(uint32_t pos)\n\n static int load_bloom_filter_from_graph(struct commit_graph *g,\n \t\t\t\t\tstruct bloom_filter *filter,\n-\t\t\t\t\tstruct commit *c)\n+\t\t\t\t\tstruct commit *c,\n+\t\t\t\t\tconst struct bloom_filter_settings *settings)\n {\n \tuint32_t lex_pos, start_index, end_index;\n \tuint32_t graph_pos = commit_graph_position(c);\n@@ -42,6 +43,21 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \tif (!g->chunk_bloom_indexes)\n \t\treturn 0;\n\n+\tif (settings) {\n+\t\tstruct bloom_filter_settings *graph_settings = g->bloom_filter_settings;\n+\t\t/*\n+\t\t * Check that the settings used to generate new Bloom filters is\n+\t\t * compatible with the settings Bloom filters in this\n+\t\t * commit-graph layer were generated with.\n+\t\t */\n+\t\tif (settings->hash_version != graph_settings->hash_version)\n+\t\t\treturn 0;\n+\t\tif (settings->num_hashes != graph_settings->num_hashes)\n+\t\t\treturn 0;\n+\t\tif (settings->bits_per_entry != graph_settings->bits_per_entry)\n+\t\t\treturn 0;\n+\t}\n+\n \tlex_pos = graph_pos - g->num_commits_in_base;\n\n \tend_index = get_be32(g->chunk_bloom_indexes + 4 * lex_pos);\n@@ -205,7 +221,8 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph,\n+\t\t\t\t\t\t filter, c, settings);\n \t}\n\n \tif (filter->data && filter->len)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex cb042bdba8..afe14af2a3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1188,7 +1188,7 @@ static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tuint32_t cur_pos = 0;\n\n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n+\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(ctx->r, *list, 0, ctx->bloom_settings, NULL);\n \t\tsize_t len = filter ? filter->len : 0;\n \t\tcur_pos += len;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1228,7 +1228,7 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n\n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n+\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(ctx->r, *list, 0, ctx->bloom_settings, NULL);\n \t\tsize_t len = filter ? filter->len : 0;\n\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n--\n2.29.0.rc1.dirty\n\n"},{"id":"407701","messageId":"103404f0-a164-2550-4730-6b8773d3ca1f@gmail.com","threadId":"53681","inReplyTo":"20201015214111.GA2176154@nand.local","subject":"Re: [PATCH v4 04/10] commit-graph: persist existence of changed-paths","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-10-16T02:18:47Z","receivedAt":"2020-10-16T02:18:52Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/15/2020 5:41 PM, Taylor Blau wrote:\n> So, we need to be more aggressively checking the Bloom filter settings\n> in any layer we want to reuse a Bloom filter out of before reusing it\n> verbatim in the current layer. The patch below the scissors line is\n> sufficient to do that, and it causes the third test to start passing.\n\nI think there are three things we should keep in mind:\n\n1. Incompatible Bloom filter settings between layers should be seen\n as _inconsistent data_ as Git should not be writing incremental\n commit-graph files with inconsistent Bloom filter settings. Thus,\n when reading the commit-graph chain we should prevent incompatible\n filters from being used. One way to do this is to notice different\n settings and completely disable Bloom filters. The other way would\n be to take the settings from the first layer with filters and then\n clear the chunk_bloom_indexes and chunk_bloom_data fields for the\n layers that don't agree. This fits with an expectation that lower\n layers are larger, so more filters can be used in that situation.\n\n2. We should be sure that Git will not agree to write incompatible\n settings between layers of a commit-graph chain. Even more, it\n should definitely not re-use filters when merging layers with\n incompatible filter values. The strategy above to ignore Bloom\n filters in incompatible upper layers is enough of a guard against\n the \"merge layers\" situation.\n\n3. Allowing users (or future Git developers) to adjust the default\n Bloom filter settings is one that is good to do for future-proofing,\n but not one that I expect to be widely used (any gains here are\n minuscule compared to the results already achieved with the defaults).\n On top of that, including incompatible settings across layers is even\n less likely to be a real use case. We should not be straining to make\n the code even worse for this imaginary scenario.\n\nWith that said...\n \n> ...But, teaching the revision machinery how to handle a walk through\n> commits in multiple commit-graph layers with incompatible Bloom filter\n> settings is trickier. Right now we compute all of the Bloom keys up\n> front using the Bloom filter settings in the top layer. I think we'd\n> probably want to change this to lazily compute those keys by:\nIt would probably be easiest to compute an array of bloom_key structs\n(per path) where the index corresponds to the depth of the commit-graph\nlayer. You can then access the correct key after you have identified\nwhich layer the commit is from.\n\n> At least, that's the best way that I can think to do it. It does get\n> kind of slow, though; we'd have to scan every commit graph layer at each\n> commit in the worst case to find the actual 'struct commit_graph *' in\n> order to get its Bloom filter settings. So, I think that's sort of\n> show-stoppingly slow, and that we should probably think more deeply\n> about it before taking up that direction.\n> \n> Maybe Stolee has some better thoughts about how we can quickly go from a\n> commit to either (a) the commit-graph struct that that commit is stored\n> in, or (b) the Bloom filter settings belonging to that struct.\n\nWe already have code that finds a commit in a commit-graph layer\nbased on its integer position by iterating until num_commits_in_base\nis below our position. The lexicographic position within that layer\nis found by subtracting num_commits_in_base.\n\nFor us, we would simply need:\n\nint get_commit_graph_layer(struct commit_graph *g, uint32_t pos)\n{\n\tuint32_t layer_index = 0;\n\n\twhile (g && pos < g->num_commits_in_base) {\n\t\tg = g->base_graph;\n\t\tlayer_index++;\n\t}\n\n\treturn layer_index;\n}\n\nYou could then use the response from get_commit_graph_layer()\nto load the correct Bloom key.\n\nAgain, I strongly suggest _not_ working hard to support this\ncase. We should only put in proper safeguards to prevent data\nlike this being written and protect a Git process that might\nstumble across data in this shape.\n\nThanks,\n-Stolee\n"},{"id":"407706","messageId":"20201016031844.GA2181783@nand.local","threadId":"53681","inReplyTo":"103404f0-a164-2550-4730-6b8773d3ca1f@gmail.com","subject":"Re: [PATCH v4 04/10] commit-graph: persist existence of changed-paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-10-16T03:18:44Z","receivedAt":"2020-10-16T03:18:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Oct 15, 2020 at 10:18:47PM -0400, Derrick Stolee wrote:\n> On 10/15/2020 5:41 PM, Taylor Blau wrote:\n> > So, we need to be more aggressively checking the Bloom filter settings\n> > in any layer we want to reuse a Bloom filter out of before reusing it\n> > verbatim in the current layer. The patch below the scissors line is\n> > sufficient to do that, and it causes the third test to start passing.\n>\n> I think there are three things we should keep in mind:\n>\n> 1. Incompatible Bloom filter settings between layers should be seen\n> as _inconsistent data_ as Git should not be writing incremental\n> commit-graph files with inconsistent Bloom filter settings. Thus,\n> when reading the commit-graph chain we should prevent incompatible\n> filters from being used. One way to do this is to notice different\n> settings and completely disable Bloom filters. The other way would\n> be to take the settings from the first layer with filters and then\n> clear the chunk_bloom_indexes and chunk_bloom_data fields for the\n> layers that don't agree. This fits with an expectation that lower\n> layers are larger, so more filters can be used in that situation.\n\nSure; I'd be fine with only allowing filters computed with the settings\npresent in the lowest or largest layer in the event that multiple layers\nexist with incompatible settings.\n\nI'm trying to point us towards a direction of not optimizing too far\nalong a direction that we're unlikely to take, while also trying to do\nsomething relatively non-invasive to make it possible for a version of\nGit to change the default Bloom settings. That is, if a user is writing\nsplit commit-graphs, and we change the default Bloom settings, they\nshouldn't have to recompute or merge down all of their Bloom filters.\n\nIf that's something that we never think is going to happen, I'm fine\nwith not thinking too hard about it. But, I also don't want to paint\nourselves into a corner, so I think something like the patch I wrote in\nthe email that you're replying to actually may be worth pursuing\nfurther. I dunno. Definitely after 2.29, though.\n\n> 2. We should be sure that Git will not agree to write incompatible\n> settings between layers of a commit-graph chain. Even more, it\n> should definitely not re-use filters when merging layers with\n> incompatible filter values. The strategy above to ignore Bloom\n> filters in incompatible upper layers is enough of a guard against\n> the \"merge layers\" situation.\n\nYeah, this would be fine with me.\n\n> 3. Allowing users (or future Git developers) to adjust the default\n> Bloom filter settings is one that is good to do for future-proofing,\n> but not one that I expect to be widely used (any gains here are\n> minuscule compared to the results already achieved with the defaults).\n> On top of that, including incompatible settings across layers is even\n> less likely to be a real use case. We should not be straining to make\n> the code even worse for this imaginary scenario.\n\nRight, I think we're pretty much in agreement here. Doing something easy\nto make sure that we don't run into a wall seems to make sense, but I\nthink modifying the revision walk machinery to keep track of multiple\nBloom keys computed with different settings corresponding to the set of\nBloom filter settings in commit-graph layers is probably too far in that\ndirection.\n\nFor what it's worth, I was mainly talking about it to say that it would\nbe more effort than it's probably worth to do. There's also nothing that\nwe're currently discussing that would prevent us from taking that same\ndirection up in six months from now.\n\nThanks,\nTaylor\n"},{"id":"407718","messageId":"6fe5de28-01f3-fc48-6bc3-aec0777ad889@gmail.com","threadId":"53681","inReplyTo":"20201016031844.GA2181783@nand.local","subject":"Re: [PATCH v4 04/10] commit-graph: persist existence of changed-paths","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-10-16T13:52:55Z","receivedAt":"2020-10-16T13:53:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 10/15/2020 11:18 PM, Taylor Blau wrote:\n> On Thu, Oct 15, 2020 at 10:18:47PM -0400, Derrick Stolee wrote:\n>> On 10/15/2020 5:41 PM, Taylor Blau wrote:\n>>> So, we need to be more aggressively checking the Bloom filter settings\n>>> in any layer we want to reuse a Bloom filter out of before reusing it\n>>> verbatim in the current layer. The patch below the scissors line is\n>>> sufficient to do that, and it causes the third test to start passing.\n>>\n>> I think there are three things we should keep in mind:\n>>\n>> 1. Incompatible Bloom filter settings between layers should be seen\n>> as _inconsistent data_ as Git should not be writing incremental\n>> commit-graph files with inconsistent Bloom filter settings. Thus,\n>> when reading the commit-graph chain we should prevent incompatible\n>> filters from being used. One way to do this is to notice different\n>> settings and completely disable Bloom filters. The other way would\n>> be to take the settings from the first layer with filters and then\n>> clear the chunk_bloom_indexes and chunk_bloom_data fields for the\n>> layers that don't agree. This fits with an expectation that lower\n>> layers are larger, so more filters can be used in that situation.\n> \n> Sure; I'd be fine with only allowing filters computed with the settings\n> present in the lowest or largest layer in the event that multiple layers\n> exist with incompatible settings.\n> \n> I'm trying to point us towards a direction of not optimizing too far\n> along a direction that we're unlikely to take, while also trying to do\n> something relatively non-invasive to make it possible for a version of\n> Git to change the default Bloom settings. That is, if a user is writing\n> split commit-graphs, and we change the default Bloom settings, they\n> shouldn't have to recompute or merge down all of their Bloom filters.\n\nThey would need to recompute when they merge layers, which introduces\na big question about how we should handle such a case.\n\n> If that's something that we never think is going to happen, I'm fine\n> with not thinking too hard about it. But, I also don't want to paint\n> ourselves into a corner, so I think something like the patch I wrote in\n> the email that you're replying to actually may be worth pursuing\n> further. I dunno. Definitely after 2.29, though.\n\nI think the proposed \"react properly to this unlikely situation\"\nis a good way to prevent getting locked into our choices now. It\nmakes it possible for \"old\" Git versions (2.30 until we decide to\nallow this mix) to interact with the inconsistent settings without\nfailure.\n\nWe don't need to do the 100% \"optimal\" case of using all filters\nin order to enable this choice in the future.\n \n[...]\n\n> For what it's worth, I was mainly talking about it to say that it would\n> be more effort than it's probably worth to do. There's also nothing that\n> we're currently discussing that would prevent us from taking that same\n> direction up in six months from now.\n\nYes, I just want to make sure that everyone agrees there is a\nmiddle ground without saying that inconsistent filter settings\nacross layers is a \"fully supported\" feature. If someone wants\nto tackle the work to make it a desirable state, then they can\ntry that (with great care).\n\nThanks,\n-Stolee\n"}]}