{"thread":{"id":"53975","subject":"[PATCH 00/10] more miscellaneous Bloom filter improvements","startedAt":"2020-08-03T18:57:10Z","lastAt":"2020-09-06T21:52:38Z","messageCount":117,"participants":["Taylor Blau","Jeff King","Derrick Stolee","Junio C Hamano","SZEDER Gábor","René Scharfe","Jakub Narębski"],"isPatch":true,"patchVersion":1,"patchTotal":10},"messages":[{"id":"402725","messageId":"cover.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":null,"subject":"[PATCH 00/10] more miscellaneous Bloom filter improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:04Z","receivedAt":"2020-08-03T18:57:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nHere are some patches that I have been sitting on while rolling out\nchanged-path Bloom filters at GitHub. I was initially going to send this\nas four separate topics, but they really make the most sense when sent\nall together.\n\nHere's an overview of what's contained:\n\n - The first patch fixes a bug where Bloom filters are only read if the\n most-recent layer in a commit-graph-chain has Bloom filters.\n\n - Patches 2-5 introduce 'commitgraph.readChangedPaths', a new\n configuration option for debugging changed-path Bloom filters. When\n disabled, the commit-graph machinery pretends as if there are no\n BIDX/BDAT chunks.\n\n This is useful when testing behavior with/without Bloom filters\n without having to regenerate the commit graph.\n\n - Patches 6-7 introduce a new 'BFXL' chunk which is a bitmap\n indicating which Bloom filters were too large to compute (ie., had\n more than 512 changed paths). This is a prerequisite for the final\n feature, --max-changed-paths.\n\n - Patches 8-10 introduces '--max-new-filters ', which allows\n callers to limit the number of Bloom filters that they are willing\n to compute from scratch when generating commit-graphs with\n --changed-paths.\n\nThe BFXL chunk is a prerequisite for '--max-new-filters' because of an\nunfortunate overloading where filters with (1) no changed paths, (2) too\nmany changed paths, and (3) ones that weren't computed at all are all\nrepresented as a length-0 filter by the BIDX chunk.\n\nThe problem is that in repositories with many commits that have too many\nchanged-paths to store Bloom filters, specifying '--max-new-filters'\nrecomputes those same large filters each commit-graph write, only to\nthrow them away. Knowing which filters are too-large allows us to skip\nover computing filters we know are a waste.\n\nThanks in advance for your review!\n\nTaylor Blau (10):\n commit-graph: introduce 'get_bloom_filter_settings()'\n commit-graph: pass a 'struct repository *' in more places\n t4216: use an '&&'-chain\n t/helper/test-read-graph.c: prepare repo settings\n commit-graph: respect 'commitgraph.readChangedPaths'\n commit-graph.c: sort index into commits list\n commit-graph: add large-filters bitmap chunk\n bloom: split 'get_bloom_filter()' in two\n commit-graph: rename 'split_commit_graph_opts'\n builtin/commit-graph.c: introduce '--max-new-filters='\n\n Documentation/config.txt | 2 +\n Documentation/config/commitgraph.txt | 8 +\n Documentation/git-commit-graph.txt | 4 +\n .../technical/commit-graph-format.txt | 9 +\n blame.c | 8 +-\n bloom.c | 34 ++-\n bloom.h | 9 +-\n builtin/commit-graph.c | 61 +++--\n commit-graph.c | 214 +++++++++++++-----\n commit-graph.h | 17 +-\n fuzz-commit-graph.c | 5 +-\n line-log.c | 2 +-\n repo-settings.c | 3 +\n repository.h | 1 +\n revision.c | 4 +-\n t/helper/test-bloom.c | 2 +-\n t/helper/test-read-graph.c | 3 +-\n t/t4216-log-bloom.sh | 54 ++++-\n t/t5324-split-commit-graph.sh | 13 ++\n 19 files changed, 357 insertions(+), 96 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\n--\n2.28.0.rc1.13.ge78abce653\n"},{"id":"402726","messageId":"08479793c1274d5ee0f063578bb0f4d93c910fa9.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 01/10] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:09Z","receivedAt":"2020-08-03T18:57:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Many places in the code often need a pointer to the commit-graph's\n'struct bloom_filter_settings', in which case they often take the value\nfrom the top-most commit-graph.\n\nIn the non-split case, this works as expected. In the split case,\nhowever, things get a little tricky. Not all layers in a chain of\nincremental commit-graphs are required to themselves have Bloom data,\nand so whether or not some part of the code uses Bloom filters depends\nentirely on whether or not the top-most level of the commit-graph chain\nhas Bloom filters.\n\nThis has been the behavior since Bloom filters were introduced, and has\nbeen codified into the tests since a759bfa9ee (t4216: add end to end\ntests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\nrequires that Bloom filters are not used in exactly the case described\nearlier.\n\nThere is no reason that this needs to be the case, since it is perfectly\nvalid for commits in an earlier layer to have Bloom filters when commits\nin a newer layer do not.\n\nSince Bloom settings are guaranteed to be the same for any layer in a\nchain that has Bloom data, it is sufficient to traverse the\n'->base_graph' pointer until either (1) a non-null 'struct\nbloom_filter_settings *' is found, or (2) until we are at the root of\nthe commit-graph chain.\n\nIntroduce a 'get_bloom_filter_settings()' function that does just this,\nand use it instead of purely dereferencing the top-most graph's\n'->bloom_filter_settings' pointer.\n\nWhile we're at it, add an additional test in t5324 to guard against code\nin the commit-graph writing machinery that doesn't correctly handle a\nNULL 'struct bloom_filter *'.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n blame.c | 6 ++++--\n bloom.c | 6 +++---\n commit-graph.c | 11 +++++++++++\n commit-graph.h | 2 ++\n revision.c | 2 +-\n t/t4216-log-bloom.sh | 9 ++++++---\n t/t5324-split-commit-graph.sh | 13 +++++++++++++\n 7 files changed, 40 insertions(+), 9 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 82fa16d658..3e5f8787bc 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -2891,16 +2891,18 @@ void setup_blame_bloom_data(struct blame_scoreboard *sb,\n \t\t\t const char *path)\n {\n \tstruct blame_bloom_data *bd;\n+\tstruct bloom_filter_settings *bs;\n \n \tif (!sb->repo->objects->commit_graph)\n \t\treturn;\n \n-\tif (!sb->repo->objects->commit_graph->bloom_filter_settings)\n+\tbs = get_bloom_filter_settings(sb->repo);\n+\tif (!bs)\n \t\treturn;\n \n \tbd = xmalloc(sizeof(struct blame_bloom_data));\n \n-\tbd->settings = sb->repo->objects->commit_graph->bloom_filter_settings;\n+\tbd->settings = bs;\n \n \tbd->alloc = 4;\n \tbd->nr = 0;\ndiff --git a/bloom.c b/bloom.c\nindex 1a573226e7..cd9380ac62 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -38,7 +38,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \twhile (graph_pos < g->num_commits_in_base)\n \t\tg = g->base_graph;\n \n-\t/* The commit graph commit 'c' lives in doesn't carry bloom filters. */\n+\t/* The commit graph commit 'c' lives in doesn't carry Bloom filters. */\n \tif (!g->chunk_bloom_indexes)\n \t\treturn 0;\n \n@@ -195,8 +195,8 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes)\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n+\t\t\t\treturn filter;\n \t}\n \n \tif (filter->data)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e51c91dd5b..d4b06811be 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -660,6 +660,17 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r)\n+{\n+\tstruct commit_graph *g = r->objects->commit_graph;\n+\twhile (g) {\n+\t\tif (g->bloom_filter_settings)\n+\t\t\treturn g->bloom_filter_settings;\n+\t\tg = g->base_graph;\n+\t}\n+\treturn NULL;\n+}\n+\n static void close_commit_graph_one(struct commit_graph *g)\n {\n \tif (!g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 09a97030dc..0677dd1031 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -87,6 +87,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n */\n int generation_numbers_enabled(struct repository *r);\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r);\n+\n enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_APPEND = (1 << 0),\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\ndiff --git a/revision.c b/revision.c\nindex 6de29cdf7a..e244beed05 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -684,7 +684,7 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \tif (!revs->repo->objects->commit_graph)\n \t\treturn;\n \n-\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n+\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n \tif (!revs->bloom_filter_settings)\n \t\treturn;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c21cc160f3..c9f9bdf1ba 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -60,7 +60,7 @@ setup () {\n \n test_bloom_filters_used () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":${2:-0},\\\"maybe\\\"\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom &&\n@@ -134,8 +134,11 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n-test_expect_success 'Do not use Bloom filters if the latest graph does not have Bloom filters.' '\n-\ttest_bloom_filters_not_used \"-- A/B\"\n+test_expect_success 'use Bloom filters even if the latest graph does not have Bloom filters' '\n+\t# Ensure that the number of empty filters is equal to the number of\n+\t# filters in the latest graph layer to prove that they are loaded (and\n+\t# ignored).\n+\ttest_bloom_filters_used \"-- A/B\" 3\n '\n \n test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 9b850ea907..5bdfd53ef9 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -425,4 +425,17 @@ done <<\\EOF\n 0600 -r--------\n EOF\n \n+test_expect_success '--split=replace with partial Bloom data' '\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/3 &&\n+\tgit rev-list -1 HEAD~2 >a &&\n+\tgit rev-list -1 HEAD~1 >b &&\n+\tgit commit-graph write --split=no-merge --stdin-commits --changed-paths graph-files &&\n+\ttest_line_count = 1 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n test_done\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402727","messageId":"52f8f7424eddc04b01882d73c39552e8e22a7178.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 02/10] commit-graph: pass a 'struct repository *' in more places","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:13Z","receivedAt":"2020-08-03T18:57:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a future commit, some commit-graph internals will want access to\n'r->settings', but we only have the 'struct object_directory *'\ncorresponding to that repository.\n\nAdd an additional parameter to pass the repository around in more\nplaces. In the next patch, we will remove the object directory (and\ninstead reference it with 'r->odb').\n\nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 2 +-\n commit-graph.c | 18 +++++++++++-------\n commit-graph.h | 6 ++++--\n fuzz-commit-graph.c | 5 +++--\n 4 files changed, 19 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 523501f217..ba5584463f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -106,7 +106,7 @@ static int graph_verify(int argc, const char **argv)\n \tFREE_AND_NULL(graph_name);\n \n \tif (open_ok)\n-\t\tgraph = load_commit_graph_one_fd_st(fd, &st, odb);\n+\t\tgraph = load_commit_graph_one_fd_st(the_repository, fd, &st, odb);\n \telse\n \t\tgraph = read_commit_graph_one(the_repository, odb);\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex d4b06811be..81a6f2a8ce 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -224,7 +224,8 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st)\n \treturn 1;\n }\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tvoid *graph_map;\n@@ -240,7 +241,7 @@ struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n \t}\n \tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n-\tret = parse_commit_graph(graph_map, graph_size);\n+\tret = parse_commit_graph(r, graph_map, graph_size);\n \n \tif (ret)\n \t\tret->odb = odb;\n@@ -280,7 +281,8 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n \tconst unsigned char *data, *chunk_lookup;\n \tuint32_t i;\n@@ -445,7 +447,9 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n \treturn NULL;\n }\n \n-static struct commit_graph *load_commit_graph_one(const char *graph_file,\n+\n+static struct commit_graph *load_commit_graph_one(struct repository *r,\n+\t\t\t\t\t\t const char *graph_file,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \n@@ -457,7 +461,7 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file,\n \tif (!open_ok)\n \t\treturn NULL;\n \n-\tg = load_commit_graph_one_fd_st(fd, &st, odb);\n+\tg = load_commit_graph_one_fd_st(r, fd, &st, odb);\n \n \tif (g)\n \t\tg->filename = xstrdup(graph_file);\n@@ -469,7 +473,7 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tchar *graph_name = get_commit_graph_filename(odb);\n-\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \tfree(graph_name);\n \n \treturn g;\n@@ -550,7 +554,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \t\tvalid = 0;\n \t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\t\tchar *graph_name = get_split_graph_filename(odb, line.buf);\n-\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \n \t\t\tfree(graph_name);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 0677dd1031..d9acb22bac 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,13 @@ struct commit_graph {\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb);\n struct commit_graph *read_commit_graph_one(struct repository *r,\n \t\t\t\t\t struct object_directory *odb);\n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n /*\n * Return 1 if and only if the repository has a commit-graph\ndiff --git a/fuzz-commit-graph.c b/fuzz-commit-graph.c\nindex 430817214d..e7cf6d5b0f 100644\n--- a/fuzz-commit-graph.c\n+++ b/fuzz-commit-graph.c\n@@ -1,7 +1,8 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n \n@@ -10,7 +11,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \tstruct commit_graph *g;\n \n \tinitialize_the_repository();\n-\tg = parse_commit_graph((void *)data, size);\n+\tg = parse_commit_graph(the_repository, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\n \n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402728","messageId":"d12fcc4a8d1637e254fa7d4a53be354cbe5199b2.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 03/10] t4216: use an '&&'-chain","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:17Z","receivedAt":"2020-08-03T18:57:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a759bfa9ee (t4216: add end to end tests for git log with Bloom\nfilters, 2020-04-06), a 'rm' invocation was added without a\ncorresponding '&&' chain.\n\nWhen 'trace.perf' already exists, everything works fine. However, the\nfunction can be executed without 'trace.perf' on disk (eg., when the\nsubset of tests run is altered with '--run'), and so the bare 'rm'\ncomplains about a missing file.\n\nTo remove some noise from the test log, invoke 'rm' with '-f', at which\npoint it is sensible to place the 'rm -f' in an '&&'-chain, which is\nboth (1) our usual style, and (2) avoids a broken chain in the future if\nmore commands are added at the beginning of the function.\n\nHelped-by: Eric Sunshine \nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c9f9bdf1ba..fe19f6a60c 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -53,7 +53,7 @@ sane_unset GIT_TRACE2_PERF_BRIEF\n sane_unset GIT_TRACE2_CONFIG_PARAMS\n \n setup () {\n-\trm \"$TRASH_DIRECTORY/trace.perf\"\n+\trm -f \"$TRASH_DIRECTORY/trace.perf\" &&\n \tgit -c core.commitGraph=false log --pretty=\"format:%s\" $1 >log_wo_bloom &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.perf\" git -c core.commitGraph=true log --pretty=\"format:%s\" $1 >log_w_bloom\n }\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402729","messageId":"42a0ca9549090c879d5c4d9cb53732d8dc4f0ef5.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 04/10] t/helper/test-read-graph.c: prepare repo settings","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:22Z","receivedAt":"2020-08-03T18:57:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The read-graph test-tool is used by a number of the commit-graph test to\nassert various properties about a commit-graph. Previously, this program\nnever ran 'prepare_repo_settings()'. There was no need to do so, since\nnone of the commit-graph machinery is affected by the repo settings.\n\nIn the next patch, the commit-graph machinery's behavior will become\ndependent on the repo settings, and so loading them before running the\nrest of the test tool is critical.\n\nAs such, teach the test tool to call 'prepare_repo_settings()'.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 6d0c962438..5f585a1725 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -12,11 +12,12 @@ int cmd__read_graph(int argc, const char **argv)\n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n \n+\tprepare_repo_settings(the_repository);\n+\n \tgraph = read_commit_graph_one(the_repository, odb);\n \tif (!graph)\n \t\treturn 1;\n \n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402730","messageId":"e61507728336ecd9de9cce3c9a74dd418e4a4215.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 05/10] commit-graph: respect 'commitgraph.readChangedPaths'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:26Z","receivedAt":"2020-08-03T18:57:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Git uses the 'core.commitGraph' configuration value to control whether\nor not the commit graph is used when parsing commits or performing a\ntraversal.\n\nNow that commit-graphs can also contain a section for changed-path Bloom\nfilters, administrators that already have commit-graphs may find it\nconvenient to use those graphs without relying on their changed-path\nBloom filters. This can happen, for example, during a staged roll-out,\nor in the event of an incident.\n\nIntroduce 'commitgraph.readChangedPaths' to control whether or not Bloom\nfilters are read. Note that this configuration is independent from both:\n\n - 'core.commitGraph', to allow flexibility in using all parts of a\n commit-graph _except_ for its Bloom filters.\n\n - The '--changed-paths' option for 'git commit-graph write', to allow\n reading and writing Bloom filters to be controlled independently.\n\nWhen the variable is set, pretend as if no Bloom data was specified at\nall. This avoids adding additional special-casing outside of the\ncommit-graph internals.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 ++\n Documentation/config/commitgraph.txt | 4 ++++\n commit-graph.c | 6 ++++--\n repo-settings.c | 3 +++\n repository.h | 1 +\n t/t4216-log-bloom.sh | 4 +++-\n 6 files changed, 17 insertions(+), 3 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ef0768b91a..78883c6e63 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -340,6 +340,8 @@ include::config/column.txt[]\n \n include::config/commit.txt[]\n \n+include::config/commitgraph.txt[]\n+\n include::config/credential.txt[]\n \n include::config/completion.txt[]\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nnew file mode 100644\nindex 0000000000..bb78e72f1b\n--- /dev/null\n+++ b/Documentation/config/commitgraph.txt\n@@ -0,0 +1,4 @@\n+commitgraph.readChangedPaths::\n+\tIf true, then git will use the changed-path Bloom filters in the\n+\tcommit-graph file (if it exists, and they are present). Defaults to\n+\ttrue. See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 81a6f2a8ce..cb9d7fea04 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -320,6 +320,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n+\tprepare_repo_settings(r);\n+\n \tgraph = alloc_commit_graph();\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -396,14 +398,14 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n \t\t\tif (graph->chunk_bloom_indexes)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n \t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n \t\t\tbreak;\n \n \t\tcase GRAPH_CHUNKID_BLOOMDATA:\n \t\t\tif (graph->chunk_bloom_data)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n \t\t\t\tuint32_t hash_version;\n \t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n \t\t\t\thash_version = get_be32(data + chunk_offset);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 0918408b34..9e551bc03d 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -17,9 +17,12 @@ void prepare_repo_settings(struct repository *r)\n \n \tif (!repo_config_get_bool(r, \"core.commitgraph\", &value))\n \t\tr->settings.core_commit_graph = value;\n+\tif (!repo_config_get_bool(r, \"commitgraph.readchangedpaths\", &value))\n+\t\tr->settings.commit_graph_read_changed_paths = value;\n \tif (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n \t\tr->settings.gc_write_commit_graph = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_commit_graph, 1);\n+\tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n \n \tif (!repo_config_get_int(r, \"index.version\", &value))\ndiff --git a/repository.h b/repository.h\nindex 3c1f7d54bd..81759b7d27 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -29,6 +29,7 @@ struct repo_settings {\n \tint initialized;\n \n \tint core_commit_graph;\n+\tint commit_graph_read_changed_paths;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fe19f6a60c..b3d1f596f8 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -90,7 +90,9 @@ do\n \t\t \"--ancestry-path side..master\"\n \tdo\n \t\ttest_expect_success \"git log option: $option for path: $path\" '\n-\t\t\ttest_bloom_filters_used \"$option -- $path\"\n+\t\t\ttest_bloom_filters_used \"$option -- $path\" &&\n+\t\t\ttest_config commitgraph.readChangedPaths false &&\n+\t\t\ttest_bloom_filters_not_used \"$option -- $path\"\n \t\t'\n \tdone\n done\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402731","messageId":"b31c60d71266c40ae23a619d5ac5fd99148e1649.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 06/10] commit-graph.c: sort index into commits list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:30Z","receivedAt":"2020-08-03T18:57:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For locality, 'compute_bloom_filters()' sorts the commits for which it\nwants to compute Bloom filters in a preferred order (cf., 3d11275505\n(commit-graph: examine commits by generation number, 2020-03-30) for\ndetails).\n\nThe subsequent patch will want to recover the new graph position of each\ncommit. Since the 'packed_commit_list' already stores a double-pointer,\navoid a 'COPY_ARRAY' and instead keep track of an index into the\noriginal list. (Use an integer index instead of a memory address, since\nthis involves a needlessly confusing triple-pointer).\n\nAlter the two sorting routines 'commit_pos_cmp' and 'commit_gen_cmp' to\ntake into account the packed_commit_list they are sorting with respect\nto. Since 'compute_bloom_filters()' is the only caller for each of those\ncomparison functions, no other call-sites need updating.\n\nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 44 ++++++++++++++++++++++++--------------------\n 1 file changed, 24 insertions(+), 20 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex cb9d7fea04..d6ea556649 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -79,10 +79,18 @@ static void set_commit_pos(struct repository *r, const struct object_id *oid)\n \t*commit_pos_at(&commit_pos, commit) = max_pos++;\n }\n \n-static int commit_pos_cmp(const void *va, const void *vb)\n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+static int commit_pos_cmp(const void *va, const void *vb, void *ctx)\n {\n-\tconst struct commit *a = *(const struct commit **)va;\n-\tconst struct commit *b = *(const struct commit **)vb;\n+\tstruct packed_commit_list *commits = ctx;\n+\n+\tconst struct commit *a = commits->list[*(int *)va];\n+\tconst struct commit *b = commits->list[*(int *)vb];\n \treturn commit_pos_at(&commit_pos, a) -\n \t commit_pos_at(&commit_pos, b);\n }\n@@ -139,10 +147,12 @@ static struct commit_graph_data *commit_graph_data_at(const struct commit *c)\n \treturn data;\n }\n \n-static int commit_gen_cmp(const void *va, const void *vb)\n+static int commit_gen_cmp(const void *va, const void *vb, void *ctx)\n {\n-\tconst struct commit *a = *(const struct commit **)va;\n-\tconst struct commit *b = *(const struct commit **)vb;\n+\tstruct packed_commit_list *commits = ctx;\n+\n+\tconst struct commit *a = commits->list[*(int *)va];\n+\tconst struct commit *b = commits->list[*(int *)vb];\n \n \tuint32_t generation_a = commit_graph_generation(a);\n \tuint32_t generation_b = commit_graph_generation(b);\n@@ -923,12 +933,6 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n-struct packed_commit_list {\n-\tstruct commit **list;\n-\tint nr;\n-\tint alloc;\n-};\n-\n struct packed_oid_list {\n \tstruct object_id *list;\n \tint nr;\n@@ -1389,7 +1393,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct progress *progress = NULL;\n-\tstruct commit **sorted_commits;\n+\tint *sorted_commits;\n \n \tinit_bloom_filters();\n \n@@ -1399,15 +1403,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->commits.nr);\n \n \tALLOC_ARRAY(sorted_commits, ctx->commits.nr);\n-\tCOPY_ARRAY(sorted_commits, ctx->commits.list, ctx->commits.nr);\n-\n-\tif (ctx->order_by_pack)\n-\t\tQSORT(sorted_commits, ctx->commits.nr, commit_pos_cmp);\n-\telse\n-\t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n+\tfor (i = 0; i < ctx->commits.nr; i++)\n+\t\tsorted_commits[i] = i;\n+\tQSORT_S(sorted_commits, ctx->commits.nr,\n+\t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n+\t\t&ctx->commits);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tstruct commit *c = sorted_commits[i];\n+\t\tint pos = sorted_commits[i];\n+\t\tstruct commit *c = ctx->commits.list[pos];\n \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402732","messageId":"ab789072331127079347742b6ce3b3415bbbe4f6.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 07/10] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:33Z","receivedAt":"2020-08-03T18:57:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a commit has 512 changed paths or greater, the commit-graph\nmachinery represents it as a zero-length filter since having many\nentries in the Bloom filter has undesirable effects on the false\npositivity rate.\n\nIn addition to these too-large filters, the commit-graph machinery also\nrepresents commits with no filter and commits with no changed paths in\nthe same way.\n\nWhen writing a commit-graph that aggregates several incremental\ncommit-graph layers (eg., with '--split=replace'), the commit-graph\nmachinery first computes all of the Bloom filters that it wants to write\nbut does not already know about from existing graph layers. Because we\noverload the zero-length filter in the above fashion, this leads to\nrecomputing large filters over and over again.\n\nThis is already undesirable, since it means that we are wasting\nconsiderable effort to discover that a commit has at least 512 changed\npaths, only to throw that effort away (and then repeat the process the\nnext time a roll-up is performed).\n\nIn a subsequent patch, we will add a '--max-new-filters' option, which\nspecifies an upper-bound on the number of new filters we are willing to\ncompute from scratch. Suppose that there are 'N' too-large filters, and\nwe specify '--max-new-filters=M'. If 'N >= M', it is unlikely that any\nfilters will be generated, since we'll spend most of our effort on\nfilters that we ultimately throw away. If 'N < M', filters will trickle\nin over time, but only at most 'M - N' per-write.\n\nTo address this, add a new chunk which encodes a bitmap where the ith\nbit is on iff the ith commit has zero or at least 512 changed paths.\nWhen computing Bloom filters, first consult the relevant bitmap (in the\ncase that we are rolling up existing layers) to see if computing the\nBloom filter from scratch would be a waste of time.\n\nThis patch implements a new chunk instead of extending the existing BIDX\nand BDAT chunks because modifying these chunks would confuse old\nclients. For example, consider setting the most-significant bit in an\nentry in the BIDX chunk to indicate that that filter is too-large. New\nclients would understand how to interpret this, but old clients would\ntreat that entry as a really large filter.\n\nBy avoiding the need to move to new versions of the BDAT and BIDX chunk,\nwe can give ourselves more time to consider whether or not other\nmodifications to these chunks are worthwhile without holding up this\nchange.\n\nAnother approach would be to introduce a new BIDX chunk (say, one\nidentified by 'BID2') which is identical to the existing BIDX chunk,\nexcept the most-significant bit of each offset is interpreted as \"this\nfilter is too big\" iff looking at a BID2 chunk. This avoids having to\nwrite a bitmap, but forces older clients to rewrite their commit-graphs\n(as well as reduces the theoretical largest Bloom filters we couldl\nwrite, and forces us to maintain the code necessary to translate BIDX\nchunks to BID2 ones). Separately from this patch, I implemented this\nalternate approach and did not find it to be advantageous.\n\nSigned-off-by: Taylor Blau \n---\n .../technical/commit-graph-format.txt | 9 ++\n commit-graph.c | 85 ++++++++++++++++++-\n commit-graph.h | 2 +\n t/t4216-log-bloom.sh | 20 ++++-\n 4 files changed, 112 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 440541045d..a7191c03d3 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -123,6 +123,15 @@ CHUNK DATA:\n of length zero.\n * The BDAT chunk is present if and only if BIDX is present.\n \n+ Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n+ * It contains a list of 'eword_t's (the length of this list is determined by\n+ the width of the chunk) which is a bitmap. The 'i'th bit is set exactly\n+ when the 'i'th commit in the graph has a changed-path Bloom filter with\n+ zero entries (either because the commit is empty, or because it contains\n+ more than 512 changed paths).\n+ * The BFXL chunk is present on newer versions of Git iff the BIDX and BDAT\n+ chunks are also present.\n+\n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n This list of H-byte hashes describe a set of B commit-graph files that\n form a commit-graph chain. The graph position for the ith commit in this\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d6ea556649..c870ffe0ed 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -41,8 +41,9 @@ void git_test_write_commit_graph_or_die(void)\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n+#define GRAPH_CHUNKID_BLOOMLARGE 0x4246584c /* \"BFXL\" */\n #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n-#define MAX_NUM_CHUNKS 7\n+#define MAX_NUM_CHUNKS 8\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -429,6 +430,17 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n \t\t\t}\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BLOOMLARGE:\n+\t\t\tif (graph->bloom_large.word_alloc)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n+\t\t\t\tuint32_t alloc = get_be64(chunk_lookup + 4) - chunk_offset;\n+\n+\t\t\t\tgraph->bloom_large.word_alloc = alloc;\n+\t\t\t\tgraph->bloom_large.words = (eword_t *)(data + chunk_offset);\n+\t\t\t}\n+\t\t\tbreak;\n \t\t}\n \n \t\tif (chunk_repeated) {\n@@ -443,6 +455,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t/* We need both the bloom chunks to exist together. Else ignore the data */\n \t\tgraph->chunk_bloom_indexes = NULL;\n \t\tgraph->chunk_bloom_data = NULL;\n+\t\tgraph->bloom_large.words = NULL;\n+\t\tgraph->bloom_large.word_alloc = 0;\n \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n \t}\n \n@@ -933,6 +947,21 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n+static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n+\t\t\t\t\t const struct commit *c)\n+{\n+\tuint32_t graph_pos = commit_graph_position(c);\n+\tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\treturn 0;\n+\n+\twhile (g && graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (!(g && g->bloom_large.word_alloc))\n+\t\treturn 0;\n+\treturn bitmap_get(&g->bloom_large, graph_pos - g->num_commits_in_base);\n+}\n+\n struct packed_oid_list {\n \tstruct object_id *list;\n \tint nr;\n@@ -969,6 +998,11 @@ struct write_commit_graph_context {\n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n+\tstruct bitmap *bloom_large;\n+\n+\tint count_bloom_filter_known_large;\n+\tint count_bloom_filter_found_large;\n+\tint count_bloom_filter_computed;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1231,6 +1265,15 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \treturn 0;\n }\n \n+static int write_graph_chunk_bloom_large(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\ttrace2_region_enter(\"commit-graph\", \"bloom_large\", ctx->r);\n+\thashwrite(f, ctx->bloom_large->words, ctx->bloom_large->word_alloc * sizeof(eword_t));\n+\ttrace2_region_leave(\"commit-graph\", \"bloom_large\", ctx->r);\n+\treturn 0;\n+}\n+\n static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n@@ -1389,6 +1432,24 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n+{\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"filter_known_large\",\n+\t\t\t ctx->count_bloom_filter_known_large);\n+\tjw_object_intmax(&jw, \"filter_found_large\",\n+\t\t\t ctx->count_bloom_filter_found_large);\n+\tjw_object_intmax(&jw, \"filter_computed\",\n+\t\t\t ctx->count_bloom_filter_computed);\n+\tjw_end(&jw);\n+\n+\ttrace2_data_json(\"commit-graph\", the_repository, \"bloom_statistics\", &jw);\n+\n+\tjw_release(&jw);\n+}\n+\n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n@@ -1396,6 +1457,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint *sorted_commits;\n \n \tinit_bloom_filters();\n+\tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD);\n \n \tif (ctx->report_progress)\n \t\tprogress = start_delayed_progress(\n@@ -1412,11 +1474,24 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n+\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\tctx->count_bloom_filter_known_large++;\n+\t\t} else {\n+\t\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n+\t\t\tif (!filter->len) {\n+\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t\t}\n+\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\t\tctx->count_bloom_filter_computed++;\n+\t\t}\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n+\tif (trace2_is_enabled())\n+\t\ttrace2_bloom_filter_write_statistics(ctx);\n+\n \tfree(sorted_commits);\n \tstop_progress(&progress);\n }\n@@ -1739,6 +1814,10 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n \t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n+\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMLARGE;\n+\t\tchunks[num_chunks].size = sizeof(eword_t) * ctx->bloom_large->word_alloc;\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_large;\n+\t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d9acb22bac..afbc5fa41e 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,6 +4,7 @@\n #include \"git-compat-util.h\"\n #include \"object-store.h\"\n #include \"oidset.h\"\n+#include \"ewah/ewok.h\"\n \n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n #define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n@@ -71,6 +72,7 @@ struct commit_graph {\n \tconst unsigned char *chunk_base_graphs;\n \tconst unsigned char *chunk_bloom_indexes;\n \tconst unsigned char *chunk_bloom_data;\n+\tstruct bitmap bloom_large;\n \n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex b3d1f596f8..231dc8a3a7 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -33,7 +33,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \tgit commit-graph write --reachable --changed-paths\n '\n graph_read_expect () {\n-\tNUM_CHUNKS=5\n+\tNUM_CHUNKS=6\n \tcat >expect <<- EOF\n \theader: 43475048 1 1 $NUM_CHUNKS 0\n \tnum_commits: $1\n@@ -195,5 +195,23 @@ test_expect_success 'correctly report changes over limit' '\n \t\tdone\n \t)\n '\n+test_bloom_filters_computed () {\n+\tcommit_graph_args=$1\n+\tbloom_trace_prefix=\"{\\\"filter_known_large\\\":$2,\\\"filter_found_large\\\":$3,\\\"filter_computed\\\":$4\"\n+\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n+\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write $commit_graph_args &&\n+\tgrep \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.event\"\n+}\n+\n+test_expect_success 'Bloom generation does not recompute too-large filters' '\n+\t(\n+\t\tcd 513changes &&\n+\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t--split=replace &&\n+\t\ttest_commit c1 filter &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n+\t\t\t1 0 1\n+\t)\n+'\n \n test_done\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402733","messageId":"a494094c10f9bddca5743973409ccb4540841116.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 08/10] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:37Z","receivedAt":"2020-08-03T18:57:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'get_bloom_filter' takes a flag to control whether it will compute a\nBloom filter if the requested one is missing. In the next patch, we'll\nadd yet another flag to this method, which would force all but one\ncaller to specify an extra 'NULL' parameter at the end.\n\nInstead of doing this, split 'get_bloom_filter' into two functions:\n'get_bloom_filter' and 'get_or_compute_bloom_filter'. The former only\nlooks up a Bloom filter (and does not compute one if it's missing,\nthus dropping the 'compute_if_not_present' flag). The latter does\ncompute missing Bloom filters, with an additional parameter to store\nwhether or not it needed to do so.\n\nThis simplifies many call-sites, since the majority of existing callers\nto 'get_bloom_filter' do not want missing Bloom filters to be computed\n(so they can drop the parameter entirely and use the simpler version of\nthe function).\n\nSigned-off-by: Taylor Blau \n---\n blame.c | 2 +-\n bloom.c | 13 ++++++++++---\n bloom.h | 9 ++++++---\n commit-graph.c | 6 +++---\n line-log.c | 2 +-\n revision.c | 2 +-\n t/helper/test-bloom.c | 2 +-\n 7 files changed, 23 insertions(+), 13 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 3e5f8787bc..756285fca7 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -1275,7 +1275,7 @@ static int maybe_changed_path(struct repository *r,\n \tif (commit_graph_generation(origin->commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn 1;\n \n-\tfilter = get_bloom_filter(r, origin->commit, 0);\n+\tfilter = get_bloom_filter(r, origin->commit);\n \n \tif (!filter)\n \t\treturn 1;\ndiff --git a/bloom.c b/bloom.c\nindex cd9380ac62..a8a21762f4 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,9 +177,10 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present)\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t int *computed)\n {\n \tstruct bloom_filter *filter;\n \tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n@@ -187,6 +188,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n+\tif (computed)\n+\t\t*computed = 0;\n+\n \tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n@@ -273,6 +277,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \t\tfilter->len = 0;\n \t}\n \n+\tif (computed)\n+\t\t*computed = 1;\n+\n \tfree(diff_queued_diff.queue);\n \tDIFF_QUEUE_CLEAR(&diff_queued_diff);\n \ndiff --git a/bloom.h b/bloom.h\nindex d8fbb0fbf1..2fdc2918f8 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -80,9 +80,12 @@ void add_key_to_filter(const struct bloom_key *key,\n \n void init_bloom_filters(void);\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present);\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t int *computed);\n+\n+#define get_bloom_filter(r, c) get_or_compute_bloom_filter((r), (c), 0, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex c870ffe0ed..2e765b26d5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1214,7 +1214,7 @@ static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tuint32_t cur_pos = 0;\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \t\tcur_pos += len;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1253,7 +1253,7 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1478,7 +1478,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tbitmap_set(ctx->bloom_large, pos);\n \t\t\tctx->count_bloom_filter_known_large++;\n \t\t} else {\n-\t\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n+\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(ctx->r, c, 1, NULL);\n \t\t\tif (!filter->len) {\n \t\t\t\tbitmap_set(ctx->bloom_large, pos);\n \t\t\t\tctx->count_bloom_filter_found_large++;\ndiff --git a/line-log.c b/line-log.c\nindex c53692834d..9e58fd185a 100644\n--- a/line-log.c\n+++ b/line-log.c\n@@ -1159,7 +1159,7 @@ static int bloom_filter_check(struct rev_info *rev,\n \t\treturn 1;\n \n \tif (!rev->bloom_filter_settings ||\n-\t !(filter = get_bloom_filter(rev->repo, commit, 0)))\n+\t !(filter = get_bloom_filter(rev->repo, commit)))\n \t\treturn 1;\n \n \tif (!range)\ndiff --git a/revision.c b/revision.c\nindex e244beed05..73b59d2134 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -756,7 +756,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \tif (commit_graph_generation(commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn -1;\n \n-\tfilter = get_bloom_filter(revs->repo, commit, 0);\n+\tfilter = get_bloom_filter(revs->repo, commit);\n \n \tif (!filter) {\n \t\tcount_bloom_filter_not_present++;\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex f0aa80b98e..76a28a2199 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -39,7 +39,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tstruct bloom_filter *filter;\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n-\tfilter = get_bloom_filter(the_repository, c, 1);\n+\tfilter = get_or_compute_bloom_filter(the_repository, c, 1, NULL);\n \tprint_bloom_filter(filter);\n }\n \n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402734","messageId":"309e76bb174d64ffae6454f2bc9968c6c2632aff.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 09/10] commit-graph: rename 'split_commit_graph_opts'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:41Z","receivedAt":"2020-08-03T18:57:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a subsequent commit, additional options will be added to the\ncommit-graph API, which themselves do not have to do with splitting.\n\nRename the 'split_commit_graph_opts' structure to the more-generic\n'commit_graph_opts' to encompass both.\n\nSuggsted-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 20 ++++++++++----------\n commit-graph.c | 40 ++++++++++++++++++++--------------------\n commit-graph.h | 6 +++---\n 3 files changed, 33 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ba5584463f..38f5f57d15 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,7 +119,7 @@ static int graph_verify(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n-static struct split_commit_graph_opts split_opts;\n+static struct commit_graph_opts write_opts;\n \n static int write_option_parse_split(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n@@ -187,24 +187,24 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n \t\t\tN_(\"enable computation for changed paths\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\twrite_option_parse_split),\n-\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\tOPT_INTEGER(0, \"max-commits\", &write_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n-\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\tOPT_INTEGER(0, \"size-multiple\", &write_opts.size_multiple,\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n-\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n \t\tOPT_END(),\n \t};\n \n \topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n-\tsplit_opts.size_multiple = 2;\n-\tsplit_opts.max_commits = 0;\n-\tsplit_opts.expire_time = 0;\n+\twrite_opts.size_multiple = 2;\n+\twrite_opts.max_commits = 0;\n+\twrite_opts.expire_time = 0;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -232,7 +232,7 @@ static int graph_write(int argc, const char **argv)\n \todb = find_odb(the_repository, opts.obj_dir);\n \n \tif (opts.reachable) {\n-\t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n+\t\tif (write_commit_graph_reachable(odb, flags, &write_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n@@ -261,7 +261,7 @@ static int graph_write(int argc, const char **argv)\n \t\t\t opts.stdin_packs ? &pack_indexes : NULL,\n \t\t\t opts.stdin_commits ? &commits : NULL,\n \t\t\t flags,\n-\t\t\t &split_opts))\n+\t\t\t &write_opts))\n \t\tresult = 1;\n \n cleanup:\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 2e765b26d5..8626024faa 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -995,7 +995,7 @@ struct write_commit_graph_context {\n \t\t changed_paths:1,\n \t\t order_by_pack:1;\n \n-\tconst struct split_commit_graph_opts *split_opts;\n+\tconst struct commit_graph_opts *opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \tstruct bitmap *bloom_large;\n@@ -1327,8 +1327,8 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1520,7 +1520,7 @@ static int add_ref_to_set(const char *refname,\n \n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t\t\t const struct commit_graph_opts *opts)\n {\n \tstruct oidset commits = OIDSET_INIT;\n \tstruct refs_cb_data data;\n@@ -1537,7 +1537,7 @@ int write_commit_graph_reachable(struct object_directory *odb,\n \tstop_progress(&data.progress);\n \n \tresult = write_commit_graph(odb, NULL, &commits,\n-\t\t\t\t flags, split_opts);\n+\t\t\t\t flags, opts);\n \n \toidset_clear(&commits);\n \treturn result;\n@@ -1652,8 +1652,8 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tctx->num_extra_edges = 0;\n \tif (ctx->report_progress)\n@@ -1951,13 +1951,13 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tint max_commits = 0;\n \tint size_mult = 2;\n \n-\tif (ctx->split_opts) {\n-\t\tmax_commits = ctx->split_opts->max_commits;\n+\tif (ctx->opts) {\n+\t\tmax_commits = ctx->opts->max_commits;\n \n-\t\tif (ctx->split_opts->size_multiple)\n-\t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t\tif (ctx->opts->size_multiple)\n+\t\t\tsize_mult = ctx->opts->size_multiple;\n \n-\t\tflags = ctx->split_opts->flags;\n+\t\tflags = ctx->opts->flags;\n \t}\n \n \tg = ctx->r->objects->commit_graph;\n@@ -2135,8 +2135,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n \n-\tif (ctx->split_opts && ctx->split_opts->expire_time)\n-\t\texpire_time = ctx->split_opts->expire_time;\n+\tif (ctx->opts && ctx->opts->expire_time)\n+\t\texpire_time = ctx->opts->expire_time;\n \tif (!ctx->split) {\n \t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n@@ -2187,7 +2187,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t const struct commit_graph_opts *opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -2203,7 +2203,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n-\tctx->split_opts = split_opts;\n+\tctx->opts = opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n \tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n@@ -2243,15 +2243,15 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\t}\n \t\t}\n \n-\t\tif (ctx->split_opts)\n-\t\t\treplace = ctx->split_opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n+\t\tif (ctx->opts)\n+\t\t\treplace = ctx->opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n \t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n-\t\tctx->oids.alloc = split_opts->max_commits;\n+\tif (ctx->split && opts && ctx->oids.alloc > opts->max_commits)\n+\t\tctx->oids.alloc = opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->odb);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex afbc5fa41e..4cd991cf26 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -107,7 +107,7 @@ enum commit_graph_split_flags {\n \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n };\n \n-struct split_commit_graph_opts {\n+struct commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n@@ -122,12 +122,12 @@ struct split_commit_graph_opts {\n */\n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t\t\t const struct commit_graph_opts *opts);\n int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t const struct commit_graph_opts *opts);\n \n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n \n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402735","messageId":"8b670571dc24973cc5e894b866a68236d8fbfa4f.1596480582.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH 10/10] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:57:44Z","receivedAt":"2020-08-03T18:57:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a command-line flag and configuration variable to fill in the\n'max_new_filters' variable introduced by the previous patch.\n\nThe command-line option '--max-new-filters' takes precedence over\n'graph.maxNewFilters', which is the default value.\n'--no-max-new-filters' can also be provided, which sets the value back\nto '-1', indicating that an unlimited number of new Bloom filters may be\ngenerated. (OPT_INTEGER only allows setting the '--no-' variant back to\n'0', hence a custom callback was used instead).\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 4 +++\n Documentation/git-commit-graph.txt | 4 +++\n bloom.c | 15 +++++++++++\n builtin/commit-graph.c | 39 +++++++++++++++++++++++++---\n commit-graph.c | 22 +++++++++++-----\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 19 ++++++++++++++\n 7 files changed, 95 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex bb78e72f1b..46f48d1aa8 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitgraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitgraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..e8034ff0ac 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,10 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Overrides the `graph.maxNewFilters` configuration.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex a8a21762f4..1742beac9e 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -51,6 +51,21 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \telse\n \t\tstart_index = 0;\n \n+\tif ((start_index == end_index) &&\n+\t (g->bloom_large.word_alloc && !bitmap_get(&g->bloom_large, lex_pos))) {\n+\t\t/*\n+\t\t * If the filter is zero-length, either (1) the filter has no\n+\t\t * changes, (2) the filter has too many changes, or (3) it\n+\t\t * wasn't computed (eg., due to '--max-new-filters').\n+\t\t *\n+\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n+\t\t * for this Bloom filter. If it is unset, then it wasn't\n+\t\t * computed. In that case, return nothing, since we don't have\n+\t\t * that filter in the graph.\n+\t\t */\n+\t\treturn 0;\n+\t}\n+\n \tfilter->len = end_index - start_index;\n \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n \t\t\t\t\tsizeof(unsigned char) * start_index +\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 38f5f57d15..aaf54a76db 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n \n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n \n@@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -270,6 +293,16 @@ static int graph_write(int argc, const char **argv)\n \treturn result;\n }\n \n+static int git_commit_graph_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, \"graph.maxnewfilters\")) {\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_commit_graph_options[] = {\n@@ -283,7 +316,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_commit_graph_usage,\n \t\t\t\t builtin_commit_graph_options);\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(git_commit_graph_config, &opts);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_commit_graph_options,\n \t\t\t builtin_commit_graph_usage,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8626024faa..87ea0437e2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1455,6 +1455,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tint *sorted_commits;\n+\tint max_new_filters;\n \n \tinit_bloom_filters();\n \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD);\n@@ -1471,6 +1472,9 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n \t\t&ctx->commits);\n \n+\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n@@ -1478,13 +1482,19 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tbitmap_set(ctx->bloom_large, pos);\n \t\t\tctx->count_bloom_filter_known_large++;\n \t\t} else {\n-\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(ctx->r, c, 1, NULL);\n-\t\t\tif (!filter->len) {\n-\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n-\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t\tint computed = 0;\n+\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\t\tctx->r, c,\n+\t\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n+\t\t\t\t&computed);\n+\t\t\tif (computed) {\n+\t\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\t\tif (filter && !filter->len) {\n+\t\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t\t\t}\n \t\t\t}\n-\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n-\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * (filter ? filter->len : 0);\n \t\t}\n \t\tdisplay_progress(progress, i + 1);\n \t}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 4cd991cf26..0842d13744 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -112,6 +112,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags flags;\n+\tint max_new_filters;\n };\n \n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 231dc8a3a7..985bea23b2 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -214,4 +214,23 @@ test_expect_success 'Bloom generation does not recompute too-large filters' '\n \t)\n '\n \n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd 513changes &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n+\t\t\t1 0 2\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t(\n+\t\tcd 513changes &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=1\" \\\n+\t\t\t1 0 1\n+\t)\n+'\n+\n test_done\n-- \n2.28.0.rc1.13.ge78abce653\n"},{"id":"402737","messageId":"20200803185947.GA67482@syl.lan","threadId":"53975","inReplyTo":"ab789072331127079347742b6ce3b3415bbbe4f6.1596480582.git.me@ttaylorr.com","subject":"Re: [PATCH 07/10] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-03T18:59:47Z","receivedAt":"2020-08-03T18:59:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Aug 03, 2020 at 02:57:33PM -0400, Taylor Blau wrote:\n> When a commit has 512 changed paths or greater, the commit-graph\n> machinery represents it as a zero-length filter since having many\n> entries in the Bloom filter has undesirable effects on the false\n> positivity rate.\n\nOne thing that occurs to me after sending this patch is that 512 may not\nbe the upper-bound on the number of changed-paths forever.\n\nMaybe we should store a uint32_t as the first four bytes of this chunk\nspecifying where we draw the line?\n\nThanks,\nTaylor\n"},{"id":"402797","messageId":"20200804072417.GA183951@coredump.intra.peff.net","threadId":"53975","inReplyTo":"08479793c1274d5ee0f063578bb0f4d93c910fa9.1596480582.git.me@ttaylorr.com","subject":"Re: [PATCH 01/10] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-08-04T07:24:17Z","receivedAt":"2020-08-04T07:24:21Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Aug 03, 2020 at 02:57:09PM -0400, Taylor Blau wrote:\n\n> diff --git a/revision.c b/revision.c\n> index 6de29cdf7a..e244beed05 100644\n> --- a/revision.c\n> +++ b/revision.c\n> @@ -684,7 +684,7 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> \tif (!revs->repo->objects->commit_graph)\n> \t\treturn;\n> \n> -\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n> +\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n> \tif (!revs->bloom_filter_settings)\n> \t\treturn;\n\nI think you could get rid of the revs->repo->objects->commit_graph check\nabove now, as get_bloom_filter_settings() would return NULL if there is\nno commit_graph at all.\n\n-Peff\n"},{"id":"402806","messageId":"8bf3b41e-a597-3691-66e0-2be2f5f7e15e@gmail.com","threadId":"53975","inReplyTo":"b31c60d71266c40ae23a619d5ac5fd99148e1649.1596480582.git.me@ttaylorr.com","subject":"Re: [PATCH 06/10] commit-graph.c: sort index into commits list","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-04T12:31:53Z","receivedAt":"2020-08-04T12:32:54Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/3/2020 2:57 PM, Taylor Blau wrote:\n> For locality, 'compute_bloom_filters()' sorts the commits for which it\n> wants to compute Bloom filters in a preferred order (cf., 3d11275505\n> (commit-graph: examine commits by generation number, 2020-03-30) for\n> details).\n> \n> The subsequent patch will want to recover the new graph position of each\n> commit. Since the 'packed_commit_list' already stores a double-pointer,\n> avoid a 'COPY_ARRAY' and instead keep track of an index into the\n> original list. (Use an integer index instead of a memory address, since\n> this involves a needlessly confusing triple-pointer).\n\nIt took me a little while to grok that we are switching from sorting\na list of commit pointers to sorting a list of integers. However, that\nmakes a lot of sense. It preserves the commit list sorted by OID for\nbinary search, which you will need soon. Perhaps another change would\nneed that at another time, too.\n\n> Alter the two sorting routines 'commit_pos_cmp' and 'commit_gen_cmp' to\n> take into account the packed_commit_list they are sorting with respect\n> to. Since 'compute_bloom_filters()' is the only caller for each of those\n> comparison functions, no other call-sites need updating.\n\nParsing the changes to these functions is the most complicated, because\nof the int-to-commit indirection. I think they are correct and as easy\nto read as possible.\n\n-Stolee\n"},{"id":"402807","messageId":"8483cbba-4a4b-030d-0ef2-0ff265470d31@gmail.com","threadId":"53975","inReplyTo":"ab789072331127079347742b6ce3b3415bbbe4f6.1596480582.git.me@ttaylorr.com","subject":"Re: [PATCH 07/10] commit-graph: add large-filters bitmap chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-04T12:57:06Z","receivedAt":"2020-08-04T12:57:12Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/3/2020 2:57 PM, Taylor Blau wrote:\n> When a commit has 512 changed paths or greater, the commit-graph\n> machinery represents it as a zero-length filter since having many\n> entries in the Bloom filter has undesirable effects on the false\n> positivity rate.\n\nYou mentioned in another message that you'd like to encode the magic\n\"512\" constant into this chunk. I think that is a good idea. Perhaps\nit is worth describing it as a MAX_CHANGED_PATHS variable or something.\n\nI don't think now is the time to expose it as a command-line argument,\nbut encoding it in the file format is a good idea. This is similar to\nthe start of the BDAT chunk starting with three values from the\nbloom_filter_settings, which are not actually customizable from arguments.\n\nFor testing, I would recommend using a GIT_TEST_* variable, just like\nGIT_TEST_BLOOM_SETTINGS_NUM_HASHES and GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\nare used in t4216-log-bloom.sh to test those BDAT chunk values.\n\n> In addition to these too-large filters, the commit-graph machinery also\n> represents commits with no filter and commits with no changed paths in\n> the same way.\n> \n> When writing a commit-graph that aggregates several incremental\n> commit-graph layers (eg., with '--split=replace'), the commit-graph\n> machinery first computes all of the Bloom filters that it wants to write\n> but does not already know about from existing graph layers. Because we\n> overload the zero-length filter in the above fashion, this leads to\n> recomputing large filters over and over again.\n> \n> This is already undesirable, since it means that we are wasting\n> considerable effort to discover that a commit has at least 512 changed\n> paths, only to throw that effort away (and then repeat the process the\n> next time a roll-up is performed).\n> \n> In a subsequent patch, we will add a '--max-new-filters' option, which\n> specifies an upper-bound on the number of new filters we are willing to\n> compute from scratch. Suppose that there are 'N' too-large filters, and\n> we specify '--max-new-filters=M'. If 'N >= M', it is unlikely that any\n> filters will be generated, since we'll spend most of our effort on\n> filters that we ultimately throw away. If 'N < M', filters will trickle\n> in over time, but only at most 'M - N' per-write.\n> \n> To address this, add a new chunk which encodes a bitmap where the ith\n> bit is on iff the ith commit has zero or at least 512 changed paths.\n> When computing Bloom filters, first consult the relevant bitmap (in the\n> case that we are rolling up existing layers) to see if computing the\n> Bloom filter from scratch would be a waste of time.\n> \n> This patch implements a new chunk instead of extending the existing BIDX\n> and BDAT chunks because modifying these chunks would confuse old\n> clients. For example, consider setting the most-significant bit in an\n> entry in the BIDX chunk to indicate that that filter is too-large. New\n> clients would understand how to interpret this, but old clients would\n> treat that entry as a really large filter.\n> \n> By avoiding the need to move to new versions of the BDAT and BIDX chunk,\n> we can give ourselves more time to consider whether or not other\n> modifications to these chunks are worthwhile without holding up this\n> change.\n> \n> Another approach would be to introduce a new BIDX chunk (say, one\n> identified by 'BID2') which is identical to the existing BIDX chunk,\n> except the most-significant bit of each offset is interpreted as \"this\n> filter is too big\" iff looking at a BID2 chunk. This avoids having to\n> write a bitmap, but forces older clients to rewrite their commit-graphs\n> (as well as reduces the theoretical largest Bloom filters we couldl\n> write, and forces us to maintain the code necessary to translate BIDX\n> chunks to BID2 ones). Separately from this patch, I implemented this\n> alternate approach and did not find it to be advantageous.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> .../technical/commit-graph-format.txt | 9 ++\n> commit-graph.c | 85 ++++++++++++++++++-\n> commit-graph.h | 2 +\n> t/t4216-log-bloom.sh | 20 ++++-\n> 4 files changed, 112 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n> index 440541045d..a7191c03d3 100644\n> --- a/Documentation/technical/commit-graph-format.txt\n> +++ b/Documentation/technical/commit-graph-format.txt\n> @@ -123,6 +123,15 @@ CHUNK DATA:\n> of length zero.\n> * The BDAT chunk is present if and only if BIDX is present.\n> \n> + Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n> + * It contains a list of 'eword_t's (the length of this list is determined by\n\nI'm not sure using 'eword_t' in this documentation is helpful. How large is that\ntype? Can we make that concrete in terms of 32- or 64-bit words? Below, you\ndescribe that this is an uncompressed bitmap whose 'i'th bit corresponds to the\n'i'th commit.\n\n> + the width of the chunk) which is a bitmap. The 'i'th bit is set exactly\n> + when the 'i'th commit in the graph has a changed-path Bloom filter with\n> + zero entries (either because the commit is empty, or because it contains\n> + more than 512 changed paths).\nAs I expect you will update this in v2, try to create a name for the \"512\" constant\nto re-use here.\n\n> + * The BFXL chunk is present on newer versions of Git iff the BIDX and BDAT\n> + chunks are also present.\n\nExpand \"iff\" to \"if and only if\". Perhaps this would be simpler to have only one\ndirection:\n\n\t* The BFXL chunk is present only when the BIDX and BDAT chunks are\n\t also present.\n\n> +\n> Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n> This list of H-byte hashes describe a set of B commit-graph files that\n> form a commit-graph chain. The graph position for the ith commit in this\n> diff --git a/commit-graph.c b/commit-graph.c\n> index d6ea556649..c870ffe0ed 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -41,8 +41,9 @@ void git_test_write_commit_graph_or_die(void)\n> #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n> #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n> #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n> +#define GRAPH_CHUNKID_BLOOMLARGE 0x4246584c /* \"BFXL\" */\n> #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n> -#define MAX_NUM_CHUNKS 7\n> +#define MAX_NUM_CHUNKS 8\n> \n> #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n> \n> @@ -429,6 +430,17 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n> \t\t\t}\n> \t\t\tbreak;\n> +\n> +\t\tcase GRAPH_CHUNKID_BLOOMLARGE:\n> +\t\t\tif (graph->bloom_large.word_alloc)\n> +\t\t\t\tchunk_repeated = 1;\n> +\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n> +\t\t\t\tuint32_t alloc = get_be64(chunk_lookup + 4) - chunk_offset;\n> +\n> +\t\t\t\tgraph->bloom_large.word_alloc = alloc;\n> +\t\t\t\tgraph->bloom_large.words = (eword_t *)(data + chunk_offset);\n> +\t\t\t}\n> +\t\t\tbreak;\n> \t\t}\n> \n> \t\tif (chunk_repeated) {\n> @@ -443,6 +455,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> \t\t/* We need both the bloom chunks to exist together. Else ignore the data */\n> \t\tgraph->chunk_bloom_indexes = NULL;\n> \t\tgraph->chunk_bloom_data = NULL;\n> +\t\tgraph->bloom_large.words = NULL;\n> +\t\tgraph->bloom_large.word_alloc = 0;\n> \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n> \t}\n> \n> @@ -933,6 +947,21 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n> }\n> \n> +static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> +\t\t\t\t\t const struct commit *c)\n> +{\n> +\tuint32_t graph_pos = commit_graph_position(c);\n> +\tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n> +\t\treturn 0;\n> +\n> +\twhile (g && graph_pos < g->num_commits_in_base)\n> +\t\tg = g->base_graph;\n> +\n> +\tif (!(g && g->bloom_large.word_alloc))\n> +\t\treturn 0;\n> +\treturn bitmap_get(&g->bloom_large, graph_pos - g->num_commits_in_base);\n> +}\n> +\n> struct packed_oid_list {\n> \tstruct object_id *list;\n> \tint nr;\n> @@ -969,6 +998,11 @@ struct write_commit_graph_context {\n> \tconst struct split_commit_graph_opts *split_opts;\n> \tsize_t total_bloom_filter_data_size;\n> \tconst struct bloom_filter_settings *bloom_settings;\n> +\tstruct bitmap *bloom_large;\n> +\n> +\tint count_bloom_filter_known_large;\n> +\tint count_bloom_filter_found_large;\n> +\tint count_bloom_filter_computed;\n> };\n> \n> static int write_graph_chunk_fanout(struct hashfile *f,\n> @@ -1231,6 +1265,15 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n> \treturn 0;\n> }\n> \n> +static int write_graph_chunk_bloom_large(struct hashfile *f,\n> +\t\t\t\t\t struct write_commit_graph_context *ctx)\n> +{\n> +\ttrace2_region_enter(\"commit-graph\", \"bloom_large\", ctx->r);\n> +\thashwrite(f, ctx->bloom_large->words, ctx->bloom_large->word_alloc * sizeof(eword_t));\n\nShould we be using word_alloc here? Or can we trust that the bitmap\nwas allocated to the exact right size to fit our number of commits?\n\nWe probably need to be careful about network-byte order here, right?\nOr, should we treat the bitmap as an array of bytes?\n\nI see that eword_t is a 64-bit type, which makes things a _little_\ntricky. We normally would use hashwrite_be32() for writing 32-bit\nwords. Likely, we will want a new hashwrite_be64() that is a wrapper\naround hashwrite_be32(). Notice that this could be used when writing\nthe chunk offsets in commit-graph.c and midx.c. For example, we\nwrite the 64-bit offsets in write_commit_graph_file() as follows:\n\n\tfor (i = 0; i <= num_chunks; i++) {\n\t\tuint32_t chunk_write[3];\n\n\t\tchunk_write[0] = htonl(chunk_ids[i]);\n\t\tchunk_write[1] = htonl(chunk_offsets[i] >> 32);\n\t\tchunk_write[2] = htonl(chunk_offsets[i] & 0xffffffff);\n\t\thashwrite(f, chunk_write, 12);\n\t}\n\nThis would be nicer as\n\n\tfor (i = 0; i <= num_chunks; i++) {\n\t\thashwrite_be32(f, chunk_ids[i]);\n\t\thashwrite_be64(f, chunk_offsets[i]);\n\t}\n\nI know this adds complexity to an already meaty series. However, this\nshould help us avoid this network-byte order issue in the future while\ncleaning up existing code.\n\n> +\ttrace2_region_leave(\"commit-graph\", \"bloom_large\", ctx->r);\n> +\treturn 0;\n> +}\n> +\n> static int oid_compare(const void *_a, const void *_b)\n> {\n> \tconst struct object_id *a = (const struct object_id *)_a;\n> @@ -1389,6 +1432,24 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n> \tstop_progress(&ctx->progress);\n> }\n> \n> +static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n> +{\n> +\tstruct json_writer jw = JSON_WRITER_INIT;\n> +\n> +\tjw_object_begin(&jw, 0);\n> +\tjw_object_intmax(&jw, \"filter_known_large\",\n> +\t\t\t ctx->count_bloom_filter_known_large);\n> +\tjw_object_intmax(&jw, \"filter_found_large\",\n> +\t\t\t ctx->count_bloom_filter_found_large);\n> +\tjw_object_intmax(&jw, \"filter_computed\",\n> +\t\t\t ctx->count_bloom_filter_computed);\n> +\tjw_end(&jw);\n\nHelpful stats.\n\n> +\n> +\ttrace2_data_json(\"commit-graph\", the_repository, \"bloom_statistics\", &jw);\n> +\n> +\tjw_release(&jw);\n> +}\n> +\n> static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> {\n> \tint i;\n> @@ -1396,6 +1457,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \tint *sorted_commits;\n> \n> \tinit_bloom_filters();\n> +\tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD);\n> \n> \tif (ctx->report_progress)\n> \t\tprogress = start_delayed_progress(\n> @@ -1412,11 +1474,24 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \tfor (i = 0; i < ctx->commits.nr; i++) {\n> \t\tint pos = sorted_commits[i];\n> \t\tstruct commit *c = ctx->commits.list[pos];\n> -\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n> -\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n> +\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n> +\t\t\tbitmap_set(ctx->bloom_large, pos);\n> +\t\t\tctx->count_bloom_filter_known_large++;\n> +\t\t} else {\n> +\t\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n> +\t\t\tif (!filter->len) {\n> +\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n> +\t\t\t\tctx->count_bloom_filter_found_large++;\n> +\t\t\t}\n> +\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n\nIs this sizeof(unsigned char) necessary? Or just a helpful reminder that\nfilter->len is a number of bytes?\n\n> +\t\t\tctx->count_bloom_filter_computed++;\n> +\t\t}\n> \t\tdisplay_progress(progress, i + 1);\n> \t}\n> \n> +\tif (trace2_is_enabled())\n> +\t\ttrace2_bloom_filter_write_statistics(ctx);\n> +\n> \tfree(sorted_commits);\n> \tstop_progress(&progress);\n> }\n> @@ -1739,6 +1814,10 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n> \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n> \t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n> \t\tnum_chunks++;\n> +\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMLARGE;\n> +\t\tchunks[num_chunks].size = sizeof(eword_t) * ctx->bloom_large->word_alloc;\n> +\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_large;\n> +\t\tnum_chunks++;\n> \t}\n> \tif (ctx->num_commit_graphs_after > 1) {\n> \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n> diff --git a/commit-graph.h b/commit-graph.h\n> index d9acb22bac..afbc5fa41e 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -4,6 +4,7 @@\n> #include \"git-compat-util.h\"\n> #include \"object-store.h\"\n> #include \"oidset.h\"\n> +#include \"ewah/ewok.h\"\n> \n> #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n> #define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n> @@ -71,6 +72,7 @@ struct commit_graph {\n> \tconst unsigned char *chunk_base_graphs;\n> \tconst unsigned char *chunk_bloom_indexes;\n> \tconst unsigned char *chunk_bloom_data;\n> +\tstruct bitmap bloom_large;\n> \n> \tstruct bloom_filter_settings *bloom_filter_settings;\n> };\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index b3d1f596f8..231dc8a3a7 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -33,7 +33,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n> \tgit commit-graph write --reachable --changed-paths\n> '\n> graph_read_expect () {\n> -\tNUM_CHUNKS=5\n> +\tNUM_CHUNKS=6\n> \tcat >expect <<- EOF\n> \theader: 43475048 1 1 $NUM_CHUNKS 0\n> \tnum_commits: $1\n> @@ -195,5 +195,23 @@ test_expect_success 'correctly report changes over limit' '\n> \t\tdone\n> \t)\n> '\n> +test_bloom_filters_computed () {\n> +\tcommit_graph_args=$1\n> +\tbloom_trace_prefix=\"{\\\"filter_known_large\\\":$2,\\\"filter_found_large\\\":$3,\\\"filter_computed\\\":$4\"\n> +\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n> +\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write $commit_graph_args &&\n> +\tgrep \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.event\"\n> +}\n\nCould this helper be moved earlier, so we can use\ntest_bloom_filters_computed in 'correctly report changes\nover limit'? We could then check that filter_known_large is\nzero while filter_found_large is 1.\n\nThese tests would be an excellent opportunity to use a\nGIT_TEST_* variable to lower the 512 limit to something\nsmaller to make the test run a tiny bit faster ;).\n\n> +\n> +test_expect_success 'Bloom generation does not recompute too-large filters' '\n> +\t(\n> +\t\tcd 513changes &&\n> +\t\tgit commit-graph write --reachable --changed-paths \\\n> +\t\t\t--split=replace &&\n> +\t\ttest_commit c1 filter &&\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n> +\t\t\t1 0 1\n> +\t)\n> +'\n\nAdding this test is valuable. Thanks.\n\n-Stolee\n\n"},{"id":"402808","messageId":"c54cc9a8-322c-6235-4170-228354c1ff33@gmail.com","threadId":"53975","inReplyTo":"a494094c10f9bddca5743973409ccb4540841116.1596480582.git.me@ttaylorr.com","subject":"Re: [PATCH 08/10] bloom: split 'get_bloom_filter()' in two","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-04T13:00:31Z","receivedAt":"2020-08-04T13:00:41Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/3/2020 2:57 PM, Taylor Blau wrote:\n> 'get_bloom_filter' takes a flag to control whether it will compute a\n> Bloom filter if the requested one is missing. In the next patch, we'll\n> add yet another flag to this method, which would force all but one\n> caller to specify an extra 'NULL' parameter at the end.\n> \n> Instead of doing this, split 'get_bloom_filter' into two functions:\n> 'get_bloom_filter' and 'get_or_compute_bloom_filter'. The former only\n> looks up a Bloom filter (and does not compute one if it's missing,\n> thus dropping the 'compute_if_not_present' flag). The latter does\n> compute missing Bloom filters, with an additional parameter to store\n> whether or not it needed to do so.\n> \n> This simplifies many call-sites, since the majority of existing callers\n> to 'get_bloom_filter' do not want missing Bloom filters to be computed\n> (so they can drop the parameter entirely and use the simpler version of\n> the function).\n\n> +struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n> +\t\t\t\t\t\t struct commit *c,\n> +\t\t\t\t\t\t int compute_if_not_present,\n> +\t\t\t\t\t\t int *computed)\n\nCould we further simplify this by letting \"computed\" be the indicator\nfor whether we should compute the filter? If \"computed\" is NULL, then\nwe won't compute it directly. This allows us to reduce the \"1, NULL)\"\nto \"NULL)\" in these callers:\n\n> +\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(ctx->r, c, 1, NULL);\n\n> +\tfilter = get_or_compute_bloom_filter(the_repository, c, 1, NULL);\n\nThanks,\n-Stolee\n"},{"id":"402809","messageId":"1c2e841b-4b04-30ba-6c94-96f8bf0cf449@gmail.com","threadId":"53975","inReplyTo":"8b670571dc24973cc5e894b866a68236d8fbfa4f.1596480582.git.me@ttaylorr.com","subject":"Re: [PATCH 10/10] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-04T13:03:36Z","receivedAt":"2020-08-04T13:03:51Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/3/2020 2:57 PM, Taylor Blau wrote:\n> Introduce a command-line flag and configuration variable to fill in the\n> 'max_new_filters' variable introduced by the previous patch.\n> \n> The command-line option '--max-new-filters' takes precedence over\n> 'graph.maxNewFilters', which is the default value.\n\ncommitGraph.maxNewFilters?\n\n> '--no-max-new-filters' can also be provided, which sets the value back\n> to '-1', indicating that an unlimited number of new Bloom filters may be\n> generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n> '0', hence a custom callback was used instead).\n\nOther than the commit message typo above, this is a good patch.\n\nThanks,\n-Stolee\n"},{"id":"402835","messageId":"20200804200849.GB89277@syl.lan","threadId":"53975","inReplyTo":"20200804072417.GA183951@coredump.intra.peff.net","subject":"Re: [PATCH 01/10] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-04T20:08:49Z","receivedAt":"2020-08-04T20:08:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 04, 2020 at 03:24:17AM -0400, Jeff King wrote:\n> On Mon, Aug 03, 2020 at 02:57:09PM -0400, Taylor Blau wrote:\n>\n> > diff --git a/revision.c b/revision.c\n> > index 6de29cdf7a..e244beed05 100644\n> > --- a/revision.c\n> > +++ b/revision.c\n> > @@ -684,7 +684,7 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n> > \tif (!revs->repo->objects->commit_graph)\n> > \t\treturn;\n> >\n> > -\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n> > +\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n> > \tif (!revs->bloom_filter_settings)\n> > \t\treturn;\n>\n> I think you could get rid of the revs->repo->objects->commit_graph check\n> above now, as get_bloom_filter_settings() would return NULL if there is\n> no commit_graph at all.\n\nYep, that's right -- and good catch. This used to not be the case (or if\nit was, relying on it was more fragile) when\n'get_bloom_filter_settings()' took a commit-graph instead of a\nrepository, but now it's no longer necessary.\n\nThanks.\n\n> -Peff\n\nTaylor\n"},{"id":"402836","messageId":"20200804201006.GC89277@syl.lan","threadId":"53975","inReplyTo":"8bf3b41e-a597-3691-66e0-2be2f5f7e15e@gmail.com","subject":"Re: [PATCH 06/10] commit-graph.c: sort index into commits list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-04T20:10:06Z","receivedAt":"2020-08-04T20:10:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 04, 2020 at 08:31:53AM -0400, Derrick Stolee wrote:\n> On 8/3/2020 2:57 PM, Taylor Blau wrote:\n> > For locality, 'compute_bloom_filters()' sorts the commits for which it\n> > wants to compute Bloom filters in a preferred order (cf., 3d11275505\n> > (commit-graph: examine commits by generation number, 2020-03-30) for\n> > details).\n> >\n> > The subsequent patch will want to recover the new graph position of each\n> > commit. Since the 'packed_commit_list' already stores a double-pointer,\n> > avoid a 'COPY_ARRAY' and instead keep track of an index into the\n> > original list. (Use an integer index instead of a memory address, since\n> > this involves a needlessly confusing triple-pointer).\n>\n> It took me a little while to grok that we are switching from sorting\n> a list of commit pointers to sorting a list of integers. However, that\n> makes a lot of sense. It preserves the commit list sorted by OID for\n> binary search, which you will need soon. Perhaps another change would\n> need that at another time, too.\n\nYeah. I had to spend some additional time with this patch (at least back\nwhen it was written in terms of 'struct commit ***'s) to convince myself\nof its correctness, too.\n\nI think that this is ultimately the right thing, and that it is probably\nas simple as I can make it without refactoring the packed_commit_list,\nwhich I think is squarely outside the scope of this (already-large)\nseries ;).\n\n> > Alter the two sorting routines 'commit_pos_cmp' and 'commit_gen_cmp' to\n> > take into account the packed_commit_list they are sorting with respect\n> > to. Since 'compute_bloom_filters()' is the only caller for each of those\n> > comparison functions, no other call-sites need updating.\n>\n> Parsing the changes to these functions is the most complicated, because\n> of the int-to-commit indirection. I think they are correct and as easy\n> to read as possible.\n>\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"402837","messageId":"20200804201247.GD89277@syl.lan","threadId":"53975","inReplyTo":"c54cc9a8-322c-6235-4170-228354c1ff33@gmail.com","subject":"Re: [PATCH 08/10] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-04T20:12:47Z","receivedAt":"2020-08-04T20:12:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 04, 2020 at 09:00:31AM -0400, Derrick Stolee wrote:\n> On 8/3/2020 2:57 PM, Taylor Blau wrote:\n> > 'get_bloom_filter' takes a flag to control whether it will compute a\n> > Bloom filter if the requested one is missing. In the next patch, we'll\n> > add yet another flag to this method, which would force all but one\n> > caller to specify an extra 'NULL' parameter at the end.\n> >\n> > Instead of doing this, split 'get_bloom_filter' into two functions:\n> > 'get_bloom_filter' and 'get_or_compute_bloom_filter'. The former only\n> > looks up a Bloom filter (and does not compute one if it's missing,\n> > thus dropping the 'compute_if_not_present' flag). The latter does\n> > compute missing Bloom filters, with an additional parameter to store\n> > whether or not it needed to do so.\n> >\n> > This simplifies many call-sites, since the majority of existing callers\n> > to 'get_bloom_filter' do not want missing Bloom filters to be computed\n> > (so they can drop the parameter entirely and use the simpler version of\n> > the function).\n>\n> > +struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n> > +\t\t\t\t\t\t struct commit *c,\n> > +\t\t\t\t\t\t int compute_if_not_present,\n> > +\t\t\t\t\t\t int *computed)\n>\n> Could we further simplify this by letting \"computed\" be the indicator\n> for whether we should compute the filter? If \"computed\" is NULL, then\n> we won't compute it directly. This allows us to reduce the \"1, NULL)\"\n> to \"NULL)\" in these callers:\n\nI like what you're getting at--that is, that we shouldn't make calling\nthis more complicated than necessary, and right now lots of callers\nalways pass \"1, NULL)\" as the last two arguments--but I'm not sure that\nI like this suggestion.\n\nI could imagine a future caller would want to compute the Bloom filters\nif missing, but not care about whether or not they were computed from\nscratch. In that case, they'd need a dummy variable. Not the worst thing\nin the world, but I think it's less clear.\n\nBy the way, I think that this suggestion only helps for \"0, NULL\" into\njust \"NULL\", not \"1, NULL\" (which requires a dummy variable with your\nsuggestion).\n\n>\n> > +\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(ctx->r, c, 1, NULL);\n>\n> > +\tfilter = get_or_compute_bloom_filter(the_repository, c, 1, NULL);\n>\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"402838","messageId":"20200804201402.GE89277@syl.lan","threadId":"53975","inReplyTo":"1c2e841b-4b04-30ba-6c94-96f8bf0cf449@gmail.com","subject":"Re: [PATCH 10/10] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-04T20:14:02Z","receivedAt":"2020-08-04T20:14:09Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 04, 2020 at 09:03:36AM -0400, Derrick Stolee wrote:\n> On 8/3/2020 2:57 PM, Taylor Blau wrote:\n> > Introduce a command-line flag and configuration variable to fill in the\n> > 'max_new_filters' variable introduced by the previous patch.\n> >\n> > The command-line option '--max-new-filters' takes precedence over\n> > 'graph.maxNewFilters', which is the default value.\n>\n> commitGraph.maxNewFilters?\n\nNice catch. There are some other places that I made this typo, but I\nfixed those up locally, too.\n\n> > '--no-max-new-filters' can also be provided, which sets the value back\n> > to '-1', indicating that an unlimited number of new Bloom filters may be\n> > generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n> > '0', hence a custom callback was used instead).\n>\n> Other than the commit message typo above, this is a good patch.\n\nThanks, and thanks for the good suggestion to write it in the first\nplace ;).\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"402913","messageId":"001f3385ffd96f24a640ff6670166e67784f1f50.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:01:37Z","receivedAt":"2020-08-05T17:03:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Many places in the code often need a pointer to the commit-graph's\n'struct bloom_filter_settings', in which case they often take the value\nfrom the top-most commit-graph.\n\nIn the non-split case, this works as expected. In the split case,\nhowever, things get a little tricky. Not all layers in a chain of\nincremental commit-graphs are required to themselves have Bloom data,\nand so whether or not some part of the code uses Bloom filters depends\nentirely on whether or not the top-most level of the commit-graph chain\nhas Bloom filters.\n\nThis has been the behavior since Bloom filters were introduced, and has\nbeen codified into the tests since a759bfa9ee (t4216: add end to end\ntests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\nrequires that Bloom filters are not used in exactly the case described\nearlier.\n\nThere is no reason that this needs to be the case, since it is perfectly\nvalid for commits in an earlier layer to have Bloom filters when commits\nin a newer layer do not.\n\nSince Bloom settings are guaranteed to be the same for any layer in a\nchain that has Bloom data, it is sufficient to traverse the\n'->base_graph' pointer until either (1) a non-null 'struct\nbloom_filter_settings *' is found, or (2) until we are at the root of\nthe commit-graph chain.\n\nIntroduce a 'get_bloom_filter_settings()' function that does just this,\nand use it instead of purely dereferencing the top-most graph's\n'->bloom_filter_settings' pointer.\n\nWhile we're at it, add an additional test in t5324 to guard against code\nin the commit-graph writing machinery that doesn't correctly handle a\nNULL 'struct bloom_filter *'.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n blame.c | 6 ++++--\n bloom.c | 6 +++---\n commit-graph.c | 11 +++++++++++\n commit-graph.h | 2 ++\n revision.c | 5 +----\n t/t4216-log-bloom.sh | 9 ++++++---\n t/t5324-split-commit-graph.sh | 13 +++++++++++++\n 7 files changed, 40 insertions(+), 12 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 82fa16d658..3e5f8787bc 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -2891,16 +2891,18 @@ void setup_blame_bloom_data(struct blame_scoreboard *sb,\n \t\t\t const char *path)\n {\n \tstruct blame_bloom_data *bd;\n+\tstruct bloom_filter_settings *bs;\n \n \tif (!sb->repo->objects->commit_graph)\n \t\treturn;\n \n-\tif (!sb->repo->objects->commit_graph->bloom_filter_settings)\n+\tbs = get_bloom_filter_settings(sb->repo);\n+\tif (!bs)\n \t\treturn;\n \n \tbd = xmalloc(sizeof(struct blame_bloom_data));\n \n-\tbd->settings = sb->repo->objects->commit_graph->bloom_filter_settings;\n+\tbd->settings = bs;\n \n \tbd->alloc = 4;\n \tbd->nr = 0;\ndiff --git a/bloom.c b/bloom.c\nindex 1a573226e7..cd9380ac62 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -38,7 +38,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \twhile (graph_pos < g->num_commits_in_base)\n \t\tg = g->base_graph;\n \n-\t/* The commit graph commit 'c' lives in doesn't carry bloom filters. */\n+\t/* The commit graph commit 'c' lives in doesn't carry Bloom filters. */\n \tif (!g->chunk_bloom_indexes)\n \t\treturn 0;\n \n@@ -195,8 +195,8 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes)\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n+\t\t\t\treturn filter;\n \t}\n \n \tif (filter->data)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e51c91dd5b..d4b06811be 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -660,6 +660,17 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r)\n+{\n+\tstruct commit_graph *g = r->objects->commit_graph;\n+\twhile (g) {\n+\t\tif (g->bloom_filter_settings)\n+\t\t\treturn g->bloom_filter_settings;\n+\t\tg = g->base_graph;\n+\t}\n+\treturn NULL;\n+}\n+\n static void close_commit_graph_one(struct commit_graph *g)\n {\n \tif (!g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 09a97030dc..0677dd1031 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -87,6 +87,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n */\n int generation_numbers_enabled(struct repository *r);\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r);\n+\n enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_APPEND = (1 << 0),\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\ndiff --git a/revision.c b/revision.c\nindex 6de29cdf7a..c45ed1076e 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -681,10 +681,7 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \trepo_parse_commit(revs->repo, revs->commits->item);\n \n-\tif (!revs->repo->objects->commit_graph)\n-\t\treturn;\n-\n-\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n+\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n \tif (!revs->bloom_filter_settings)\n \t\treturn;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c21cc160f3..c9f9bdf1ba 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -60,7 +60,7 @@ setup () {\n \n test_bloom_filters_used () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":${2:-0},\\\"maybe\\\"\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom &&\n@@ -134,8 +134,11 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n-test_expect_success 'Do not use Bloom filters if the latest graph does not have Bloom filters.' '\n-\ttest_bloom_filters_not_used \"-- A/B\"\n+test_expect_success 'use Bloom filters even if the latest graph does not have Bloom filters' '\n+\t# Ensure that the number of empty filters is equal to the number of\n+\t# filters in the latest graph layer to prove that they are loaded (and\n+\t# ignored).\n+\ttest_bloom_filters_used \"-- A/B\" 3\n '\n \n test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 9b850ea907..5bdfd53ef9 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -425,4 +425,17 @@ done <<\\EOF\n 0600 -r--------\n EOF\n \n+test_expect_success '--split=replace with partial Bloom data' '\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/3 &&\n+\tgit rev-list -1 HEAD~2 >a &&\n+\tgit rev-list -1 HEAD~1 >b &&\n+\tgit commit-graph write --split=no-merge --stdin-commits --changed-paths graph-files &&\n+\ttest_line_count = 1 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n test_done\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402914","messageId":"e4d068a4784214d15fa2c5d284704c85050e34b8.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 02/14] t4216: use an '&&'-chain","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:20Z","receivedAt":"2020-08-05T17:03:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a759bfa9ee (t4216: add end to end tests for git log with Bloom\nfilters, 2020-04-06), a 'rm' invocation was added without a\ncorresponding '&&' chain.\n\nWhen 'trace.perf' already exists, everything works fine. However, the\nfunction can be executed without 'trace.perf' on disk (eg., when the\nsubset of tests run is altered with '--run'), and so the bare 'rm'\ncomplains about a missing file.\n\nTo remove some noise from the test log, invoke 'rm' with '-f', at which\npoint it is sensible to place the 'rm -f' in an '&&'-chain, which is\nboth (1) our usual style, and (2) avoids a broken chain in the future if\nmore commands are added at the beginning of the function.\n\nHelped-by: Eric Sunshine \nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c9f9bdf1ba..fe19f6a60c 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -53,7 +53,7 @@ sane_unset GIT_TRACE2_PERF_BRIEF\n sane_unset GIT_TRACE2_CONFIG_PARAMS\n \n setup () {\n-\trm \"$TRASH_DIRECTORY/trace.perf\"\n+\trm -f \"$TRASH_DIRECTORY/trace.perf\" &&\n \tgit -c core.commitGraph=false log --pretty=\"format:%s\" $1 >log_wo_bloom &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.perf\" git -c core.commitGraph=true log --pretty=\"format:%s\" $1 >log_w_bloom\n }\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402915","messageId":"d1c4bbcaa9627068d5d9fbd0e4a2e8c8834a4bd3.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 09/14] bloom/diff: properly short-circuit on max_changes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:47Z","receivedAt":"2020-08-05T17:05:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nCommit e3696980 (diff: halt tree-diff early after max_changes,\n2020-03-30) intended to create a mechanism to short-circuit a diff\ncalculation after a certain number of paths were modified. By\nincrementing a \"num_changes\" counter throughout the recursive\nll_diff_tree_paths(), this was supposed to match the number of changes\nthat would be written into the changed-path Bloom filters.\nUnfortunately, this was not implemented correctly and instead misses\nsimple cases like file modifications. This then does not stop very\nlarge changed-path filters from being written (unless they add or remove\nmany files).\n\nTo start, change the implementation in ll_diff_tree_paths() to instead\nuse the global diff_queue_diff struct's 'nr' member as the count. This\nis a way to simplify the logic instead of making more mistakes in the\ncomplicated diff code.\n\nThis has a drawback: the diff_queue_diff struct only lists the paths\ncorresponding to blob changes, not their leading directories. Thus,\nget_or_compute_bloom_filter() needs an additional check to see if the\nhashmap with the leading directories becomes too large.\n\nOne reason why this was not caught by test cases was that the test in\nt4216-log-bloom.sh that was supposed to check this \"too many changes\"\ncondition only checked this on the initial commit of a repository. The\nold logic counted these values correctly. Update this test in a few\nways:\n\n1. Use GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS to reduce the limit,\n allowing smaller commits to engage with this logic.\n\n2. Create several interesting cases of edits, adds, removes, and mode\n changes (in the second commit). By testing both sides of the\n inequality with the *_MAX_CHANGED_PATHS variable, we can see that\n the count is exactly correct, so none of these changes are missed\n or over-counted.\n\n3. Use the trace2 data value filter_found_large to verify that these\n commits are on the correct side of the limit.\n\nAnother way to verify the behavior is correct is through performance\ntests. By testing on my local copies of the Git repository and the Linux\nkernel repository, I could measure the effect of these short-circuits\nwhen computing a fresh commit-graph file with changed-path Bloom filters\nusing the command\n\n GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=N time \\\n git commit-graph write --reachable --changed-paths\n\nand reporting the wall time and resulting commit-graph size.\n\nFor Git, the results are\n\n| | N=1 | N=10 | N=512 |\n|--------|----------------|----------------|----------------|\n| HEAD~1 | 10.90s 9.18MB | 11.11s 9.34MB | 11.31s 9.35MB |\n| HEAD | 9.21s 8.62MB | 11.11s 9.29MB | 11.29s 9.34MB |\n\nFor Linux, the results are\n\n| | N=1 | N=20 | N=512 |\n|--------|----------------|---------------|---------------|\n| HEAD~1 | 61.28s 64.3MB | 76.9s 72.6MB | 77.6s 72.6MB |\n| HEAD | 49.44s 56.3MB | 68.7s 65.9MB | 69.2s 65.9MB |\n\nNaturally, the improvement becomes much less as the limit grows, as\nfewer commits satisfy the short-circuit.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n bloom.c | 6 +++-\n diff.h | 2 --\n t/t4216-log-bloom.sh | 85 +++++++++++++++++++++++++++++++++++++++-----\n tree-diff.c | 5 +--\n 4 files changed, 82 insertions(+), 16 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 0cf1962dc5..ed54e96e57 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -222,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= settings->max_changed_paths) {\n+\tif (diff_queued_diff.nr <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -259,6 +259,9 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n+\t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths)\n+\t\t\tgoto cleanup;\n+\n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n@@ -268,6 +271,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n+\tcleanup:\n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\ndiff --git a/diff.h b/diff.h\nindex e0c0af6286..1d32b71885 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -287,8 +287,6 @@ struct diff_options {\n \n \t/* If non-zero, then stop computing after this many changes. */\n \tint max_changes;\n-\t/* For internal use only. */\n-\tint num_changes;\n \n \tint ita_invisible_in_index;\n /* white-space error highlighting */\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex eb2bcc51f0..21b67677ef 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -177,20 +177,87 @@ test_expect_success 'persist filter settings' '\n '\n \n test_expect_success 'correctly report changes over limit' '\n-\tgit init 513changes &&\n+\tgit init limits &&\n \t(\n-\t\tcd 513changes &&\n-\t\tfor i in $(test_seq 1 513)\n+\t\tcd limits &&\n+\t\tmkdir d &&\n+\t\tmkdir d/e &&\n+\n+\t\tfor i in $(test_seq 1 2)\n \t\tdo\n-\t\t\techo $i >file$i.txt || return 1\n+\t\t\tprintf $i >d/file$i.txt &&\n+\t\t\tprintf $i >d/e/file$i.txt || return 1\n \t\tdone &&\n-\t\tgit add . &&\n+\n+\t\tmkdir mode &&\n+\t\tprintf bash >mode/script.sh &&\n+\n+\t\tmkdir foo &&\n+\t\ttouch foo/bar &&\n+\t\ttouch foo.txt &&\n+\n+\t\tgit add d foo foo.txt mode &&\n \t\tgit commit -m \"files\" &&\n-\t\tgit commit-graph write --reachable --changed-paths &&\n-\t\tfor i in $(test_seq 1 513)\n+\n+\t\t# Commit has 7 file and 4 directory adds\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":10\" trace &&\n+\t\tgrep \"\\\"filter_found_large\\\":1\" trace &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n \t\tdo\n-\t\t\tgit -c core.commitGraph=false log -- file$i.txt >expect &&\n-\t\t\tgit log -- file$i.txt >actual &&\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# Make a variety of path changes\n+\t\tprintf new1 >d/e/file1.txt &&\n+\t\tprintf new2 >d/file2.txt &&\n+\t\trm d/e/file2.txt &&\n+\t\trm -r foo &&\n+\t\tprintf text >foo &&\n+\t\tmkdir f &&\n+\t\tprintf new1 >f/file1.txt &&\n+\n+\t\t# including a mode-only change (counts as modified)\n+\t\tgit update-index --chmod=+x mode/script.sh &&\n+\n+\t\tgit add foo d f &&\n+\t\tgit commit -m \"complicated\" &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-edit\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":10\" trace-edit &&\n+\t\tgrep \"\\\"filter_found_large\\\":2\" trace-edit &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=11 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-update\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":11\" trace-update &&\n+\t\tgrep \"\\\"filter_found_large\\\":0\" trace-update &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n \t\t\ttest_cmp expect actual || return 1\n \t\tdone\n \t)\ndiff --git a/tree-diff.c b/tree-diff.c\nindex 6ebad1a46f..7cebbb327e 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -434,7 +434,7 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tif (diff_can_quit_early(opt))\n \t\t\tbreak;\n \n-\t\tif (opt->max_changes && opt->num_changes > opt->max_changes)\n+\t\tif (opt->max_changes && diff_queued_diff.nr > opt->max_changes)\n \t\t\tbreak;\n \n \t\tif (opt->pathspec.nr) {\n@@ -521,7 +521,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \n \t\t\t/* t↓ */\n \t\t\tupdate_tree_entry(&t);\n-\t\t\topt->num_changes++;\n \t\t}\n \n \t\t/* t > p[imin] */\n@@ -539,7 +538,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tskip_emit_tp:\n \t\t\t/* ∀ pi=p[imin] pi↓ */\n \t\t\tupdate_tp_entries(tp, nparent);\n-\t\t\topt->num_changes++;\n \t\t}\n \t}\n \n@@ -557,7 +555,6 @@ struct combine_diff_path *diff_tree_paths(\n \tconst struct object_id **parents_oid, int nparent,\n \tstruct strbuf *base, struct diff_options *opt)\n {\n-\topt->num_changes = 0;\n \tp = ll_diff_tree_paths(p, oid, parents_oid, nparent, base, opt);\n \n \t/*\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402916","messageId":"e92ccafcf7e0efe6e7e1b3af8f2e2e8634a1816b.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 10/14] commit-graph.c: sort index into commits list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:50Z","receivedAt":"2020-08-05T17:05:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For locality, 'compute_bloom_filters()' sorts the commits for which it\nwants to compute Bloom filters in a preferred order (cf., 3d11275505\n(commit-graph: examine commits by generation number, 2020-03-30) for\ndetails).\n\nA future patch will want to recover the new graph position of each\ncommit. Since the 'packed_commit_list' already stores a double-pointer,\navoid a 'COPY_ARRAY' and instead keep track of an index into the\noriginal list. (Use an integer index instead of a memory address, since\nthis involves a needlessly confusing triple-pointer).\n\nAlter the two sorting routines 'commit_pos_cmp' and 'commit_gen_cmp' to\ntake into account the packed_commit_list they are sorting with respect\nto. Since 'compute_bloom_filters()' is the only caller for each of those\ncomparison functions, no other call-sites need updating.\n\nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 43 ++++++++++++++++++++++++-------------------\n 1 file changed, 24 insertions(+), 19 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 48d4697f54..0d70545149 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -79,10 +79,18 @@ static void set_commit_pos(struct repository *r, const struct object_id *oid)\n \t*commit_pos_at(&commit_pos, commit) = max_pos++;\n }\n \n-static int commit_pos_cmp(const void *va, const void *vb)\n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+static int commit_pos_cmp(const void *va, const void *vb, void *ctx)\n {\n-\tconst struct commit *a = *(const struct commit **)va;\n-\tconst struct commit *b = *(const struct commit **)vb;\n+\tstruct packed_commit_list *commits = ctx;\n+\n+\tconst struct commit *a = commits->list[*(int *)va];\n+\tconst struct commit *b = commits->list[*(int *)vb];\n \treturn commit_pos_at(&commit_pos, a) -\n \t commit_pos_at(&commit_pos, b);\n }\n@@ -139,10 +147,12 @@ static struct commit_graph_data *commit_graph_data_at(const struct commit *c)\n \treturn data;\n }\n \n-static int commit_gen_cmp(const void *va, const void *vb)\n+static int commit_gen_cmp(const void *va, const void *vb, void *ctx)\n {\n-\tconst struct commit *a = *(const struct commit **)va;\n-\tconst struct commit *b = *(const struct commit **)vb;\n+\tstruct packed_commit_list *commits = ctx;\n+\n+\tconst struct commit *a = commits->list[*(int *)va];\n+\tconst struct commit *b = commits->list[*(int *)vb];\n \n \tuint32_t generation_a = commit_graph_generation(a);\n \tuint32_t generation_b = commit_graph_generation(b);\n@@ -922,11 +932,6 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n-struct packed_commit_list {\n-\tstruct commit **list;\n-\tint nr;\n-\tint alloc;\n-};\n \n struct packed_oid_list {\n \tstruct object_id *list;\n@@ -1408,7 +1413,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct progress *progress = NULL;\n-\tstruct commit **sorted_commits;\n+\tint *sorted_commits;\n \n \tinit_bloom_filters();\n \n@@ -1418,16 +1423,16 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->commits.nr);\n \n \tALLOC_ARRAY(sorted_commits, ctx->commits.nr);\n-\tCOPY_ARRAY(sorted_commits, ctx->commits.list, ctx->commits.nr);\n-\n-\tif (ctx->order_by_pack)\n-\t\tQSORT(sorted_commits, ctx->commits.nr, commit_pos_cmp);\n-\telse\n-\t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n+\tfor (i = 0; i < ctx->commits.nr; i++)\n+\t\tsorted_commits[i] = i;\n+\tQSORT_S(sorted_commits, ctx->commits.nr,\n+\t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n+\t\t&ctx->commits);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tint computed = 0;\n-\t\tstruct commit *c = sorted_commits[i];\n+\t\tint pos = sorted_commits[i];\n+\t\tstruct commit *c = ctx->commits.list[pos];\n \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\tctx->r,\n \t\t\tc,\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402917","messageId":"4deb724fc13e580430628f0ac408685394c8689f.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 08/14] bloom: use provided 'struct bloom_filter_settings'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:42Z","receivedAt":"2020-08-05T17:05:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When 'get_or_compute_bloom_filter()' needs to compute a Bloom filter\nfrom scratch, it looks to the default 'struct bloom_filter_settings' in\norder to determine the maximum number of changed paths, number of bits\nper entry, and so on.\n\nAll of these values have so far been constant, and so there was no need\nto pass in a pointer from the caller (eg., the one that is stored in the\n'struct write_commit_graph_context').\n\nStart passing in a 'struct bloom_filter_settings *' instead of using the\ndefault values to respect graph-specific settings (eg., in the case of\nsetting 'GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS').\n\nIn order to have an initialized value for these settings, move its\ninitialization to earlier in the commit-graph write.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 13 ++++++-------\n bloom.h | 3 ++-\n commit-graph.c | 21 ++++++++++-----------\n t/helper/test-bloom.c | 1 +\n 4 files changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex a8a21762f4..0cf1962dc5 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -180,13 +180,12 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t int *computed)\n {\n \tstruct bloom_filter *filter;\n-\tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \tint i;\n \tstruct diff_options diffopt;\n-\tint max_changes = 512;\n \n \tif (computed)\n \t\t*computed = 0;\n@@ -211,7 +210,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n \tdiffopt.detect_rename = 0;\n-\tdiffopt.max_changes = max_changes;\n+\tdiffopt.max_changes = settings->max_changed_paths;\n \tdiff_setup_done(&diffopt);\n \n \t/* ensure commit is parsed so we have parent information */\n@@ -223,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= max_changes) {\n+\tif (diffopt.num_changes <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -260,13 +259,13 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n-\t\tfilter->len = (hashmap_get_size(&pathmap) * settings.bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n \t\t\tstruct bloom_key key;\n-\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, &settings);\n-\t\t\tadd_key_to_filter(&key, filter, &settings);\n+\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, settings);\n+\t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\ndiff --git a/bloom.h b/bloom.h\nindex c4107a6415..2353c74a24 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -92,11 +92,12 @@ void init_bloom_filters(void);\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t int *computed);\n \n #define DEFAULT_BLOOM_MAX_CHANGES 512\n #define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n-\t(r), (c), 0, NULL)\n+\t(r), (c), 0, NULL, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ba2a2cfb22..48d4697f54 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1432,6 +1432,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->r,\n \t\t\tc,\n \t\t\t1,\n+\t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed) {\n \t\t\tctx->count_bloom_filter_computed++;\n@@ -1688,17 +1689,6 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tint num_chunks = 3;\n \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n-\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n-\n-\tif (!ctx->bloom_settings) {\n-\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n-\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n-\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n-\t\t\t\t\t\t\t bloom_settings.num_hashes);\n-\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n-\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n-\t\tctx->bloom_settings = &bloom_settings;\n-\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -2144,6 +2134,7 @@ int write_commit_graph(struct object_directory *odb,\n \tuint32_t i, count_distinct = 0;\n \tint res = 0;\n \tint replace = 0;\n+\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n@@ -2157,6 +2148,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split_opts = split_opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n+\t\t\t\t\t\t bloom_settings.bits_per_entry);\n+\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n+\t\t\t\t\t\t bloom_settings.num_hashes);\n+\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n+\tctx->bloom_settings = &bloom_settings;\n+\n \tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n \t\tctx->changed_paths = 1;\n \tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 531af439c2..4af949164c 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -40,6 +40,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n \tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t &settings,\n \t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402918","messageId":"23525947c8c3bf9fc5faf59d9a7b6da65cec4e12.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:39Z","receivedAt":"2020-08-05T17:06:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'get_bloom_filter' takes a flag to control whether it will compute a\nBloom filter if the requested one is missing. In the next patch, we'll\nadd yet another parameter to this method, which would force all but one\ncaller to specify an extra 'NULL' parameter at the end.\n\nInstead of doing this, split 'get_bloom_filter' into two functions:\n'get_bloom_filter' and 'get_or_compute_bloom_filter'. The former only\nlooks up a Bloom filter (and does not compute one if it's missing,\nthus dropping the 'compute_if_not_present' flag). The latter does\ncompute missing Bloom filters, with an additional parameter to store\nwhether or not it needed to do so.\n\nThis simplifies many call-sites, since the majority of existing callers\nto 'get_bloom_filter' do not want missing Bloom filters to be computed\n(so they can drop the parameter entirely and use the simpler version of\nthe function).\n\nWhile we're at it, instrument the new 'get_or_compute_bloom_filter()'\nwith two counters in the 'write_commit_graph_context' struct which store\nthe number of filters that we computed, and the number of those which\nwere too large to store.\n\nIt would be nice to drop the 'compute_if_not_present' flag entirely,\nsince all remaining callers of 'get_or_compute_bloom_filter' pass it as\n'1', but this will change in a future patch and hence cannot be removed.\n\nSigned-off-by: Taylor Blau \n---\n blame.c | 2 +-\n bloom.c | 13 ++++++++++---\n bloom.h | 11 ++++++++---\n commit-graph.c | 38 +++++++++++++++++++++++++++++++++++---\n line-log.c | 2 +-\n revision.c | 2 +-\n t/helper/test-bloom.c | 3 ++-\n 7 files changed, 58 insertions(+), 13 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 3e5f8787bc..756285fca7 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -1275,7 +1275,7 @@ static int maybe_changed_path(struct repository *r,\n \tif (commit_graph_generation(origin->commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn 1;\n \n-\tfilter = get_bloom_filter(r, origin->commit, 0);\n+\tfilter = get_bloom_filter(r, origin->commit);\n \n \tif (!filter)\n \t\treturn 1;\ndiff --git a/bloom.c b/bloom.c\nindex cd9380ac62..a8a21762f4 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,9 +177,10 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present)\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t int *computed)\n {\n \tstruct bloom_filter *filter;\n \tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n@@ -187,6 +188,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n+\tif (computed)\n+\t\t*computed = 0;\n+\n \tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n@@ -273,6 +277,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \t\tfilter->len = 0;\n \t}\n \n+\tif (computed)\n+\t\t*computed = 1;\n+\n \tfree(diff_queued_diff.queue);\n \tDIFF_QUEUE_CLEAR(&diff_queued_diff);\n \ndiff --git a/bloom.h b/bloom.h\nindex 0b9b59a6fe..c4107a6415 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -89,9 +89,14 @@ void add_key_to_filter(const struct bloom_key *key,\n \n void init_bloom_filters(void);\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present);\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t int *computed);\n+\n+#define DEFAULT_BLOOM_MAX_CHANGES 512\n+#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n+\t(r), (c), 0, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 86dd4b979e..ba2a2cfb22 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -964,6 +964,9 @@ struct write_commit_graph_context {\n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n+\n+\tint count_bloom_filter_found_large;\n+\tint count_bloom_filter_computed;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1175,7 +1178,7 @@ static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tuint32_t cur_pos = 0;\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \t\tcur_pos += len;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1215,7 +1218,7 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1385,6 +1388,22 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n+{\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"filter_found_large\",\n+\t\t\t ctx->count_bloom_filter_found_large);\n+\tjw_object_intmax(&jw, \"filter_computed\",\n+\t\t\t ctx->count_bloom_filter_computed);\n+\tjw_end(&jw);\n+\n+\ttrace2_data_json(\"commit-graph\", the_repository, \"bloom_statistics\", &jw);\n+\n+\tjw_release(&jw);\n+}\n+\n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n@@ -1407,12 +1426,25 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tint computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n+\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\tctx->r,\n+\t\t\tc,\n+\t\t\t1,\n+\t\t\t&computed);\n+\t\tif (computed) {\n+\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (filter && !filter->len)\n+\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t}\n \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n+\tif (trace2_is_enabled())\n+\t\ttrace2_bloom_filter_write_statistics(ctx);\n+\n \tfree(sorted_commits);\n \tstop_progress(&progress);\n }\ndiff --git a/line-log.c b/line-log.c\nindex c53692834d..9e58fd185a 100644\n--- a/line-log.c\n+++ b/line-log.c\n@@ -1159,7 +1159,7 @@ static int bloom_filter_check(struct rev_info *rev,\n \t\treturn 1;\n \n \tif (!rev->bloom_filter_settings ||\n-\t !(filter = get_bloom_filter(rev->repo, commit, 0)))\n+\t !(filter = get_bloom_filter(rev->repo, commit)))\n \t\treturn 1;\n \n \tif (!range)\ndiff --git a/revision.c b/revision.c\nindex c45ed1076e..b7ec712755 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -753,7 +753,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \tif (commit_graph_generation(commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn -1;\n \n-\tfilter = get_bloom_filter(revs->repo, commit, 0);\n+\tfilter = get_bloom_filter(revs->repo, commit);\n \n \tif (!filter) {\n \t\tcount_bloom_filter_not_present++;\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex f0aa80b98e..531af439c2 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -39,7 +39,8 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tstruct bloom_filter *filter;\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n-\tfilter = get_bloom_filter(the_repository, c, 1);\n+\tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n \n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402919","messageId":"100b26d7c8a5e7ce21a950f33791eadab74e8e70.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:58Z","receivedAt":"2020-08-05T17:06:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a commit has more than a certain number of changed paths (commonly\n512), the commit-graph machinery represents it as a zero-length filter.\nThis is done since having many entries in the Bloom filter has\nundesirable effects on the false positivity rate.\n\nIn addition to these too-large filters, the commit-graph machinery also\nrepresents commits with no filter and commits with no changed paths in\nthe same way.\n\nWhen writing a commit-graph that aggregates several incremental\ncommit-graph layers (eg., with '--split=replace'), the commit-graph\nmachinery first computes all of the Bloom filters that it wants to write\nbut does not already know about from existing graph layers. Because we\noverload the zero-length filter in the above fashion, this leads to\nrecomputing large filters over and over again.\n\nThis is already undesirable, since it means that we are wasting\nconsiderable effort to discover that a commit with too many changed\npaths, only to throw that effort away (and then repeat the process the\nnext time a roll-up is performed).\n\nIn a subsequent patch, we will add a '--max-new-filters=' option,\nwhich specifies an upper-bound on the number of new filters we are\nwilling to compute from scratch. Suppose that there are 'N' too-large\nfilters, and we specify '--max-new-filters=M'. If 'N >= M', it is\nunlikely that any filters will be generated, since we'll spend most of\nour effort on filters that we ultimately throw away. If 'N < M', filters\nwill trickle in over time, but only at most 'M - N' per-write.\n\nTo address this, add a new chunk which encodes a bitmap where the ith\nbit is on iff the ith commit has zero or at least 512 changed paths.\nLikewise store the maximum number of changed paths we are willing to\nstore in order to prepare for eventually making this value more easily\ncustomizable. When computing Bloom filters, first consult the relevant\nbitmap (in the case that we are rolling up existing layers) to see if\ncomputing the Bloom filter from scratch would be a waste of time.\n\nThis patch implements a new chunk instead of extending the existing BIDX\nand BDAT chunks because modifying these chunks would confuse old\nclients. (Eg., setting the most-significant bit in the BIDX chunk would\nconfuse old clients and require a version bump).\n\nTo allow using the existing bitmap code with 64-bit words, we write the\ndata in network byte order from the 64-bit words. This means we also\nneed to read the array from the commit-graph file by translating each\nword from network byte order using get_be64() upon first use of the\nbitmap. This is only used when writing the commit-graph, so this is a\nrelatively small operation compared to the other writing code.\n\nBy avoiding the need to move to new versions of the BDAT and BIDX chunk,\nwe can give ourselves more time to consider whether or not other\nmodifications to these chunks are worthwhile without holding up this\nchange.\n\nAnother approach would be to introduce a new BIDX chunk (say, one\nidentified by 'BID2') which is identical to the existing BIDX chunk,\nexcept the most-significant bit of each offset is interpreted as \"this\nfilter is too big\" iff looking at a BID2 chunk. This avoids having to\nwrite a bitmap, but forces older clients to rewrite their commit-graphs\n(as well as reduces the theoretical largest Bloom filters we couldl\nwrite, and forces us to maintain the code necessary to translate BIDX\nchunks to BID2 ones). Separately from this patch, I implemented this\nalternate approach and did not find it to be advantageous.\n\nSigned-off-by: Taylor Blau \n---\n .../technical/commit-graph-format.txt | 12 +++\n bloom.h | 2 +-\n commit-graph.c | 101 +++++++++++++++---\n commit-graph.h | 5 +\n t/t4216-log-bloom.sh | 25 ++++-\n 5 files changed, 130 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 440541045d..5f2d9ab4d7 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -123,6 +123,18 @@ CHUNK DATA:\n of length zero.\n * The BDAT chunk is present if and only if BIDX is present.\n \n+ Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n+ * It starts with a 32-bit unsigned integer specifying the maximum number of\n+ changed-paths that can be stored in a single Bloom filter.\n+ * It then contains a list of 64-bit words (the length of this list is\n+ determined by the width of the chunk) which is a bitmap. The 'i'th bit is\n+ set exactly when the 'i'th commit in the graph has a changed-path Bloom\n+ filter with zero entries (either because the commit is empty, or because\n+ it contains more than 512 changed paths).\n+ * The BFXL chunk is present only when the BIDX and BDAT chunks are\n+ also present.\n+\n+\n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n This list of H-byte hashes describe a set of B commit-graph files that\n form a commit-graph chain. The graph position for the ith commit in this\ndiff --git a/bloom.h b/bloom.h\nindex 2353c74a24..73238276bc 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -33,7 +33,7 @@ struct bloom_filter_settings {\n \t * The maximum number of changed paths per commit\n \t * before declaring a Bloom filter to be too-large.\n \t *\n-\t * Not written to the commit-graph file.\n+\t * Written to the 'BFXL' chunk (instead of 'BDAT').\n \t */\n \tuint32_t max_changed_paths;\n };\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8964453433..1fee49d171 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -41,8 +41,9 @@ void git_test_write_commit_graph_or_die(void)\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n+#define GRAPH_CHUNKID_BLOOMLARGE 0x4246584c /* \"BFXL\" */\n #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n-#define MAX_NUM_CHUNKS 7\n+#define MAX_NUM_CHUNKS 8\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -429,6 +430,19 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n \t\t\t}\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BLOOMLARGE:\n+\t\t\tif (graph->chunk_bloom_large_filters)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n+\t\t\t\tgraph->bloom_large_to_alloc = get_be64(chunk_lookup + 4)\n+\t\t\t\t\t\t\t - chunk_offset - sizeof(uint32_t);\n+\n+\t\t\t\tgraph->bloom_large.word_alloc = 0; /* populate when necessary */\n+\t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n+\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n+\t\t\t}\n+\t\t\tbreak;\n \t\t}\n \n \t\tif (chunk_repeated) {\n@@ -443,6 +457,7 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t/* We need both the bloom chunks to exist together. Else ignore the data */\n \t\tgraph->chunk_bloom_indexes = NULL;\n \t\tgraph->chunk_bloom_data = NULL;\n+\t\tgraph->chunk_bloom_large_filters = NULL;\n \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n \t}\n \n@@ -932,6 +947,31 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n+static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n+\t\t\t\t\t const struct commit *c)\n+{\n+\tuint32_t graph_pos = commit_graph_position(c);\n+\tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\treturn 0;\n+\n+\twhile (g && graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (!g || !g->bloom_large_to_alloc)\n+\t\treturn 0;\n+\n+\tif (!g->bloom_large.word_alloc) {\n+\t\tsize_t i;\n+\t\tg->bloom_large.word_alloc = g->bloom_large_to_alloc;\n+\t\tg->bloom_large.words = xmalloc(g->bloom_large_to_alloc * sizeof(eword_t));\n+\n+\t\tfor (i = 0; i < g->bloom_large_to_alloc; i++)\n+\t\t\tg->bloom_large.words[i] = get_be64(g->chunk_bloom_large_filters\n+\t\t\t\t\t\t\t + i * sizeof(eword_t));\n+\t}\n+\n+\treturn bitmap_get(&g->bloom_large, graph_pos - g->num_commits_in_base);\n+}\n \n struct packed_oid_list {\n \tstruct object_id *list;\n@@ -970,8 +1010,10 @@ struct write_commit_graph_context {\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \n+\tint count_bloom_filter_known_large;\n \tint count_bloom_filter_found_large;\n \tint count_bloom_filter_computed;\n+\tstruct bitmap *bloom_large;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1235,6 +1277,23 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \treturn 0;\n }\n \n+static int write_graph_chunk_bloom_large(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\tsize_t i, alloc = ctx->commits.nr / BITS_IN_EWORD;\n+\tif (ctx->commits.nr % BITS_IN_EWORD)\n+\t\talloc++;\n+\tif (alloc > ctx->bloom_large->word_alloc)\n+\t\tBUG(\"write_graph_chunk_bloom_large: bitmap not large enough\");\n+\n+\ttrace2_region_enter(\"commit-graph\", \"bloom_large\", ctx->r);\n+\thashwrite_be32(f, ctx->bloom_settings->max_changed_paths);\n+\tfor (i = 0; i < ctx->bloom_large->word_alloc; i++)\n+\t\thashwrite_be64(f, ctx->bloom_large->words[i]);\n+\ttrace2_region_leave(\"commit-graph\", \"bloom_large\", ctx->r);\n+\treturn 0;\n+}\n+\n static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n@@ -1398,6 +1457,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \tstruct json_writer jw = JSON_WRITER_INIT;\n \n \tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"filter_known_large\",\n+\t\t\t ctx->count_bloom_filter_known_large);\n \tjw_object_intmax(&jw, \"filter_found_large\",\n \t\t\t ctx->count_bloom_filter_found_large);\n \tjw_object_intmax(&jw, \"filter_computed\",\n@@ -1416,6 +1477,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint *sorted_commits;\n \n \tinit_bloom_filters();\n+\tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n \n \tif (ctx->report_progress)\n \t\tprogress = start_delayed_progress(\n@@ -1430,21 +1492,28 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t&ctx->commits);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tint computed = 0;\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n-\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n-\t\t\tctx->r,\n-\t\t\tc,\n-\t\t\t1,\n-\t\t\tctx->bloom_settings,\n-\t\t\t&computed);\n-\t\tif (computed) {\n-\t\t\tctx->count_bloom_filter_computed++;\n-\t\t\tif (filter && !filter->len)\n-\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n+\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\tctx->count_bloom_filter_known_large++;\n+\t\t} else {\n+\t\t\tint computed = 0;\n+\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\t\tctx->r,\n+\t\t\t\tc,\n+\t\t\t\t1,\n+\t\t\t\tctx->bloom_settings,\n+\t\t\t\t&computed);\n+\t\t\tif (computed) {\n+\t\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\t\tif (filter && !filter->len) {\n+\t\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\t}\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n@@ -1764,6 +1833,11 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n \t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n+\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMLARGE;\n+\t\tchunks[num_chunks].size = sizeof(eword_t) * ctx->bloom_large->word_alloc\n+\t\t\t\t\t+ sizeof(uint32_t);\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_large;\n+\t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n@@ -2503,6 +2577,7 @@ void free_commit_graph(struct commit_graph *g)\n \t}\n \tfree(g->filename);\n \tfree(g->bloom_filter_settings);\n+\tbitmap_free(g->bloom_large);\n \tfree(g);\n }\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex d9acb22bac..f4fb996dd5 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,6 +4,7 @@\n #include \"git-compat-util.h\"\n #include \"object-store.h\"\n #include \"oidset.h\"\n+#include \"ewah/ewok.h\"\n \n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n #define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n@@ -71,6 +72,10 @@ struct commit_graph {\n \tconst unsigned char *chunk_base_graphs;\n \tconst unsigned char *chunk_bloom_indexes;\n \tconst unsigned char *chunk_bloom_data;\n+\tconst unsigned char *chunk_bloom_large_filters;\n+\n+\tsize_t bloom_large_to_alloc;\n+\tstruct bitmap bloom_large;\n \n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 21b67677ef..6859d85369 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -33,7 +33,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \tgit commit-graph write --reachable --changed-paths\n '\n graph_read_expect () {\n-\tNUM_CHUNKS=5\n+\tNUM_CHUNKS=6\n \tcat >expect <<- EOF\n \theader: 43475048 1 1 $NUM_CHUNKS 0\n \tnum_commits: $1\n@@ -262,5 +262,28 @@ test_expect_success 'correctly report changes over limit' '\n \t\tdone\n \t)\n '\n+test_bloom_filters_computed () {\n+\tcommit_graph_args=$1\n+\tbloom_trace_prefix=\"{\\\"filter_known_large\\\":$2,\\\"filter_found_large\\\":$3,\\\"filter_computed\\\":$4\"\n+\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n+\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write $commit_graph_args &&\n+\tgrep \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.event\"\n+}\n+\n+test_expect_success 'Bloom generation does not recompute too-large filters' '\n+\t(\n+\t\tcd limits &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t--split=replace &&\n+\t\ttest_commit c1 filter &&\n+\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n+\t\t\t2 0 1\n+\t)\n+'\n \n test_done\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402920","messageId":"c42d678714c16271fc02bc4e80c7c5a942930322.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 11/14] csum-file.h: introduce 'hashwrite_be64()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:54Z","receivedAt":"2020-08-05T17:06:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A small handful of writers who wish to encode 64-bit values in network\norder have worked around the lack of such a helper by calling the 32-bit\nvariant twice.\n\nThe subsequent commit will add another caller who wants to write a\n64-bit value. To ease their (and the existing caller's) pain, introduce\na helper to do just that, and convert existing call-sites.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 8 ++------\n csum-file.h | 6 ++++++\n midx.c | 3 +--\n 3 files changed, 9 insertions(+), 8 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0d70545149..8964453433 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1784,12 +1784,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tchunk_offset = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunks[i].id);\n-\t\tchunk_write[1] = htonl(chunk_offset >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offset & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n+\t\thashwrite_be32(f, chunks[i].id);\n+\t\thashwrite_be64(f, chunk_offset);\n \n \t\tchunk_offset += chunks[i].size;\n \t}\ndiff --git a/csum-file.h b/csum-file.h\nindex f9cbd317fb..b026ec7766 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -62,4 +62,10 @@ static inline void hashwrite_be32(struct hashfile *f, uint32_t data)\n \thashwrite(f, &data, sizeof(data));\n }\n \n+static inline void hashwrite_be64(struct hashfile *f, uint64_t data)\n+{\n+\thashwrite_be32(f, data >> 32);\n+\thashwrite_be32(f, data & 0xffffffffUL);\n+}\n+\n #endif\ndiff --git a/midx.c b/midx.c\nindex 6d1584ca51..8b33149ce4 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -775,8 +775,7 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \t\tif (!(offset >> 31))\n \t\t\tcontinue;\n \n-\t\thashwrite_be32(f, offset >> 32);\n-\t\thashwrite_be32(f, offset & 0xffffffffUL);\n+\t\thashwrite_be64(f, offset);\n \t\twritten += 2 * sizeof(uint32_t);\n \n \t\tnr_large_offset--;\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402921","messageId":"2ee0b8435168226f91ef60916413be0a5da5290d.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 13/14] commit-graph: rename 'split_commit_graph_opts'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:03:01Z","receivedAt":"2020-08-05T17:06:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the subsequent commit, additional options will be added to the\ncommit-graph API which have nothing to do with splitting.\n\nRename the 'split_commit_graph_opts' structure to the more-generic\n'commit_graph_opts' to encompass both.\n\nSuggsted-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 20 ++++++++++----------\n commit-graph.c | 40 ++++++++++++++++++++--------------------\n commit-graph.h | 6 +++---\n 3 files changed, 33 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ba5584463f..38f5f57d15 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,7 +119,7 @@ static int graph_verify(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n-static struct split_commit_graph_opts split_opts;\n+static struct commit_graph_opts write_opts;\n \n static int write_option_parse_split(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n@@ -187,24 +187,24 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n \t\t\tN_(\"enable computation for changed paths\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\twrite_option_parse_split),\n-\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\tOPT_INTEGER(0, \"max-commits\", &write_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n-\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\tOPT_INTEGER(0, \"size-multiple\", &write_opts.size_multiple,\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n-\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n \t\tOPT_END(),\n \t};\n \n \topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n-\tsplit_opts.size_multiple = 2;\n-\tsplit_opts.max_commits = 0;\n-\tsplit_opts.expire_time = 0;\n+\twrite_opts.size_multiple = 2;\n+\twrite_opts.max_commits = 0;\n+\twrite_opts.expire_time = 0;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -232,7 +232,7 @@ static int graph_write(int argc, const char **argv)\n \todb = find_odb(the_repository, opts.obj_dir);\n \n \tif (opts.reachable) {\n-\t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n+\t\tif (write_commit_graph_reachable(odb, flags, &write_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n@@ -261,7 +261,7 @@ static int graph_write(int argc, const char **argv)\n \t\t\t opts.stdin_packs ? &pack_indexes : NULL,\n \t\t\t opts.stdin_commits ? &commits : NULL,\n \t\t\t flags,\n-\t\t\t &split_opts))\n+\t\t\t &write_opts))\n \t\tresult = 1;\n \n cleanup:\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1fee49d171..82fca07579 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1006,7 +1006,7 @@ struct write_commit_graph_context {\n \t\t changed_paths:1,\n \t\t order_by_pack:1;\n \n-\tconst struct split_commit_graph_opts *split_opts;\n+\tconst struct commit_graph_opts *opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \n@@ -1347,8 +1347,8 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1548,7 +1548,7 @@ static int add_ref_to_set(const char *refname,\n \n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t\t\t const struct commit_graph_opts *opts)\n {\n \tstruct oidset commits = OIDSET_INIT;\n \tstruct refs_cb_data data;\n@@ -1565,7 +1565,7 @@ int write_commit_graph_reachable(struct object_directory *odb,\n \tstop_progress(&data.progress);\n \n \tresult = write_commit_graph(odb, NULL, &commits,\n-\t\t\t\t flags, split_opts);\n+\t\t\t\t flags, opts);\n \n \toidset_clear(&commits);\n \treturn result;\n@@ -1680,8 +1680,8 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tctx->num_extra_edges = 0;\n \tif (ctx->report_progress)\n@@ -1967,13 +1967,13 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tint max_commits = 0;\n \tint size_mult = 2;\n \n-\tif (ctx->split_opts) {\n-\t\tmax_commits = ctx->split_opts->max_commits;\n+\tif (ctx->opts) {\n+\t\tmax_commits = ctx->opts->max_commits;\n \n-\t\tif (ctx->split_opts->size_multiple)\n-\t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t\tif (ctx->opts->size_multiple)\n+\t\t\tsize_mult = ctx->opts->size_multiple;\n \n-\t\tflags = ctx->split_opts->flags;\n+\t\tflags = ctx->opts->flags;\n \t}\n \n \tg = ctx->r->objects->commit_graph;\n@@ -2151,8 +2151,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n \n-\tif (ctx->split_opts && ctx->split_opts->expire_time)\n-\t\texpire_time = ctx->split_opts->expire_time;\n+\tif (ctx->opts && ctx->opts->expire_time)\n+\t\texpire_time = ctx->opts->expire_time;\n \tif (!ctx->split) {\n \t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n@@ -2203,7 +2203,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t const struct commit_graph_opts *opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -2220,7 +2220,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n-\tctx->split_opts = split_opts;\n+\tctx->opts = opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n@@ -2268,15 +2268,15 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\t}\n \t\t}\n \n-\t\tif (ctx->split_opts)\n-\t\t\treplace = ctx->split_opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n+\t\tif (ctx->opts)\n+\t\t\treplace = ctx->opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n \t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n-\t\tctx->oids.alloc = split_opts->max_commits;\n+\tif (ctx->split && opts && ctx->oids.alloc > opts->max_commits)\n+\t\tctx->oids.alloc = opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->odb);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex f4fb996dd5..1d147b7b76 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -110,7 +110,7 @@ enum commit_graph_split_flags {\n \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n };\n \n-struct split_commit_graph_opts {\n+struct commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n@@ -125,12 +125,12 @@ struct split_commit_graph_opts {\n */\n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t\t\t const struct commit_graph_opts *opts);\n int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t const struct commit_graph_opts *opts);\n \n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n \n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402922","messageId":"afdc614c0d146aa132775022c896030a0b2f2c32.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 03/14] commit-graph: pass a 'struct repository *' in more places","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:24Z","receivedAt":"2020-08-05T17:06:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a future commit, some commit-graph internals will want access to\n'r->settings', but we only have the 'struct object_directory *'\ncorresponding to that repository.\n\nAdd an additional parameter to pass the repository around in more\nplaces.\n\nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 2 +-\n commit-graph.c | 17 ++++++++++-------\n commit-graph.h | 6 ++++--\n fuzz-commit-graph.c | 5 +++--\n 4 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 523501f217..ba5584463f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -106,7 +106,7 @@ static int graph_verify(int argc, const char **argv)\n \tFREE_AND_NULL(graph_name);\n \n \tif (open_ok)\n-\t\tgraph = load_commit_graph_one_fd_st(fd, &st, odb);\n+\t\tgraph = load_commit_graph_one_fd_st(the_repository, fd, &st, odb);\n \telse\n \t\tgraph = read_commit_graph_one(the_repository, odb);\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex d4b06811be..0c1030641c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -224,7 +224,8 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st)\n \treturn 1;\n }\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tvoid *graph_map;\n@@ -240,7 +241,7 @@ struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n \t}\n \tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n-\tret = parse_commit_graph(graph_map, graph_size);\n+\tret = parse_commit_graph(r, graph_map, graph_size);\n \n \tif (ret)\n \t\tret->odb = odb;\n@@ -280,7 +281,8 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n \tconst unsigned char *data, *chunk_lookup;\n \tuint32_t i;\n@@ -445,7 +447,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n \treturn NULL;\n }\n \n-static struct commit_graph *load_commit_graph_one(const char *graph_file,\n+static struct commit_graph *load_commit_graph_one(struct repository *r,\n+\t\t\t\t\t\t const char *graph_file,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \n@@ -457,7 +460,7 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file,\n \tif (!open_ok)\n \t\treturn NULL;\n \n-\tg = load_commit_graph_one_fd_st(fd, &st, odb);\n+\tg = load_commit_graph_one_fd_st(r, fd, &st, odb);\n \n \tif (g)\n \t\tg->filename = xstrdup(graph_file);\n@@ -469,7 +472,7 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tchar *graph_name = get_commit_graph_filename(odb);\n-\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \tfree(graph_name);\n \n \treturn g;\n@@ -550,7 +553,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \t\tvalid = 0;\n \t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\t\tchar *graph_name = get_split_graph_filename(odb, line.buf);\n-\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \n \t\t\tfree(graph_name);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 0677dd1031..d9acb22bac 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,13 @@ struct commit_graph {\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb);\n struct commit_graph *read_commit_graph_one(struct repository *r,\n \t\t\t\t\t struct object_directory *odb);\n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n /*\n * Return 1 if and only if the repository has a commit-graph\ndiff --git a/fuzz-commit-graph.c b/fuzz-commit-graph.c\nindex 430817214d..e7cf6d5b0f 100644\n--- a/fuzz-commit-graph.c\n+++ b/fuzz-commit-graph.c\n@@ -1,7 +1,8 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n \n@@ -10,7 +11,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \tstruct commit_graph *g;\n \n \tinitialize_the_repository();\n-\tg = parse_commit_graph((void *)data, size);\n+\tg = parse_commit_graph(the_repository, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\n \n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402923","messageId":"cover.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH v2 00/14] more miscellaneous Bloom filter improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:01:30Z","receivedAt":"2020-08-05T17:07:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nHere's a re-roll of mine and Stolee's series containing a grab bag of\nchanged-path Bloom filter improvements. The contents are described in\n[1], but the culmination of this series is in adding the new 'BFXL'\nchunk, and allowing callers to pass '--max-new-filters ' to\n'git commit-graph write' commands.\n\nLots of little things have changed since last time, most notably a\nmoderate restructuring of the patches to slot in Stolee's fix for\n'max_changes' not working properly in the diff machinery [2].\n\nA range-diff is included below for convening, although I'm not sure how\nilluminating it is to read compared to starting from scratch reading the\npatches one by one...\n\nThanks again for all of the helpful review since last time.\n\n[1]: https://lore.kernel.org/git/cover.1596480582.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/a08c26bb-54ec-13af-e503-fccd68727cf3@gmail.com/\n\nDerrick Stolee (1):\n bloom/diff: properly short-circuit on max_changes\n\nTaylor Blau (13):\n commit-graph: introduce 'get_bloom_filter_settings()'\n t4216: use an '&&'-chain\n commit-graph: pass a 'struct repository *' in more places\n t/helper/test-read-graph.c: prepare repo settings\n commit-graph: respect 'commitGraph.readChangedPaths'\n commit-graph.c: store maximum changed paths\n bloom: split 'get_bloom_filter()' in two\n bloom: use provided 'struct bloom_filter_settings'\n commit-graph.c: sort index into commits list\n csum-file.h: introduce 'hashwrite_be64()'\n commit-graph: add large-filters bitmap chunk\n commit-graph: rename 'split_commit_graph_opts'\n builtin/commit-graph.c: introduce '--max-new-filters='\n\n Documentation/config.txt | 2 +\n Documentation/config/commitgraph.txt | 8 +\n Documentation/git-commit-graph.txt | 4 +\n .../technical/commit-graph-format.txt | 12 +\n blame.c | 8 +-\n bloom.c | 51 +++-\n bloom.h | 23 +-\n builtin/commit-graph.c | 61 +++-\n commit-graph.c | 268 +++++++++++++-----\n commit-graph.h | 20 +-\n csum-file.h | 6 +\n diff.h | 2 -\n fuzz-commit-graph.c | 5 +-\n line-log.c | 2 +-\n midx.c | 3 +-\n repo-settings.c | 3 +\n repository.h | 1 +\n revision.c | 7 +-\n t/helper/test-bloom.c | 4 +-\n t/helper/test-read-graph.c | 3 +-\n t/t4216-log-bloom.sh | 148 ++++++++--\n t/t5324-split-commit-graph.sh | 13 +\n tree-diff.c | 5 +-\n 23 files changed, 518 insertions(+), 141 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\nRange-diff against v1:\n [ ... rebased onto 'master' ]\n 1: 08479793c1 ! 11: 001f3385ff commit-graph: introduce 'get_bloom_filter_settings()'\n @@ commit-graph.h: struct commit_graph *parse_commit_graph(void *graph_map, size_t\n\n ## revision.c ##\n @@ revision.c: static void prepare_to_use_bloom_filter(struct rev_info *revs)\n - \tif (!revs->repo->objects->commit_graph)\n - \t\treturn;\n\n + \trepo_parse_commit(revs->repo, revs->commits->item);\n +\n +-\tif (!revs->repo->objects->commit_graph)\n +-\t\treturn;\n +-\n -\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n +\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n \tif (!revs->bloom_filter_settings)\n 3: d12fcc4a8d = 12: e4d068a478 t4216: use an '&&'-chain\n 2: 52f8f7424e ! 13: afdc614c0d commit-graph: pass a 'struct repository *' in more places\n @@ Commit message\n corresponding to that repository.\n\n Add an additional parameter to pass the repository around in more\n - places. In the next patch, we will remove the object directory (and\n - instead reference it with 'r->odb').\n + places.\n\n Signed-off-by: Taylor Blau \n\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(void *graph_map, size_t\n }\n\n -static struct commit_graph *load_commit_graph_one(const char *graph_file,\n -+\n +static struct commit_graph *load_commit_graph_one(struct repository *r,\n +\t\t\t\t\t\t const char *graph_file,\n \t\t\t\t\t\t struct object_directory *odb)\n 4: 42a0ca9549 = 14: 038e996ced t/helper/test-read-graph.c: prepare repo settings\n 5: e615077283 ! 15: 404f10319a commit-graph: respect 'commitgraph.readChangedPaths'\n @@ Metadata\n Author: Taylor Blau \n\n ## Commit message ##\n - commit-graph: respect 'commitgraph.readChangedPaths'\n + commit-graph: respect 'commitGraph.readChangedPaths'\n\n Git uses the 'core.commitGraph' configuration value to control whether\n or not the commit graph is used when parsing commits or performing a\n @@ Commit message\n Bloom filters. This can happen, for example, during a staged roll-out,\n or in the event of an incident.\n\n - Introduce 'commitgraph.readChangedPaths' to control whether or not Bloom\n + Introduce 'commitGraph.readChangedPaths' to control whether or not Bloom\n filters are read. Note that this configuration is independent from both:\n\n - 'core.commitGraph', to allow flexibility in using all parts of a\n @@ Documentation/config.txt: include::config/column.txt[]\n\n ## Documentation/config/commitgraph.txt (new) ##\n @@\n -+commitgraph.readChangedPaths::\n ++commitGraph.readChangedPaths::\n +\tIf true, then git will use the changed-path Bloom filters in the\n +\tcommit-graph file (if it exists, and they are present). Defaults to\n +\ttrue. See linkgit:git-commit-graph[1] for more information.\n -: ---------- > 16: 053991f048 commit-graph.c: store maximum changed paths\n 8: a494094c10 ! 17: 23525947c8 bloom: split 'get_bloom_filter()' in two\n @@ Commit message\n\n 'get_bloom_filter' takes a flag to control whether it will compute a\n Bloom filter if the requested one is missing. In the next patch, we'll\n - add yet another flag to this method, which would force all but one\n + add yet another parameter to this method, which would force all but one\n caller to specify an extra 'NULL' parameter at the end.\n\n Instead of doing this, split 'get_bloom_filter' into two functions:\n @@ Commit message\n (so they can drop the parameter entirely and use the simpler version of\n the function).\n\n + While we're at it, instrument the new 'get_or_compute_bloom_filter()'\n + with two counters in the 'write_commit_graph_context' struct which store\n + the number of filters that we computed, and the number of those which\n + were too large to store.\n +\n + It would be nice to drop the 'compute_if_not_present' flag entirely,\n + since all remaining callers of 'get_or_compute_bloom_filter' pass it as\n + '1', but this will change in a future patch and hence cannot be removed.\n +\n Signed-off-by: Taylor Blau \n\n ## blame.c ##\n @@ bloom.h: void add_key_to_filter(const struct bloom_key *key,\n +\t\t\t\t\t\t int compute_if_not_present,\n +\t\t\t\t\t\t int *computed);\n +\n -+#define get_bloom_filter(r, c) get_or_compute_bloom_filter((r), (c), 0, NULL)\n ++#define DEFAULT_BLOOM_MAX_CHANGES 512\n ++#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n ++\t(r), (c), 0, NULL)\n\n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\n\n ## commit-graph.c ##\n +@@ commit-graph.c: struct write_commit_graph_context {\n + \tconst struct split_commit_graph_opts *split_opts;\n + \tsize_t total_bloom_filter_data_size;\n + \tconst struct bloom_filter_settings *bloom_settings;\n ++\n ++\tint count_bloom_filter_found_large;\n ++\tint count_bloom_filter_computed;\n + };\n +\n + static int write_graph_chunk_fanout(struct hashfile *f,\n @@ commit-graph.c: static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tuint32_t cur_pos = 0;\n\n @@ commit-graph.c: static int write_graph_chunk_bloom_data(struct hashfile *f,\n \t\tsize_t len = filter ? filter->len : 0;\n\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n +@@ commit-graph.c: static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n + \tstop_progress(&ctx->progress);\n + }\n +\n ++static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n ++{\n ++\tstruct json_writer jw = JSON_WRITER_INIT;\n ++\n ++\tjw_object_begin(&jw, 0);\n ++\tjw_object_intmax(&jw, \"filter_found_large\",\n ++\t\t\t ctx->count_bloom_filter_found_large);\n ++\tjw_object_intmax(&jw, \"filter_computed\",\n ++\t\t\t ctx->count_bloom_filter_computed);\n ++\tjw_end(&jw);\n ++\n ++\ttrace2_data_json(\"commit-graph\", the_repository, \"bloom_statistics\", &jw);\n ++\n ++\tjw_release(&jw);\n ++}\n ++\n + static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n + {\n + \tint i;\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n - \t\t\tbitmap_set(ctx->bloom_large, pos);\n - \t\t\tctx->count_bloom_filter_known_large++;\n - \t\t} else {\n --\t\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n -+\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(ctx->r, c, 1, NULL);\n - \t\t\tif (!filter->len) {\n - \t\t\t\tbitmap_set(ctx->bloom_large, pos);\n - \t\t\t\tctx->count_bloom_filter_found_large++;\n + \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n +\n + \tfor (i = 0; i < ctx->commits.nr; i++) {\n ++\t\tint computed = 0;\n + \t\tstruct commit *c = sorted_commits[i];\n +-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n ++\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n ++\t\t\tctx->r,\n ++\t\t\tc,\n ++\t\t\t1,\n ++\t\t\t&computed);\n ++\t\tif (computed) {\n ++\t\t\tctx->count_bloom_filter_computed++;\n ++\t\t\tif (filter && !filter->len)\n ++\t\t\t\tctx->count_bloom_filter_found_large++;\n ++\t\t}\n + \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n + \t\tdisplay_progress(progress, i + 1);\n + \t}\n +\n ++\tif (trace2_is_enabled())\n ++\t\ttrace2_bloom_filter_write_statistics(ctx);\n ++\n + \tfree(sorted_commits);\n + \tstop_progress(&progress);\n + }\n\n ## line-log.c ##\n @@ line-log.c: static int bloom_filter_check(struct rev_info *rev,\n @@ t/helper/test-bloom.c: static void get_bloom_filter_for_commit(const struct obje\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n -\tfilter = get_bloom_filter(the_repository, c, 1);\n -+\tfilter = get_or_compute_bloom_filter(the_repository, c, 1, NULL);\n ++\tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n ++\t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n\n -: ---------- > 18: 4deb724fc1 bloom: use provided 'struct bloom_filter_settings'\n -: ---------- > 19: d1c4bbcaa9 bloom/diff: properly short-circuit on max_changes\n 6: b31c60d712 ! 20: e92ccafcf7 commit-graph.c: sort index into commits list\n @@ Commit message\n (commit-graph: examine commits by generation number, 2020-03-30) for\n details).\n\n - The subsequent patch will want to recover the new graph position of each\n + A future patch will want to recover the new graph position of each\n commit. Since the 'packed_commit_list' already stores a double-pointer,\n avoid a 'COPY_ARRAY' and instead keep track of an index into the\n original list. (Use an integer index instead of a memory address, since\n @@ commit-graph.c: struct tree *get_commit_tree_in_graph(struct repository *r, cons\n -\tint nr;\n -\tint alloc;\n -};\n --\n +\n struct packed_oid_list {\n \tstruct object_id *list;\n - \tint nr;\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_cont\n +\t\t&ctx->commits);\n\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n + \t\tint computed = 0;\n -\t\tstruct commit *c = sorted_commits[i];\n +\t\tint pos = sorted_commits[i];\n +\t\tstruct commit *c = ctx->commits.list[pos];\n - \t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n - \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n - \t\tdisplay_progress(progress, i + 1);\n + \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n + \t\t\tctx->r,\n + \t\t\tc,\n -: ---------- > 21: c42d678714 csum-file.h: introduce 'hashwrite_be64()'\n 7: ab78907233 ! 22: 100b26d7c8 commit-graph: add large-filters bitmap chunk\n @@ Metadata\n ## Commit message ##\n commit-graph: add large-filters bitmap chunk\n\n - When a commit has 512 changed paths or greater, the commit-graph\n - machinery represents it as a zero-length filter since having many\n - entries in the Bloom filter has undesirable effects on the false\n - positivity rate.\n + When a commit has more than a certain number of changed paths (commonly\n + 512), the commit-graph machinery represents it as a zero-length filter.\n + This is done since having many entries in the Bloom filter has\n + undesirable effects on the false positivity rate.\n\n In addition to these too-large filters, the commit-graph machinery also\n represents commits with no filter and commits with no changed paths in\n @@ Commit message\n recomputing large filters over and over again.\n\n This is already undesirable, since it means that we are wasting\n - considerable effort to discover that a commit has at least 512 changed\n + considerable effort to discover that a commit with too many changed\n paths, only to throw that effort away (and then repeat the process the\n next time a roll-up is performed).\n\n - In a subsequent patch, we will add a '--max-new-filters' option, which\n - specifies an upper-bound on the number of new filters we are willing to\n - compute from scratch. Suppose that there are 'N' too-large filters, and\n - we specify '--max-new-filters=M'. If 'N >= M', it is unlikely that any\n - filters will be generated, since we'll spend most of our effort on\n - filters that we ultimately throw away. If 'N < M', filters will trickle\n - in over time, but only at most 'M - N' per-write.\n + In a subsequent patch, we will add a '--max-new-filters=' option,\n + which specifies an upper-bound on the number of new filters we are\n + willing to compute from scratch. Suppose that there are 'N' too-large\n + filters, and we specify '--max-new-filters=M'. If 'N >= M', it is\n + unlikely that any filters will be generated, since we'll spend most of\n + our effort on filters that we ultimately throw away. If 'N < M', filters\n + will trickle in over time, but only at most 'M - N' per-write.\n\n To address this, add a new chunk which encodes a bitmap where the ith\n bit is on iff the ith commit has zero or at least 512 changed paths.\n - When computing Bloom filters, first consult the relevant bitmap (in the\n - case that we are rolling up existing layers) to see if computing the\n - Bloom filter from scratch would be a waste of time.\n + Likewise store the maximum number of changed paths we are willing to\n + store in order to prepare for eventually making this value more easily\n + customizable. When computing Bloom filters, first consult the relevant\n + bitmap (in the case that we are rolling up existing layers) to see if\n + computing the Bloom filter from scratch would be a waste of time.\n\n This patch implements a new chunk instead of extending the existing BIDX\n and BDAT chunks because modifying these chunks would confuse old\n - clients. For example, consider setting the most-significant bit in an\n - entry in the BIDX chunk to indicate that that filter is too-large. New\n - clients would understand how to interpret this, but old clients would\n - treat that entry as a really large filter.\n + clients. (Eg., setting the most-significant bit in the BIDX chunk would\n + confuse old clients and require a version bump).\n +\n + To allow using the existing bitmap code with 64-bit words, we write the\n + data in network byte order from the 64-bit words. This means we also\n + need to read the array from the commit-graph file by translating each\n + word from network byte order using get_be64() upon first use of the\n + bitmap. This is only used when writing the commit-graph, so this is a\n + relatively small operation compared to the other writing code.\n\n By avoiding the need to move to new versions of the BDAT and BIDX chunk,\n we can give ourselves more time to consider whether or not other\n @@ Commit message\n write a bitmap, but forces older clients to rewrite their commit-graphs\n (as well as reduces the theoretical largest Bloom filters we couldl\n write, and forces us to maintain the code necessary to translate BIDX\n - chunks to BID2 ones). Separately from this patch, I implemented this\n + chunks to BID2 ones). Separately from this patch, I implemented this\n alternate approach and did not find it to be advantageous.\n\n Signed-off-by: Taylor Blau \n @@ Documentation/technical/commit-graph-format.txt: CHUNK DATA:\n * The BDAT chunk is present if and only if BIDX is present.\n\n + Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n -+ * It contains a list of 'eword_t's (the length of this list is determined by\n -+ the width of the chunk) which is a bitmap. The 'i'th bit is set exactly\n -+ when the 'i'th commit in the graph has a changed-path Bloom filter with\n -+ zero entries (either because the commit is empty, or because it contains\n -+ more than 512 changed paths).\n -+ * The BFXL chunk is present on newer versions of Git iff the BIDX and BDAT\n -+ chunks are also present.\n ++ * It starts with a 32-bit unsigned integer specifying the maximum number of\n ++ changed-paths that can be stored in a single Bloom filter.\n ++ * It then contains a list of 64-bit words (the length of this list is\n ++ determined by the width of the chunk) which is a bitmap. The 'i'th bit is\n ++ set exactly when the 'i'th commit in the graph has a changed-path Bloom\n ++ filter with zero entries (either because the commit is empty, or because\n ++ it contains more than 512 changed paths).\n ++ * The BFXL chunk is present only when the BIDX and BDAT chunks are\n ++ also present.\n ++\n +\n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n This list of H-byte hashes describe a set of B commit-graph files that\n form a commit-graph chain. The graph position for the ith commit in this\n\n + ## bloom.h ##\n +@@ bloom.h: struct bloom_filter_settings {\n + \t * The maximum number of changed paths per commit\n + \t * before declaring a Bloom filter to be too-large.\n + \t *\n +-\t * Not written to the commit-graph file.\n ++\t * Written to the 'BFXL' chunk (instead of 'BDAT').\n + \t */\n + \tuint32_t max_changed_paths;\n + };\n +\n ## commit-graph.c ##\n @@ commit-graph.c: void git_test_write_commit_graph_or_die(void)\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\tbreak;\n +\n +\t\tcase GRAPH_CHUNKID_BLOOMLARGE:\n -+\t\t\tif (graph->bloom_large.word_alloc)\n ++\t\t\tif (graph->chunk_bloom_large_filters)\n +\t\t\t\tchunk_repeated = 1;\n +\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n -+\t\t\t\tuint32_t alloc = get_be64(chunk_lookup + 4) - chunk_offset;\n ++\t\t\t\tgraph->bloom_large_to_alloc = get_be64(chunk_lookup + 4)\n ++\t\t\t\t\t\t\t - chunk_offset - sizeof(uint32_t);\n +\n -+\t\t\t\tgraph->bloom_large.word_alloc = alloc;\n -+\t\t\t\tgraph->bloom_large.words = (eword_t *)(data + chunk_offset);\n ++\t\t\t\tgraph->bloom_large.word_alloc = 0; /* populate when necessary */\n ++\t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n ++\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n +\t\t\t}\n +\t\t\tbreak;\n \t\t}\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t/* We need both the bloom chunks to exist together. Else ignore the data */\n \t\tgraph->chunk_bloom_indexes = NULL;\n \t\tgraph->chunk_bloom_data = NULL;\n -+\t\tgraph->bloom_large.words = NULL;\n -+\t\tgraph->bloom_large.word_alloc = 0;\n ++\t\tgraph->chunk_bloom_large_filters = NULL;\n \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n \t}\n\n @@ commit-graph.c: struct tree *get_commit_tree_in_graph(struct repository *r, cons\n +\twhile (g && graph_pos < g->num_commits_in_base)\n +\t\tg = g->base_graph;\n +\n -+\tif (!(g && g->bloom_large.word_alloc))\n ++\tif (!g || !g->bloom_large_to_alloc)\n +\t\treturn 0;\n ++\n ++\tif (!g->bloom_large.word_alloc) {\n ++\t\tsize_t i;\n ++\t\tg->bloom_large.word_alloc = g->bloom_large_to_alloc;\n ++\t\tg->bloom_large.words = xmalloc(g->bloom_large_to_alloc * sizeof(eword_t));\n ++\n ++\t\tfor (i = 0; i < g->bloom_large_to_alloc; i++)\n ++\t\t\tg->bloom_large.words[i] = get_be64(g->chunk_bloom_large_filters\n ++\t\t\t\t\t\t\t + i * sizeof(eword_t));\n ++\t}\n ++\n +\treturn bitmap_get(&g->bloom_large, graph_pos - g->num_commits_in_base);\n +}\n -+\n +\n struct packed_oid_list {\n \tstruct object_id *list;\n - \tint nr;\n @@ commit-graph.c: struct write_commit_graph_context {\n - \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n -+\tstruct bitmap *bloom_large;\n -+\n +\n +\tint count_bloom_filter_known_large;\n -+\tint count_bloom_filter_found_large;\n -+\tint count_bloom_filter_computed;\n + \tint count_bloom_filter_found_large;\n + \tint count_bloom_filter_computed;\n ++\tstruct bitmap *bloom_large;\n };\n\n static int write_graph_chunk_fanout(struct hashfile *f,\n @@ commit-graph.c: static int write_graph_chunk_bloom_data(struct hashfile *f,\n }\n\n +static int write_graph_chunk_bloom_large(struct hashfile *f,\n -+\t\t\t\t\t struct write_commit_graph_context *ctx)\n ++\t\t\t\t\t struct write_commit_graph_context *ctx)\n +{\n ++\tsize_t i, alloc = ctx->commits.nr / BITS_IN_EWORD;\n ++\tif (ctx->commits.nr % BITS_IN_EWORD)\n ++\t\talloc++;\n ++\tif (alloc > ctx->bloom_large->word_alloc)\n ++\t\tBUG(\"write_graph_chunk_bloom_large: bitmap not large enough\");\n ++\n +\ttrace2_region_enter(\"commit-graph\", \"bloom_large\", ctx->r);\n -+\thashwrite(f, ctx->bloom_large->words, ctx->bloom_large->word_alloc * sizeof(eword_t));\n ++\thashwrite_be32(f, ctx->bloom_settings->max_changed_paths);\n ++\tfor (i = 0; i < ctx->bloom_large->word_alloc; i++)\n ++\t\thashwrite_be64(f, ctx->bloom_large->words[i]);\n +\ttrace2_region_leave(\"commit-graph\", \"bloom_large\", ctx->r);\n +\treturn 0;\n +}\n @@ commit-graph.c: static int write_graph_chunk_bloom_data(struct hashfile *f,\n static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n -@@ commit-graph.c: static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n - \tstop_progress(&ctx->progress);\n - }\n +@@ commit-graph.c: static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n + \tstruct json_writer jw = JSON_WRITER_INIT;\n\n -+static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n -+{\n -+\tstruct json_writer jw = JSON_WRITER_INIT;\n -+\n -+\tjw_object_begin(&jw, 0);\n + \tjw_object_begin(&jw, 0);\n +\tjw_object_intmax(&jw, \"filter_known_large\",\n +\t\t\t ctx->count_bloom_filter_known_large);\n -+\tjw_object_intmax(&jw, \"filter_found_large\",\n -+\t\t\t ctx->count_bloom_filter_found_large);\n -+\tjw_object_intmax(&jw, \"filter_computed\",\n -+\t\t\t ctx->count_bloom_filter_computed);\n -+\tjw_end(&jw);\n -+\n -+\ttrace2_data_json(\"commit-graph\", the_repository, \"bloom_statistics\", &jw);\n -+\n -+\tjw_release(&jw);\n -+}\n -+\n - static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n - {\n - \tint i;\n + \tjw_object_intmax(&jw, \"filter_found_large\",\n + \t\t\t ctx->count_bloom_filter_found_large);\n + \tjw_object_intmax(&jw, \"filter_computed\",\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint *sorted_commits;\n\n \tinit_bloom_filters();\n -+\tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD);\n ++\tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n\n \tif (ctx->report_progress)\n \t\tprogress = start_delayed_progress(\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n + \t\t&ctx->commits);\n +\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n +-\t\tint computed = 0;\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n --\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n --\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n +-\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n +-\t\t\tctx->r,\n +-\t\t\tc,\n +-\t\t\t1,\n +-\t\t\tctx->bloom_settings,\n +-\t\t\t&computed);\n +-\t\tif (computed) {\n +-\t\t\tctx->count_bloom_filter_computed++;\n +-\t\t\tif (filter && !filter->len)\n +-\t\t\t\tctx->count_bloom_filter_found_large++;\n +\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n +\t\t\tbitmap_set(ctx->bloom_large, pos);\n +\t\t\tctx->count_bloom_filter_known_large++;\n +\t\t} else {\n -+\t\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n -+\t\t\tif (!filter->len) {\n -+\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n -+\t\t\t\tctx->count_bloom_filter_found_large++;\n ++\t\t\tint computed = 0;\n ++\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n ++\t\t\t\tctx->r,\n ++\t\t\t\tc,\n ++\t\t\t\t1,\n ++\t\t\t\tctx->bloom_settings,\n ++\t\t\t\t&computed);\n ++\t\t\tif (computed) {\n ++\t\t\t\tctx->count_bloom_filter_computed++;\n ++\t\t\t\tif (filter && !filter->len) {\n ++\t\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n ++\t\t\t\t\tctx->count_bloom_filter_found_large++;\n ++\t\t\t\t}\n +\t\t\t}\n +\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n -+\t\t\tctx->count_bloom_filter_computed++;\n -+\t\t}\n + \t\t}\n +-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n\n -+\tif (trace2_is_enabled())\n -+\t\ttrace2_bloom_filter_write_statistics(ctx);\n -+\n - \tfree(sorted_commits);\n - \tstop_progress(&progress);\n - }\n @@ commit-graph.c: static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n \t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n +\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMLARGE;\n -+\t\tchunks[num_chunks].size = sizeof(eword_t) * ctx->bloom_large->word_alloc;\n ++\t\tchunks[num_chunks].size = sizeof(eword_t) * ctx->bloom_large->word_alloc\n ++\t\t\t\t\t+ sizeof(uint32_t);\n +\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_large;\n +\t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n +@@ commit-graph.c: void free_commit_graph(struct commit_graph *g)\n + \t}\n + \tfree(g->filename);\n + \tfree(g->bloom_filter_settings);\n ++\tbitmap_free(g->bloom_large);\n + \tfree(g);\n + }\n +\n\n ## commit-graph.h ##\n @@\n @@ commit-graph.h: struct commit_graph {\n \tconst unsigned char *chunk_base_graphs;\n \tconst unsigned char *chunk_bloom_indexes;\n \tconst unsigned char *chunk_bloom_data;\n ++\tconst unsigned char *chunk_bloom_large_filters;\n ++\n ++\tsize_t bloom_large_to_alloc;\n +\tstruct bitmap bloom_large;\n\n \tstruct bloom_filter_settings *bloom_filter_settings;\n @@ t/t4216-log-bloom.sh: test_expect_success 'correctly report changes over limit'\n +\n +test_expect_success 'Bloom generation does not recompute too-large filters' '\n +\t(\n -+\t\tcd 513changes &&\n -+\t\tgit commit-graph write --reachable --changed-paths \\\n ++\t\tcd limits &&\n ++\n ++\t\t# start from scratch and rebuild\n ++\t\trm -f .git/objects/info/commit-graph &&\n ++\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n ++\t\t\tgit commit-graph write --reachable --changed-paths \\\n +\t\t\t--split=replace &&\n +\t\ttest_commit c1 filter &&\n ++\n +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n -+\t\t\t1 0 1\n ++\t\t\t2 0 1\n +\t)\n +'\n\n 9: 309e76bb17 ! 23: 2ee0b84351 commit-graph: rename 'split_commit_graph_opts'\n @@ Metadata\n ## Commit message ##\n commit-graph: rename 'split_commit_graph_opts'\n\n - In a subsequent commit, additional options will be added to the\n - commit-graph API, which themselves do not have to do with splitting.\n + In the subsequent commit, additional options will be added to the\n + commit-graph API which have nothing to do with splitting.\n\n Rename the 'split_commit_graph_opts' structure to the more-generic\n 'commit_graph_opts' to encompass both.\n @@ commit-graph.c: struct write_commit_graph_context {\n +\tconst struct commit_graph_opts *opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n - \tstruct bitmap *bloom_large;\n +\n @@ commit-graph.c: static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n @@ commit-graph.c: int write_commit_graph(struct object_directory *odb,\n +\tctx->opts = opts;\n \tctx->total_bloom_filter_data_size = 0;\n\n - \tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n + \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n @@ commit-graph.c: int write_commit_graph(struct object_directory *odb,\n \t\t\t}\n \t\t}\n10: 8b670571dc ! 24: 3b66ae4a9c builtin/commit-graph.c: introduce '--max-new-filters='\n @@ Commit message\n 'max_new_filters' variable introduced by the previous patch.\n\n The command-line option '--max-new-filters' takes precedence over\n - 'graph.maxNewFilters', which is the default value.\n + 'commitGraph.maxNewFilters', which is the default value.\n '--no-max-new-filters' can also be provided, which sets the value back\n to '-1', indicating that an unlimited number of new Bloom filters may be\n generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n @@ Commit message\n\n ## Documentation/config/commitgraph.txt ##\n @@\n -+commitgraph.maxNewFilters::\n ++commitGraph.maxNewFilters::\n +\tSpecifies the default value for the `--max-new-filters` option of `git\n +\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n +\n - commitgraph.readChangedPaths::\n + commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\n\n @@ Documentation/git-commit-graph.txt: this option is given, future commit-graph wr\n +\n +With the `--max-new-filters=` option, generate at most `n` new Bloom\n +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n -+enforced. Overrides the `graph.maxNewFilters` configuration.\n ++enforced. Overrides the `commitGraph.maxNewFilters` configuration.\n ++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n @@ builtin/commit-graph.c: static int graph_write(int argc, const char **argv)\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n +\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n -+\t\t\tNULL, N_(\"maximum number of Bloom filters to compute\"),\n ++\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n +\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n @@ builtin/commit-graph.c: static int graph_write(int argc, const char **argv)\n\n +static int git_commit_graph_config(const char *var, const char *value, void *cb)\n +{\n -+\tif (!strcmp(var, \"graph.maxnewfilters\")) {\n ++\tif (!strcmp(var, \"commitgraph.maxnewfilters\")) {\n +\t\twrite_opts.max_new_filters = git_config_int(var, value);\n +\t\treturn 0;\n +\t}\n @@ builtin/commit-graph.c: int cmd_commit_graph(int argc, const char **argv, const\n \t\t\t builtin_commit_graph_usage,\n\n ## commit-graph.c ##\n +@@ commit-graph.c: struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n + }\n +\n + static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n +-\t\t\t\t\t const struct commit *c)\n ++\t\t\t\t\t const struct commit *c,\n ++\t\t\t\t\t uint32_t max_changed_paths)\n + {\n + \tuint32_t graph_pos = commit_graph_position(c);\n + \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_cont\n +\tint max_new_filters;\n\n \tinit_bloom_filters();\n - \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD);\n + \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n \t\t&ctx->commits);\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_cont\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n -@@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n +-\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n ++\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph,\n ++\t\t\t\t\t\t c,\n ++\t\t\t\t\t\t ctx->bloom_settings->max_changed_paths)) {\n \t\t\tbitmap_set(ctx->bloom_large, pos);\n \t\t\tctx->count_bloom_filter_known_large++;\n \t\t} else {\n --\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(ctx->r, c, 1, NULL);\n --\t\t\tif (!filter->len) {\n --\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n --\t\t\t\tctx->count_bloom_filter_found_large++;\n -+\t\t\tint computed = 0;\n -+\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n -+\t\t\t\tctx->r, c,\n +@@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n + \t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n + \t\t\t\tctx->r,\n + \t\t\t\tc,\n +-\t\t\t\t1,\n +\t\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n -+\t\t\t\t&computed);\n -+\t\t\tif (computed) {\n -+\t\t\t\tctx->count_bloom_filter_computed++;\n -+\t\t\t\tif (filter && !filter->len) {\n -+\t\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n -+\t\t\t\t\tctx->count_bloom_filter_found_large++;\n -+\t\t\t\t}\n + \t\t\t\tctx->bloom_settings,\n + \t\t\t\t&computed);\n + \t\t\tif (computed) {\n +@@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n + \t\t\t\t\tctx->count_bloom_filter_found_large++;\n + \t\t\t\t}\n \t\t\t}\n -\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n --\t\t\tctx->count_bloom_filter_computed++;\n -+\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * (filter ? filter->len : 0);\n ++\t\t\tif (filter)\n ++\t\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\t}\n \t\tdisplay_progress(progress, i + 1);\n \t}\n @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation does not recompute t\n\n +test_expect_success 'Bloom generation is limited by --max-new-filters' '\n +\t(\n -+\t\tcd 513changes &&\n ++\t\tcd limits &&\n +\t\ttest_commit c2 filter &&\n +\t\ttest_commit c3 filter &&\n +\t\ttest_commit c4 no-filter &&\n +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n -+\t\t\t1 0 2\n ++\t\t\t2 0 2\n +\t)\n +'\n +\n +test_expect_success 'Bloom generation backfills previously-skipped filters' '\n +\t(\n -+\t\tcd 513changes &&\n ++\t\tcd limits &&\n +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=1\" \\\n -+\t\t\t1 0 1\n ++\t\t\t2 0 1\n +\t)\n +'\n +\n--\n2.28.0.rc1.13.ge78abce653\n"},{"id":"402924","messageId":"053991f0482b248c3c5fbaf604edfab796ac2540.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 06/14] commit-graph.c: store maximum changed paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:35Z","receivedAt":"2020-08-05T17:07:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For now, we assume that there is a fixed constant describing the\nmaximum number of changed paths we are willing to store in a Bloom\nfilter.\n\nPrepare for that to (at least partially) not be the case by making it a\nmember of the 'struct bloom_filter_settings'. This will be helpful in\nthe subsequent patches by reducing the size of test cases that exercise\nstoring too many changed paths, as well as preparing for an eventual\nfuture in which this value might change.\n\nThis patch alone does not cause newly generated Bloom filters to use\na custom upper-bound on the maximum number of changed paths a single\nBloom filter can hold, that will occur in a later patch.\n\nSigned-off-by: Taylor Blau \n---\n bloom.h | 11 ++++++++++-\n commit-graph.c | 3 +++\n t/t4216-log-bloom.sh | 4 ++--\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.h b/bloom.h\nindex d8fbb0fbf1..0b9b59a6fe 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -28,9 +28,18 @@ struct bloom_filter_settings {\n \t * that contain n*b bits.\n \t */\n \tuint32_t bits_per_entry;\n+\n+\t/*\n+\t * The maximum number of changed paths per commit\n+\t * before declaring a Bloom filter to be too-large.\n+\t *\n+\t * Not written to the commit-graph file.\n+\t */\n+\tuint32_t max_changed_paths;\n };\n \n-#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10 }\n+#define DEFAULT_BLOOM_MAX_CHANGES 512\n+#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10, DEFAULT_BLOOM_MAX_CHANGES }\n #define BITS_PER_WORD 8\n #define BLOOMDATA_CHUNK_HEADER_SIZE 3 * sizeof(uint32_t)\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex a516e93d71..86dd4b979e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1194,6 +1194,7 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n \tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n \tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_object_intmax(&jw, \"max_changed_paths\", ctx->bloom_settings->max_changed_paths);\n \tjw_end(&jw);\n \n \ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n@@ -1662,6 +1663,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n \t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n \t\t\t\t\t\t\t bloom_settings.num_hashes);\n+\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n \t\tctx->bloom_settings = &bloom_settings;\n \t}\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex b3d1f596f8..eb2bcc51f0 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -169,11 +169,11 @@ test_expect_success 'persist filter settings' '\n \t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n \t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n \t\tGIT_TRACE2_EVENT_NESTING=5 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2-auto.txt\n '\n \n test_expect_success 'correctly report changes over limit' '\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402925","messageId":"038e996ced8b7c6418894a791caeb7f80b2456be.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 04/14] t/helper/test-read-graph.c: prepare repo settings","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:28Z","receivedAt":"2020-08-05T17:07:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The read-graph test-tool is used by a number of the commit-graph test to\nassert various properties about a commit-graph. Previously, this program\nnever ran 'prepare_repo_settings()'. There was no need to do so, since\nnone of the commit-graph machinery is affected by the repo settings.\n\nIn the next patch, the commit-graph machinery's behavior will become\ndependent on the repo settings, and so loading them before running the\nrest of the test tool is critical.\n\nAs such, teach the test tool to call 'prepare_repo_settings()'.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 6d0c962438..5f585a1725 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -12,11 +12,12 @@ int cmd__read_graph(int argc, const char **argv)\n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n \n+\tprepare_repo_settings(the_repository);\n+\n \tgraph = read_commit_graph_one(the_repository, odb);\n \tif (!graph)\n \t\treturn 1;\n \n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402926","messageId":"404f10319af210a2edf05bd1097ab5a844cc5c37.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 05/14] commit-graph: respect 'commitGraph.readChangedPaths'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:02:31Z","receivedAt":"2020-08-05T17:07:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Git uses the 'core.commitGraph' configuration value to control whether\nor not the commit graph is used when parsing commits or performing a\ntraversal.\n\nNow that commit-graphs can also contain a section for changed-path Bloom\nfilters, administrators that already have commit-graphs may find it\nconvenient to use those graphs without relying on their changed-path\nBloom filters. This can happen, for example, during a staged roll-out,\nor in the event of an incident.\n\nIntroduce 'commitGraph.readChangedPaths' to control whether or not Bloom\nfilters are read. Note that this configuration is independent from both:\n\n - 'core.commitGraph', to allow flexibility in using all parts of a\n commit-graph _except_ for its Bloom filters.\n\n - The '--changed-paths' option for 'git commit-graph write', to allow\n reading and writing Bloom filters to be controlled independently.\n\nWhen the variable is set, pretend as if no Bloom data was specified at\nall. This avoids adding additional special-casing outside of the\ncommit-graph internals.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 ++\n Documentation/config/commitgraph.txt | 4 ++++\n commit-graph.c | 6 ++++--\n repo-settings.c | 3 +++\n repository.h | 1 +\n t/t4216-log-bloom.sh | 4 +++-\n 6 files changed, 17 insertions(+), 3 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ef0768b91a..78883c6e63 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -340,6 +340,8 @@ include::config/column.txt[]\n \n include::config/commit.txt[]\n \n+include::config/commitgraph.txt[]\n+\n include::config/credential.txt[]\n \n include::config/completion.txt[]\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nnew file mode 100644\nindex 0000000000..cff0797b54\n--- /dev/null\n+++ b/Documentation/config/commitgraph.txt\n@@ -0,0 +1,4 @@\n+commitGraph.readChangedPaths::\n+\tIf true, then git will use the changed-path Bloom filters in the\n+\tcommit-graph file (if it exists, and they are present). Defaults to\n+\ttrue. See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0c1030641c..a516e93d71 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -320,6 +320,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n+\tprepare_repo_settings(r);\n+\n \tgraph = alloc_commit_graph();\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -396,14 +398,14 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n \t\t\tif (graph->chunk_bloom_indexes)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n \t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n \t\t\tbreak;\n \n \t\tcase GRAPH_CHUNKID_BLOOMDATA:\n \t\t\tif (graph->chunk_bloom_data)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n \t\t\t\tuint32_t hash_version;\n \t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n \t\t\t\thash_version = get_be32(data + chunk_offset);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 0918408b34..9e551bc03d 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -17,9 +17,12 @@ void prepare_repo_settings(struct repository *r)\n \n \tif (!repo_config_get_bool(r, \"core.commitgraph\", &value))\n \t\tr->settings.core_commit_graph = value;\n+\tif (!repo_config_get_bool(r, \"commitgraph.readchangedpaths\", &value))\n+\t\tr->settings.commit_graph_read_changed_paths = value;\n \tif (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n \t\tr->settings.gc_write_commit_graph = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_commit_graph, 1);\n+\tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n \n \tif (!repo_config_get_int(r, \"index.version\", &value))\ndiff --git a/repository.h b/repository.h\nindex 3c1f7d54bd..81759b7d27 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -29,6 +29,7 @@ struct repo_settings {\n \tint initialized;\n \n \tint core_commit_graph;\n+\tint commit_graph_read_changed_paths;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fe19f6a60c..b3d1f596f8 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -90,7 +90,9 @@ do\n \t\t \"--ancestry-path side..master\"\n \tdo\n \t\ttest_expect_success \"git log option: $option for path: $path\" '\n-\t\t\ttest_bloom_filters_used \"$option -- $path\"\n+\t\t\ttest_bloom_filters_used \"$option -- $path\" &&\n+\t\t\ttest_config commitgraph.readChangedPaths false &&\n+\t\t\ttest_bloom_filters_not_used \"$option -- $path\"\n \t\t'\n \tdone\n done\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402927","messageId":"3b66ae4a9c2aa97dd64f88904ad2bf2756ccd9ef.1596646576.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596646576.git.me@ttaylorr.com","subject":"[PATCH v2 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T17:03:05Z","receivedAt":"2020-08-05T17:10:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a command-line flag and configuration variable to fill in the\n'max_new_filters' variable introduced by the previous patch.\n\nThe command-line option '--max-new-filters' takes precedence over\n'commitGraph.maxNewFilters', which is the default value.\n'--no-max-new-filters' can also be provided, which sets the value back\nto '-1', indicating that an unlimited number of new Bloom filters may be\ngenerated. (OPT_INTEGER only allows setting the '--no-' variant back to\n'0', hence a custom callback was used instead).\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 4 +++\n Documentation/git-commit-graph.txt | 4 +++\n bloom.c | 15 +++++++++++\n builtin/commit-graph.c | 39 +++++++++++++++++++++++++---\n commit-graph.c | 16 +++++++++---\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 19 ++++++++++++++\n 7 files changed, 91 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex cff0797b54..4582c39fc4 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitGraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..9c887d5d79 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,10 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Overrides the `commitGraph.maxNewFilters` configuration.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex ed54e96e57..d0c0fd049d 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -51,6 +51,21 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \telse\n \t\tstart_index = 0;\n \n+\tif ((start_index == end_index) &&\n+\t (g->bloom_large.word_alloc && !bitmap_get(&g->bloom_large, lex_pos))) {\n+\t\t/*\n+\t\t * If the filter is zero-length, either (1) the filter has no\n+\t\t * changes, (2) the filter has too many changes, or (3) it\n+\t\t * wasn't computed (eg., due to '--max-new-filters').\n+\t\t *\n+\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n+\t\t * for this Bloom filter. If it is unset, then it wasn't\n+\t\t * computed. In that case, return nothing, since we don't have\n+\t\t * that filter in the graph.\n+\t\t */\n+\t\treturn 0;\n+\t}\n+\n \tfilter->len = end_index - start_index;\n \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n \t\t\t\t\tsizeof(unsigned char) * start_index +\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 38f5f57d15..3500a6e1f1 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n \n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n \n@@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -270,6 +293,16 @@ static int graph_write(int argc, const char **argv)\n \treturn result;\n }\n \n+static int git_commit_graph_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, \"commitgraph.maxnewfilters\")) {\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_commit_graph_options[] = {\n@@ -283,7 +316,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_commit_graph_usage,\n \t\t\t\t builtin_commit_graph_options);\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(git_commit_graph_config, &opts);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_commit_graph_options,\n \t\t\t builtin_commit_graph_usage,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 82fca07579..76b1238262 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -948,7 +948,8 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n }\n \n static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n-\t\t\t\t\t const struct commit *c)\n+\t\t\t\t\t const struct commit *c,\n+\t\t\t\t\t uint32_t max_changed_paths)\n {\n \tuint32_t graph_pos = commit_graph_position(c);\n \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n@@ -1475,6 +1476,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tint *sorted_commits;\n+\tint max_new_filters;\n \n \tinit_bloom_filters();\n \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n@@ -1491,10 +1493,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n \t\t&ctx->commits);\n \n+\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n-\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n+\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph,\n+\t\t\t\t\t\t c,\n+\t\t\t\t\t\t ctx->bloom_settings->max_changed_paths)) {\n \t\t\tbitmap_set(ctx->bloom_large, pos);\n \t\t\tctx->count_bloom_filter_known_large++;\n \t\t} else {\n@@ -1502,7 +1509,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\t\tctx->r,\n \t\t\t\tc,\n-\t\t\t\t1,\n+\t\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n \t\t\t\tctx->bloom_settings,\n \t\t\t\t&computed);\n \t\t\tif (computed) {\n@@ -1512,7 +1519,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\t\tctx->count_bloom_filter_found_large++;\n \t\t\t\t}\n \t\t\t}\n-\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\t\tif (filter)\n+\t\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\t}\n \t\tdisplay_progress(progress, i + 1);\n \t}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 1d147b7b76..47d99ea4bf 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -115,6 +115,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags flags;\n+\tint max_new_filters;\n };\n \n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 6859d85369..3aab8ffbe3 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -286,4 +286,23 @@ test_expect_success 'Bloom generation does not recompute too-large filters' '\n \t)\n '\n \n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n+\t\t\t2 0 2\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=1\" \\\n+\t\t\t2 0 1\n+\t)\n+'\n+\n test_done\n-- \n2.28.0.rc1.13.ge78abce653\n"},{"id":"402967","messageId":"xmqqbljoolhy.fsf@gitster.c.googlers.com","threadId":"53975","inReplyTo":"100b26d7c8a5e7ce21a950f33791eadab74e8e70.1596646576.git.me@ttaylorr.com","subject":"Re: [PATCH v2 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-05T21:01:29Z","receivedAt":"2020-08-05T21:01:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> @@ -71,6 +72,10 @@ struct commit_graph {\n> \tconst unsigned char *chunk_base_graphs;\n> \tconst unsigned char *chunk_bloom_indexes;\n> \tconst unsigned char *chunk_bloom_data;\n> +\tconst unsigned char *chunk_bloom_large_filters;\n> +\n> +\tsize_t bloom_large_to_alloc;\n> +\tstruct bitmap bloom_large;\n\nHmph, is the API rich enough to allow users to release the resource\nused by such an embedded bitmap? I ask becuase...\n\n> @@ -2503,6 +2577,7 @@ void free_commit_graph(struct commit_graph *g)\n> \t}\n> \tfree(g->filename);\n> \tfree(g->bloom_filter_settings);\n> +\tbitmap_free(g->bloom_large);\n> \tfree(g);\n> }\n\n... this hunk cannot be possibly correct as-is, and cannot be made\ncorrect without changing g->bloom_large to a pointer into a heap\nallocated bitmap, because bitmap_free() wants to not just release\nthe resource held by the bitmap but the bitmap itself.\n"},{"id":"402971","messageId":"20200805211730.GJ9546@syl.lan","threadId":"53975","inReplyTo":"xmqqbljoolhy.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T21:17:30Z","receivedAt":"2020-08-05T21:17:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 05, 2020 at 02:01:29PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > @@ -71,6 +72,10 @@ struct commit_graph {\n> > \tconst unsigned char *chunk_base_graphs;\n> > \tconst unsigned char *chunk_bloom_indexes;\n> > \tconst unsigned char *chunk_bloom_data;\n> > +\tconst unsigned char *chunk_bloom_large_filters;\n> > +\n> > +\tsize_t bloom_large_to_alloc;\n> > +\tstruct bitmap bloom_large;\n>\n> Hmph, is the API rich enough to allow users to release the resource\n> used by such an embedded bitmap? I ask becuase...\n>\n> > @@ -2503,6 +2577,7 @@ void free_commit_graph(struct commit_graph *g)\n> > \t}\n> > \tfree(g->filename);\n> > \tfree(g->bloom_filter_settings);\n> > +\tbitmap_free(g->bloom_large);\n> > \tfree(g);\n> > }\n>\n> ... this hunk cannot be possibly correct as-is, and cannot be made\n> correct without changing g->bloom_large to a pointer into a heap\n> allocated bitmap, because bitmap_free() wants to not just release\n> the resource held by the bitmap but the bitmap itself.\n\nYuck, that's definitely wrong. Serves me right for sneaking this in\nafter I had run `git rebase -x 'make -j40 DEVELOPER=1 test'\nupstream/master` ;-).\n\nBelow the scissors line should do the trick. It should apply cleanly at\nthis point in the series, but it'll produce a compilation failure on the\nvery last patch (fixing it is straightforward and looks like the\nfollowing diff):\n\ndiff --git a/bloom.c b/bloom.c\nindex d0c0fd049d..8d07209c6b 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -52,7 +52,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n start_index = 0;\n\n if ((start_index == end_index) &&\n- (g->bloom_large.word_alloc && !bitmap_get(&g->bloom_large, lex_pos))) {\n+ (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n /*\n * If the filter is zero-length, either (1) the filter has no\n * changes, (2) the filter has too many changes, or (3) it\n\nIn either case, this will fix the bad free():\n\n--- >8 ---\n\nSubject: [PATCH] fixup! commit-graph: add large-filters bitmap chunk\n\nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 18 ++++++++++--------\n commit-graph.h | 2 +-\n 2 files changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1fee49d171..add76f1824 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -438,7 +438,10 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\tgraph->bloom_large_to_alloc = get_be64(chunk_lookup + 4)\n \t\t\t\t\t\t\t - chunk_offset - sizeof(uint32_t);\n\n-\t\t\t\tgraph->bloom_large.word_alloc = 0; /* populate when necessary */\n+\t\t\t\t/*\n+\t\t\t\t * leave 'bloom_large' uninitialized, and\n+\t\t\t\t * populate when necessary\n+\t\t\t\t */\n \t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n \t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n \t\t\t}\n@@ -960,17 +963,15 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n \tif (!g || !g->bloom_large_to_alloc)\n \t\treturn 0;\n\n-\tif (!g->bloom_large.word_alloc) {\n+\tif (!g->bloom_large) {\n \t\tsize_t i;\n-\t\tg->bloom_large.word_alloc = g->bloom_large_to_alloc;\n-\t\tg->bloom_large.words = xmalloc(g->bloom_large_to_alloc * sizeof(eword_t));\n-\n+\t\tg->bloom_large = bitmap_word_alloc(g->bloom_large_to_alloc);\n \t\tfor (i = 0; i < g->bloom_large_to_alloc; i++)\n-\t\t\tg->bloom_large.words[i] = get_be64(g->chunk_bloom_large_filters\n-\t\t\t\t\t\t\t + i * sizeof(eword_t));\n+\t\t\tg->bloom_large->words[i] = get_be64(g->chunk_bloom_large_filters\n+\t\t\t\t\t\t\t + i * sizeof(eword_t));\n \t}\n\n-\treturn bitmap_get(&g->bloom_large, graph_pos - g->num_commits_in_base);\n+\treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n }\n\n struct packed_oid_list {\n@@ -2360,6 +2361,7 @@ int write_commit_graph(struct object_directory *odb,\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n+\tfree(ctx->bloom_large);\n\n \tif (ctx->commit_graph_filenames_after) {\n \t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\ndiff --git a/commit-graph.h b/commit-graph.h\nindex f4fb996dd5..b1ab86a3c8 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,7 +75,7 @@ struct commit_graph {\n \tconst unsigned char *chunk_bloom_large_filters;\n\n \tsize_t bloom_large_to_alloc;\n-\tstruct bitmap bloom_large;\n+\tstruct bitmap *bloom_large;\n\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n--\n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"402995","messageId":"xmqqh7tgn37w.fsf@gitster.c.googlers.com","threadId":"53975","inReplyTo":"20200805211730.GJ9546@syl.lan","subject":"Re: [PATCH v2 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-05T22:21:39Z","receivedAt":"2020-08-05T22:21:45Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Wed, Aug 05, 2020 at 02:01:29PM -0700, Junio C Hamano wrote:\n>> Taylor Blau writes:\n>>\n>> > @@ -71,6 +72,10 @@ struct commit_graph {\n>> > \tconst unsigned char *chunk_base_graphs;\n>> > \tconst unsigned char *chunk_bloom_indexes;\n>> > \tconst unsigned char *chunk_bloom_data;\n>> > +\tconst unsigned char *chunk_bloom_large_filters;\n>> > +\n>> > +\tsize_t bloom_large_to_alloc;\n>> > +\tstruct bitmap bloom_large;\n>>\n>> Hmph, is the API rich enough to allow users to release the resource\n>> used by such an embedded bitmap? I ask becuase...\n>>\n>> > @@ -2503,6 +2577,7 @@ void free_commit_graph(struct commit_graph *g)\n>> > \t}\n>> > \tfree(g->filename);\n>> > \tfree(g->bloom_filter_settings);\n>> > +\tbitmap_free(g->bloom_large);\n>> > \tfree(g);\n>> > }\n>>\n>> ... this hunk cannot be possibly correct as-is, and cannot be made\n>> correct without changing g->bloom_large to a pointer into a heap\n>> allocated bitmap, because bitmap_free() wants to not just release\n>> the resource held by the bitmap but the bitmap itself.\n>\n> Yuck, that's definitely wrong. Serves me right for sneaking this in\n> after I had run `git rebase -x 'make -j40 DEVELOPER=1 test'\n> upstream/master` ;-).\n>\n> Below the scissors line should do the trick. It should apply cleanly at\n> this point in the series, but it'll produce a compilation failure on the\n> very last patch (fixing it is straightforward and looks like the\n> following diff):\n>\n> diff --git a/bloom.c b/bloom.c\n> index d0c0fd049d..8d07209c6b 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -52,7 +52,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n> start_index = 0;\n>\n> if ((start_index == end_index) &&\n> - (g->bloom_large.word_alloc && !bitmap_get(&g->bloom_large, lex_pos))) {\n> + (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n> /*\n> * If the filter is zero-length, either (1) the filter has no\n> * changes, (2) the filter has too many changes, or (3) it\n>\n> In either case, this will fix the bad free():\n>\n> --- >8 ---\n>\n> Subject: [PATCH] fixup! commit-graph: add large-filters bitmap chunk\n>\n> Signed-off-by: Taylor Blau \n> ---\n> commit-graph.c | 18 ++++++++++--------\n> commit-graph.h | 2 +-\n> 2 files changed, 11 insertions(+), 9 deletions(-)\n> ...\n> @@ -2360,6 +2361,7 @@ int write_commit_graph(struct object_directory *odb,\n> \tfree(ctx->graph_name);\n> \tfree(ctx->commits.list);\n> \tfree(ctx->oids.list);\n> +\tfree(ctx->bloom_large);\n\nIs this correct, or shouldn't it be bitmap_free() instead?\n"},{"id":"402997","messageId":"20200805222531.GA30420@syl.lan","threadId":"53975","inReplyTo":"xmqqh7tgn37w.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-05T22:25:31Z","receivedAt":"2020-08-05T22:25:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 05, 2020 at 03:21:39PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > On Wed, Aug 05, 2020 at 02:01:29PM -0700, Junio C Hamano wrote:\n> >> Taylor Blau writes:\n> >>\n> >> > @@ -71,6 +72,10 @@ struct commit_graph {\n> >> > \tconst unsigned char *chunk_base_graphs;\n> >> > \tconst unsigned char *chunk_bloom_indexes;\n> >> > \tconst unsigned char *chunk_bloom_data;\n> >> > +\tconst unsigned char *chunk_bloom_large_filters;\n> >> > +\n> >> > +\tsize_t bloom_large_to_alloc;\n> >> > +\tstruct bitmap bloom_large;\n> >>\n> >> Hmph, is the API rich enough to allow users to release the resource\n> >> used by such an embedded bitmap? I ask becuase...\n> >>\n> >> > @@ -2503,6 +2577,7 @@ void free_commit_graph(struct commit_graph *g)\n> >> > \t}\n> >> > \tfree(g->filename);\n> >> > \tfree(g->bloom_filter_settings);\n> >> > +\tbitmap_free(g->bloom_large);\n> >> > \tfree(g);\n> >> > }\n> >>\n> >> ... this hunk cannot be possibly correct as-is, and cannot be made\n> >> correct without changing g->bloom_large to a pointer into a heap\n> >> allocated bitmap, because bitmap_free() wants to not just release\n> >> the resource held by the bitmap but the bitmap itself.\n> >\n> > Yuck, that's definitely wrong. Serves me right for sneaking this in\n> > after I had run `git rebase -x 'make -j40 DEVELOPER=1 test'\n> > upstream/master` ;-).\n> >\n> > Below the scissors line should do the trick. It should apply cleanly at\n> > this point in the series, but it'll produce a compilation failure on the\n> > very last patch (fixing it is straightforward and looks like the\n> > following diff):\n> >\n> > diff --git a/bloom.c b/bloom.c\n> > index d0c0fd049d..8d07209c6b 100644\n> > --- a/bloom.c\n> > +++ b/bloom.c\n> > @@ -52,7 +52,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n> > start_index = 0;\n> >\n> > if ((start_index == end_index) &&\n> > - (g->bloom_large.word_alloc && !bitmap_get(&g->bloom_large, lex_pos))) {\n> > + (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n> > /*\n> > * If the filter is zero-length, either (1) the filter has no\n> > * changes, (2) the filter has too many changes, or (3) it\n> >\n> > In either case, this will fix the bad free():\n> >\n> > --- >8 ---\n> >\n> > Subject: [PATCH] fixup! commit-graph: add large-filters bitmap chunk\n> >\n> > Signed-off-by: Taylor Blau \n> > ---\n> > commit-graph.c | 18 ++++++++++--------\n> > commit-graph.h | 2 +-\n> > 2 files changed, 11 insertions(+), 9 deletions(-)\n> > ...\n> > @@ -2360,6 +2361,7 @@ int write_commit_graph(struct object_directory *odb,\n> > \tfree(ctx->graph_name);\n> > \tfree(ctx->commits.list);\n> > \tfree(ctx->oids.list);\n> > +\tfree(ctx->bloom_large);\n>\n> Is this correct, or shouldn't it be bitmap_free() instead?\n\nAck, that should be 'bitmap_free()'. Double checking, 'bitmap_free' does\nhandle a 'NULL' argument like 'free', so dealing with an old\ncommit-graph lacking this chunk will work fine.\n\nThanks for catching my mistake. I'm off tomorrow, Friday, and Monday,\nso my responses from now on may be intermittent, but I should have some\ntime.\n\nThanks,\nTaylor\n"},{"id":"403353","messageId":"20200811134807.GA26715@syl.lan","threadId":"53975","inReplyTo":"20200805222531.GA30420@syl.lan","subject":"Re: [PATCH v2 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T13:48:07Z","receivedAt":"2020-08-11T13:48:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 05, 2020 at 06:25:31PM -0400, Taylor Blau wrote:\n> On Wed, Aug 05, 2020 at 03:21:39PM -0700, Junio C Hamano wrote:\n> > Taylor Blau writes:\n> >\n> > > On Wed, Aug 05, 2020 at 02:01:29PM -0700, Junio C Hamano wrote:\n> > >> Taylor Blau writes:\n> > >>\n> > >> > @@ -71,6 +72,10 @@ struct commit_graph {\n> > >> > \tconst unsigned char *chunk_base_graphs;\n> > >> > \tconst unsigned char *chunk_bloom_indexes;\n> > >> > \tconst unsigned char *chunk_bloom_data;\n> > >> > +\tconst unsigned char *chunk_bloom_large_filters;\n> > >> > +\n> > >> > +\tsize_t bloom_large_to_alloc;\n> > >> > +\tstruct bitmap bloom_large;\n> > >>\n> > >> Hmph, is the API rich enough to allow users to release the resource\n> > >> used by such an embedded bitmap? I ask becuase...\n> > >>\n> > >> > @@ -2503,6 +2577,7 @@ void free_commit_graph(struct commit_graph *g)\n> > >> > \t}\n> > >> > \tfree(g->filename);\n> > >> > \tfree(g->bloom_filter_settings);\n> > >> > +\tbitmap_free(g->bloom_large);\n> > >> > \tfree(g);\n> > >> > }\n> > >>\n> > >> ... this hunk cannot be possibly correct as-is, and cannot be made\n> > >> correct without changing g->bloom_large to a pointer into a heap\n> > >> allocated bitmap, because bitmap_free() wants to not just release\n> > >> the resource held by the bitmap but the bitmap itself.\n> > >\n> > > Yuck, that's definitely wrong. Serves me right for sneaking this in\n> > > after I had run `git rebase -x 'make -j40 DEVELOPER=1 test'\n> > > upstream/master` ;-).\n> > >\n> > > Below the scissors line should do the trick. It should apply cleanly at\n> > > this point in the series, but it'll produce a compilation failure on the\n> > > very last patch (fixing it is straightforward and looks like the\n> > > following diff):\n> > >\n> > > diff --git a/bloom.c b/bloom.c\n> > > index d0c0fd049d..8d07209c6b 100644\n> > > --- a/bloom.c\n> > > +++ b/bloom.c\n> > > @@ -52,7 +52,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n> > > start_index = 0;\n> > >\n> > > if ((start_index == end_index) &&\n> > > - (g->bloom_large.word_alloc && !bitmap_get(&g->bloom_large, lex_pos))) {\n> > > + (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n> > > /*\n> > > * If the filter is zero-length, either (1) the filter has no\n> > > * changes, (2) the filter has too many changes, or (3) it\n> > >\n> > > In either case, this will fix the bad free():\n> > >\n> > > --- >8 ---\n> > >\n> > > Subject: [PATCH] fixup! commit-graph: add large-filters bitmap chunk\n> > >\n> > > Signed-off-by: Taylor Blau \n> > > ---\n> > > commit-graph.c | 18 ++++++++++--------\n> > > commit-graph.h | 2 +-\n> > > 2 files changed, 11 insertions(+), 9 deletions(-)\n> > > ...\n> > > @@ -2360,6 +2361,7 @@ int write_commit_graph(struct object_directory *odb,\n> > > \tfree(ctx->graph_name);\n> > > \tfree(ctx->commits.list);\n> > > \tfree(ctx->oids.list);\n> > > +\tfree(ctx->bloom_large);\n> >\n> > Is this correct, or shouldn't it be bitmap_free() instead?\n>\n> Ack, that should be 'bitmap_free()'. Double checking, 'bitmap_free' does\n> handle a 'NULL' argument like 'free', so dealing with an old\n> commit-graph lacking this chunk will work fine.\n>\n> Thanks for catching my mistake. I'm off tomorrow, Friday, and Monday,\n> so my responses from now on may be intermittent, but I should have some\n> time.\n\nI'm back :). Let's squash the following into patch (bearing in mind that\nyou'll have to drop a '&' in the final patch of this series as a result\nof this change):\n\n--- >8 ---\n\nSubject: [PATCH] fixup! commit-graph: add large-filters bitmap chunk\n\nMakes the commit-graph's 'bloom_large' bitmap a pointer so that it can\nbe managed with the standard bitmap APIs.\n\nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 18 ++++++++++--------\n commit-graph.h | 2 +-\n 2 files changed, 11 insertions(+), 9 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1fee49d171..384089be87 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -438,7 +438,10 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\tgraph->bloom_large_to_alloc = get_be64(chunk_lookup + 4)\n \t\t\t\t\t\t\t - chunk_offset - sizeof(uint32_t);\n\n-\t\t\t\tgraph->bloom_large.word_alloc = 0; /* populate when necessary */\n+\t\t\t\t/*\n+\t\t\t\t * leave 'bloom_large' uninitialized, and\n+\t\t\t\t * populate when necessary\n+\t\t\t\t */\n \t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n \t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n \t\t\t}\n@@ -960,17 +963,15 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n \tif (!g || !g->bloom_large_to_alloc)\n \t\treturn 0;\n\n-\tif (!g->bloom_large.word_alloc) {\n+\tif (!g->bloom_large) {\n \t\tsize_t i;\n-\t\tg->bloom_large.word_alloc = g->bloom_large_to_alloc;\n-\t\tg->bloom_large.words = xmalloc(g->bloom_large_to_alloc * sizeof(eword_t));\n-\n+\t\tg->bloom_large = bitmap_word_alloc(g->bloom_large_to_alloc);\n \t\tfor (i = 0; i < g->bloom_large_to_alloc; i++)\n-\t\t\tg->bloom_large.words[i] = get_be64(g->chunk_bloom_large_filters\n-\t\t\t\t\t\t\t + i * sizeof(eword_t));\n+\t\t\tg->bloom_large->words[i] = get_be64(g->chunk_bloom_large_filters\n+\t\t\t\t\t\t\t + i * sizeof(eword_t));\n \t}\n\n-\treturn bitmap_get(&g->bloom_large, graph_pos - g->num_commits_in_base);\n+\treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n }\n\n struct packed_oid_list {\n@@ -2360,6 +2361,7 @@ int write_commit_graph(struct object_directory *odb,\n \tfree(ctx->graph_name);\n \tfree(ctx->commits.list);\n \tfree(ctx->oids.list);\n+\tbitmap_free(ctx->bloom_large);\n\n \tif (ctx->commit_graph_filenames_after) {\n \t\tfor (i = 0; i < ctx->num_commit_graphs_after; i++) {\ndiff --git a/commit-graph.h b/commit-graph.h\nindex f4fb996dd5..b1ab86a3c8 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,7 +75,7 @@ struct commit_graph {\n \tconst unsigned char *chunk_bloom_large_filters;\n\n \tsize_t bloom_large_to_alloc;\n-\tstruct bitmap bloom_large;\n+\tstruct bitmap *bloom_large;\n\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n--\n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403377","messageId":"xmqqo8nh9ffd.fsf@gitster.c.googlers.com","threadId":"53975","inReplyTo":"20200811134807.GA26715@syl.lan","subject":"Re: [PATCH v2 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-11T18:59:50Z","receivedAt":"2020-08-11T18:59:55Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> I'm back :). Let's squash the following into patch (bearing in mind that\n> you'll have to drop a '&' in the final patch of this series as a result\n> of this change):\n\nLet's have replacement patches for those steps (or a full series is\nalso fine). Don't make me squash and clean-up.\n\nThat would give us (well, mostly the contributor) a chance for the\nfinal proofreading before sending.\n\nThanks.\n"},{"id":"403389","messageId":"cover.1597178914.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH v3 00/14] more miscellaneous Bloom filter improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:13Z","receivedAt":"2020-08-11T20:51:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nHere's a(nother) re-roll of mine and Stolee's series to introduce the\nnew BFXL commit-graph chunk, along with the '--max-new-filters' option\nto 'git commit-graph write'.\n\nNot really much has changed since v2, other than a rebase onto the\nlatest from master (the fifth 2.29 batch, at the time of writing), and\nto squash in a few fixups that I sent in response to my v2 series.\n\nHopefully this should be ready for queueing. (Stolee has looked at this\na lot off-list, but it would be great to get an ack from him on list,\ntoo).\n\nDerrick Stolee (1):\n bloom/diff: properly short-circuit on max_changes\n\nTaylor Blau (13):\n commit-graph: introduce 'get_bloom_filter_settings()'\n t4216: use an '&&'-chain\n commit-graph: pass a 'struct repository *' in more places\n t/helper/test-read-graph.c: prepare repo settings\n commit-graph: respect 'commitGraph.readChangedPaths'\n commit-graph.c: store maximum changed paths\n bloom: split 'get_bloom_filter()' in two\n bloom: use provided 'struct bloom_filter_settings'\n commit-graph.c: sort index into commits list\n csum-file.h: introduce 'hashwrite_be64()'\n commit-graph: add large-filters bitmap chunk\n commit-graph: rename 'split_commit_graph_opts'\n builtin/commit-graph.c: introduce '--max-new-filters='\n\n Documentation/config.txt | 2 +\n Documentation/config/commitgraph.txt | 8 +\n Documentation/git-commit-graph.txt | 4 +\n .../technical/commit-graph-format.txt | 12 +\n blame.c | 8 +-\n bloom.c | 51 +++-\n bloom.h | 22 +-\n builtin/commit-graph.c | 61 +++-\n commit-graph.c | 274 +++++++++++++-----\n commit-graph.h | 19 +-\n csum-file.h | 6 +\n diff.h | 2 -\n fuzz-commit-graph.c | 5 +-\n line-log.c | 2 +-\n midx.c | 3 +-\n repo-settings.c | 3 +\n repository.h | 1 +\n revision.c | 7 +-\n t/helper/test-bloom.c | 4 +-\n t/helper/test-read-graph.c | 3 +-\n t/t4216-log-bloom.sh | 148 ++++++++--\n t/t5324-split-commit-graph.sh | 13 +\n tree-diff.c | 5 +-\n 23 files changed, 522 insertions(+), 141 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\nRange-diff against v2:\n [ ... rebase onto 'master' ... ]\n 1: 001f3385ff = 34: e714e54240 commit-graph: introduce 'get_bloom_filter_settings()'\n 2: e4d068a478 = 35: 9fc8b17d6f t4216: use an '&&'-chain\n 3: afdc614c0d = 36: 8dbe4838b7 commit-graph: pass a 'struct repository *' in more places\n 4: 038e996ced = 37: f59db1e30d t/helper/test-read-graph.c: prepare repo settings\n 5: 404f10319a = 38: daae6788c0 commit-graph: respect 'commitGraph.readChangedPaths'\n 6: 053991f048 = 39: bf498844ef commit-graph.c: store maximum changed paths\n 7: 23525947c8 ! 40: eba2794873 bloom: split 'get_bloom_filter()' in two\n @@ bloom.h: void add_key_to_filter(const struct bloom_key *key,\n +\t\t\t\t\t\t int compute_if_not_present,\n +\t\t\t\t\t\t int *computed);\n +\n -+#define DEFAULT_BLOOM_MAX_CHANGES 512\n +#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n +\t(r), (c), 0, NULL)\n\n 8: 4deb724fc1 ! 41: 4f08177dbe bloom: use provided 'struct bloom_filter_settings'\n @@ bloom.h: void init_bloom_filters(void);\n +\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t int *computed);\n\n - #define DEFAULT_BLOOM_MAX_CHANGES 512\n #define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n -\t(r), (c), 0, NULL)\n +\t(r), (c), 0, NULL, NULL)\n 9: d1c4bbcaa9 = 42: cc1dc8b121 bloom/diff: properly short-circuit on max_changes\n10: e92ccafcf7 = 43: 23fd52c3b8 commit-graph.c: sort index into commits list\n11: c42d678714 = 44: 4800cd373e csum-file.h: introduce 'hashwrite_be64()'\n12: 100b26d7c8 ! 45: 619e0c619d commit-graph: add large-filters bitmap chunk\n @@ Commit message\n To allow using the existing bitmap code with 64-bit words, we write the\n data in network byte order from the 64-bit words. This means we also\n need to read the array from the commit-graph file by translating each\n - word from network byte order using get_be64() upon first use of the\n - bitmap. This is only used when writing the commit-graph, so this is a\n - relatively small operation compared to the other writing code.\n + word from network byte order using get_be64() when loading the commit\n + graph. (Note that this *could* be delayed until first-use, but a later\n + patch will rely on this being initialized early, so we assume the\n + up-front cost when parsing instead of delaying initialization).\n\n By avoiding the need to move to new versions of the BDAT and BIDX chunk,\n we can give ourselves more time to consider whether or not other\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n +\t\t\tif (graph->chunk_bloom_large_filters)\n +\t\t\t\tchunk_repeated = 1;\n +\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n -+\t\t\t\tgraph->bloom_large_to_alloc = get_be64(chunk_lookup + 4)\n -+\t\t\t\t\t\t\t - chunk_offset - sizeof(uint32_t);\n -+\n -+\t\t\t\tgraph->bloom_large.word_alloc = 0; /* populate when necessary */\n ++\t\t\t\tsize_t alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n +\t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n +\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n ++\t\t\t\tif (alloc) {\n ++\t\t\t\t\tsize_t j;\n ++\t\t\t\t\tgraph->bloom_large = bitmap_word_alloc(alloc);\n ++\n ++\t\t\t\t\tfor (j = 0; j < graph->bloom_large->word_alloc; j++)\n ++\t\t\t\t\t\tgraph->bloom_large->words[j] = get_be64(\n ++\t\t\t\t\t\t\tgraph->chunk_bloom_large_filters + j * sizeof(eword_t));\n ++\t\t\t\t}\n +\t\t\t}\n +\t\t\tbreak;\n \t\t}\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n \t\tgraph->chunk_bloom_data = NULL;\n +\t\tgraph->chunk_bloom_large_filters = NULL;\n \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n ++\t\tbitmap_free(graph->bloom_large);\n \t}\n\n + \thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n @@ commit-graph.c: struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n @@ commit-graph.c: struct tree *get_commit_tree_in_graph(struct repository *r, cons\n +\twhile (g && graph_pos < g->num_commits_in_base)\n +\t\tg = g->base_graph;\n +\n -+\tif (!g || !g->bloom_large_to_alloc)\n ++\tif (!(g && g->bloom_large))\n +\t\treturn 0;\n -+\n -+\tif (!g->bloom_large.word_alloc) {\n -+\t\tsize_t i;\n -+\t\tg->bloom_large.word_alloc = g->bloom_large_to_alloc;\n -+\t\tg->bloom_large.words = xmalloc(g->bloom_large_to_alloc * sizeof(eword_t));\n -+\n -+\t\tfor (i = 0; i < g->bloom_large_to_alloc; i++)\n -+\t\t\tg->bloom_large.words[i] = get_be64(g->chunk_bloom_large_filters\n -+\t\t\t\t\t\t\t + i * sizeof(eword_t));\n -+\t}\n -+\n -+\treturn bitmap_get(&g->bloom_large, graph_pos - g->num_commits_in_base);\n ++\treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n +}\n\n struct packed_oid_list {\n @@ commit-graph.h: struct commit_graph {\n \tconst unsigned char *chunk_bloom_data;\n +\tconst unsigned char *chunk_bloom_large_filters;\n +\n -+\tsize_t bloom_large_to_alloc;\n -+\tstruct bitmap bloom_large;\n ++\tstruct bitmap *bloom_large;\n\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n13: 2ee0b84351 = 46: b2e33ecba8 commit-graph: rename 'split_commit_graph_opts'\n14: 3b66ae4a9c ! 47: 09f6871f66 builtin/commit-graph.c: introduce '--max-new-filters='\n @@ bloom.c: static int load_bloom_filter_from_graph(struct commit_graph *g,\n \t\tstart_index = 0;\n\n +\tif ((start_index == end_index) &&\n -+\t (g->bloom_large.word_alloc && !bitmap_get(&g->bloom_large, lex_pos))) {\n ++\t (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n +\t\t/*\n +\t\t * If the filter is zero-length, either (1) the filter has no\n +\t\t * changes, (2) the filter has too many changes, or (3) it\n @@ commit-graph.c: struct tree *get_commit_tree_in_graph(struct repository *r, cons\n {\n \tuint32_t graph_pos = commit_graph_position(c);\n \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n +@@ commit-graph.c: static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n +\n + \tif (!(g && g->bloom_large))\n + \t\treturn 0;\n ++\tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n ++\t\t/*\n ++\t\t * Force all commits which are subject to a different\n ++\t\t * 'max_changed_paths' limit to be recomputed from scratch.\n ++\t\t *\n ++\t\t * Note that this could likely be improved, but is ignored since\n ++\t\t * all real-world graphs set the maximum number of changed paths\n ++\t\t * at 512.\n ++\t\t */\n ++\t\treturn 0;\n ++\t}\n + \treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n + }\n +\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n--\n2.28.0.rc1.13.ge78abce653\n"},{"id":"403390","messageId":"e714e54240bb339d1ecebcea157b734abf1c14ef.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:19Z","receivedAt":"2020-08-11T20:51:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Many places in the code often need a pointer to the commit-graph's\n'struct bloom_filter_settings', in which case they often take the value\nfrom the top-most commit-graph.\n\nIn the non-split case, this works as expected. In the split case,\nhowever, things get a little tricky. Not all layers in a chain of\nincremental commit-graphs are required to themselves have Bloom data,\nand so whether or not some part of the code uses Bloom filters depends\nentirely on whether or not the top-most level of the commit-graph chain\nhas Bloom filters.\n\nThis has been the behavior since Bloom filters were introduced, and has\nbeen codified into the tests since a759bfa9ee (t4216: add end to end\ntests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\nrequires that Bloom filters are not used in exactly the case described\nearlier.\n\nThere is no reason that this needs to be the case, since it is perfectly\nvalid for commits in an earlier layer to have Bloom filters when commits\nin a newer layer do not.\n\nSince Bloom settings are guaranteed to be the same for any layer in a\nchain that has Bloom data, it is sufficient to traverse the\n'->base_graph' pointer until either (1) a non-null 'struct\nbloom_filter_settings *' is found, or (2) until we are at the root of\nthe commit-graph chain.\n\nIntroduce a 'get_bloom_filter_settings()' function that does just this,\nand use it instead of purely dereferencing the top-most graph's\n'->bloom_filter_settings' pointer.\n\nWhile we're at it, add an additional test in t5324 to guard against code\nin the commit-graph writing machinery that doesn't correctly handle a\nNULL 'struct bloom_filter *'.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n blame.c | 6 ++++--\n bloom.c | 6 +++---\n commit-graph.c | 11 +++++++++++\n commit-graph.h | 2 ++\n revision.c | 5 +----\n t/t4216-log-bloom.sh | 9 ++++++---\n t/t5324-split-commit-graph.sh | 13 +++++++++++++\n 7 files changed, 40 insertions(+), 12 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 82fa16d658..3e5f8787bc 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -2891,16 +2891,18 @@ void setup_blame_bloom_data(struct blame_scoreboard *sb,\n \t\t\t const char *path)\n {\n \tstruct blame_bloom_data *bd;\n+\tstruct bloom_filter_settings *bs;\n \n \tif (!sb->repo->objects->commit_graph)\n \t\treturn;\n \n-\tif (!sb->repo->objects->commit_graph->bloom_filter_settings)\n+\tbs = get_bloom_filter_settings(sb->repo);\n+\tif (!bs)\n \t\treturn;\n \n \tbd = xmalloc(sizeof(struct blame_bloom_data));\n \n-\tbd->settings = sb->repo->objects->commit_graph->bloom_filter_settings;\n+\tbd->settings = bs;\n \n \tbd->alloc = 4;\n \tbd->nr = 0;\ndiff --git a/bloom.c b/bloom.c\nindex 1a573226e7..cd9380ac62 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -38,7 +38,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \twhile (graph_pos < g->num_commits_in_base)\n \t\tg = g->base_graph;\n \n-\t/* The commit graph commit 'c' lives in doesn't carry bloom filters. */\n+\t/* The commit graph commit 'c' lives in doesn't carry Bloom filters. */\n \tif (!g->chunk_bloom_indexes)\n \t\treturn 0;\n \n@@ -195,8 +195,8 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes)\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n+\t\t\t\treturn filter;\n \t}\n \n \tif (filter->data)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex e51c91dd5b..d4b06811be 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -660,6 +660,17 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r)\n+{\n+\tstruct commit_graph *g = r->objects->commit_graph;\n+\twhile (g) {\n+\t\tif (g->bloom_filter_settings)\n+\t\t\treturn g->bloom_filter_settings;\n+\t\tg = g->base_graph;\n+\t}\n+\treturn NULL;\n+}\n+\n static void close_commit_graph_one(struct commit_graph *g)\n {\n \tif (!g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 09a97030dc..0677dd1031 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -87,6 +87,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n */\n int generation_numbers_enabled(struct repository *r);\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r);\n+\n enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_APPEND = (1 << 0),\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\ndiff --git a/revision.c b/revision.c\nindex 3dcf689341..be600186ee 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -680,10 +680,7 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \trepo_parse_commit(revs->repo, revs->commits->item);\n \n-\tif (!revs->repo->objects->commit_graph)\n-\t\treturn;\n-\n-\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n+\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n \tif (!revs->bloom_filter_settings)\n \t\treturn;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c21cc160f3..c9f9bdf1ba 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -60,7 +60,7 @@ setup () {\n \n test_bloom_filters_used () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":${2:-0},\\\"maybe\\\"\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom &&\n@@ -134,8 +134,11 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n-test_expect_success 'Do not use Bloom filters if the latest graph does not have Bloom filters.' '\n-\ttest_bloom_filters_not_used \"-- A/B\"\n+test_expect_success 'use Bloom filters even if the latest graph does not have Bloom filters' '\n+\t# Ensure that the number of empty filters is equal to the number of\n+\t# filters in the latest graph layer to prove that they are loaded (and\n+\t# ignored).\n+\ttest_bloom_filters_used \"-- A/B\" 3\n '\n \n test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 9b850ea907..5bdfd53ef9 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -425,4 +425,17 @@ done <<\\EOF\n 0600 -r--------\n EOF\n \n+test_expect_success '--split=replace with partial Bloom data' '\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/3 &&\n+\tgit rev-list -1 HEAD~2 >a &&\n+\tgit rev-list -1 HEAD~1 >b &&\n+\tgit commit-graph write --split=no-merge --stdin-commits --changed-paths graph-files &&\n+\ttest_line_count = 1 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n test_done\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403391","messageId":"9fc8b17d6fe9e1f7b177627e67420443e7a2e98c.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 02/14] t4216: use an '&&'-chain","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:23Z","receivedAt":"2020-08-11T20:51:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a759bfa9ee (t4216: add end to end tests for git log with Bloom\nfilters, 2020-04-06), a 'rm' invocation was added without a\ncorresponding '&&' chain.\n\nWhen 'trace.perf' already exists, everything works fine. However, the\nfunction can be executed without 'trace.perf' on disk (eg., when the\nsubset of tests run is altered with '--run'), and so the bare 'rm'\ncomplains about a missing file.\n\nTo remove some noise from the test log, invoke 'rm' with '-f', at which\npoint it is sensible to place the 'rm -f' in an '&&'-chain, which is\nboth (1) our usual style, and (2) avoids a broken chain in the future if\nmore commands are added at the beginning of the function.\n\nHelped-by: Eric Sunshine \nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex c9f9bdf1ba..fe19f6a60c 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -53,7 +53,7 @@ sane_unset GIT_TRACE2_PERF_BRIEF\n sane_unset GIT_TRACE2_CONFIG_PARAMS\n \n setup () {\n-\trm \"$TRASH_DIRECTORY/trace.perf\"\n+\trm -f \"$TRASH_DIRECTORY/trace.perf\" &&\n \tgit -c core.commitGraph=false log --pretty=\"format:%s\" $1 >log_wo_bloom &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.perf\" git -c core.commitGraph=true log --pretty=\"format:%s\" $1 >log_w_bloom\n }\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403392","messageId":"8dbe4838b725e21d7bfe16eb2c4e15e68f620dd0.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 03/14] commit-graph: pass a 'struct repository *' in more places","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:27Z","receivedAt":"2020-08-11T20:51:32Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a future commit, some commit-graph internals will want access to\n'r->settings', but we only have the 'struct object_directory *'\ncorresponding to that repository.\n\nAdd an additional parameter to pass the repository around in more\nplaces.\n\nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 2 +-\n commit-graph.c | 17 ++++++++++-------\n commit-graph.h | 6 ++++--\n fuzz-commit-graph.c | 5 +++--\n 4 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 523501f217..ba5584463f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -106,7 +106,7 @@ static int graph_verify(int argc, const char **argv)\n \tFREE_AND_NULL(graph_name);\n \n \tif (open_ok)\n-\t\tgraph = load_commit_graph_one_fd_st(fd, &st, odb);\n+\t\tgraph = load_commit_graph_one_fd_st(the_repository, fd, &st, odb);\n \telse\n \t\tgraph = read_commit_graph_one(the_repository, odb);\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex d4b06811be..0c1030641c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -224,7 +224,8 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st)\n \treturn 1;\n }\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tvoid *graph_map;\n@@ -240,7 +241,7 @@ struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n \t}\n \tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n-\tret = parse_commit_graph(graph_map, graph_size);\n+\tret = parse_commit_graph(r, graph_map, graph_size);\n \n \tif (ret)\n \t\tret->odb = odb;\n@@ -280,7 +281,8 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n \tconst unsigned char *data, *chunk_lookup;\n \tuint32_t i;\n@@ -445,7 +447,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n \treturn NULL;\n }\n \n-static struct commit_graph *load_commit_graph_one(const char *graph_file,\n+static struct commit_graph *load_commit_graph_one(struct repository *r,\n+\t\t\t\t\t\t const char *graph_file,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \n@@ -457,7 +460,7 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file,\n \tif (!open_ok)\n \t\treturn NULL;\n \n-\tg = load_commit_graph_one_fd_st(fd, &st, odb);\n+\tg = load_commit_graph_one_fd_st(r, fd, &st, odb);\n \n \tif (g)\n \t\tg->filename = xstrdup(graph_file);\n@@ -469,7 +472,7 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tchar *graph_name = get_commit_graph_filename(odb);\n-\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \tfree(graph_name);\n \n \treturn g;\n@@ -550,7 +553,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \t\tvalid = 0;\n \t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\t\tchar *graph_name = get_split_graph_filename(odb, line.buf);\n-\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \n \t\t\tfree(graph_name);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 0677dd1031..d9acb22bac 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,13 @@ struct commit_graph {\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb);\n struct commit_graph *read_commit_graph_one(struct repository *r,\n \t\t\t\t\t struct object_directory *odb);\n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n /*\n * Return 1 if and only if the repository has a commit-graph\ndiff --git a/fuzz-commit-graph.c b/fuzz-commit-graph.c\nindex 430817214d..e7cf6d5b0f 100644\n--- a/fuzz-commit-graph.c\n+++ b/fuzz-commit-graph.c\n@@ -1,7 +1,8 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n \n@@ -10,7 +11,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \tstruct commit_graph *g;\n \n \tinitialize_the_repository();\n-\tg = parse_commit_graph((void *)data, size);\n+\tg = parse_commit_graph(the_repository, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\n \n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403393","messageId":"f59db1e30dfcf69f6d6ab961b42766c62724324d.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 04/14] t/helper/test-read-graph.c: prepare repo settings","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:32Z","receivedAt":"2020-08-11T20:51:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The read-graph test-tool is used by a number of the commit-graph test to\nassert various properties about a commit-graph. Previously, this program\nnever ran 'prepare_repo_settings()'. There was no need to do so, since\nnone of the commit-graph machinery is affected by the repo settings.\n\nIn the next patch, the commit-graph machinery's behavior will become\ndependent on the repo settings, and so loading them before running the\nrest of the test tool is critical.\n\nAs such, teach the test tool to call 'prepare_repo_settings()'.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 6d0c962438..5f585a1725 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -12,11 +12,12 @@ int cmd__read_graph(int argc, const char **argv)\n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n \n+\tprepare_repo_settings(the_repository);\n+\n \tgraph = read_commit_graph_one(the_repository, odb);\n \tif (!graph)\n \t\treturn 1;\n \n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403394","messageId":"daae6788c0a0d9414158483922aa4459b446683e.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 05/14] commit-graph: respect 'commitGraph.readChangedPaths'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:36Z","receivedAt":"2020-08-11T20:51:41Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Git uses the 'core.commitGraph' configuration value to control whether\nor not the commit graph is used when parsing commits or performing a\ntraversal.\n\nNow that commit-graphs can also contain a section for changed-path Bloom\nfilters, administrators that already have commit-graphs may find it\nconvenient to use those graphs without relying on their changed-path\nBloom filters. This can happen, for example, during a staged roll-out,\nor in the event of an incident.\n\nIntroduce 'commitGraph.readChangedPaths' to control whether or not Bloom\nfilters are read. Note that this configuration is independent from both:\n\n - 'core.commitGraph', to allow flexibility in using all parts of a\n commit-graph _except_ for its Bloom filters.\n\n - The '--changed-paths' option for 'git commit-graph write', to allow\n reading and writing Bloom filters to be controlled independently.\n\nWhen the variable is set, pretend as if no Bloom data was specified at\nall. This avoids adding additional special-casing outside of the\ncommit-graph internals.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 ++\n Documentation/config/commitgraph.txt | 4 ++++\n commit-graph.c | 6 ++++--\n repo-settings.c | 3 +++\n repository.h | 1 +\n t/t4216-log-bloom.sh | 4 +++-\n 6 files changed, 17 insertions(+), 3 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex ef0768b91a..78883c6e63 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -340,6 +340,8 @@ include::config/column.txt[]\n \n include::config/commit.txt[]\n \n+include::config/commitgraph.txt[]\n+\n include::config/credential.txt[]\n \n include::config/completion.txt[]\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nnew file mode 100644\nindex 0000000000..cff0797b54\n--- /dev/null\n+++ b/Documentation/config/commitgraph.txt\n@@ -0,0 +1,4 @@\n+commitGraph.readChangedPaths::\n+\tIf true, then git will use the changed-path Bloom filters in the\n+\tcommit-graph file (if it exists, and they are present). Defaults to\n+\ttrue. See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0c1030641c..a516e93d71 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -320,6 +320,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n+\tprepare_repo_settings(r);\n+\n \tgraph = alloc_commit_graph();\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -396,14 +398,14 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n \t\t\tif (graph->chunk_bloom_indexes)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n \t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n \t\t\tbreak;\n \n \t\tcase GRAPH_CHUNKID_BLOOMDATA:\n \t\t\tif (graph->chunk_bloom_data)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n \t\t\t\tuint32_t hash_version;\n \t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n \t\t\t\thash_version = get_be32(data + chunk_offset);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex 0918408b34..9e551bc03d 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -17,9 +17,12 @@ void prepare_repo_settings(struct repository *r)\n \n \tif (!repo_config_get_bool(r, \"core.commitgraph\", &value))\n \t\tr->settings.core_commit_graph = value;\n+\tif (!repo_config_get_bool(r, \"commitgraph.readchangedpaths\", &value))\n+\t\tr->settings.commit_graph_read_changed_paths = value;\n \tif (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n \t\tr->settings.gc_write_commit_graph = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_commit_graph, 1);\n+\tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n \n \tif (!repo_config_get_int(r, \"index.version\", &value))\ndiff --git a/repository.h b/repository.h\nindex 3c1f7d54bd..81759b7d27 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -29,6 +29,7 @@ struct repo_settings {\n \tint initialized;\n \n \tint core_commit_graph;\n+\tint commit_graph_read_changed_paths;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fe19f6a60c..b3d1f596f8 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -90,7 +90,9 @@ do\n \t\t \"--ancestry-path side..master\"\n \tdo\n \t\ttest_expect_success \"git log option: $option for path: $path\" '\n-\t\t\ttest_bloom_filters_used \"$option -- $path\"\n+\t\t\ttest_bloom_filters_used \"$option -- $path\" &&\n+\t\t\ttest_config commitgraph.readChangedPaths false &&\n+\t\t\ttest_bloom_filters_not_used \"$option -- $path\"\n \t\t'\n \tdone\n done\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403395","messageId":"bf498844ef1134a0f762f6df449b9431dc10fc3b.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 06/14] commit-graph.c: store maximum changed paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:41Z","receivedAt":"2020-08-11T20:51:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For now, we assume that there is a fixed constant describing the\nmaximum number of changed paths we are willing to store in a Bloom\nfilter.\n\nPrepare for that to (at least partially) not be the case by making it a\nmember of the 'struct bloom_filter_settings'. This will be helpful in\nthe subsequent patches by reducing the size of test cases that exercise\nstoring too many changed paths, as well as preparing for an eventual\nfuture in which this value might change.\n\nThis patch alone does not cause newly generated Bloom filters to use\na custom upper-bound on the maximum number of changed paths a single\nBloom filter can hold, that will occur in a later patch.\n\nSigned-off-by: Taylor Blau \n---\n bloom.h | 11 ++++++++++-\n commit-graph.c | 3 +++\n t/t4216-log-bloom.sh | 4 ++--\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.h b/bloom.h\nindex d8fbb0fbf1..0b9b59a6fe 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -28,9 +28,18 @@ struct bloom_filter_settings {\n \t * that contain n*b bits.\n \t */\n \tuint32_t bits_per_entry;\n+\n+\t/*\n+\t * The maximum number of changed paths per commit\n+\t * before declaring a Bloom filter to be too-large.\n+\t *\n+\t * Not written to the commit-graph file.\n+\t */\n+\tuint32_t max_changed_paths;\n };\n \n-#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10 }\n+#define DEFAULT_BLOOM_MAX_CHANGES 512\n+#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10, DEFAULT_BLOOM_MAX_CHANGES }\n #define BITS_PER_WORD 8\n #define BLOOMDATA_CHUNK_HEADER_SIZE 3 * sizeof(uint32_t)\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex a516e93d71..86dd4b979e 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1194,6 +1194,7 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n \tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n \tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_object_intmax(&jw, \"max_changed_paths\", ctx->bloom_settings->max_changed_paths);\n \tjw_end(&jw);\n \n \ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n@@ -1662,6 +1663,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n \t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n \t\t\t\t\t\t\t bloom_settings.num_hashes);\n+\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n \t\tctx->bloom_settings = &bloom_settings;\n \t}\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex b3d1f596f8..eb2bcc51f0 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -169,11 +169,11 @@ test_expect_success 'persist filter settings' '\n \t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n \t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n \t\tGIT_TRACE2_EVENT_NESTING=5 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2-auto.txt\n '\n \n test_expect_success 'correctly report changes over limit' '\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403396","messageId":"eba279487318137064eef776c779ca537a4a711d.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:45Z","receivedAt":"2020-08-11T20:51:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'get_bloom_filter' takes a flag to control whether it will compute a\nBloom filter if the requested one is missing. In the next patch, we'll\nadd yet another parameter to this method, which would force all but one\ncaller to specify an extra 'NULL' parameter at the end.\n\nInstead of doing this, split 'get_bloom_filter' into two functions:\n'get_bloom_filter' and 'get_or_compute_bloom_filter'. The former only\nlooks up a Bloom filter (and does not compute one if it's missing,\nthus dropping the 'compute_if_not_present' flag). The latter does\ncompute missing Bloom filters, with an additional parameter to store\nwhether or not it needed to do so.\n\nThis simplifies many call-sites, since the majority of existing callers\nto 'get_bloom_filter' do not want missing Bloom filters to be computed\n(so they can drop the parameter entirely and use the simpler version of\nthe function).\n\nWhile we're at it, instrument the new 'get_or_compute_bloom_filter()'\nwith two counters in the 'write_commit_graph_context' struct which store\nthe number of filters that we computed, and the number of those which\nwere too large to store.\n\nIt would be nice to drop the 'compute_if_not_present' flag entirely,\nsince all remaining callers of 'get_or_compute_bloom_filter' pass it as\n'1', but this will change in a future patch and hence cannot be removed.\n\nSigned-off-by: Taylor Blau \n---\n blame.c | 2 +-\n bloom.c | 13 ++++++++++---\n bloom.h | 10 +++++++---\n commit-graph.c | 38 +++++++++++++++++++++++++++++++++++---\n line-log.c | 2 +-\n revision.c | 2 +-\n t/helper/test-bloom.c | 3 ++-\n 7 files changed, 57 insertions(+), 13 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 3e5f8787bc..756285fca7 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -1275,7 +1275,7 @@ static int maybe_changed_path(struct repository *r,\n \tif (commit_graph_generation(origin->commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn 1;\n \n-\tfilter = get_bloom_filter(r, origin->commit, 0);\n+\tfilter = get_bloom_filter(r, origin->commit);\n \n \tif (!filter)\n \t\treturn 1;\ndiff --git a/bloom.c b/bloom.c\nindex cd9380ac62..a8a21762f4 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,9 +177,10 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present)\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t int *computed)\n {\n \tstruct bloom_filter *filter;\n \tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n@@ -187,6 +188,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n+\tif (computed)\n+\t\t*computed = 0;\n+\n \tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n@@ -273,6 +277,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \t\tfilter->len = 0;\n \t}\n \n+\tif (computed)\n+\t\t*computed = 1;\n+\n \tfree(diff_queued_diff.queue);\n \tDIFF_QUEUE_CLEAR(&diff_queued_diff);\n \ndiff --git a/bloom.h b/bloom.h\nindex 0b9b59a6fe..baa91926db 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -89,9 +89,13 @@ void add_key_to_filter(const struct bloom_key *key,\n \n void init_bloom_filters(void);\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present);\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t int *computed);\n+\n+#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n+\t(r), (c), 0, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 86dd4b979e..ba2a2cfb22 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -964,6 +964,9 @@ struct write_commit_graph_context {\n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n+\n+\tint count_bloom_filter_found_large;\n+\tint count_bloom_filter_computed;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1175,7 +1178,7 @@ static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tuint32_t cur_pos = 0;\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \t\tcur_pos += len;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1215,7 +1218,7 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1385,6 +1388,22 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n+{\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"filter_found_large\",\n+\t\t\t ctx->count_bloom_filter_found_large);\n+\tjw_object_intmax(&jw, \"filter_computed\",\n+\t\t\t ctx->count_bloom_filter_computed);\n+\tjw_end(&jw);\n+\n+\ttrace2_data_json(\"commit-graph\", the_repository, \"bloom_statistics\", &jw);\n+\n+\tjw_release(&jw);\n+}\n+\n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n@@ -1407,12 +1426,25 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tint computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n+\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\tctx->r,\n+\t\t\tc,\n+\t\t\t1,\n+\t\t\t&computed);\n+\t\tif (computed) {\n+\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (filter && !filter->len)\n+\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t}\n \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n+\tif (trace2_is_enabled())\n+\t\ttrace2_bloom_filter_write_statistics(ctx);\n+\n \tfree(sorted_commits);\n \tstop_progress(&progress);\n }\ndiff --git a/line-log.c b/line-log.c\nindex bf73ea95ac..68eeb425f8 100644\n--- a/line-log.c\n+++ b/line-log.c\n@@ -1159,7 +1159,7 @@ static int bloom_filter_check(struct rev_info *rev,\n \t\treturn 1;\n \n \tif (!rev->bloom_filter_settings ||\n-\t !(filter = get_bloom_filter(rev->repo, commit, 0)))\n+\t !(filter = get_bloom_filter(rev->repo, commit)))\n \t\treturn 1;\n \n \tif (!range)\ndiff --git a/revision.c b/revision.c\nindex be600186ee..7f58ecc411 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -751,7 +751,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \tif (commit_graph_generation(commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn -1;\n \n-\tfilter = get_bloom_filter(revs->repo, commit, 0);\n+\tfilter = get_bloom_filter(revs->repo, commit);\n \n \tif (!filter) {\n \t\tcount_bloom_filter_not_present++;\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex f0aa80b98e..531af439c2 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -39,7 +39,8 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tstruct bloom_filter *filter;\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n-\tfilter = get_bloom_filter(the_repository, c, 1);\n+\tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n \n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403397","messageId":"4800cd373eefb8d71094254d90fd0304067e464f.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 11/14] csum-file.h: introduce 'hashwrite_be64()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:49Z","receivedAt":"2020-08-11T20:51:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A small handful of writers who wish to encode 64-bit values in network\norder have worked around the lack of such a helper by calling the 32-bit\nvariant twice.\n\nThe subsequent commit will add another caller who wants to write a\n64-bit value. To ease their (and the existing caller's) pain, introduce\na helper to do just that, and convert existing call-sites.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 8 ++------\n csum-file.h | 6 ++++++\n midx.c | 3 +--\n 3 files changed, 9 insertions(+), 8 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0d70545149..8964453433 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1784,12 +1784,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tchunk_offset = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunks[i].id);\n-\t\tchunk_write[1] = htonl(chunk_offset >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offset & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n+\t\thashwrite_be32(f, chunks[i].id);\n+\t\thashwrite_be64(f, chunk_offset);\n \n \t\tchunk_offset += chunks[i].size;\n \t}\ndiff --git a/csum-file.h b/csum-file.h\nindex f9cbd317fb..b026ec7766 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -62,4 +62,10 @@ static inline void hashwrite_be32(struct hashfile *f, uint32_t data)\n \thashwrite(f, &data, sizeof(data));\n }\n \n+static inline void hashwrite_be64(struct hashfile *f, uint64_t data)\n+{\n+\thashwrite_be32(f, data >> 32);\n+\thashwrite_be32(f, data & 0xffffffffUL);\n+}\n+\n #endif\ndiff --git a/midx.c b/midx.c\nindex a5fb797ede..51ca27cf34 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -775,8 +775,7 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \t\tif (!(offset >> 31))\n \t\t\tcontinue;\n \n-\t\thashwrite_be32(f, offset >> 32);\n-\t\thashwrite_be32(f, offset & 0xffffffffUL);\n+\t\thashwrite_be64(f, offset);\n \t\twritten += 2 * sizeof(uint32_t);\n \n \t\tnr_large_offset--;\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403398","messageId":"4f08177dbe0f3cddfcc7cddbdb65446f410b6ecf.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 08/14] bloom: use provided 'struct bloom_filter_settings'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:53Z","receivedAt":"2020-08-11T20:51:58Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When 'get_or_compute_bloom_filter()' needs to compute a Bloom filter\nfrom scratch, it looks to the default 'struct bloom_filter_settings' in\norder to determine the maximum number of changed paths, number of bits\nper entry, and so on.\n\nAll of these values have so far been constant, and so there was no need\nto pass in a pointer from the caller (eg., the one that is stored in the\n'struct write_commit_graph_context').\n\nStart passing in a 'struct bloom_filter_settings *' instead of using the\ndefault values to respect graph-specific settings (eg., in the case of\nsetting 'GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS').\n\nIn order to have an initialized value for these settings, move its\ninitialization to earlier in the commit-graph write.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 13 ++++++-------\n bloom.h | 3 ++-\n commit-graph.c | 21 ++++++++++-----------\n t/helper/test-bloom.c | 1 +\n 4 files changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex a8a21762f4..0cf1962dc5 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -180,13 +180,12 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t int *computed)\n {\n \tstruct bloom_filter *filter;\n-\tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \tint i;\n \tstruct diff_options diffopt;\n-\tint max_changes = 512;\n \n \tif (computed)\n \t\t*computed = 0;\n@@ -211,7 +210,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n \tdiffopt.detect_rename = 0;\n-\tdiffopt.max_changes = max_changes;\n+\tdiffopt.max_changes = settings->max_changed_paths;\n \tdiff_setup_done(&diffopt);\n \n \t/* ensure commit is parsed so we have parent information */\n@@ -223,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= max_changes) {\n+\tif (diffopt.num_changes <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -260,13 +259,13 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n-\t\tfilter->len = (hashmap_get_size(&pathmap) * settings.bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n \t\t\tstruct bloom_key key;\n-\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, &settings);\n-\t\t\tadd_key_to_filter(&key, filter, &settings);\n+\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, settings);\n+\t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\ndiff --git a/bloom.h b/bloom.h\nindex baa91926db..3f19e3fca4 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -92,10 +92,11 @@ void init_bloom_filters(void);\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t int *computed);\n \n #define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n-\t(r), (c), 0, NULL)\n+\t(r), (c), 0, NULL, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ba2a2cfb22..48d4697f54 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1432,6 +1432,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->r,\n \t\t\tc,\n \t\t\t1,\n+\t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed) {\n \t\t\tctx->count_bloom_filter_computed++;\n@@ -1688,17 +1689,6 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tint num_chunks = 3;\n \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n-\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n-\n-\tif (!ctx->bloom_settings) {\n-\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n-\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n-\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n-\t\t\t\t\t\t\t bloom_settings.num_hashes);\n-\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n-\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n-\t\tctx->bloom_settings = &bloom_settings;\n-\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -2144,6 +2134,7 @@ int write_commit_graph(struct object_directory *odb,\n \tuint32_t i, count_distinct = 0;\n \tint res = 0;\n \tint replace = 0;\n+\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n@@ -2157,6 +2148,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split_opts = split_opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n+\t\t\t\t\t\t bloom_settings.bits_per_entry);\n+\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n+\t\t\t\t\t\t bloom_settings.num_hashes);\n+\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n+\tctx->bloom_settings = &bloom_settings;\n+\n \tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n \t\tctx->changed_paths = 1;\n \tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 531af439c2..4af949164c 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -40,6 +40,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n \tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t &settings,\n \t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403399","messageId":"cc1dc8b121c66247742f58b38af457bc1eed3208.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 09/14] bloom/diff: properly short-circuit on max_changes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:51:56Z","receivedAt":"2020-08-11T20:52:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nCommit e3696980 (diff: halt tree-diff early after max_changes,\n2020-03-30) intended to create a mechanism to short-circuit a diff\ncalculation after a certain number of paths were modified. By\nincrementing a \"num_changes\" counter throughout the recursive\nll_diff_tree_paths(), this was supposed to match the number of changes\nthat would be written into the changed-path Bloom filters.\nUnfortunately, this was not implemented correctly and instead misses\nsimple cases like file modifications. This then does not stop very\nlarge changed-path filters from being written (unless they add or remove\nmany files).\n\nTo start, change the implementation in ll_diff_tree_paths() to instead\nuse the global diff_queue_diff struct's 'nr' member as the count. This\nis a way to simplify the logic instead of making more mistakes in the\ncomplicated diff code.\n\nThis has a drawback: the diff_queue_diff struct only lists the paths\ncorresponding to blob changes, not their leading directories. Thus,\nget_or_compute_bloom_filter() needs an additional check to see if the\nhashmap with the leading directories becomes too large.\n\nOne reason why this was not caught by test cases was that the test in\nt4216-log-bloom.sh that was supposed to check this \"too many changes\"\ncondition only checked this on the initial commit of a repository. The\nold logic counted these values correctly. Update this test in a few\nways:\n\n1. Use GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS to reduce the limit,\n allowing smaller commits to engage with this logic.\n\n2. Create several interesting cases of edits, adds, removes, and mode\n changes (in the second commit). By testing both sides of the\n inequality with the *_MAX_CHANGED_PATHS variable, we can see that\n the count is exactly correct, so none of these changes are missed\n or over-counted.\n\n3. Use the trace2 data value filter_found_large to verify that these\n commits are on the correct side of the limit.\n\nAnother way to verify the behavior is correct is through performance\ntests. By testing on my local copies of the Git repository and the Linux\nkernel repository, I could measure the effect of these short-circuits\nwhen computing a fresh commit-graph file with changed-path Bloom filters\nusing the command\n\n GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=N time \\\n git commit-graph write --reachable --changed-paths\n\nand reporting the wall time and resulting commit-graph size.\n\nFor Git, the results are\n\n| | N=1 | N=10 | N=512 |\n|--------|----------------|----------------|----------------|\n| HEAD~1 | 10.90s 9.18MB | 11.11s 9.34MB | 11.31s 9.35MB |\n| HEAD | 9.21s 8.62MB | 11.11s 9.29MB | 11.29s 9.34MB |\n\nFor Linux, the results are\n\n| | N=1 | N=20 | N=512 |\n|--------|----------------|---------------|---------------|\n| HEAD~1 | 61.28s 64.3MB | 76.9s 72.6MB | 77.6s 72.6MB |\n| HEAD | 49.44s 56.3MB | 68.7s 65.9MB | 69.2s 65.9MB |\n\nNaturally, the improvement becomes much less as the limit grows, as\nfewer commits satisfy the short-circuit.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n bloom.c | 6 +++-\n diff.h | 2 --\n t/t4216-log-bloom.sh | 85 +++++++++++++++++++++++++++++++++++++++-----\n tree-diff.c | 5 +--\n 4 files changed, 82 insertions(+), 16 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 0cf1962dc5..ed54e96e57 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -222,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= settings->max_changed_paths) {\n+\tif (diff_queued_diff.nr <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -259,6 +259,9 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n+\t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths)\n+\t\t\tgoto cleanup;\n+\n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n@@ -268,6 +271,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n+\tcleanup:\n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\ndiff --git a/diff.h b/diff.h\nindex e0c0af6286..1d32b71885 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -287,8 +287,6 @@ struct diff_options {\n \n \t/* If non-zero, then stop computing after this many changes. */\n \tint max_changes;\n-\t/* For internal use only. */\n-\tint num_changes;\n \n \tint ita_invisible_in_index;\n /* white-space error highlighting */\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex eb2bcc51f0..21b67677ef 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -177,20 +177,87 @@ test_expect_success 'persist filter settings' '\n '\n \n test_expect_success 'correctly report changes over limit' '\n-\tgit init 513changes &&\n+\tgit init limits &&\n \t(\n-\t\tcd 513changes &&\n-\t\tfor i in $(test_seq 1 513)\n+\t\tcd limits &&\n+\t\tmkdir d &&\n+\t\tmkdir d/e &&\n+\n+\t\tfor i in $(test_seq 1 2)\n \t\tdo\n-\t\t\techo $i >file$i.txt || return 1\n+\t\t\tprintf $i >d/file$i.txt &&\n+\t\t\tprintf $i >d/e/file$i.txt || return 1\n \t\tdone &&\n-\t\tgit add . &&\n+\n+\t\tmkdir mode &&\n+\t\tprintf bash >mode/script.sh &&\n+\n+\t\tmkdir foo &&\n+\t\ttouch foo/bar &&\n+\t\ttouch foo.txt &&\n+\n+\t\tgit add d foo foo.txt mode &&\n \t\tgit commit -m \"files\" &&\n-\t\tgit commit-graph write --reachable --changed-paths &&\n-\t\tfor i in $(test_seq 1 513)\n+\n+\t\t# Commit has 7 file and 4 directory adds\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":10\" trace &&\n+\t\tgrep \"\\\"filter_found_large\\\":1\" trace &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n \t\tdo\n-\t\t\tgit -c core.commitGraph=false log -- file$i.txt >expect &&\n-\t\t\tgit log -- file$i.txt >actual &&\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# Make a variety of path changes\n+\t\tprintf new1 >d/e/file1.txt &&\n+\t\tprintf new2 >d/file2.txt &&\n+\t\trm d/e/file2.txt &&\n+\t\trm -r foo &&\n+\t\tprintf text >foo &&\n+\t\tmkdir f &&\n+\t\tprintf new1 >f/file1.txt &&\n+\n+\t\t# including a mode-only change (counts as modified)\n+\t\tgit update-index --chmod=+x mode/script.sh &&\n+\n+\t\tgit add foo d f &&\n+\t\tgit commit -m \"complicated\" &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-edit\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":10\" trace-edit &&\n+\t\tgrep \"\\\"filter_found_large\\\":2\" trace-edit &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=11 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-update\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":11\" trace-update &&\n+\t\tgrep \"\\\"filter_found_large\\\":0\" trace-update &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n \t\t\ttest_cmp expect actual || return 1\n \t\tdone\n \t)\ndiff --git a/tree-diff.c b/tree-diff.c\nindex 6ebad1a46f..7cebbb327e 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -434,7 +434,7 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tif (diff_can_quit_early(opt))\n \t\t\tbreak;\n \n-\t\tif (opt->max_changes && opt->num_changes > opt->max_changes)\n+\t\tif (opt->max_changes && diff_queued_diff.nr > opt->max_changes)\n \t\t\tbreak;\n \n \t\tif (opt->pathspec.nr) {\n@@ -521,7 +521,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \n \t\t\t/* t↓ */\n \t\t\tupdate_tree_entry(&t);\n-\t\t\topt->num_changes++;\n \t\t}\n \n \t\t/* t > p[imin] */\n@@ -539,7 +538,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tskip_emit_tp:\n \t\t\t/* ∀ pi=p[imin] pi↓ */\n \t\t\tupdate_tp_entries(tp, nparent);\n-\t\t\topt->num_changes++;\n \t\t}\n \t}\n \n@@ -557,7 +555,6 @@ struct combine_diff_path *diff_tree_paths(\n \tconst struct object_id **parents_oid, int nparent,\n \tstruct strbuf *base, struct diff_options *opt)\n {\n-\topt->num_changes = 0;\n \tp = ll_diff_tree_paths(p, oid, parents_oid, nparent, base, opt);\n \n \t/*\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403400","messageId":"23fd52c3b84e518af7c22aa3edd0447dbaeac3d7.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 10/14] commit-graph.c: sort index into commits list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:52:01Z","receivedAt":"2020-08-11T20:52:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For locality, 'compute_bloom_filters()' sorts the commits for which it\nwants to compute Bloom filters in a preferred order (cf., 3d11275505\n(commit-graph: examine commits by generation number, 2020-03-30) for\ndetails).\n\nA future patch will want to recover the new graph position of each\ncommit. Since the 'packed_commit_list' already stores a double-pointer,\navoid a 'COPY_ARRAY' and instead keep track of an index into the\noriginal list. (Use an integer index instead of a memory address, since\nthis involves a needlessly confusing triple-pointer).\n\nAlter the two sorting routines 'commit_pos_cmp' and 'commit_gen_cmp' to\ntake into account the packed_commit_list they are sorting with respect\nto. Since 'compute_bloom_filters()' is the only caller for each of those\ncomparison functions, no other call-sites need updating.\n\nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 43 ++++++++++++++++++++++++-------------------\n 1 file changed, 24 insertions(+), 19 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 48d4697f54..0d70545149 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -79,10 +79,18 @@ static void set_commit_pos(struct repository *r, const struct object_id *oid)\n \t*commit_pos_at(&commit_pos, commit) = max_pos++;\n }\n \n-static int commit_pos_cmp(const void *va, const void *vb)\n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+static int commit_pos_cmp(const void *va, const void *vb, void *ctx)\n {\n-\tconst struct commit *a = *(const struct commit **)va;\n-\tconst struct commit *b = *(const struct commit **)vb;\n+\tstruct packed_commit_list *commits = ctx;\n+\n+\tconst struct commit *a = commits->list[*(int *)va];\n+\tconst struct commit *b = commits->list[*(int *)vb];\n \treturn commit_pos_at(&commit_pos, a) -\n \t commit_pos_at(&commit_pos, b);\n }\n@@ -139,10 +147,12 @@ static struct commit_graph_data *commit_graph_data_at(const struct commit *c)\n \treturn data;\n }\n \n-static int commit_gen_cmp(const void *va, const void *vb)\n+static int commit_gen_cmp(const void *va, const void *vb, void *ctx)\n {\n-\tconst struct commit *a = *(const struct commit **)va;\n-\tconst struct commit *b = *(const struct commit **)vb;\n+\tstruct packed_commit_list *commits = ctx;\n+\n+\tconst struct commit *a = commits->list[*(int *)va];\n+\tconst struct commit *b = commits->list[*(int *)vb];\n \n \tuint32_t generation_a = commit_graph_generation(a);\n \tuint32_t generation_b = commit_graph_generation(b);\n@@ -922,11 +932,6 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n-struct packed_commit_list {\n-\tstruct commit **list;\n-\tint nr;\n-\tint alloc;\n-};\n \n struct packed_oid_list {\n \tstruct object_id *list;\n@@ -1408,7 +1413,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct progress *progress = NULL;\n-\tstruct commit **sorted_commits;\n+\tint *sorted_commits;\n \n \tinit_bloom_filters();\n \n@@ -1418,16 +1423,16 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->commits.nr);\n \n \tALLOC_ARRAY(sorted_commits, ctx->commits.nr);\n-\tCOPY_ARRAY(sorted_commits, ctx->commits.list, ctx->commits.nr);\n-\n-\tif (ctx->order_by_pack)\n-\t\tQSORT(sorted_commits, ctx->commits.nr, commit_pos_cmp);\n-\telse\n-\t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n+\tfor (i = 0; i < ctx->commits.nr; i++)\n+\t\tsorted_commits[i] = i;\n+\tQSORT_S(sorted_commits, ctx->commits.nr,\n+\t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n+\t\t&ctx->commits);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tint computed = 0;\n-\t\tstruct commit *c = sorted_commits[i];\n+\t\tint pos = sorted_commits[i];\n+\t\tstruct commit *c = ctx->commits.list[pos];\n \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\tctx->r,\n \t\t\tc,\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403401","messageId":"619e0c619dd12e2bea2b80c7d249ba66fe2a315c.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:52:07Z","receivedAt":"2020-08-11T20:52:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a commit has more than a certain number of changed paths (commonly\n512), the commit-graph machinery represents it as a zero-length filter.\nThis is done since having many entries in the Bloom filter has\nundesirable effects on the false positivity rate.\n\nIn addition to these too-large filters, the commit-graph machinery also\nrepresents commits with no filter and commits with no changed paths in\nthe same way.\n\nWhen writing a commit-graph that aggregates several incremental\ncommit-graph layers (eg., with '--split=replace'), the commit-graph\nmachinery first computes all of the Bloom filters that it wants to write\nbut does not already know about from existing graph layers. Because we\noverload the zero-length filter in the above fashion, this leads to\nrecomputing large filters over and over again.\n\nThis is already undesirable, since it means that we are wasting\nconsiderable effort to discover that a commit with too many changed\npaths, only to throw that effort away (and then repeat the process the\nnext time a roll-up is performed).\n\nIn a subsequent patch, we will add a '--max-new-filters=' option,\nwhich specifies an upper-bound on the number of new filters we are\nwilling to compute from scratch. Suppose that there are 'N' too-large\nfilters, and we specify '--max-new-filters=M'. If 'N >= M', it is\nunlikely that any filters will be generated, since we'll spend most of\nour effort on filters that we ultimately throw away. If 'N < M', filters\nwill trickle in over time, but only at most 'M - N' per-write.\n\nTo address this, add a new chunk which encodes a bitmap where the ith\nbit is on iff the ith commit has zero or at least 512 changed paths.\nLikewise store the maximum number of changed paths we are willing to\nstore in order to prepare for eventually making this value more easily\ncustomizable. When computing Bloom filters, first consult the relevant\nbitmap (in the case that we are rolling up existing layers) to see if\ncomputing the Bloom filter from scratch would be a waste of time.\n\nThis patch implements a new chunk instead of extending the existing BIDX\nand BDAT chunks because modifying these chunks would confuse old\nclients. (Eg., setting the most-significant bit in the BIDX chunk would\nconfuse old clients and require a version bump).\n\nTo allow using the existing bitmap code with 64-bit words, we write the\ndata in network byte order from the 64-bit words. This means we also\nneed to read the array from the commit-graph file by translating each\nword from network byte order using get_be64() when loading the commit\ngraph. (Note that this *could* be delayed until first-use, but a later\npatch will rely on this being initialized early, so we assume the\nup-front cost when parsing instead of delaying initialization).\n\nBy avoiding the need to move to new versions of the BDAT and BIDX chunk,\nwe can give ourselves more time to consider whether or not other\nmodifications to these chunks are worthwhile without holding up this\nchange.\n\nAnother approach would be to introduce a new BIDX chunk (say, one\nidentified by 'BID2') which is identical to the existing BIDX chunk,\nexcept the most-significant bit of each offset is interpreted as \"this\nfilter is too big\" iff looking at a BID2 chunk. This avoids having to\nwrite a bitmap, but forces older clients to rewrite their commit-graphs\n(as well as reduces the theoretical largest Bloom filters we couldl\nwrite, and forces us to maintain the code necessary to translate BIDX\nchunks to BID2 ones). Separately from this patch, I implemented this\nalternate approach and did not find it to be advantageous.\n\nSigned-off-by: Taylor Blau \n---\n .../technical/commit-graph-format.txt | 12 +++\n bloom.h | 2 +-\n commit-graph.c | 96 ++++++++++++++++---\n commit-graph.h | 4 +\n t/t4216-log-bloom.sh | 25 ++++-\n 5 files changed, 124 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 440541045d..5f2d9ab4d7 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -123,6 +123,18 @@ CHUNK DATA:\n of length zero.\n * The BDAT chunk is present if and only if BIDX is present.\n \n+ Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n+ * It starts with a 32-bit unsigned integer specifying the maximum number of\n+ changed-paths that can be stored in a single Bloom filter.\n+ * It then contains a list of 64-bit words (the length of this list is\n+ determined by the width of the chunk) which is a bitmap. The 'i'th bit is\n+ set exactly when the 'i'th commit in the graph has a changed-path Bloom\n+ filter with zero entries (either because the commit is empty, or because\n+ it contains more than 512 changed paths).\n+ * The BFXL chunk is present only when the BIDX and BDAT chunks are\n+ also present.\n+\n+\n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n This list of H-byte hashes describe a set of B commit-graph files that\n form a commit-graph chain. The graph position for the ith commit in this\ndiff --git a/bloom.h b/bloom.h\nindex 3f19e3fca4..464d9b57de 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -33,7 +33,7 @@ struct bloom_filter_settings {\n \t * The maximum number of changed paths per commit\n \t * before declaring a Bloom filter to be too-large.\n \t *\n-\t * Not written to the commit-graph file.\n+\t * Written to the 'BFXL' chunk (instead of 'BDAT').\n \t */\n \tuint32_t max_changed_paths;\n };\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8964453433..ea0583298c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -41,8 +41,9 @@ void git_test_write_commit_graph_or_die(void)\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n+#define GRAPH_CHUNKID_BLOOMLARGE 0x4246584c /* \"BFXL\" */\n #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n-#define MAX_NUM_CHUNKS 7\n+#define MAX_NUM_CHUNKS 8\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -429,6 +430,24 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n \t\t\t}\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BLOOMLARGE:\n+\t\t\tif (graph->chunk_bloom_large_filters)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n+\t\t\t\tsize_t alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n+\t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n+\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n+\t\t\t\tif (alloc) {\n+\t\t\t\t\tsize_t j;\n+\t\t\t\t\tgraph->bloom_large = bitmap_word_alloc(alloc);\n+\n+\t\t\t\t\tfor (j = 0; j < graph->bloom_large->word_alloc; j++)\n+\t\t\t\t\t\tgraph->bloom_large->words[j] = get_be64(\n+\t\t\t\t\t\t\tgraph->chunk_bloom_large_filters + j * sizeof(eword_t));\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tbreak;\n \t\t}\n \n \t\tif (chunk_repeated) {\n@@ -443,7 +462,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t/* We need both the bloom chunks to exist together. Else ignore the data */\n \t\tgraph->chunk_bloom_indexes = NULL;\n \t\tgraph->chunk_bloom_data = NULL;\n+\t\tgraph->chunk_bloom_large_filters = NULL;\n \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n+\t\tbitmap_free(graph->bloom_large);\n \t}\n \n \thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n@@ -932,6 +953,20 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n+static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n+\t\t\t\t\t const struct commit *c)\n+{\n+\tuint32_t graph_pos = commit_graph_position(c);\n+\tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\treturn 0;\n+\n+\twhile (g && graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (!(g && g->bloom_large))\n+\t\treturn 0;\n+\treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n+}\n \n struct packed_oid_list {\n \tstruct object_id *list;\n@@ -970,8 +1005,10 @@ struct write_commit_graph_context {\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \n+\tint count_bloom_filter_known_large;\n \tint count_bloom_filter_found_large;\n \tint count_bloom_filter_computed;\n+\tstruct bitmap *bloom_large;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1235,6 +1272,23 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \treturn 0;\n }\n \n+static int write_graph_chunk_bloom_large(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\tsize_t i, alloc = ctx->commits.nr / BITS_IN_EWORD;\n+\tif (ctx->commits.nr % BITS_IN_EWORD)\n+\t\talloc++;\n+\tif (alloc > ctx->bloom_large->word_alloc)\n+\t\tBUG(\"write_graph_chunk_bloom_large: bitmap not large enough\");\n+\n+\ttrace2_region_enter(\"commit-graph\", \"bloom_large\", ctx->r);\n+\thashwrite_be32(f, ctx->bloom_settings->max_changed_paths);\n+\tfor (i = 0; i < ctx->bloom_large->word_alloc; i++)\n+\t\thashwrite_be64(f, ctx->bloom_large->words[i]);\n+\ttrace2_region_leave(\"commit-graph\", \"bloom_large\", ctx->r);\n+\treturn 0;\n+}\n+\n static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n@@ -1398,6 +1452,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \tstruct json_writer jw = JSON_WRITER_INIT;\n \n \tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"filter_known_large\",\n+\t\t\t ctx->count_bloom_filter_known_large);\n \tjw_object_intmax(&jw, \"filter_found_large\",\n \t\t\t ctx->count_bloom_filter_found_large);\n \tjw_object_intmax(&jw, \"filter_computed\",\n@@ -1416,6 +1472,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint *sorted_commits;\n \n \tinit_bloom_filters();\n+\tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n \n \tif (ctx->report_progress)\n \t\tprogress = start_delayed_progress(\n@@ -1430,21 +1487,28 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t&ctx->commits);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tint computed = 0;\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n-\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n-\t\t\tctx->r,\n-\t\t\tc,\n-\t\t\t1,\n-\t\t\tctx->bloom_settings,\n-\t\t\t&computed);\n-\t\tif (computed) {\n-\t\t\tctx->count_bloom_filter_computed++;\n-\t\t\tif (filter && !filter->len)\n-\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n+\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\tctx->count_bloom_filter_known_large++;\n+\t\t} else {\n+\t\t\tint computed = 0;\n+\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\t\tctx->r,\n+\t\t\t\tc,\n+\t\t\t\t1,\n+\t\t\t\tctx->bloom_settings,\n+\t\t\t\t&computed);\n+\t\t\tif (computed) {\n+\t\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\t\tif (filter && !filter->len) {\n+\t\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\t}\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n@@ -1764,6 +1828,11 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n \t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n+\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMLARGE;\n+\t\tchunks[num_chunks].size = sizeof(eword_t) * ctx->bloom_large->word_alloc\n+\t\t\t\t\t+ sizeof(uint32_t);\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_large;\n+\t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n@@ -2503,6 +2572,7 @@ void free_commit_graph(struct commit_graph *g)\n \t}\n \tfree(g->filename);\n \tfree(g->bloom_filter_settings);\n+\tbitmap_free(g->bloom_large);\n \tfree(g);\n }\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex d9acb22bac..ddbca1b59d 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,6 +4,7 @@\n #include \"git-compat-util.h\"\n #include \"object-store.h\"\n #include \"oidset.h\"\n+#include \"ewah/ewok.h\"\n \n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n #define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n@@ -71,6 +72,9 @@ struct commit_graph {\n \tconst unsigned char *chunk_base_graphs;\n \tconst unsigned char *chunk_bloom_indexes;\n \tconst unsigned char *chunk_bloom_data;\n+\tconst unsigned char *chunk_bloom_large_filters;\n+\n+\tstruct bitmap *bloom_large;\n \n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 21b67677ef..6859d85369 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -33,7 +33,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \tgit commit-graph write --reachable --changed-paths\n '\n graph_read_expect () {\n-\tNUM_CHUNKS=5\n+\tNUM_CHUNKS=6\n \tcat >expect <<- EOF\n \theader: 43475048 1 1 $NUM_CHUNKS 0\n \tnum_commits: $1\n@@ -262,5 +262,28 @@ test_expect_success 'correctly report changes over limit' '\n \t\tdone\n \t)\n '\n+test_bloom_filters_computed () {\n+\tcommit_graph_args=$1\n+\tbloom_trace_prefix=\"{\\\"filter_known_large\\\":$2,\\\"filter_found_large\\\":$3,\\\"filter_computed\\\":$4\"\n+\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n+\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write $commit_graph_args &&\n+\tgrep \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.event\"\n+}\n+\n+test_expect_success 'Bloom generation does not recompute too-large filters' '\n+\t(\n+\t\tcd limits &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t--split=replace &&\n+\t\ttest_commit c1 filter &&\n+\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n+\t\t\t2 0 1\n+\t)\n+'\n \n test_done\n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403402","messageId":"b2e33ecba880f4b49af7a94fb0decb08929dff36.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 13/14] commit-graph: rename 'split_commit_graph_opts'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:52:11Z","receivedAt":"2020-08-11T20:52:16Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the subsequent commit, additional options will be added to the\ncommit-graph API which have nothing to do with splitting.\n\nRename the 'split_commit_graph_opts' structure to the more-generic\n'commit_graph_opts' to encompass both.\n\nSuggsted-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 20 ++++++++++----------\n commit-graph.c | 40 ++++++++++++++++++++--------------------\n commit-graph.h | 6 +++---\n 3 files changed, 33 insertions(+), 33 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ba5584463f..38f5f57d15 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,7 +119,7 @@ static int graph_verify(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n-static struct split_commit_graph_opts split_opts;\n+static struct commit_graph_opts write_opts;\n \n static int write_option_parse_split(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n@@ -187,24 +187,24 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n \t\t\tN_(\"enable computation for changed paths\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\twrite_option_parse_split),\n-\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\tOPT_INTEGER(0, \"max-commits\", &write_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n-\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\tOPT_INTEGER(0, \"size-multiple\", &write_opts.size_multiple,\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n-\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n \t\tOPT_END(),\n \t};\n \n \topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n-\tsplit_opts.size_multiple = 2;\n-\tsplit_opts.max_commits = 0;\n-\tsplit_opts.expire_time = 0;\n+\twrite_opts.size_multiple = 2;\n+\twrite_opts.max_commits = 0;\n+\twrite_opts.expire_time = 0;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -232,7 +232,7 @@ static int graph_write(int argc, const char **argv)\n \todb = find_odb(the_repository, opts.obj_dir);\n \n \tif (opts.reachable) {\n-\t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n+\t\tif (write_commit_graph_reachable(odb, flags, &write_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n@@ -261,7 +261,7 @@ static int graph_write(int argc, const char **argv)\n \t\t\t opts.stdin_packs ? &pack_indexes : NULL,\n \t\t\t opts.stdin_commits ? &commits : NULL,\n \t\t\t flags,\n-\t\t\t &split_opts))\n+\t\t\t &write_opts))\n \t\tresult = 1;\n \n cleanup:\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ea0583298c..6886f319a5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1001,7 +1001,7 @@ struct write_commit_graph_context {\n \t\t changed_paths:1,\n \t\t order_by_pack:1;\n \n-\tconst struct split_commit_graph_opts *split_opts;\n+\tconst struct commit_graph_opts *opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \n@@ -1342,8 +1342,8 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1543,7 +1543,7 @@ static int add_ref_to_set(const char *refname,\n \n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t\t\t const struct commit_graph_opts *opts)\n {\n \tstruct oidset commits = OIDSET_INIT;\n \tstruct refs_cb_data data;\n@@ -1560,7 +1560,7 @@ int write_commit_graph_reachable(struct object_directory *odb,\n \tstop_progress(&data.progress);\n \n \tresult = write_commit_graph(odb, NULL, &commits,\n-\t\t\t\t flags, split_opts);\n+\t\t\t\t flags, opts);\n \n \toidset_clear(&commits);\n \treturn result;\n@@ -1675,8 +1675,8 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tctx->num_extra_edges = 0;\n \tif (ctx->report_progress)\n@@ -1962,13 +1962,13 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tint max_commits = 0;\n \tint size_mult = 2;\n \n-\tif (ctx->split_opts) {\n-\t\tmax_commits = ctx->split_opts->max_commits;\n+\tif (ctx->opts) {\n+\t\tmax_commits = ctx->opts->max_commits;\n \n-\t\tif (ctx->split_opts->size_multiple)\n-\t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t\tif (ctx->opts->size_multiple)\n+\t\t\tsize_mult = ctx->opts->size_multiple;\n \n-\t\tflags = ctx->split_opts->flags;\n+\t\tflags = ctx->opts->flags;\n \t}\n \n \tg = ctx->r->objects->commit_graph;\n@@ -2146,8 +2146,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n \n-\tif (ctx->split_opts && ctx->split_opts->expire_time)\n-\t\texpire_time = ctx->split_opts->expire_time;\n+\tif (ctx->opts && ctx->opts->expire_time)\n+\t\texpire_time = ctx->opts->expire_time;\n \tif (!ctx->split) {\n \t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n@@ -2198,7 +2198,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t const struct commit_graph_opts *opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -2215,7 +2215,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n-\tctx->split_opts = split_opts;\n+\tctx->opts = opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n@@ -2263,15 +2263,15 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\t}\n \t\t}\n \n-\t\tif (ctx->split_opts)\n-\t\t\treplace = ctx->split_opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n+\t\tif (ctx->opts)\n+\t\t\treplace = ctx->opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n \t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n-\t\tctx->oids.alloc = split_opts->max_commits;\n+\tif (ctx->split && opts && ctx->oids.alloc > opts->max_commits)\n+\t\tctx->oids.alloc = opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->odb);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex ddbca1b59d..af08c4505d 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -109,7 +109,7 @@ enum commit_graph_split_flags {\n \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n };\n \n-struct split_commit_graph_opts {\n+struct commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n@@ -124,12 +124,12 @@ struct split_commit_graph_opts {\n */\n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t\t\t const struct commit_graph_opts *opts);\n int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t const struct commit_graph_opts *opts);\n \n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n \n-- \n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403403","messageId":"09f6871f66bff838c067a3e0d23cd4622171f3bd.1597178915.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"[PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T20:52:14Z","receivedAt":"2020-08-11T20:52:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a command-line flag and configuration variable to fill in the\n'max_new_filters' variable introduced by the previous patch.\n\nThe command-line option '--max-new-filters' takes precedence over\n'commitGraph.maxNewFilters', which is the default value.\n'--no-max-new-filters' can also be provided, which sets the value back\nto '-1', indicating that an unlimited number of new Bloom filters may be\ngenerated. (OPT_INTEGER only allows setting the '--no-' variant back to\n'0', hence a custom callback was used instead).\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 4 +++\n Documentation/git-commit-graph.txt | 4 +++\n bloom.c | 15 +++++++++++\n builtin/commit-graph.c | 39 +++++++++++++++++++++++++---\n commit-graph.c | 27 ++++++++++++++++---\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 19 ++++++++++++++\n 7 files changed, 102 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex cff0797b54..4582c39fc4 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitGraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..9c887d5d79 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,10 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Overrides the `commitGraph.maxNewFilters` configuration.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex ed54e96e57..8d07209c6b 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -51,6 +51,21 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \telse\n \t\tstart_index = 0;\n \n+\tif ((start_index == end_index) &&\n+\t (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n+\t\t/*\n+\t\t * If the filter is zero-length, either (1) the filter has no\n+\t\t * changes, (2) the filter has too many changes, or (3) it\n+\t\t * wasn't computed (eg., due to '--max-new-filters').\n+\t\t *\n+\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n+\t\t * for this Bloom filter. If it is unset, then it wasn't\n+\t\t * computed. In that case, return nothing, since we don't have\n+\t\t * that filter in the graph.\n+\t\t */\n+\t\treturn 0;\n+\t}\n+\n \tfilter->len = end_index - start_index;\n \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n \t\t\t\t\tsizeof(unsigned char) * start_index +\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 38f5f57d15..3500a6e1f1 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n \n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n \n@@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -270,6 +293,16 @@ static int graph_write(int argc, const char **argv)\n \treturn result;\n }\n \n+static int git_commit_graph_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, \"commitgraph.maxnewfilters\")) {\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_commit_graph_options[] = {\n@@ -283,7 +316,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_commit_graph_usage,\n \t\t\t\t builtin_commit_graph_options);\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(git_commit_graph_config, &opts);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_commit_graph_options,\n \t\t\t builtin_commit_graph_usage,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6886f319a5..4aae5471e3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -954,7 +954,8 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n }\n \n static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n-\t\t\t\t\t const struct commit *c)\n+\t\t\t\t\t const struct commit *c,\n+\t\t\t\t\t uint32_t max_changed_paths)\n {\n \tuint32_t graph_pos = commit_graph_position(c);\n \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n@@ -965,6 +966,17 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n \n \tif (!(g && g->bloom_large))\n \t\treturn 0;\n+\tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n+\t\t/*\n+\t\t * Force all commits which are subject to a different\n+\t\t * 'max_changed_paths' limit to be recomputed from scratch.\n+\t\t *\n+\t\t * Note that this could likely be improved, but is ignored since\n+\t\t * all real-world graphs set the maximum number of changed paths\n+\t\t * at 512.\n+\t\t */\n+\t\treturn 0;\n+\t}\n \treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n }\n \n@@ -1470,6 +1482,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tint *sorted_commits;\n+\tint max_new_filters;\n \n \tinit_bloom_filters();\n \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n@@ -1486,10 +1499,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n \t\t&ctx->commits);\n \n+\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n-\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n+\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph,\n+\t\t\t\t\t\t c,\n+\t\t\t\t\t\t ctx->bloom_settings->max_changed_paths)) {\n \t\t\tbitmap_set(ctx->bloom_large, pos);\n \t\t\tctx->count_bloom_filter_known_large++;\n \t\t} else {\n@@ -1497,7 +1515,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\t\tctx->r,\n \t\t\t\tc,\n-\t\t\t\t1,\n+\t\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n \t\t\t\tctx->bloom_settings,\n \t\t\t\t&computed);\n \t\t\tif (computed) {\n@@ -1507,7 +1525,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\t\tctx->count_bloom_filter_found_large++;\n \t\t\t\t}\n \t\t\t}\n-\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\t\tif (filter)\n+\t\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\t}\n \t\tdisplay_progress(progress, i + 1);\n \t}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex af08c4505d..75ef83708b 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -114,6 +114,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags flags;\n+\tint max_new_filters;\n };\n \n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 6859d85369..3aab8ffbe3 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -286,4 +286,23 @@ test_expect_success 'Bloom generation does not recompute too-large filters' '\n \t)\n '\n \n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n+\t\t\t2 0 2\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=1\" \\\n+\t\t\t2 0 1\n+\t)\n+'\n+\n test_done\n-- \n2.28.0.rc1.13.ge78abce653\n"},{"id":"403408","messageId":"4f487dca-a938-b4c4-8b77-a202209459cb@gmail.com","threadId":"53975","inReplyTo":"619e0c619dd12e2bea2b80c7d249ba66fe2a315c.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-11T21:11:49Z","receivedAt":"2020-08-11T21:11:53Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/11/20 4:52 PM, Taylor Blau wrote:\n> To allow using the existing bitmap code with 64-bit words, we write the\n> data in network byte order from the 64-bit words. This means we also\n> need to read the array from the commit-graph file by translating each\n> word from network byte order using get_be64() when loading the commit\n> graph. (Note that this *could* be delayed until first-use, but a later\n> patch will rely on this being initialized early, so we assume the\n> up-front cost when parsing instead of delaying initialization).\n\nI don't think this is correct to do. This means that every commit-graph\nload will load the entire large bloom filter chunk into memory before\nallowing a single commit to be read from the graph.\n\nIn the case of a very large commit-graph (> 1 million commits), this\nwould cause a significant initial cost that is not necessarily needed\nfor anything. For example, \"git log -1\" will be delayed by this action.\n\nIf I understand correctly, this bloom_large bitmap is only used when\nwriting Bloom filters. At that point, reading the entire bitmap from\ndisk into memory is inexpensive compared to the time saved by the\nfeature.\n\n> @@ -429,6 +430,24 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n> \t\t\t}\n> \t\t\tbreak;\n> +\n> +\t\tcase GRAPH_CHUNKID_BLOOMLARGE:\n> +\t\t\tif (graph->chunk_bloom_large_filters)\n> +\t\t\t\tchunk_repeated = 1;\n> +\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n\nThis is guarded against commitGraph.readChangedPaths, which is good,\nbut that's not enough to claim that we need the bloom_large bitmap\nin this process.\n\n> +\t\t\t\tsize_t alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n\nIf we store this inside the commit_graph struct, we can save this\nsize for later so...\n\n> +\t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n> +\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n\n...this portion can be done only when we are about to read from the\nbitmap.\n\n> +\t\t\t\tif (alloc) {\n> +\t\t\t\t\tsize_t j;\n> +\t\t\t\t\tgraph->bloom_large = bitmap_word_alloc(alloc);\n> +\n> +\t\t\t\t\tfor (j = 0; j < graph->bloom_large->word_alloc; j++)\n> +\t\t\t\t\t\tgraph->bloom_large->words[j] = get_be64(\n> +\t\t\t\t\t\t\tgraph->chunk_bloom_large_filters + j * sizeof(eword_t));\n> +\t\t\t\t}\n\n\n\n> +\t\t\t}\n> +\t\t\tbreak;\n> \t\t}\n> \n> \t\tif (chunk_repeated) {\n> @@ -443,7 +462,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> \t\t/* We need both the bloom chunks to exist together. Else ignore the data */\n> \t\tgraph->chunk_bloom_indexes = NULL;\n> \t\tgraph->chunk_bloom_data = NULL;\n> +\t\tgraph->chunk_bloom_large_filters = NULL;\n> \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n> +\t\tbitmap_free(graph->bloom_large);\n\nPerhaps this bitmap_free() needs a check to see if we\ninitialized it (with my recommended lazy-loading), but\nmaybe not?\n\n> \t}\n> \n> \thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n> @@ -932,6 +953,20 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n> }\n> \n> +static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> +\t\t\t\t\t const struct commit *c)\n> +{\n> +\tuint32_t graph_pos = commit_graph_position(c);\n> +\tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n> +\t\treturn 0;\n> +\n> +\twhile (g && graph_pos < g->num_commits_in_base)\n> +\t\tg = g->base_graph;\n> +\n> +\tif (!(g && g->bloom_large))\n> +\t\treturn 0;\n\nHere is where we can check if the size of the chunk is\nnon-zero and if the g->bloom_large bitmap is uninitialized.\nSince we are caring about this information, it is now time\nto do the network-byte transition of the full bitmap.\n\n> +\treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n> +}\n> \nThanks,\n-Stolee\n"},{"id":"403409","messageId":"20200811211815.GA65971@syl.lan","threadId":"53975","inReplyTo":"4f487dca-a938-b4c4-8b77-a202209459cb@gmail.com","subject":"Re: [PATCH v3 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T21:18:15Z","receivedAt":"2020-08-11T21:18:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 11, 2020 at 05:11:49PM -0400, Derrick Stolee wrote:\n> On 8/11/20 4:52 PM, Taylor Blau wrote:\n> > To allow using the existing bitmap code with 64-bit words, we write the\n> > data in network byte order from the 64-bit words. This means we also\n> > need to read the array from the commit-graph file by translating each\n> > word from network byte order using get_be64() when loading the commit\n> > graph. (Note that this *could* be delayed until first-use, but a later\n> > patch will rely on this being initialized early, so we assume the\n> > up-front cost when parsing instead of delaying initialization).\n>\n> I don't think this is correct to do. This means that every commit-graph\n> load will load the entire large bloom filter chunk into memory before\n> allowing a single commit to be read from the graph.\n>\n> In the case of a very large commit-graph (> 1 million commits), this\n> would cause a significant initial cost that is not necessarily needed\n> for anything. For example, \"git log -1\" will be delayed by this action.\n>\n> If I understand correctly, this bloom_large bitmap is only used when\n> writing Bloom filters. At that point, reading the entire bitmap from\n> disk into memory is inexpensive compared to the time saved by the\n> feature.\n\nThat's not quite correct. By this point in the patch series, we only use\nthis bitmap for writing, but in the final patch, we will use it in\n'load_bloom_filter_from_graph()' (see that patch for the details of why,\nthere is an explanatory comment to that effect).\n\nSo, the way that this lazy initialization was written was subtly\nincorrect, since calling 'load_bloom_filter_from_graph' would return the\nwrong value depending on what was or wasn't called before it (namely\nwhether or not we had initialized the bitmap).\n\n> > @@ -429,6 +430,24 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> > \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n> > \t\t\t}\n> > \t\t\tbreak;\n> > +\n> > +\t\tcase GRAPH_CHUNKID_BLOOMLARGE:\n> > +\t\t\tif (graph->chunk_bloom_large_filters)\n> > +\t\t\t\tchunk_repeated = 1;\n> > +\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n>\n> This is guarded against commitGraph.readChangedPaths, which is good,\n> but that's not enough to claim that we need the bloom_large bitmap\n> in this process.\n>\n> > +\t\t\t\tsize_t alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n>\n> If we store this inside the commit_graph struct, we can save this\n> size for later so...\n>\n> > +\t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n> > +\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n>\n> ...this portion can be done only when we are about to read from the\n> bitmap.\n\nRight, that all is clear.\n\n>\n> > +\t\t\t\tif (alloc) {\n> > +\t\t\t\t\tsize_t j;\n> > +\t\t\t\t\tgraph->bloom_large = bitmap_word_alloc(alloc);\n> > +\n> > +\t\t\t\t\tfor (j = 0; j < graph->bloom_large->word_alloc; j++)\n> > +\t\t\t\t\t\tgraph->bloom_large->words[j] = get_be64(\n> > +\t\t\t\t\t\t\tgraph->chunk_bloom_large_filters + j * sizeof(eword_t));\n> > +\t\t\t\t}\n>\n>\n>\n> > +\t\t\t}\n> > +\t\t\tbreak;\n> > \t\t}\n> >\n> > \t\tif (chunk_repeated) {\n> > @@ -443,7 +462,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> > \t\t/* We need both the bloom chunks to exist together. Else ignore the data */\n> > \t\tgraph->chunk_bloom_indexes = NULL;\n> > \t\tgraph->chunk_bloom_data = NULL;\n> > +\t\tgraph->chunk_bloom_large_filters = NULL;\n> > \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n> > +\t\tbitmap_free(graph->bloom_large);\n>\n> Perhaps this bitmap_free() needs a check to see if we\n> initialized it (with my recommended lazy-loading), but\n> maybe not?\n\nIt does not ('bitmap_free()' understands that a NULL argument is a noop,\nlike 'free()').\n>\n> > \t}\n> >\n> > \thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n> > @@ -932,6 +953,20 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> > \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n> > }\n> >\n> > +static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> > +\t\t\t\t\t const struct commit *c)\n> > +{\n> > +\tuint32_t graph_pos = commit_graph_position(c);\n> > +\tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n> > +\t\treturn 0;\n> > +\n> > +\twhile (g && graph_pos < g->num_commits_in_base)\n> > +\t\tg = g->base_graph;\n> > +\n> > +\tif (!(g && g->bloom_large))\n> > +\t\treturn 0;\n>\n> Here is where we can check if the size of the chunk is\n> non-zero and if the g->bloom_large bitmap is uninitialized.\n> Since we are caring about this information, it is now time\n> to do the network-byte transition of the full bitmap.\n\nYes, that's right. But again, we'll need to move this out to a\nnon-static helper function so that we can call it from bloom.c in the\nfinal patch.\n\nI'm admittedly a little unsure of what to do next, since the changes are\nscoped to this and the remaining patches (really just this and the final\npatch).\n\nI guess I'll try the other approach of replacing these two patches\nby sending emails in response so that Junio can take those (I'll avoid\nsending a fixup patch).\n\n> > +\treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n> > +}\n> >\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"403410","messageId":"20200811211830.GS2898@szeder.dev","threadId":"53975","inReplyTo":"e714e54240bb339d1ecebcea157b734abf1c14ef.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-08-11T21:18:30Z","receivedAt":"2020-08-11T21:18:35Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 04:51:19PM -0400, Taylor Blau wrote:\n> Many places in the code often need a pointer to the commit-graph's\n> 'struct bloom_filter_settings', in which case they often take the value\n> from the top-most commit-graph.\n> \n> In the non-split case, this works as expected. In the split case,\n> however, things get a little tricky. Not all layers in a chain of\n> incremental commit-graphs are required to themselves have Bloom data,\n> and so whether or not some part of the code uses Bloom filters depends\n> entirely on whether or not the top-most level of the commit-graph chain\n> has Bloom filters.\n> \n> This has been the behavior since Bloom filters were introduced, and has\n> been codified into the tests since a759bfa9ee (t4216: add end to end\n> tests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\n> requires that Bloom filters are not used in exactly the case described\n> earlier.\n> \n> There is no reason that this needs to be the case, since it is perfectly\n> valid for commits in an earlier layer to have Bloom filters when commits\n> in a newer layer do not.\n> \n> Since Bloom settings are guaranteed to be the same for any layer in a\n> chain that has Bloom data,\n\nIs it? Where is that guaranteed?\n\n> it is sufficient to traverse the\n> '->base_graph' pointer until either (1) a non-null 'struct\n> bloom_filter_settings *' is found, or (2) until we are at the root of\n> the commit-graph chain.\n> \n> Introduce a 'get_bloom_filter_settings()' function that does just this,\n> and use it instead of purely dereferencing the top-most graph's\n> '->bloom_filter_settings' pointer.\n> \n> While we're at it, add an additional test in t5324 to guard against code\n> in the commit-graph writing machinery that doesn't correctly handle a\n> NULL 'struct bloom_filter *'.\n> \n> Co-authored-by: Derrick Stolee \n> Signed-off-by: Taylor Blau \n"},{"id":"403411","messageId":"20200811212118.GA66447@syl.lan","threadId":"53975","inReplyTo":"20200811211830.GS2898@szeder.dev","subject":"Re: [PATCH v3 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T21:21:18Z","receivedAt":"2020-08-11T21:21:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 11, 2020 at 11:18:30PM +0200, SZEDER Gábor wrote:\n> On Tue, Aug 11, 2020 at 04:51:19PM -0400, Taylor Blau wrote:\n> > Many places in the code often need a pointer to the commit-graph's\n> > 'struct bloom_filter_settings', in which case they often take the value\n> > from the top-most commit-graph.\n> >\n> > In the non-split case, this works as expected. In the split case,\n> > however, things get a little tricky. Not all layers in a chain of\n> > incremental commit-graphs are required to themselves have Bloom data,\n> > and so whether or not some part of the code uses Bloom filters depends\n> > entirely on whether or not the top-most level of the commit-graph chain\n> > has Bloom filters.\n> >\n> > This has been the behavior since Bloom filters were introduced, and has\n> > been codified into the tests since a759bfa9ee (t4216: add end to end\n> > tests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\n> > requires that Bloom filters are not used in exactly the case described\n> > earlier.\n> >\n> > There is no reason that this needs to be the case, since it is perfectly\n> > valid for commits in an earlier layer to have Bloom filters when commits\n> > in a newer layer do not.\n> >\n> > Since Bloom settings are guaranteed to be the same for any layer in a\n> > chain that has Bloom data,\n>\n> Is it? Where is that guaranteed?\n\nThere is no mechanism whatsoever to customize these settings that is\nexposed to the user (except for the undocumented 'GIT_TEST' environment\nvariables).\n\n> > it is sufficient to traverse the\n> > '->base_graph' pointer until either (1) a non-null 'struct\n> > bloom_filter_settings *' is found, or (2) until we are at the root of\n> > the commit-graph chain.\n> >\n> > Introduce a 'get_bloom_filter_settings()' function that does just this,\n> > and use it instead of purely dereferencing the top-most graph's\n> > '->bloom_filter_settings' pointer.\n> >\n> > While we're at it, add an additional test in t5324 to guard against code\n> > in the commit-graph writing machinery that doesn't correctly handle a\n> > NULL 'struct bloom_filter *'.\n> >\n> > Co-authored-by: Derrick Stolee \n> > Signed-off-by: Taylor Blau \nThanks,\nTaylor\n"},{"id":"403413","messageId":"20200811212716.GA13422@szeder.dev","threadId":"53975","inReplyTo":"20200811212118.GA66447@syl.lan","subject":"Re: [PATCH v3 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-08-11T21:27:16Z","receivedAt":"2020-08-11T21:27:24Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 05:21:18PM -0400, Taylor Blau wrote:\n> On Tue, Aug 11, 2020 at 11:18:30PM +0200, SZEDER Gábor wrote:\n> > On Tue, Aug 11, 2020 at 04:51:19PM -0400, Taylor Blau wrote:\n> > > Many places in the code often need a pointer to the commit-graph's\n> > > 'struct bloom_filter_settings', in which case they often take the value\n> > > from the top-most commit-graph.\n> > >\n> > > In the non-split case, this works as expected. In the split case,\n> > > however, things get a little tricky. Not all layers in a chain of\n> > > incremental commit-graphs are required to themselves have Bloom data,\n> > > and so whether or not some part of the code uses Bloom filters depends\n> > > entirely on whether or not the top-most level of the commit-graph chain\n> > > has Bloom filters.\n> > >\n> > > This has been the behavior since Bloom filters were introduced, and has\n> > > been codified into the tests since a759bfa9ee (t4216: add end to end\n> > > tests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\n> > > requires that Bloom filters are not used in exactly the case described\n> > > earlier.\n> > >\n> > > There is no reason that this needs to be the case, since it is perfectly\n> > > valid for commits in an earlier layer to have Bloom filters when commits\n> > > in a newer layer do not.\n> > >\n> > > Since Bloom settings are guaranteed to be the same for any layer in a\n> > > chain that has Bloom data,\n> >\n> > Is it? Where is that guaranteed?\n> \n> There is no mechanism whatsoever to customize these settings that is\n> exposed to the user (except for the undocumented 'GIT_TEST' environment\n> variables).\n\nLet me rephrase it, then: where is it written in the commit-graph\nformat specification that these must be the same in all layers?\n\nNowhere.\n\n> > > it is sufficient to traverse the\n> > > '->base_graph' pointer until either (1) a non-null 'struct\n> > > bloom_filter_settings *' is found, or (2) until we are at the root of\n> > > the commit-graph chain.\n> > >\n> > > Introduce a 'get_bloom_filter_settings()' function that does just this,\n> > > and use it instead of purely dereferencing the top-most graph's\n> > > '->bloom_filter_settings' pointer.\n> > >\n> > > While we're at it, add an additional test in t5324 to guard against code\n> > > in the commit-graph writing machinery that doesn't correctly handle a\n> > > NULL 'struct bloom_filter *'.\n> > >\n> > > Co-authored-by: Derrick Stolee \n> > > Signed-off-by: Taylor Blau \n> Thanks,\n> Taylor\n"},{"id":"403414","messageId":"20200811213411.GB66656@syl.lan","threadId":"53975","inReplyTo":"20200811212716.GA13422@szeder.dev","subject":"Re: [PATCH v3 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T21:34:11Z","receivedAt":"2020-08-11T21:34:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 11, 2020 at 11:27:16PM +0200, SZEDER Gábor wrote:\n> On Tue, Aug 11, 2020 at 05:21:18PM -0400, Taylor Blau wrote:\n> > On Tue, Aug 11, 2020 at 11:18:30PM +0200, SZEDER Gábor wrote:\n> > > On Tue, Aug 11, 2020 at 04:51:19PM -0400, Taylor Blau wrote:\n> > > > Many places in the code often need a pointer to the commit-graph's\n> > > > 'struct bloom_filter_settings', in which case they often take the value\n> > > > from the top-most commit-graph.\n> > > >\n> > > > In the non-split case, this works as expected. In the split case,\n> > > > however, things get a little tricky. Not all layers in a chain of\n> > > > incremental commit-graphs are required to themselves have Bloom data,\n> > > > and so whether or not some part of the code uses Bloom filters depends\n> > > > entirely on whether or not the top-most level of the commit-graph chain\n> > > > has Bloom filters.\n> > > >\n> > > > This has been the behavior since Bloom filters were introduced, and has\n> > > > been codified into the tests since a759bfa9ee (t4216: add end to end\n> > > > tests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\n> > > > requires that Bloom filters are not used in exactly the case described\n> > > > earlier.\n> > > >\n> > > > There is no reason that this needs to be the case, since it is perfectly\n> > > > valid for commits in an earlier layer to have Bloom filters when commits\n> > > > in a newer layer do not.\n> > > >\n> > > > Since Bloom settings are guaranteed to be the same for any layer in a\n> > > > chain that has Bloom data,\n> > >\n> > > Is it? Where is that guaranteed?\n> >\n> > There is no mechanism whatsoever to customize these settings that is\n> > exposed to the user (except for the undocumented 'GIT_TEST' environment\n> > variables).\n>\n> Let me rephrase it, then: where is it written in the commit-graph\n> format specification that these must be the same in all layers?\n>\n> Nowhere.\n\nOK. We can certainly document that this is the case. For this purpose,\nall we really care about is that the graph _has_ Bloom filters anywhere.\nIf you wanted to return the exact matching settings, you could also\nprovide a commit and return the settings belonging to the graph that\ncontains that commit.\n\nIn the case where we don't have a commit, we could use the default\nsettings instead.\n\nI think that we are a little bit dealing with a problem that doesn't\nexist, since we do not document the sole method by which you would\nchange these settings. So, maybe we can think more about this, but my\npreference would be to leave this patch alone.\n\nMaybe Stolee can chime in, too.\n\n> > > > it is sufficient to traverse the\n> > > > '->base_graph' pointer until either (1) a non-null 'struct\n> > > > bloom_filter_settings *' is found, or (2) until we are at the root of\n> > > > the commit-graph chain.\n> > > >\n> > > > Introduce a 'get_bloom_filter_settings()' function that does just this,\n> > > > and use it instead of purely dereferencing the top-most graph's\n> > > > '->bloom_filter_settings' pointer.\n> > > >\n> > > > While we're at it, add an additional test in t5324 to guard against code\n> > > > in the commit-graph writing machinery that doesn't correctly handle a\n> > > > NULL 'struct bloom_filter *'.\n> > > >\n> > > > Co-authored-by: Derrick Stolee \n> > > > Signed-off-by: Taylor Blau \n> > Thanks,\n> > Taylor\nThanks,\nTaylor\n"},{"id":"403416","messageId":"20200811220503.GC66656@syl.lan","threadId":"53975","inReplyTo":"20200811211815.GA65971@syl.lan","subject":"Re: [PATCH v3 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-11T22:05:03Z","receivedAt":"2020-08-11T22:05:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 11, 2020 at 05:18:15PM -0400, Taylor Blau wrote:\n> On Tue, Aug 11, 2020 at 05:11:49PM -0400, Derrick Stolee wrote:\n> > On 8/11/20 4:52 PM, Taylor Blau wrote:\n> > > To allow using the existing bitmap code with 64-bit words, we write the\n> > > data in network byte order from the 64-bit words. This means we also\n> > > need to read the array from the commit-graph file by translating each\n> > > word from network byte order using get_be64() when loading the commit\n> > > graph. (Note that this *could* be delayed until first-use, but a later\n> > > patch will rely on this being initialized early, so we assume the\n> > > up-front cost when parsing instead of delaying initialization).\n> >\n> > I don't think this is correct to do. This means that every commit-graph\n> > load will load the entire large bloom filter chunk into memory before\n> > allowing a single commit to be read from the graph.\n> >\n> > In the case of a very large commit-graph (> 1 million commits), this\n> > would cause a significant initial cost that is not necessarily needed\n> > for anything. For example, \"git log -1\" will be delayed by this action.\n> >\n> > If I understand correctly, this bloom_large bitmap is only used when\n> > writing Bloom filters. At that point, reading the entire bitmap from\n> > disk into memory is inexpensive compared to the time saved by the\n> > feature.\n>\n> That's not quite correct. By this point in the patch series, we only use\n> this bitmap for writing, but in the final patch, we will use it in\n> 'load_bloom_filter_from_graph()' (see that patch for the details of why,\n> there is an explanatory comment to that effect).\n>\n> So, the way that this lazy initialization was written was subtly\n> incorrect, since calling 'load_bloom_filter_from_graph' would return the\n> wrong value depending on what was or wasn't called before it (namely\n> whether or not we had initialized the bitmap).\n\nAck, this is totally wrong. I had the wrong impression of where you were\nactually initializing the bitmaps, which is what motivated the change.\nWhoops, totally my mistake. Here's a version of the patch that does what\nit says.\n\nThere is a small adjustment that needs to be made to the final patch, so\nI can send a version of that, too, if we decide that this is the\ndirection we want to go (and that there are no other changes to the\nseries). The change to the final patch basically boils down to using\n'get_bloom_filter_large_in_graph()' instead of manually checking the\nbitmap (so that the aforementioned function can do its initialization\nahead of time).\n\nThanks for catching this subtle issue, I'm glad that we got it resolved\nbefore going further.\n\n--- >8 ---\n\nSubject: [PATCH] commit-graph: add large-filters bitmap chunk\n\nWhen a commit has more than a certain number of changed paths (commonly\n512), the commit-graph machinery represents it as a zero-length filter.\nThis is done since having many entries in the Bloom filter has\nundesirable effects on the false positivity rate.\n\nIn addition to these too-large filters, the commit-graph machinery also\nrepresents commits with no filter and commits with no changed paths in\nthe same way.\n\nWhen writing a commit-graph that aggregates several incremental\ncommit-graph layers (eg., with '--split=replace'), the commit-graph\nmachinery first computes all of the Bloom filters that it wants to write\nbut does not already know about from existing graph layers. Because we\noverload the zero-length filter in the above fashion, this leads to\nrecomputing large filters over and over again.\n\nThis is already undesirable, since it means that we are wasting\nconsiderable effort to discover that a commit with too many changed\npaths, only to throw that effort away (and then repeat the process the\nnext time a roll-up is performed).\n\nIn a subsequent patch, we will add a '--max-new-filters=' option,\nwhich specifies an upper-bound on the number of new filters we are\nwilling to compute from scratch. Suppose that there are 'N' too-large\nfilters, and we specify '--max-new-filters=M'. If 'N >= M', it is\nunlikely that any filters will be generated, since we'll spend most of\nour effort on filters that we ultimately throw away. If 'N < M', filters\nwill trickle in over time, but only at most 'M - N' per-write.\n\nTo address this, add a new chunk which encodes a bitmap where the ith\nbit is on iff the ith commit has zero or at least 512 changed paths.\nLikewise store the maximum number of changed paths we are willing to\nstore in order to prepare for eventually making this value more easily\ncustomizable. When computing Bloom filters, first consult the relevant\nbitmap (in the case that we are rolling up existing layers) to see if\ncomputing the Bloom filter from scratch would be a waste of time.\n\nThis patch implements a new chunk instead of extending the existing BIDX\nand BDAT chunks because modifying these chunks would confuse old\nclients. (Eg., setting the most-significant bit in the BIDX chunk would\nconfuse old clients and require a version bump).\n\nTo allow using the existing bitmap code with 64-bit words, we write the\ndata in network byte order from the 64-bit words. This means we also\nneed to read the array from the commit-graph file by translating each\nword from network byte order using get_be64() when loading the commit\ngraph. Initialize this bitmap lazily to avoid paying a linear-time cost\nupon each commit-graph load even if we do not need the bitmaps\nthemselves.\n\nBy avoiding the need to move to new versions of the BDAT and BIDX chunk,\nwe can give ourselves more time to consider whether or not other\nmodifications to these chunks are worthwhile without holding up this\nchange.\n\nAnother approach would be to introduce a new BIDX chunk (say, one\nidentified by 'BID2') which is identical to the existing BIDX chunk,\nexcept the most-significant bit of each offset is interpreted as \"this\nfilter is too big\" iff looking at a BID2 chunk. This avoids having to\nwrite a bitmap, but forces older clients to rewrite their commit-graphs\n(as well as reduces the theoretical largest Bloom filters we could\nwrite, and forces us to maintain the code necessary to translate BIDX\nchunks to BID2 ones). Separately from this patch, I implemented this\nalternate approach and did not find it to be advantageous.\n\nSigned-off-by: Taylor Blau \n---\n .../technical/commit-graph-format.txt | 12 +++\n bloom.h | 2 +-\n commit-graph.c | 100 +++++++++++++++---\n commit-graph.h | 10 ++\n t/t4216-log-bloom.sh | 25 ++++-\n 5 files changed, 134 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 440541045d..5f2d9ab4d7 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -123,6 +123,18 @@ CHUNK DATA:\n of length zero.\n * The BDAT chunk is present if and only if BIDX is present.\n\n+ Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n+ * It starts with a 32-bit unsigned integer specifying the maximum number of\n+ changed-paths that can be stored in a single Bloom filter.\n+ * It then contains a list of 64-bit words (the length of this list is\n+ determined by the width of the chunk) which is a bitmap. The 'i'th bit is\n+ set exactly when the 'i'th commit in the graph has a changed-path Bloom\n+ filter with zero entries (either because the commit is empty, or because\n+ it contains more than 512 changed paths).\n+ * The BFXL chunk is present only when the BIDX and BDAT chunks are\n+ also present.\n+\n+\n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n This list of H-byte hashes describe a set of B commit-graph files that\n form a commit-graph chain. The graph position for the ith commit in this\ndiff --git a/bloom.h b/bloom.h\nindex 3f19e3fca4..464d9b57de 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -33,7 +33,7 @@ struct bloom_filter_settings {\n \t * The maximum number of changed paths per commit\n \t * before declaring a Bloom filter to be too-large.\n \t *\n-\t * Not written to the commit-graph file.\n+\t * Written to the 'BFXL' chunk (instead of 'BDAT').\n \t */\n \tuint32_t max_changed_paths;\n };\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8964453433..4ccc7a3e56 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -41,8 +41,9 @@ void git_test_write_commit_graph_or_die(void)\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n+#define GRAPH_CHUNKID_BLOOMLARGE 0x4246584c /* \"BFXL\" */\n #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n-#define MAX_NUM_CHUNKS 7\n+#define MAX_NUM_CHUNKS 8\n\n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n\n@@ -429,6 +430,16 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n \t\t\t}\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BLOOMLARGE:\n+\t\t\tif (graph->chunk_bloom_large_filters)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n+\t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n+\t\t\t\tgraph->bloom_large_alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n+\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n+\t\t\t}\n+\t\t\tbreak;\n \t\t}\n\n \t\tif (chunk_repeated) {\n@@ -443,6 +454,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t/* We need both the bloom chunks to exist together. Else ignore the data */\n \t\tgraph->chunk_bloom_indexes = NULL;\n \t\tgraph->chunk_bloom_data = NULL;\n+\t\tgraph->chunk_bloom_large_filters = NULL;\n+\t\tgraph->bloom_large_alloc = 0;\n \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n \t}\n\n@@ -932,6 +945,32 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n\n+int get_bloom_filter_large_in_graph(struct commit_graph *g,\n+\t\t\t\t const struct commit *c)\n+{\n+\tuint32_t graph_pos = commit_graph_position(c);\n+\tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n+\t\treturn 0;\n+\n+\twhile (g && graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (!g)\n+\t\treturn 0;\n+\n+\tif (!g->bloom_large && g->bloom_large_alloc) {\n+\t\tsize_t i;\n+\t\tg->bloom_large = bitmap_word_alloc(g->bloom_large_alloc);\n+\n+\t\tfor (i = 0; i < g->bloom_large->word_alloc; i++)\n+\t\t\tg->bloom_large->words[i] = get_be64(\n+\t\t\t\tg->chunk_bloom_large_filters + i * sizeof(eword_t));\n+\t}\n+\n+\tif (!g->bloom_large)\n+\t\treturn 0;\n+\treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n+}\n\n struct packed_oid_list {\n \tstruct object_id *list;\n@@ -970,8 +1009,10 @@ struct write_commit_graph_context {\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n\n+\tint count_bloom_filter_known_large;\n \tint count_bloom_filter_found_large;\n \tint count_bloom_filter_computed;\n+\tstruct bitmap *bloom_large;\n };\n\n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1235,6 +1276,23 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \treturn 0;\n }\n\n+static int write_graph_chunk_bloom_large(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\tsize_t i, alloc = ctx->commits.nr / BITS_IN_EWORD;\n+\tif (ctx->commits.nr % BITS_IN_EWORD)\n+\t\talloc++;\n+\tif (alloc > ctx->bloom_large->word_alloc)\n+\t\tBUG(\"write_graph_chunk_bloom_large: bitmap not large enough\");\n+\n+\ttrace2_region_enter(\"commit-graph\", \"bloom_large\", ctx->r);\n+\thashwrite_be32(f, ctx->bloom_settings->max_changed_paths);\n+\tfor (i = 0; i < ctx->bloom_large->word_alloc; i++)\n+\t\thashwrite_be64(f, ctx->bloom_large->words[i]);\n+\ttrace2_region_leave(\"commit-graph\", \"bloom_large\", ctx->r);\n+\treturn 0;\n+}\n+\n static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n@@ -1398,6 +1456,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \tstruct json_writer jw = JSON_WRITER_INIT;\n\n \tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"filter_known_large\",\n+\t\t\t ctx->count_bloom_filter_known_large);\n \tjw_object_intmax(&jw, \"filter_found_large\",\n \t\t\t ctx->count_bloom_filter_found_large);\n \tjw_object_intmax(&jw, \"filter_computed\",\n@@ -1416,6 +1476,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint *sorted_commits;\n\n \tinit_bloom_filters();\n+\tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n\n \tif (ctx->report_progress)\n \t\tprogress = start_delayed_progress(\n@@ -1430,21 +1491,28 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t&ctx->commits);\n\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tint computed = 0;\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n-\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n-\t\t\tctx->r,\n-\t\t\tc,\n-\t\t\t1,\n-\t\t\tctx->bloom_settings,\n-\t\t\t&computed);\n-\t\tif (computed) {\n-\t\t\tctx->count_bloom_filter_computed++;\n-\t\t\tif (filter && !filter->len)\n-\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n+\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\tctx->count_bloom_filter_known_large++;\n+\t\t} else {\n+\t\t\tint computed = 0;\n+\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\t\tctx->r,\n+\t\t\t\tc,\n+\t\t\t\t1,\n+\t\t\t\tctx->bloom_settings,\n+\t\t\t\t&computed);\n+\t\t\tif (computed) {\n+\t\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\t\tif (filter && !filter->len) {\n+\t\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\t}\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n\n@@ -1764,6 +1832,11 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n \t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n+\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMLARGE;\n+\t\tchunks[num_chunks].size = sizeof(eword_t) * ctx->bloom_large->word_alloc\n+\t\t\t\t\t+ sizeof(uint32_t);\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_large;\n+\t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n@@ -2503,6 +2576,7 @@ void free_commit_graph(struct commit_graph *g)\n \t}\n \tfree(g->filename);\n \tfree(g->bloom_filter_settings);\n+\tbitmap_free(g->bloom_large);\n \tfree(g);\n }\n\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d9acb22bac..9afb1477d5 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,6 +4,7 @@\n #include \"git-compat-util.h\"\n #include \"object-store.h\"\n #include \"oidset.h\"\n+#include \"ewah/ewok.h\"\n\n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n #define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n@@ -50,6 +51,9 @@ void load_commit_graph_info(struct repository *r, struct commit *item);\n struct tree *get_commit_tree_in_graph(struct repository *r,\n \t\t\t\t const struct commit *c);\n\n+int get_bloom_filter_large_in_graph(struct commit_graph *g,\n+\t\t\t\t const struct commit *c);\n+\n struct commit_graph {\n \tconst unsigned char *data;\n \tsize_t data_len;\n@@ -71,6 +75,10 @@ struct commit_graph {\n \tconst unsigned char *chunk_base_graphs;\n \tconst unsigned char *chunk_bloom_indexes;\n \tconst unsigned char *chunk_bloom_data;\n+\tconst unsigned char *chunk_bloom_large_filters;\n+\n+\tstruct bitmap *bloom_large;\n+\tsize_t bloom_large_alloc;\n\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n@@ -83,6 +91,8 @@ struct commit_graph *read_commit_graph_one(struct repository *r,\n struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\t\tvoid *graph_map, size_t graph_size);\n\n+void prepare_commit_graph_bloom_large(struct commit_graph *g);\n+\n /*\n * Return 1 if and only if the repository has a commit-graph\n * file and generation numbers are computed in that file.\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 21b67677ef..6859d85369 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -33,7 +33,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \tgit commit-graph write --reachable --changed-paths\n '\n graph_read_expect () {\n-\tNUM_CHUNKS=5\n+\tNUM_CHUNKS=6\n \tcat >expect <<- EOF\n \theader: 43475048 1 1 $NUM_CHUNKS 0\n \tnum_commits: $1\n@@ -262,5 +262,28 @@ test_expect_success 'correctly report changes over limit' '\n \t\tdone\n \t)\n '\n+test_bloom_filters_computed () {\n+\tcommit_graph_args=$1\n+\tbloom_trace_prefix=\"{\\\"filter_known_large\\\":$2,\\\"filter_found_large\\\":$3,\\\"filter_computed\\\":$4\"\n+\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n+\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write $commit_graph_args &&\n+\tgrep \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.event\"\n+}\n+\n+test_expect_success 'Bloom generation does not recompute too-large filters' '\n+\t(\n+\t\tcd limits &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t--split=replace &&\n+\t\ttest_commit c1 filter &&\n+\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n+\t\t\t2 0 1\n+\t)\n+'\n\n test_done\n--\n2.28.0.rc1.13.ge78abce653\n\n"},{"id":"403426","messageId":"20200811235537.GC13422@szeder.dev","threadId":"53975","inReplyTo":"20200811213411.GB66656@syl.lan","subject":"Re: [PATCH v3 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-08-11T23:55:37Z","receivedAt":"2020-08-11T23:55:43Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 05:34:11PM -0400, Taylor Blau wrote:\n> On Tue, Aug 11, 2020 at 11:27:16PM +0200, SZEDER Gábor wrote:\n> > On Tue, Aug 11, 2020 at 05:21:18PM -0400, Taylor Blau wrote:\n> > > On Tue, Aug 11, 2020 at 11:18:30PM +0200, SZEDER Gábor wrote:\n> > > > On Tue, Aug 11, 2020 at 04:51:19PM -0400, Taylor Blau wrote:\n> > > > > Many places in the code often need a pointer to the commit-graph's\n> > > > > 'struct bloom_filter_settings', in which case they often take the value\n> > > > > from the top-most commit-graph.\n> > > > >\n> > > > > In the non-split case, this works as expected. In the split case,\n> > > > > however, things get a little tricky. Not all layers in a chain of\n> > > > > incremental commit-graphs are required to themselves have Bloom data,\n> > > > > and so whether or not some part of the code uses Bloom filters depends\n> > > > > entirely on whether or not the top-most level of the commit-graph chain\n> > > > > has Bloom filters.\n> > > > >\n> > > > > This has been the behavior since Bloom filters were introduced, and has\n> > > > > been codified into the tests since a759bfa9ee (t4216: add end to end\n> > > > > tests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\n> > > > > requires that Bloom filters are not used in exactly the case described\n> > > > > earlier.\n> > > > >\n> > > > > There is no reason that this needs to be the case, since it is perfectly\n> > > > > valid for commits in an earlier layer to have Bloom filters when commits\n> > > > > in a newer layer do not.\n> > > > >\n> > > > > Since Bloom settings are guaranteed to be the same for any layer in a\n> > > > > chain that has Bloom data,\n> > > >\n> > > > Is it? Where is that guaranteed?\n> > >\n> > > There is no mechanism whatsoever to customize these settings that is\n> > > exposed to the user (except for the undocumented 'GIT_TEST' environment\n> > > variables).\n> >\n> > Let me rephrase it, then: where is it written in the commit-graph\n> > format specification that these must be the same in all layers?\n> >\n> > Nowhere.\n> \n> OK. We can certainly document that this is the case.\n\nIMO we absolutely must document this first; ideally it should have\nbeen carefully considered and documented right from the start.\n\nSome thougths about this:\n\n https://public-inbox.org/git/20200619140230.GB22200@szeder.dev/\n\n> For this purpose,\n> all we really care about is that the graph _has_ Bloom filters anywhere.\n> If you wanted to return the exact matching settings, you could also\n> provide a commit and return the settings belonging to the graph that\n> contains that commit.\n> \n> In the case where we don't have a commit, we could use the default\n> settings instead.\n> \n> I think that we are a little bit dealing with a problem that doesn't\n> exist, since we do not document the sole method by which you would\n> change these settings. So, maybe we can think more about this, but my\n> preference would be to leave this patch alone.\n\nOther implementations can write split commit-graphs with modified path\nBloom filters as well, and at the moment there is nothing in the specs\nthat tells them not to use different Bloom filter settings in\ndifferent layers.\n\n> Maybe Stolee can chime in, too.\n> \n> > > > > it is sufficient to traverse the\n> > > > > '->base_graph' pointer until either (1) a non-null 'struct\n> > > > > bloom_filter_settings *' is found, or (2) until we are at the root of\n> > > > > the commit-graph chain.\n> > > > >\n> > > > > Introduce a 'get_bloom_filter_settings()' function that does just this,\n> > > > > and use it instead of purely dereferencing the top-most graph's\n> > > > > '->bloom_filter_settings' pointer.\n> > > > >\n> > > > > While we're at it, add an additional test in t5324 to guard against code\n> > > > > in the commit-graph writing machinery that doesn't correctly handle a\n> > > > > NULL 'struct bloom_filter *'.\n> > > > >\n> > > > > Co-authored-by: Derrick Stolee \n> > > > > Signed-off-by: Taylor Blau \n> > > Thanks,\n> > > Taylor\n> Thanks,\n> Taylor\n"},{"id":"403439","messageId":"d525c35e-12fb-1918-c02b-d0323ee9b664@gmail.com","threadId":"53975","inReplyTo":"20200811235537.GC13422@szeder.dev","subject":"Re: [PATCH v3 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-12T11:48:55Z","receivedAt":"2020-08-12T11:49:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/11/2020 7:55 PM, SZEDER Gábor wrote:\n> On Tue, Aug 11, 2020 at 05:34:11PM -0400, Taylor Blau wrote:\n>> On Tue, Aug 11, 2020 at 11:27:16PM +0200, SZEDER Gábor wrote:\n>>> On Tue, Aug 11, 2020 at 05:21:18PM -0400, Taylor Blau wrote:\n>>>> On Tue, Aug 11, 2020 at 11:18:30PM +0200, SZEDER Gábor wrote:\n>>>>> On Tue, Aug 11, 2020 at 04:51:19PM -0400, Taylor Blau wrote:\n>>>>>> Many places in the code often need a pointer to the commit-graph's\n>>>>>> 'struct bloom_filter_settings', in which case they often take the value\n>>>>>> from the top-most commit-graph.\n>>>>>>\n>>>>>> In the non-split case, this works as expected. In the split case,\n>>>>>> however, things get a little tricky. Not all layers in a chain of\n>>>>>> incremental commit-graphs are required to themselves have Bloom data,\n>>>>>> and so whether or not some part of the code uses Bloom filters depends\n>>>>>> entirely on whether or not the top-most level of the commit-graph chain\n>>>>>> has Bloom filters.\n>>>>>>\n>>>>>> This has been the behavior since Bloom filters were introduced, and has\n>>>>>> been codified into the tests since a759bfa9ee (t4216: add end to end\n>>>>>> tests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\n>>>>>> requires that Bloom filters are not used in exactly the case described\n>>>>>> earlier.\n>>>>>>\n>>>>>> There is no reason that this needs to be the case, since it is perfectly\n>>>>>> valid for commits in an earlier layer to have Bloom filters when commits\n>>>>>> in a newer layer do not.\n>>>>>>\n>>>>>> Since Bloom settings are guaranteed to be the same for any layer in a\n>>>>>> chain that has Bloom data,\n>>>>>\n>>>>> Is it? Where is that guaranteed?\n\nPerhaps instead of \"guaranteed\" we could say \"Git never writes\na commit-graph chain with different settings between layers.\"\n\n>>>> There is no mechanism whatsoever to customize these settings that is\n>>>> exposed to the user (except for the undocumented 'GIT_TEST' environment\n>>>> variables).\n>>>\n>>> Let me rephrase it, then: where is it written in the commit-graph\n>>> format specification that these must be the same in all layers?\n>>>\n>>> Nowhere.\n>>\n>> OK. We can certainly document that this is the case.\n> \n> IMO we absolutely must document this first; ideally it should have\n> been carefully considered and documented right from the start.\n\nYou're right. One of the major difficulties in bringing a Bloom\nfilter implementation to the commit-graph feature when we did was\nthat the split commit-graph was introduced between our initial\nprototypes and when it was finally prepared for a full submission.\n\nThere certainly are gaps in the implementation and documentation.\nI think Taylor is doing a great job by addressing one of those gaps\nin a focused, thoughtful way.\n\n> Some thougths about this:\n> \n> https://public-inbox.org/git/20200619140230.GB22200@szeder.dev/\n\nI appreciate your attention to detail. Your comments on the existing\nimplementation do point out some of its shortcomings, and that is a\nvaluable contribution.\n\nActually converting those thoughts into patches is a lot of work.\n\n>> For this purpose,\n>> all we really care about is that the graph _has_ Bloom filters anywhere.\n>> If you wanted to return the exact matching settings, you could also\n>> provide a commit and return the settings belonging to the graph that\n>> contains that commit.\n>>\n>> In the case where we don't have a commit, we could use the default\n>> settings instead.\n>>\n>> I think that we are a little bit dealing with a problem that doesn't\n>> exist, since we do not document the sole method by which you would\n>> change these settings. So, maybe we can think more about this, but my\n>> preference would be to leave this patch alone.\n> \n> Other implementations can write split commit-graphs with modified path\n> Bloom filters as well, and at the moment there is nothing in the specs\n> that tells them not to use different Bloom filter settings in\n> different layers.\n\nYou are 100% correct that there is a gap in documentation. That should\nbe corrected at some point. (I don't consider it a blocker for this\nseries.)\n\nBut also: Git itself is the true test of a \"correct\" third-party\nimplementation. libgit2 and JGit try to match Git, \"warts and all\".\nIf another implementation wrote data that results in incorrect\nbehavior by Git, then that implementation is wrong.\n\nImproving documentation can make those errors less likely.\n\nWe also must design with \"future Git\" in mind, presenting it with\nenough flexibility to improve formats. The custom Bloom filter\nsettings do allow that flexibility, but the requirement that all\nlayers have identical settings exists for a reason (despite not\nbeing documented). It is important that any commit walk that intends\nto use the changed-path Bloom filters can compute the bloom keys\nfor the test paths only once.\n\nThanks,\n-Stolee\n"},{"id":"403440","messageId":"20200812114929.GA19082@szeder.dev","threadId":"53975","inReplyTo":"09f6871f66bff838c067a3e0d23cd4622171f3bd.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-08-12T11:49:29Z","receivedAt":"2020-08-12T11:49:39Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 04:52:14PM -0400, Taylor Blau wrote:\n\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 6886f319a5..4aae5471e3 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -954,7 +954,8 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> }\n> \n> static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> -\t\t\t\t\t const struct commit *c)\n> +\t\t\t\t\t const struct commit *c,\n> +\t\t\t\t\t uint32_t max_changed_paths)\n> {\n> \tuint32_t graph_pos = commit_graph_position(c);\n> \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n> @@ -965,6 +966,17 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> \n> \tif (!(g && g->bloom_large))\n> \t\treturn 0;\n> +\tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n> +\t\t/*\n> +\t\t * Force all commits which are subject to a different\n> +\t\t * 'max_changed_paths' limit to be recomputed from scratch.\n> +\t\t *\n> +\t\t * Note that this could likely be improved, but is ignored since\n> +\t\t * all real-world graphs set the maximum number of changed paths\n> +\t\t * at 512.\n> +\t\t */\n> +\t\treturn 0;\n> +\t}\n> \treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n> }\n> \n> @@ -1470,6 +1482,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \tint i;\n> \tstruct progress *progress = NULL;\n> \tint *sorted_commits;\n> +\tint max_new_filters;\n> \n> \tinit_bloom_filters();\n> \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n> @@ -1486,10 +1499,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n> \t\t&ctx->commits);\n> \n> +\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n> +\t\tctx->opts->max_new_filters : ctx->commits.nr;\n\ngit_test_write_commit_graph_or_die() invokes\nwrite_commit_graph_reachable() with opts=0x0, so 'ctx->opts' is NULL,\nand we get segfault. This breaks a lot of tests when run with\nGIT_TEST_COMMIT_GRAPH=1 GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1.\n\n"},{"id":"403441","messageId":"ee5c0db0-6582-f039-96e1-be64ee4fe7a6@gmail.com","threadId":"53975","inReplyTo":"09f6871f66bff838c067a3e0d23cd4622171f3bd.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-12T12:29:06Z","receivedAt":"2020-08-12T12:29:13Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/11/2020 4:52 PM, Taylor Blau wrote:\n> Introduce a command-line flag and configuration variable to fill in the\n> 'max_new_filters' variable introduced by the previous patch.\n> \n> The command-line option '--max-new-filters' takes precedence over\n> 'commitGraph.maxNewFilters', which is the default value.\n> '--no-max-new-filters' can also be provided, which sets the value back\n> to '-1', indicating that an unlimited number of new Bloom filters may be\n> generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n> '0', hence a custom callback was used instead).\n> \n> Signed-off-by: Taylor Blau \n\n...\n\n> diff --git a/bloom.c b/bloom.c\n> index ed54e96e57..8d07209c6b 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -51,6 +51,21 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n> \telse\n> \t\tstart_index = 0;\n> \n> +\tif ((start_index == end_index) &&\n> +\t (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n> +\t\t/*\n> +\t\t * If the filter is zero-length, either (1) the filter has no\n> +\t\t * changes, (2) the filter has too many changes, or (3) it\n> +\t\t * wasn't computed (eg., due to '--max-new-filters').\n> +\t\t *\n> +\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n> +\t\t * for this Bloom filter. If it is unset, then it wasn't\n> +\t\t * computed. In that case, return nothing, since we don't have\n> +\t\t * that filter in the graph.\n> +\t\t */\n> +\t\treturn 0;\n> +\t}\n> +\n\nHere, you are creating a distinction between an \"empty\" filter and\na \"too large\" filter at a place that I don't think is important.\n\nFor instance, this code will be triggered by \"git log -- \"\nbut you only care about the filter being too large when writing the\ncommit-graph. I think this check for a \"too large\" filter should\ninstead be inside get_or_compute_bloom_filter(). I include a patch\nbelow that applies on top of tb/bloom-improvements that gets to\nmy point (and how minor the issue might be).\n\nThanks,\n-Stolee\n\n--- >8 ---\n\nFrom 92a7a3d0f769fad7617426730cb97584a3d07794 Mon Sep 17 00:00:00 2001\nFrom: Derrick Stolee \nDate: Wed, 12 Aug 2020 08:20:03 -0400\nSubject: [PATCH] commit-graph: lazy-load large Bloom filter bitmap\n\nThe bloom_large bitmap in struct commit_graph is currently loaded\nimmediately upon first parse of the commit-graph file (when the chunk\nexists). This is needed because the current implementation of\nget_bloom_filter_from_graph() is special-cased to return NULL when the\nfilter is marked as \"too large\".\n\nThis has a slight drawback: before we can read a single commit out of\nthe commit-graph file, we need to load this entire chunk into memory.\nThis happens even for commands that don't need changed-path Bloom\nfilters, such as \"git log -1\".\n\nThis \"too large\" information is only used when writing a commit-graph\nfile, so we can delay the check for a large filter until after we check\ncompute_if_not_present in get_or_compute_bloom_filter(). Also, place\nthat lazy-load directly in the get_bloom_filter_large_in_graph() method,\nso we ensure it is ready when needed.\n\nThis may be overkill. For a repository with one million commits, this\nfilter size is approximately 125 *kilobytes* of data. My local\nmeasurements found that this took between 1 and 2 milliseconds to load\ninto memory. Even for repositories with 10 million commits, this\ndifference would not be noticeable for end-user commands.\n\nOn the other hand, these handfuls of milliseconds could add up when\nrunning a hosting service using Git, so this extra effort is probably\nworth it.\n\nSigned-off-by: Derrick Stolee \n---\n bloom.c | 27 ++++++++-------------------\n commit-graph.c | 30 +++++++++++++++++-------------\n commit-graph.h | 8 ++++++++\n 3 files changed, 33 insertions(+), 32 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 8d07209c6b..6d0884fa19 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -51,21 +51,6 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \telse\n \t\tstart_index = 0;\n \n-\tif ((start_index == end_index) &&\n-\t (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n-\t\t/*\n-\t\t * If the filter is zero-length, either (1) the filter has no\n-\t\t * changes, (2) the filter has too many changes, or (3) it\n-\t\t * wasn't computed (eg., due to '--max-new-filters').\n-\t\t *\n-\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n-\t\t * for this Bloom filter. If it is unset, then it wasn't\n-\t\t * computed. In that case, return nothing, since we don't have\n-\t\t * that filter in the graph.\n-\t\t */\n-\t\treturn 0;\n-\t}\n-\n \tfilter->len = end_index - start_index;\n \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n \t\t\t\t\tsizeof(unsigned char) * start_index +\n@@ -212,16 +197,20 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n-\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n+\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n \n-\tif (filter->data)\n+\tif (filter && filter->len)\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\n \n+\tif (filter && !filter->len &&\n+\t get_bloom_filter_large_in_graph(r->objects->commit_graph, c,\n+\t \t\t\t\t settings->max_changed_paths))\n+\t\treturn filter;\n+\n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n \tdiffopt.detect_rename = 0;\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 4aae5471e3..ea89f431cc 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -435,17 +435,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\tif (graph->chunk_bloom_large_filters)\n \t\t\t\tchunk_repeated = 1;\n \t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n-\t\t\t\tsize_t alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n+\t\t\t\tgraph->bloom_large_alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n \t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n \t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n-\t\t\t\tif (alloc) {\n-\t\t\t\t\tsize_t j;\n-\t\t\t\t\tgraph->bloom_large = bitmap_word_alloc(alloc);\n-\n-\t\t\t\t\tfor (j = 0; j < graph->bloom_large->word_alloc; j++)\n-\t\t\t\t\t\tgraph->bloom_large->words[j] = get_be64(\n-\t\t\t\t\t\t\tgraph->chunk_bloom_large_filters + j * sizeof(eword_t));\n-\t\t\t\t}\n \t\t\t}\n \t\t\tbreak;\n \t\t}\n@@ -953,9 +945,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n-static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n-\t\t\t\t\t const struct commit *c,\n-\t\t\t\t\t uint32_t max_changed_paths)\n+int get_bloom_filter_large_in_graph(struct commit_graph *g,\n+\t\t\t\t const struct commit *c,\n+\t\t\t\t uint32_t max_changed_paths)\n {\n \tuint32_t graph_pos = commit_graph_position(c);\n \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n@@ -964,8 +956,20 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n \twhile (g && graph_pos < g->num_commits_in_base)\n \t\tg = g->base_graph;\n \n-\tif (!(g && g->bloom_large))\n+\tif (!g || !g->bloom_large_alloc)\n \t\treturn 0;\n+\n+\tif (!g->bloom_large) {\n+\t\tsize_t j;\n+\t\tg->bloom_large = bitmap_word_alloc(g->bloom_large_alloc);\n+\n+\t\tfor (j = 0; j < g->bloom_large->word_alloc; j++) {\n+\t\t\tconst void *data = g->chunk_bloom_large_filters +\n+\t\t\t\t\t j * sizeof(eword_t);\n+\t\t\tg->bloom_large->words[j] = get_be64(data);\n+\t\t}\n+\t}\n+\n \tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n \t\t/*\n \t\t * Force all commits which are subject to a different\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 75ef83708b..126fd43380 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -51,6 +51,13 @@ void load_commit_graph_info(struct repository *r, struct commit *item);\n struct tree *get_commit_tree_in_graph(struct repository *r,\n \t\t\t\t const struct commit *c);\n \n+/*\n+ * Returns 1 if this commit was marked in the BFXL chunk as having more\n+ * than max_changed_paths changes.\n+ */\n+int get_bloom_filter_large_in_graph(struct commit_graph *g,\n+\t\t\t\t const struct commit *c,\n+\t\t\t\t uint32_t max_changed_paths);\n struct commit_graph {\n \tconst unsigned char *data;\n \tsize_t data_len;\n@@ -74,6 +81,7 @@ struct commit_graph {\n \tconst unsigned char *chunk_bloom_data;\n \tconst unsigned char *chunk_bloom_large_filters;\n \n+\tsize_t bloom_large_alloc;\n \tstruct bitmap *bloom_large;\n \n \tstruct bloom_filter_settings *bloom_filter_settings;\n-- \n2.28.0.38.gc6f546511c1\n\n\n"},{"id":"403705","messageId":"20200814201028.GC30103@syl.lan","threadId":"53975","inReplyTo":"ee5c0db0-6582-f039-96e1-be64ee4fe7a6@gmail.com","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-14T20:10:28Z","receivedAt":"2020-08-14T20:10:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 12, 2020 at 08:29:06AM -0400, Derrick Stolee wrote:\n> On 8/11/2020 4:52 PM, Taylor Blau wrote:\n> > Introduce a command-line flag and configuration variable to fill in the\n> > 'max_new_filters' variable introduced by the previous patch.\n> >\n> > The command-line option '--max-new-filters' takes precedence over\n> > 'commitGraph.maxNewFilters', which is the default value.\n> > '--no-max-new-filters' can also be provided, which sets the value back\n> > to '-1', indicating that an unlimited number of new Bloom filters may be\n> > generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n> > '0', hence a custom callback was used instead).\n> >\n> > Signed-off-by: Taylor Blau \n>\n> ...\n>\n> > diff --git a/bloom.c b/bloom.c\n> > index ed54e96e57..8d07209c6b 100644\n> > --- a/bloom.c\n> > +++ b/bloom.c\n> > @@ -51,6 +51,21 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n> > \telse\n> > \t\tstart_index = 0;\n> >\n> > +\tif ((start_index == end_index) &&\n> > +\t (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n> > +\t\t/*\n> > +\t\t * If the filter is zero-length, either (1) the filter has no\n> > +\t\t * changes, (2) the filter has too many changes, or (3) it\n> > +\t\t * wasn't computed (eg., due to '--max-new-filters').\n> > +\t\t *\n> > +\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n> > +\t\t * for this Bloom filter. If it is unset, then it wasn't\n> > +\t\t * computed. In that case, return nothing, since we don't have\n> > +\t\t * that filter in the graph.\n> > +\t\t */\n> > +\t\treturn 0;\n> > +\t}\n> > +\n>\n> Here, you are creating a distinction between an \"empty\" filter and\n> a \"too large\" filter at a place that I don't think is important.\n>\n> For instance, this code will be triggered by \"git log -- \"\n> but you only care about the filter being too large when writing the\n> commit-graph. I think this check for a \"too large\" filter should\n> instead be inside get_or_compute_bloom_filter(). I include a patch\n> below that applies on top of tb/bloom-improvements that gets to\n> my point (and how minor the issue might be).\n\nThat's a good point. I factored the lazy-loading out in basically the\nsame way that you did below, but didn't move this call out of\n'load_bloom_filter_from_graph'. I think that you're right that this is\nbetter at the calling end (replacing the 'start_index == end_index'\ncheck with a '!filter->len' one instead).\n\nI'm going to \"ignore\" this patch below since I (a) already have it\nlocally, and (b) would rather just do this correctly the first time than\nintroduce and subsequently fix a performance regression.\n\n> Thanks,\n> -Stolee\n>\n> --- >8 ---\n>\n> From 92a7a3d0f769fad7617426730cb97584a3d07794 Mon Sep 17 00:00:00 2001\n> From: Derrick Stolee \n> Date: Wed, 12 Aug 2020 08:20:03 -0400\n> Subject: [PATCH] commit-graph: lazy-load large Bloom filter bitmap\n>\n> The bloom_large bitmap in struct commit_graph is currently loaded\n> immediately upon first parse of the commit-graph file (when the chunk\n> exists). This is needed because the current implementation of\n> get_bloom_filter_from_graph() is special-cased to return NULL when the\n> filter is marked as \"too large\".\n>\n> This has a slight drawback: before we can read a single commit out of\n> the commit-graph file, we need to load this entire chunk into memory.\n> This happens even for commands that don't need changed-path Bloom\n> filters, such as \"git log -1\".\n>\n> This \"too large\" information is only used when writing a commit-graph\n> file, so we can delay the check for a large filter until after we check\n> compute_if_not_present in get_or_compute_bloom_filter(). Also, place\n> that lazy-load directly in the get_bloom_filter_large_in_graph() method,\n> so we ensure it is ready when needed.\n>\n> This may be overkill. For a repository with one million commits, this\n> filter size is approximately 125 *kilobytes* of data. My local\n> measurements found that this took between 1 and 2 milliseconds to load\n> into memory. Even for repositories with 10 million commits, this\n> difference would not be noticeable for end-user commands.\n>\n> On the other hand, these handfuls of milliseconds could add up when\n> running a hosting service using Git, so this extra effort is probably\n> worth it.\n>\n> Signed-off-by: Derrick Stolee \n> ---\n> bloom.c | 27 ++++++++-------------------\n> commit-graph.c | 30 +++++++++++++++++-------------\n> commit-graph.h | 8 ++++++++\n> 3 files changed, 33 insertions(+), 32 deletions(-)\n>\n> diff --git a/bloom.c b/bloom.c\n> index 8d07209c6b..6d0884fa19 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -51,21 +51,6 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n> \telse\n> \t\tstart_index = 0;\n>\n> -\tif ((start_index == end_index) &&\n> -\t (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n> -\t\t/*\n> -\t\t * If the filter is zero-length, either (1) the filter has no\n> -\t\t * changes, (2) the filter has too many changes, or (3) it\n> -\t\t * wasn't computed (eg., due to '--max-new-filters').\n> -\t\t *\n> -\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n> -\t\t * for this Bloom filter. If it is unset, then it wasn't\n> -\t\t * computed. In that case, return nothing, since we don't have\n> -\t\t * that filter in the graph.\n> -\t\t */\n> -\t\treturn 0;\n> -\t}\n> -\n> \tfilter->len = end_index - start_index;\n> \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n> \t\t\t\t\tsizeof(unsigned char) * start_index +\n> @@ -212,16 +197,20 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n>\n> \tif (!filter->data) {\n> \t\tload_commit_graph_info(r, c);\n> -\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n> -\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n> -\t\t\t\treturn filter;\n> +\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n> +\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n> \t}\n>\n> -\tif (filter->data)\n> +\tif (filter && filter->len)\n> \t\treturn filter;\n> \tif (!compute_if_not_present)\n> \t\treturn NULL;\n>\n> +\tif (filter && !filter->len &&\n> +\t get_bloom_filter_large_in_graph(r->objects->commit_graph, c,\n> +\t \t\t\t\t settings->max_changed_paths))\n> +\t\treturn filter;\n> +\n> \trepo_diff_setup(r, &diffopt);\n> \tdiffopt.flags.recursive = 1;\n> \tdiffopt.detect_rename = 0;\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 4aae5471e3..ea89f431cc 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -435,17 +435,9 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> \t\t\tif (graph->chunk_bloom_large_filters)\n> \t\t\t\tchunk_repeated = 1;\n> \t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n> -\t\t\t\tsize_t alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n> +\t\t\t\tgraph->bloom_large_alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n> \t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n> \t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n> -\t\t\t\tif (alloc) {\n> -\t\t\t\t\tsize_t j;\n> -\t\t\t\t\tgraph->bloom_large = bitmap_word_alloc(alloc);\n> -\n> -\t\t\t\t\tfor (j = 0; j < graph->bloom_large->word_alloc; j++)\n> -\t\t\t\t\t\tgraph->bloom_large->words[j] = get_be64(\n> -\t\t\t\t\t\t\tgraph->chunk_bloom_large_filters + j * sizeof(eword_t));\n> -\t\t\t\t}\n> \t\t\t}\n> \t\t\tbreak;\n> \t\t}\n> @@ -953,9 +945,9 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n> }\n>\n> -static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> -\t\t\t\t\t const struct commit *c,\n> -\t\t\t\t\t uint32_t max_changed_paths)\n> +int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> +\t\t\t\t const struct commit *c,\n> +\t\t\t\t uint32_t max_changed_paths)\n> {\n> \tuint32_t graph_pos = commit_graph_position(c);\n> \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n> @@ -964,8 +956,20 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> \twhile (g && graph_pos < g->num_commits_in_base)\n> \t\tg = g->base_graph;\n>\n> -\tif (!(g && g->bloom_large))\n> +\tif (!g || !g->bloom_large_alloc)\n> \t\treturn 0;\n> +\n> +\tif (!g->bloom_large) {\n> +\t\tsize_t j;\n> +\t\tg->bloom_large = bitmap_word_alloc(g->bloom_large_alloc);\n> +\n> +\t\tfor (j = 0; j < g->bloom_large->word_alloc; j++) {\n> +\t\t\tconst void *data = g->chunk_bloom_large_filters +\n> +\t\t\t\t\t j * sizeof(eword_t);\n> +\t\t\tg->bloom_large->words[j] = get_be64(data);\n> +\t\t}\n> +\t}\n> +\n> \tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n> \t\t/*\n> \t\t * Force all commits which are subject to a different\n> diff --git a/commit-graph.h b/commit-graph.h\n> index 75ef83708b..126fd43380 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -51,6 +51,13 @@ void load_commit_graph_info(struct repository *r, struct commit *item);\n> struct tree *get_commit_tree_in_graph(struct repository *r,\n> \t\t\t\t const struct commit *c);\n>\n> +/*\n> + * Returns 1 if this commit was marked in the BFXL chunk as having more\n> + * than max_changed_paths changes.\n> + */\n> +int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> +\t\t\t\t const struct commit *c,\n> +\t\t\t\t uint32_t max_changed_paths);\n> struct commit_graph {\n> \tconst unsigned char *data;\n> \tsize_t data_len;\n> @@ -74,6 +81,7 @@ struct commit_graph {\n> \tconst unsigned char *chunk_bloom_data;\n> \tconst unsigned char *chunk_bloom_large_filters;\n>\n> +\tsize_t bloom_large_alloc;\n> \tstruct bitmap *bloom_large;\n>\n> \tstruct bloom_filter_settings *bloom_filter_settings;\n> --\n> 2.28.0.38.gc6f546511c1\n>\n>\n\nThanks,\nTaylor\n"},{"id":"403708","messageId":"20200814201732.GD30103@syl.lan","threadId":"53975","inReplyTo":"d525c35e-12fb-1918-c02b-d0323ee9b664@gmail.com","subject":"Re: [PATCH v3 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-14T20:17:32Z","receivedAt":"2020-08-14T20:17:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 12, 2020 at 07:48:55AM -0400, Derrick Stolee wrote:\n> On 8/11/2020 7:55 PM, SZEDER Gábor wrote:\n> > On Tue, Aug 11, 2020 at 05:34:11PM -0400, Taylor Blau wrote:\n> >> On Tue, Aug 11, 2020 at 11:27:16PM +0200, SZEDER Gábor wrote:\n> >>> On Tue, Aug 11, 2020 at 05:21:18PM -0400, Taylor Blau wrote:\n> >>>> On Tue, Aug 11, 2020 at 11:18:30PM +0200, SZEDER Gábor wrote:\n> >>>>> On Tue, Aug 11, 2020 at 04:51:19PM -0400, Taylor Blau wrote:\n> >>>>>> Many places in the code often need a pointer to the commit-graph's\n> >>>>>> 'struct bloom_filter_settings', in which case they often take the value\n> >>>>>> from the top-most commit-graph.\n> >>>>>>\n> >>>>>> In the non-split case, this works as expected. In the split case,\n> >>>>>> however, things get a little tricky. Not all layers in a chain of\n> >>>>>> incremental commit-graphs are required to themselves have Bloom data,\n> >>>>>> and so whether or not some part of the code uses Bloom filters depends\n> >>>>>> entirely on whether or not the top-most level of the commit-graph chain\n> >>>>>> has Bloom filters.\n> >>>>>>\n> >>>>>> This has been the behavior since Bloom filters were introduced, and has\n> >>>>>> been codified into the tests since a759bfa9ee (t4216: add end to end\n> >>>>>> tests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\n> >>>>>> requires that Bloom filters are not used in exactly the case described\n> >>>>>> earlier.\n> >>>>>>\n> >>>>>> There is no reason that this needs to be the case, since it is perfectly\n> >>>>>> valid for commits in an earlier layer to have Bloom filters when commits\n> >>>>>> in a newer layer do not.\n> >>>>>>\n> >>>>>> Since Bloom settings are guaranteed to be the same for any layer in a\n> >>>>>> chain that has Bloom data,\n> >>>>>\n> >>>>> Is it? Where is that guaranteed?\n>\n> Perhaps instead of \"guaranteed\" we could say \"Git never writes\n> a commit-graph chain with different settings between layers.\"\n>\n> >>>> There is no mechanism whatsoever to customize these settings that is\n> >>>> exposed to the user (except for the undocumented 'GIT_TEST' environment\n> >>>> variables).\n> >>>\n> >>> Let me rephrase it, then: where is it written in the commit-graph\n> >>> format specification that these must be the same in all layers?\n> >>>\n> >>> Nowhere.\n> >>\n> >> OK. We can certainly document that this is the case.\n> >\n> > IMO we absolutely must document this first; ideally it should have\n> > been carefully considered and documented right from the start.\n>\n> You're right. One of the major difficulties in bringing a Bloom\n> filter implementation to the commit-graph feature when we did was\n> that the split commit-graph was introduced between our initial\n> prototypes and when it was finally prepared for a full submission.\n>\n> There certainly are gaps in the implementation and documentation.\n> I think Taylor is doing a great job by addressing one of those gaps\n> in a focused, thoughtful way.\n>\n> > Some thougths about this:\n> >\n> > https://public-inbox.org/git/20200619140230.GB22200@szeder.dev/\n>\n> I appreciate your attention to detail. Your comments on the existing\n> implementation do point out some of its shortcomings, and that is a\n> valuable contribution.\n>\n> Actually converting those thoughts into patches is a lot of work.\n>\n> >> For this purpose,\n> >> all we really care about is that the graph _has_ Bloom filters anywhere.\n> >> If you wanted to return the exact matching settings, you could also\n> >> provide a commit and return the settings belonging to the graph that\n> >> contains that commit.\n> >>\n> >> In the case where we don't have a commit, we could use the default\n> >> settings instead.\n> >>\n> >> I think that we are a little bit dealing with a problem that doesn't\n> >> exist, since we do not document the sole method by which you would\n> >> change these settings. So, maybe we can think more about this, but my\n> >> preference would be to leave this patch alone.\n> >\n> > Other implementations can write split commit-graphs with modified path\n> > Bloom filters as well, and at the moment there is nothing in the specs\n> > that tells them not to use different Bloom filter settings in\n> > different layers.\n>\n> You are 100% correct that there is a gap in documentation. That should\n> be corrected at some point. (I don't consider it a blocker for this\n> series.)\n>\n> But also: Git itself is the true test of a \"correct\" third-party\n> implementation. libgit2 and JGit try to match Git, \"warts and all\".\n> If another implementation wrote data that results in incorrect\n> behavior by Git, then that implementation is wrong.\n>\n> Improving documentation can make those errors less likely.\n\nI agree with this reasoning. Would anybody object to moving forward with\nthis series without a change in documentation today (but rather down the\nroad)?\n\n> We also must design with \"future Git\" in mind, presenting it with\n> enough flexibility to improve formats. The custom Bloom filter\n> settings do allow that flexibility, but the requirement that all\n> layers have identical settings exists for a reason (despite not\n> being documented). It is important that any commit walk that intends\n> to use the changed-path Bloom filters can compute the bloom keys\n> for the test paths only once.\n>\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"403709","messageId":"20200814202021.GE30103@syl.lan","threadId":"53975","inReplyTo":"20200812114929.GA19082@szeder.dev","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-08-14T20:20:21Z","receivedAt":"2020-08-14T20:20:27Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 12, 2020 at 01:49:29PM +0200, SZEDER Gábor wrote:\n> On Tue, Aug 11, 2020 at 04:52:14PM -0400, Taylor Blau wrote:\n>\n> > diff --git a/commit-graph.c b/commit-graph.c\n> > index 6886f319a5..4aae5471e3 100644\n> > --- a/commit-graph.c\n> > +++ b/commit-graph.c\n> > @@ -954,7 +954,8 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> > }\n> >\n> > static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> > -\t\t\t\t\t const struct commit *c)\n> > +\t\t\t\t\t const struct commit *c,\n> > +\t\t\t\t\t uint32_t max_changed_paths)\n> > {\n> > \tuint32_t graph_pos = commit_graph_position(c);\n> > \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n> > @@ -965,6 +966,17 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> >\n> > \tif (!(g && g->bloom_large))\n> > \t\treturn 0;\n> > +\tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n> > +\t\t/*\n> > +\t\t * Force all commits which are subject to a different\n> > +\t\t * 'max_changed_paths' limit to be recomputed from scratch.\n> > +\t\t *\n> > +\t\t * Note that this could likely be improved, but is ignored since\n> > +\t\t * all real-world graphs set the maximum number of changed paths\n> > +\t\t * at 512.\n> > +\t\t */\n> > +\t\treturn 0;\n> > +\t}\n> > \treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n> > }\n> >\n> > @@ -1470,6 +1482,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> > \tint i;\n> > \tstruct progress *progress = NULL;\n> > \tint *sorted_commits;\n> > +\tint max_new_filters;\n> >\n> > \tinit_bloom_filters();\n> > \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n> > @@ -1486,10 +1499,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> > \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n> > \t\t&ctx->commits);\n> >\n> > +\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n> > +\t\tctx->opts->max_new_filters : ctx->commits.nr;\n>\n> git_test_write_commit_graph_or_die() invokes\n> write_commit_graph_reachable() with opts=0x0, so 'ctx->opts' is NULL,\n> and we get segfault. This breaks a lot of tests when run with\n> GIT_TEST_COMMIT_GRAPH=1 GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1.\n\nGreat catch, thanks. Fixing this is as simple as adding 'ctx->opts &&'\nright before we dereference 'ctx->opts', since setting this variable\nequal to 'ctx->commits.nr' is the right thing to do in that case.\n\nUnrelated to this comment, I am hoping to send out a final version of\nthis series sometime next week so that we can keep moving forward with\nBloom filter improvements.\n\nHave you had a chance to review the rest of the patches? I'll happily\nwait until you have had a chance to do so before sending v5 so that we\ncan avoid a v6.\n\nThanks,\nTaylor\n"},{"id":"403905","messageId":"20200817225004.GB29528@szeder.dev","threadId":"53975","inReplyTo":"20200814202021.GE30103@syl.lan","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-08-17T22:50:04Z","receivedAt":"2020-08-17T22:50:12Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Fri, Aug 14, 2020 at 04:20:21PM -0400, Taylor Blau wrote:\n> > > @@ -1486,10 +1499,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> > > \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n> > > \t\t&ctx->commits);\n> > >\n> > > +\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n> > > +\t\tctx->opts->max_new_filters : ctx->commits.nr;\n> >\n> > git_test_write_commit_graph_or_die() invokes\n> > write_commit_graph_reachable() with opts=0x0, so 'ctx->opts' is NULL,\n> > and we get segfault. This breaks a lot of tests when run with\n> > GIT_TEST_COMMIT_GRAPH=1 GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1.\n> \n> Great catch, thanks. Fixing this is as simple as adding 'ctx->opts &&'\n> right before we dereference 'ctx->opts', since setting this variable\n> equal to 'ctx->commits.nr' is the right thing to do in that case.\n\nThat would avoid the segfault, sure, but I would rather see all\ncallers of write_commit_graph{_reachable}() passing a valid opts\ninstance. Just like we don't call the diff machinery with a NULL\ndiff_options, or the revision walking machinery with a NULL rev_info.\n\n> Unrelated to this comment, I am hoping to send out a final version of\n> this series sometime next week so that we can keep moving forward with\n> Bloom filter improvements.\n> \n> Have you had a chance to review the rest of the patches? I'll happily\n> wait until you have had a chance to do so before sending v5 so that we\n\nv5? This is v3, and I'm unable to a find a v4.\n\n> can avoid a v6.\n> \n> Thanks,\n> Taylor\n"},{"id":"403977","messageId":"20200818222329.GC29528@szeder.dev","threadId":"53975","inReplyTo":"09f6871f66bff838c067a3e0d23cd4622171f3bd.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-08-18T22:23:29Z","receivedAt":"2020-08-18T22:23:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 04:52:14PM -0400, Taylor Blau wrote:\n> Introduce a command-line flag and configuration variable to fill in the\n> 'max_new_filters' variable introduced by the previous patch.\n\n'max_new_filters' is introduced in this patch.\n\n> The command-line option '--max-new-filters' takes precedence over\n> 'commitGraph.maxNewFilters', which is the default value.\n\nWhat \"filters\"? While misnamed, the '--changed-paths' options did a\ngood job at hiding the implementation detail that we use Bloom filters\nto speed up pathspec-limited revision walks; as far as I remember this\nwas a conscious design decision. Including \"filter\" in the name of\nthe option and corresponding config variable goes against this\ndecision. Furthermore, by not being specific we might end up in abad\nsituation when adding some other filters to the commit-graph format.\nUnfortunately, I can't offhand propose a better option name, all my\nideas were horrible.\n\n> '--no-max-new-filters' can also be provided, which sets the value back\n> to '-1', indicating that an unlimited number of new Bloom filters may be\n> generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n> '0', hence a custom callback was used instead).\n> \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/config/commitgraph.txt | 4 +++\n> Documentation/git-commit-graph.txt | 4 +++\n> bloom.c | 15 +++++++++++\n> builtin/commit-graph.c | 39 +++++++++++++++++++++++++---\n> commit-graph.c | 27 ++++++++++++++++---\n> commit-graph.h | 1 +\n> t/t4216-log-bloom.sh | 19 ++++++++++++++\n> 7 files changed, 102 insertions(+), 7 deletions(-)\n> \n> diff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\n> index cff0797b54..4582c39fc4 100644\n> --- a/Documentation/config/commitgraph.txt\n> +++ b/Documentation/config/commitgraph.txt\n> @@ -1,3 +1,7 @@\n> +commitGraph.maxNewFilters::\n> +\tSpecifies the default value for the `--max-new-filters` option of `git\n> +\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n> +\n> commitGraph.readChangedPaths::\n> \tIf true, then git will use the changed-path Bloom filters in the\n> \tcommit-graph file (if it exists, and they are present). Defaults to\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 17405c73a9..9c887d5d79 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -67,6 +67,10 @@ this option is given, future commit-graph writes will automatically assume\n> that this option was intended. Use `--no-changed-paths` to stop storing this\n> data.\n> +\n> +With the `--max-new-filters=` option, generate at most `n` new Bloom\n> +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n> +enforced. Overrides the `commitGraph.maxNewFilters` configuration.\n\nI think this description should also detail what happens with those\ncommits for which no modified path Bloom filters are calculated, and\nwhich commands will calculate them (even implicitly).\n\n> ++\n> With the `--split[=]` option, write the commit-graph as a\n> chain of multiple commit-graph files stored in\n> `/info/commit-graphs`. Commit-graph layers are merged based on the\n> diff --git a/bloom.c b/bloom.c\n> index ed54e96e57..8d07209c6b 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -51,6 +51,21 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n> \telse\n> \t\tstart_index = 0;\n> \n> +\tif ((start_index == end_index) &&\n> +\t (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n> +\t\t/*\n> +\t\t * If the filter is zero-length, either (1) the filter has no\n> +\t\t * changes, (2) the filter has too many changes, or (3) it\n> +\t\t * wasn't computed (eg., due to '--max-new-filters').\n> +\t\t *\n> +\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n> +\t\t * for this Bloom filter. If it is unset, then it wasn't\n> +\t\t * computed. In that case, return nothing, since we don't have\n> +\t\t * that filter in the graph.\n> +\t\t */\n> +\t\treturn 0;\n> +\t}\n> +\n> \tfilter->len = end_index - start_index;\n> \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n> \t\t\t\t\tsizeof(unsigned char) * start_index +\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 38f5f57d15..3500a6e1f1 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n> \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> -\t \"[--changed-paths] [--[no-]progress] \"),\n> +\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n> +\t \"\"),\n> \tNULL\n> };\n> \n> @@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n> static const char * const builtin_commit_graph_write_usage[] = {\n> \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> -\t \"[--changed-paths] [--[no-]progress] \"),\n> +\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n> +\t \"\"),\n> \tNULL\n> };\n> \n> @@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n> \treturn 0;\n> }\n> \n> +static int write_option_max_new_filters(const struct option *opt,\n> +\t\t\t\t\tconst char *arg,\n> +\t\t\t\t\tint unset)\n> +{\n> +\tint *to = opt->value;\n> +\tif (unset)\n> +\t\t*to = -1;\n> +\telse {\n> +\t\tconst char *s;\n> +\t\t*to = strtol(arg, (char **)&s, 10);\n> +\t\tif (*s)\n> +\t\t\treturn error(_(\"%s expects a numerical value\"),\n> +\t\t\t\t optname(opt, opt->flags));\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> static int graph_write(int argc, const char **argv)\n> {\n> \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n> @@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n> \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n> \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n> \t\t\tN_(\"only expire files older than a given date-time\")),\n> +\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n> +\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n> +\t\t\t0, write_option_max_new_filters),\n> \t\tOPT_END(),\n> \t};\n> \n> @@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n> \twrite_opts.size_multiple = 2;\n> \twrite_opts.max_commits = 0;\n> \twrite_opts.expire_time = 0;\n> +\twrite_opts.max_new_filters = -1;\n> \n> \ttrace2_cmd_mode(\"write\");\n> \n> @@ -270,6 +293,16 @@ static int graph_write(int argc, const char **argv)\n> \treturn result;\n> }\n> \n> +static int git_commit_graph_config(const char *var, const char *value, void *cb)\n> +{\n> +\tif (!strcmp(var, \"commitgraph.maxnewfilters\")) {\n> +\t\twrite_opts.max_new_filters = git_config_int(var, value);\n> +\t\treturn 0;\n> +\t}\n> +\n> +\treturn git_default_config(var, value, cb);\n> +}\n> +\n> int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> {\n> \tstatic struct option builtin_commit_graph_options[] = {\n> @@ -283,7 +316,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\tusage_with_options(builtin_commit_graph_usage,\n> \t\t\t\t builtin_commit_graph_options);\n> \n> -\tgit_config(git_default_config, NULL);\n> +\tgit_config(git_commit_graph_config, &opts);\n> \targc = parse_options(argc, argv, prefix,\n> \t\t\t builtin_commit_graph_options,\n> \t\t\t builtin_commit_graph_usage,\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 6886f319a5..4aae5471e3 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -954,7 +954,8 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> }\n> \n> static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> -\t\t\t\t\t const struct commit *c)\n> +\t\t\t\t\t const struct commit *c,\n> +\t\t\t\t\t uint32_t max_changed_paths)\n> {\n> \tuint32_t graph_pos = commit_graph_position(c);\n> \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n> @@ -965,6 +966,17 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> \n> \tif (!(g && g->bloom_large))\n> \t\treturn 0;\n> +\tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n> +\t\t/*\n> +\t\t * Force all commits which are subject to a different\n> +\t\t * 'max_changed_paths' limit to be recomputed from scratch.\n> +\t\t *\n> +\t\t * Note that this could likely be improved, but is ignored since\n> +\t\t * all real-world graphs set the maximum number of changed paths\n> +\t\t * at 512.\n\nI don't understand what the second part of this comment is trying to\nsay; and real-world graphs might very well contain Bloom filters with\nmore than 512 modified paths, because the applying that limit was\nbuggy.\n\n> +\t\t */\n> +\t\treturn 0;\n> +\t}\n> \treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n> }\n> \n> @@ -1470,6 +1482,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \tint i;\n> \tstruct progress *progress = NULL;\n> \tint *sorted_commits;\n> +\tint max_new_filters;\n> \n> \tinit_bloom_filters();\n> \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n> @@ -1486,10 +1499,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n> \t\t&ctx->commits);\n> \n> +\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n> +\t\tctx->opts->max_new_filters : ctx->commits.nr;\n> +\n> \tfor (i = 0; i < ctx->commits.nr; i++) {\n> \t\tint pos = sorted_commits[i];\n> \t\tstruct commit *c = ctx->commits.list[pos];\n> -\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n> +\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph,\n> +\t\t\t\t\t\t c,\n> +\t\t\t\t\t\t ctx->bloom_settings->max_changed_paths)) {\n> \t\t\tbitmap_set(ctx->bloom_large, pos);\n> \t\t\tctx->count_bloom_filter_known_large++;\n> \t\t} else {\n> @@ -1497,7 +1515,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n> \t\t\t\tctx->r,\n> \t\t\t\tc,\n> -\t\t\t\t1,\n> +\t\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n> \t\t\t\tctx->bloom_settings,\n> \t\t\t\t&computed);\n> \t\t\tif (computed) {\n> @@ -1507,7 +1525,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \t\t\t\t\tctx->count_bloom_filter_found_large++;\n> \t\t\t\t}\n> \t\t\t}\n> -\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n> +\t\t\tif (filter)\n> +\t\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n> \t\t}\n> \t\tdisplay_progress(progress, i + 1);\n> \t}\n> diff --git a/commit-graph.h b/commit-graph.h\n> index af08c4505d..75ef83708b 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -114,6 +114,7 @@ struct commit_graph_opts {\n> \tint max_commits;\n> \ttimestamp_t expire_time;\n> \tenum commit_graph_split_flags flags;\n> +\tint max_new_filters;\n> };\n> \n> /*\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index 6859d85369..3aab8ffbe3 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -286,4 +286,23 @@ test_expect_success 'Bloom generation does not recompute too-large filters' '\n> \t)\n> '\n> \n> +test_expect_success 'Bloom generation is limited by --max-new-filters' '\n> +\t(\n> +\t\tcd limits &&\n> +\t\ttest_commit c2 filter &&\n> +\t\ttest_commit c3 filter &&\n> +\t\ttest_commit c4 no-filter &&\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n> +\t\t\t2 0 2\n> +\t)\n> +'\n> +\n> +test_expect_success 'Bloom generation backfills previously-skipped filters' '\n> +\t(\n> +\t\tcd limits &&\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=1\" \\\n> +\t\t\t2 0 1\n> +\t)\n> +'\n> +\n> test_done\n> -- \n> 2.28.0.rc1.13.ge78abce653\n"},{"id":"403994","messageId":"20200819082021.GA5238@szeder.dev","threadId":"53975","inReplyTo":"09f6871f66bff838c067a3e0d23cd4622171f3bd.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-08-19T08:20:21Z","receivedAt":"2020-08-19T08:20:32Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 04:52:14PM -0400, Taylor Blau wrote:\n> Introduce a command-line flag and configuration variable to fill in the\n> 'max_new_filters' variable introduced by the previous patch.\n> \n> The command-line option '--max-new-filters' takes precedence over\n> 'commitGraph.maxNewFilters', which is the default value.\n> '--no-max-new-filters' can also be provided, which sets the value back\n> to '-1', indicating that an unlimited number of new Bloom filters may be\n> generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n> '0', hence a custom callback was used instead).\n\nForgot the most important thing: Why? Please explain in the commit\nmessage why this option is necesary, what problems does it solve,\nhow it is supposed to interact with other options and why so.\n\n"},{"id":"403996","messageId":"20200819095656.GD29528@szeder.dev","threadId":"53975","inReplyTo":"b2e33ecba880f4b49af7a94fb0decb08929dff36.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 13/14] commit-graph: rename 'split_commit_graph_opts'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-08-19T09:56:56Z","receivedAt":"2020-08-19T09:57:06Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 04:52:11PM -0400, Taylor Blau wrote:\n> In the subsequent commit, additional options will be added to the\n> commit-graph API which have nothing to do with splitting.\n> \n> Rename the 'split_commit_graph_opts' structure to the more-generic\n> 'commit_graph_opts' to encompass both.\n\nGood. Note, however, that write_commit_graph() has a 'flags'\nparameter as well, and when a feature is enabled via this 'flags',\nthen first a corresponding 'ctx->foo' field is set, and that\n'ctx->foo' is checked while computing and writing the commit-graph.\nWith the generic options struct some other feature will be enabled via\nthe 'opts->bar' field, so simply 'ctx->opts->bar' is checked while\nwriting the commit-graph.\n\nWith the generic options struct there really is no need for a separate\nflags parameter, the values in the flags can be stored in the options\nstruct, and we can eliminate this inconsistency instead of adding even\nmore.\n\n\n> diff --git a/commit-graph.h b/commit-graph.h\n> index ddbca1b59d..af08c4505d 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -109,7 +109,7 @@ enum commit_graph_split_flags {\n> \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n> };\n> \n> -struct split_commit_graph_opts {\n> +struct commit_graph_opts {\n> \tint size_multiple;\n> \tint max_commits;\n> \ttimestamp_t expire_time;\n enum commit_graph_split_flags flags;\n\nWhile this was 'struct split_commit_graph_opts *split_opts' it was\nclear what kind of flags were in this 'flags' field. Now that the\nstruct is generic it's not clear anymore, so perhaps it should be\nrenamed as well (e.g. 'split_flags'), or even turned into a couple of\nbit fields.\n\n> @@ -124,12 +124,12 @@ struct split_commit_graph_opts {\n> */\n> int write_commit_graph_reachable(struct object_directory *odb,\n> \t\t\t\t enum commit_graph_write_flags flags,\n> -\t\t\t\t const struct split_commit_graph_opts *split_opts);\n> +\t\t\t\t const struct commit_graph_opts *opts);\n> int write_commit_graph(struct object_directory *odb,\n> \t\t struct string_list *pack_indexes,\n> \t\t struct oidset *commits,\n> \t\t enum commit_graph_write_flags flags,\n> -\t\t const struct split_commit_graph_opts *split_opts);\n> +\t\t const struct commit_graph_opts *opts);\n> \n> #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n> \n> -- \n> 2.28.0.rc1.13.ge78abce653\n> \n"},{"id":"404007","messageId":"20200819133526.GE29528@szeder.dev","threadId":"53975","inReplyTo":"619e0c619dd12e2bea2b80c7d249ba66fe2a315c.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 12/14] commit-graph: add large-filters bitmap chunk","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-08-19T13:35:26Z","receivedAt":"2020-08-19T13:37:23Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 04:52:07PM -0400, Taylor Blau wrote:\n> When a commit has more than a certain number of changed paths (commonly\n> 512), the commit-graph machinery represents it as a zero-length filter.\n> This is done since having many entries in the Bloom filter has\n> undesirable effects on the false positivity rate.\n\nThis is not the case, the false positive probability depends on the\nratio of the Bloom filter's size and the number of elements it\ncontains, and we size the filters proportional to the number of\nelements they contain, so the number of elements shouldn't affect the \nfalse positive rate.\n\nOn the contrary, it's the small filters, up to around 30-35 bytes, \nthat tend to have larger than expected false positive rate when using\ndouble hashing.\n\n"},{"id":"404865","messageId":"20200901143546.GA5678@szeder.dev","threadId":"53975","inReplyTo":"619e0c619dd12e2bea2b80c7d249ba66fe2a315c.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 12/14] commit-graph: add large-filters bitmap chunk","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-09-01T14:35:46Z","receivedAt":"2020-09-01T14:36:30Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 04:52:07PM -0400, Taylor Blau wrote:\n> When a commit has more than a certain number of changed paths (commonly\n> 512), the commit-graph machinery represents it as a zero-length filter.\n> This is done since having many entries in the Bloom filter has\n> undesirable effects on the false positivity rate.\n> \n> In addition to these too-large filters, the commit-graph machinery also\n> represents commits with no filter and commits with no changed paths in\n> the same way.\n> \n> When writing a commit-graph that aggregates several incremental\n> commit-graph layers (eg., with '--split=replace'), the commit-graph\n> machinery first computes all of the Bloom filters that it wants to write\n> but does not already know about from existing graph layers. Because we\n> overload the zero-length filter in the above fashion, this leads to\n> recomputing large filters over and over again.\n> \n> This is already undesirable, since it means that we are wasting\n> considerable effort to discover that a commit with too many changed\n> paths, only to throw that effort away (and then repeat the process the\n> next time a roll-up is performed).\n\nIs this really the case?\n\nIf a commit has a corresponding entry in the Bloom filters chunks,\nthen the commit-graph machinery does know the Bloom filter associated\nwith that commit. The size of that filter should not matter, i.e.\neven if it is a zero-length filter, the commit-graph machinery should\nknow about it all the same. And as far as I can tell it does indeed,\nbecause load_bloom_filter_from_graph() sets a non-NULL 'filter->data'\npointer even if 'filter->len' is zero, which get_bloom_filter() treats\nas \"we know about this\", and returns early without (re)computing the\nfilter. Even the test 'Bloom generation does not recompute too-large\nfilters' added in this patch is expected to succeed, and my\nsuperficial and makeshift testing seems to corroborate this; at least\nI couldn't find a combination of commands and options that would\nrecompute any existing zero-length Bloom filters.\n\nAm I missing something?\n\n> In a subsequent patch, we will add a '--max-new-filters=' option,\n> which specifies an upper-bound on the number of new filters we are\n> willing to compute from scratch. Suppose that there are 'N' too-large\n> filters, and we specify '--max-new-filters=M'. If 'N >= M', it is\n> unlikely that any filters will be generated, since we'll spend most of\n> our effort on filters that we ultimately throw away. If 'N < M', filters\n> will trickle in over time, but only at most 'M - N' per-write.\n> \n> To address this, add a new chunk which encodes a bitmap where the ith\n> bit is on iff the ith commit has zero or at least 512 changed paths.\n> Likewise store the maximum number of changed paths we are willing to\n> store in order to prepare for eventually making this value more easily\n> customizable.\n\nI don't understand how storing this value would make it any easier to\ncustomize.\n\n> When computing Bloom filters, first consult the relevant\n> bitmap (in the case that we are rolling up existing layers) to see if\n> computing the Bloom filter from scratch would be a waste of time.\n> \n> This patch implements a new chunk instead of extending the existing BIDX\n> and BDAT chunks because modifying these chunks would confuse old\n> clients. (Eg., setting the most-significant bit in the BIDX chunk would\n> confuse old clients and require a version bump).\n> \n> To allow using the existing bitmap code with 64-bit words, we write the\n> data in network byte order from the 64-bit words. This means we also\n> need to read the array from the commit-graph file by translating each\n> word from network byte order using get_be64() when loading the commit\n> graph. (Note that this *could* be delayed until first-use, but a later\n> patch will rely on this being initialized early, so we assume the\n> up-front cost when parsing instead of delaying initialization).\n> \n> By avoiding the need to move to new versions of the BDAT and BIDX chunk,\n> we can give ourselves more time to consider whether or not other\n> modifications to these chunks are worthwhile without holding up this\n> change.\n> \n> Another approach would be to introduce a new BIDX chunk (say, one\n> identified by 'BID2') which is identical to the existing BIDX chunk,\n> except the most-significant bit of each offset is interpreted as \"this\n> filter is too big\" iff looking at a BID2 chunk. This avoids having to\n> write a bitmap, but forces older clients to rewrite their commit-graphs\n> (as well as reduces the theoretical largest Bloom filters we couldl\n\nAnd it reduces the max possible size of the BDAT chunk, and thus the\nmax number of commits with Bloom filters as well.\n\ns/couldl/could/\n\n> write, and forces us to maintain the code necessary to translate BIDX\n> chunks to BID2 ones). Separately from this patch, I implemented this\n> alternate approach and did not find it to be advantageous.\n\nLet's take a step back to reconsider what should be stored in this\nbitmap for a moment. Sure, setting a bit for each commit that doesn't\nmodify any paths or modifies too many makes it possible to repliably\nidentify commits that don't have Bloom filters yet. But isn't it a\nbit roundabout way? I think it would be better to directly track\nwhich commits don't have Bloom filters yet. IOW what you really want\nis a, say, BNCY \"Bloom filter Not Computed Yet\" chunk, where we set\nthe corresponding bit for each commit which has an entry in the BIDX\nchunk but for which a Bloom filter hasn't been computed yet.\n\n - It's simpler and easier to explain (IMO).\n\n - This bitmap chunk can easily be made optional: if all Bloom\n filters have been computed, then the bitmap will contain all\n zeros. So why bother writing it, when we can save a bit of space\n instead?\n\n - It avoids the unpleasentness of setting a bit in the _Large_ Bloom\n Filters chunks for commits _not_ modifying any paths.\n\n - Less incentive to spill implementation details to the format\n specification (e.g. 512 modified paths).\n\nNow, let's take another step back: is such a bitmap really necessary?\nWe could write a single-byte Bloom filter with no bits set for commits\nnot modifying any paths, and a single-byte Bloom filter with all bits\nset for commits modifying too many paths. This is compatible with the\nspecs and any existing implementation should do the right thing when\nreading such filters, this would allow us to interpret zero-length\nfilters as \"not computed yet\", and if that bitmap chunk won't be\noptional, then this would save space as long as less than 1/8 of\ncommits modify no or too many paths. Unfortunately, however, all\nexisting zero-length Bloom filters have to be recomputed.\n\n\n> Signed-off-by: Taylor Blau \n> ---\n> .../technical/commit-graph-format.txt | 12 +++\n> bloom.h | 2 +-\n> commit-graph.c | 96 ++++++++++++++++---\n> commit-graph.h | 4 +\n> t/t4216-log-bloom.sh | 25 ++++-\n> 5 files changed, 124 insertions(+), 15 deletions(-)\n> \n> diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n> index 440541045d..5f2d9ab4d7 100644\n> --- a/Documentation/technical/commit-graph-format.txt\n> +++ b/Documentation/technical/commit-graph-format.txt\n> @@ -123,6 +123,18 @@ CHUNK DATA:\n> of length zero.\n> * The BDAT chunk is present if and only if BIDX is present.\n> \n> + Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n> + * It starts with a 32-bit unsigned integer specifying the maximum number of\n> + changed-paths that can be stored in a single Bloom filter.\n\nShould this value be the same in all elements of a commit-graph chain?\nNote that in this case having different values won't affect revision\nwalks using modified path Bloom filters.\n\n> + * It then contains a list of 64-bit words (the length of this list is\n> + determined by the width of the chunk) which is a bitmap. The 'i'th bit is\n> + set exactly when the 'i'th commit in the graph has a changed-path Bloom\n> + filter with zero entries (either because the commit is empty, or because\n> + it contains more than 512 changed paths).\n\nPlease make clear the byte order of these 64 bit words in the specs as\nwell.\n\nFurthermore, that 512 path limit is an implementation detail, so it\nwould be better if it didn't leak into the specification of this new\nchunk.\n\n> + * The BFXL chunk is present only when the BIDX and BDAT chunks are\n> + also present.\n> +\n> +\n> Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n> This list of H-byte hashes describe a set of B commit-graph files that\n> form a commit-graph chain. The graph position for the ith commit in this\n\n\n\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index 21b67677ef..6859d85369 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -33,7 +33,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n> \tgit commit-graph write --reachable --changed-paths\n> '\n> graph_read_expect () {\n> -\tNUM_CHUNKS=5\n> +\tNUM_CHUNKS=6\n> \tcat >expect <<- EOF\n> \theader: 43475048 1 1 $NUM_CHUNKS 0\n> \tnum_commits: $1\n> @@ -262,5 +262,28 @@ test_expect_success 'correctly report changes over limit' '\n> \t\tdone\n> \t)\n> '\n> +test_bloom_filters_computed () {\n> +\tcommit_graph_args=$1\n> +\tbloom_trace_prefix=\"{\\\"filter_known_large\\\":$2,\\\"filter_found_large\\\":$3,\\\"filter_computed\\\":$4\"\n> +\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n> +\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write $commit_graph_args &&\n> +\tgrep \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.event\"\n> +}\n> +\n> +test_expect_success 'Bloom generation does not recompute too-large filters' '\n> +\t(\n> +\t\tcd limits &&\n> +\n> +\t\t# start from scratch and rebuild\n> +\t\trm -f .git/objects/info/commit-graph &&\n> +\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n> +\t\t\tgit commit-graph write --reachable --changed-paths \\\n> +\t\t\t--split=replace &&\n> +\t\ttest_commit c1 filter &&\n> +\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n> +\t\t\t2 0 1\n> +\t)\n> +'\n> \n> test_done\n> -- \n> 2.28.0.rc1.13.ge78abce653\n> \n"},{"id":"404866","messageId":"20200901143650.GA22631@szeder.dev","threadId":"53975","inReplyTo":"09f6871f66bff838c067a3e0d23cd4622171f3bd.1597178915.git.me@ttaylorr.com","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-09-01T14:36:50Z","receivedAt":"2020-09-01T14:37:08Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Tue, Aug 11, 2020 at 04:52:14PM -0400, Taylor Blau wrote:\n> Introduce a command-line flag and configuration variable to fill in the\n> 'max_new_filters' variable introduced by the previous patch.\n> \n> The command-line option '--max-new-filters' takes precedence over\n> 'commitGraph.maxNewFilters', which is the default value.\n> '--no-max-new-filters' can also be provided, which sets the value back\n> to '-1', indicating that an unlimited number of new Bloom filters may be\n> generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n> '0', hence a custom callback was used instead).\n> \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/config/commitgraph.txt | 4 +++\n> Documentation/git-commit-graph.txt | 4 +++\n> bloom.c | 15 +++++++++++\n> builtin/commit-graph.c | 39 +++++++++++++++++++++++++---\n> commit-graph.c | 27 ++++++++++++++++---\n> commit-graph.h | 1 +\n> t/t4216-log-bloom.sh | 19 ++++++++++++++\n> 7 files changed, 102 insertions(+), 7 deletions(-)\n> \n> diff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\n> index cff0797b54..4582c39fc4 100644\n> --- a/Documentation/config/commitgraph.txt\n> +++ b/Documentation/config/commitgraph.txt\n> @@ -1,3 +1,7 @@\n> +commitGraph.maxNewFilters::\n> +\tSpecifies the default value for the `--max-new-filters` option of `git\n> +\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n> +\n> commitGraph.readChangedPaths::\n> \tIf true, then git will use the changed-path Bloom filters in the\n> \tcommit-graph file (if it exists, and they are present). Defaults to\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 17405c73a9..9c887d5d79 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -67,6 +67,10 @@ this option is given, future commit-graph writes will automatically assume\n> that this option was intended. Use `--no-changed-paths` to stop storing this\n> data.\n> +\n> +With the `--max-new-filters=` option, generate at most `n` new Bloom\n> +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n> +enforced. Overrides the `commitGraph.maxNewFilters` configuration.\n> ++\n> With the `--split[=]` option, write the commit-graph as a\n> chain of multiple commit-graph files stored in\n> `/info/commit-graphs`. Commit-graph layers are merged based on the\n> diff --git a/bloom.c b/bloom.c\n> index ed54e96e57..8d07209c6b 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -51,6 +51,21 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n> \telse\n> \t\tstart_index = 0;\n> \n> +\tif ((start_index == end_index) &&\n> +\t (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n> +\t\t/*\n> +\t\t * If the filter is zero-length, either (1) the filter has no\n> +\t\t * changes, (2) the filter has too many changes, or (3) it\n> +\t\t * wasn't computed (eg., due to '--max-new-filters').\n> +\t\t *\n> +\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n> +\t\t * for this Bloom filter. If it is unset, then it wasn't\n> +\t\t * computed. In that case, return nothing, since we don't have\n> +\t\t * that filter in the graph.\n> +\t\t */\n> +\t\treturn 0;\n> +\t}\n> +\n> \tfilter->len = end_index - start_index;\n> \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n> \t\t\t\t\tsizeof(unsigned char) * start_index +\n> diff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\n> index 38f5f57d15..3500a6e1f1 100644\n> --- a/builtin/commit-graph.c\n> +++ b/builtin/commit-graph.c\n> @@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n> \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n> \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> -\t \"[--changed-paths] [--[no-]progress] \"),\n> +\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n> +\t \"\"),\n> \tNULL\n> };\n> \n> @@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n> static const char * const builtin_commit_graph_write_usage[] = {\n> \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n> \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n> -\t \"[--changed-paths] [--[no-]progress] \"),\n> +\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n> +\t \"\"),\n> \tNULL\n> };\n> \n> @@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n> \treturn 0;\n> }\n> \n> +static int write_option_max_new_filters(const struct option *opt,\n> +\t\t\t\t\tconst char *arg,\n> +\t\t\t\t\tint unset)\n> +{\n> +\tint *to = opt->value;\n> +\tif (unset)\n> +\t\t*to = -1;\n> +\telse {\n> +\t\tconst char *s;\n> +\t\t*to = strtol(arg, (char **)&s, 10);\n> +\t\tif (*s)\n> +\t\t\treturn error(_(\"%s expects a numerical value\"),\n> +\t\t\t\t optname(opt, opt->flags));\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> static int graph_write(int argc, const char **argv)\n> {\n> \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n> @@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n> \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n> \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n> \t\t\tN_(\"only expire files older than a given date-time\")),\n> +\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n> +\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n> +\t\t\t0, write_option_max_new_filters),\n> \t\tOPT_END(),\n> \t};\n> \n> @@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n> \twrite_opts.size_multiple = 2;\n> \twrite_opts.max_commits = 0;\n> \twrite_opts.expire_time = 0;\n> +\twrite_opts.max_new_filters = -1;\n> \n> \ttrace2_cmd_mode(\"write\");\n> \n> @@ -270,6 +293,16 @@ static int graph_write(int argc, const char **argv)\n> \treturn result;\n> }\n> \n> +static int git_commit_graph_config(const char *var, const char *value, void *cb)\n> +{\n> +\tif (!strcmp(var, \"commitgraph.maxnewfilters\")) {\n> +\t\twrite_opts.max_new_filters = git_config_int(var, value);\n> +\t\treturn 0;\n> +\t}\n> +\n> +\treturn git_default_config(var, value, cb);\n> +}\n> +\n> int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> {\n> \tstatic struct option builtin_commit_graph_options[] = {\n> @@ -283,7 +316,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n> \t\tusage_with_options(builtin_commit_graph_usage,\n> \t\t\t\t builtin_commit_graph_options);\n> \n> -\tgit_config(git_default_config, NULL);\n> +\tgit_config(git_commit_graph_config, &opts);\n> \targc = parse_options(argc, argv, prefix,\n> \t\t\t builtin_commit_graph_options,\n> \t\t\t builtin_commit_graph_usage,\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 6886f319a5..4aae5471e3 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -954,7 +954,8 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n> }\n> \n> static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> -\t\t\t\t\t const struct commit *c)\n> +\t\t\t\t\t const struct commit *c,\n> +\t\t\t\t\t uint32_t max_changed_paths)\n> {\n> \tuint32_t graph_pos = commit_graph_position(c);\n> \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n> @@ -965,6 +966,17 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> \n> \tif (!(g && g->bloom_large))\n> \t\treturn 0;\n> +\tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n> +\t\t/*\n> +\t\t * Force all commits which are subject to a different\n> +\t\t * 'max_changed_paths' limit to be recomputed from scratch.\n> +\t\t *\n> +\t\t * Note that this could likely be improved, but is ignored since\n> +\t\t * all real-world graphs set the maximum number of changed paths\n> +\t\t * at 512.\n> +\t\t */\n> +\t\treturn 0;\n> +\t}\n> \treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n> }\n> \n> @@ -1470,6 +1482,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \tint i;\n> \tstruct progress *progress = NULL;\n> \tint *sorted_commits;\n> +\tint max_new_filters;\n> \n> \tinit_bloom_filters();\n> \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n> @@ -1486,10 +1499,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n> \t\t&ctx->commits);\n> \n> +\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n> +\t\tctx->opts->max_new_filters : ctx->commits.nr;\n> +\n> \tfor (i = 0; i < ctx->commits.nr; i++) {\n> \t\tint pos = sorted_commits[i];\n> \t\tstruct commit *c = ctx->commits.list[pos];\n> -\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n> +\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph,\n> +\t\t\t\t\t\t c,\n> +\t\t\t\t\t\t ctx->bloom_settings->max_changed_paths)) {\n> \t\t\tbitmap_set(ctx->bloom_large, pos);\n> \t\t\tctx->count_bloom_filter_known_large++;\n> \t\t} else {\n> @@ -1497,7 +1515,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n> \t\t\t\tctx->r,\n> \t\t\t\tc,\n> -\t\t\t\t1,\n> +\t\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n> \t\t\t\tctx->bloom_settings,\n> \t\t\t\t&computed);\n> \t\t\tif (computed) {\n> @@ -1507,7 +1525,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \t\t\t\t\tctx->count_bloom_filter_found_large++;\n> \t\t\t\t}\n> \t\t\t}\n> -\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n> +\t\t\tif (filter)\n> +\t\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n> \t\t}\n> \t\tdisplay_progress(progress, i + 1);\n> \t}\n> diff --git a/commit-graph.h b/commit-graph.h\n> index af08c4505d..75ef83708b 100644\n> --- a/commit-graph.h\n> +++ b/commit-graph.h\n> @@ -114,6 +114,7 @@ struct commit_graph_opts {\n> \tint max_commits;\n> \ttimestamp_t expire_time;\n> \tenum commit_graph_split_flags flags;\n> +\tint max_new_filters;\n> };\n> \n> /*\n> diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n> index 6859d85369..3aab8ffbe3 100755\n> --- a/t/t4216-log-bloom.sh\n> +++ b/t/t4216-log-bloom.sh\n> @@ -286,4 +286,23 @@ test_expect_success 'Bloom generation does not recompute too-large filters' '\n> \t)\n> '\n> \n> +test_expect_success 'Bloom generation is limited by --max-new-filters' '\n> +\t(\n> +\t\tcd limits &&\n> +\t\ttest_commit c2 filter &&\n> +\t\ttest_commit c3 filter &&\n> +\t\ttest_commit c4 no-filter &&\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n> +\t\t\t2 0 2\n> +\t)\n> +'\n> +\n> +test_expect_success 'Bloom generation backfills previously-skipped filters' '\n> +\t(\n> +\t\tcd limits &&\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=1\" \\\n> +\t\t\t2 0 1\n> +\t)\n> +'\n> +\n> test_done\n> -- \n> 2.28.0.rc1.13.ge78abce653\n\nSomething seems to be wrong in this patch, though I haven't looked\ncloser. Consider this test with a bit of makeshift tracing:\n\n --- >8 ---\n\ndiff --git a/bloom.c b/bloom.c\nindex 8d07209c6b..1a0dec35cd 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -222,6 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \tif (!compute_if_not_present)\n \t\treturn NULL;\n \n+\tprintf(\"get_or_compute_bloom_filter(): diff: %s\\n\", oid_to_hex(&c->object.oid));\n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n \tdiffopt.detect_rename = 0;\ndiff --git a/t/t9999-test.sh b/t/t9999-test.sh\nnew file mode 100755\nindex 0000000000..0833e6ff7e\n--- /dev/null\n+++ b/t/t9999-test.sh\n@@ -0,0 +1,25 @@\n+#!/bin/sh\n+\n+test_description='test'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'test' '\n+\tfor i in 1 2 3 4 5 6\n+\tdo\n+\t\tgit commit -q --allow-empty -m $i || return 1\n+\tdone &&\n+\tgit log --oneline &&\n+\n+\t# We have 6 commits and compute 2 Bloom filters per execution,\n+\t# so 3 executions should be enough... but, alas, it isnt.\n+\tfor i in 1 2 3 4 5\n+\tdo\n+\t\t# No --split=replace!\n+\t\tgit commit-graph write --reachable --changed-paths --max-new-filters=2 || return 1\n+\tdone &&\n+\n+\tgit commit-graph write --reachable --changed-paths --max-new-filters=4\n+'\n+\n+test_done\n\n --- >8 ---\n\nIt's output looks like:\n\n [...]\n + git log --oneline\n 13fcefa (HEAD -> master) 6\n 0c71516 5\n a82a61c 4\n 54c6b2c 3\n fc99def 2\n a3a8cd3 1\n + git commit-graph write --reachable --changed-paths --max-new-filters=2\n get_or_compute_bloom_filter(): diff: 0c71516945bf0a23813e80205961d29ebc1020e0\n get_or_compute_bloom_filter(): diff: 13fcefa4bb859a15c4edc6bb01b8b6c91b4f32b6\n + git commit-graph write --reachable --changed-paths --max-new-filters=2\n get_or_compute_bloom_filter(): diff: 54c6b2cd4fb50066683a197cc6d677689618505a\n get_or_compute_bloom_filter(): diff: a3a8cd3c82028671bf51502d77277baf14a2f528\n + git commit-graph write --reachable --changed-paths --max-new-filters=2\n get_or_compute_bloom_filter(): diff: 0c71516945bf0a23813e80205961d29ebc1020e0\n get_or_compute_bloom_filter(): diff: 13fcefa4bb859a15c4edc6bb01b8b6c91b4f32b6\n + git commit-graph write --reachable --changed-paths --max-new-filters=2\n get_or_compute_bloom_filter(): diff: 54c6b2cd4fb50066683a197cc6d677689618505a\n get_or_compute_bloom_filter(): diff: a3a8cd3c82028671bf51502d77277baf14a2f528\n + git commit-graph write --reachable --changed-paths --max-new-filters=2\n get_or_compute_bloom_filter(): diff: 0c71516945bf0a23813e80205961d29ebc1020e0\n get_or_compute_bloom_filter(): diff: 13fcefa4bb859a15c4edc6bb01b8b6c91b4f32b6\n + git commit-graph write --reachable --changed-paths\n get_or_compute_bloom_filter(): diff: 54c6b2cd4fb50066683a197cc6d677689618505a\n get_or_compute_bloom_filter(): diff: a3a8cd3c82028671bf51502d77277baf14a2f528\n get_or_compute_bloom_filter(): diff: a82a61c79b2b07c4440e292613e11a69e33ef7a2\n get_or_compute_bloom_filter(): diff: fc99def8b1df27bcab7d1f4b7ced73239f9bd7ec\n\nSee how the third write with '--max-new-filters=2' computes the\nfilters that have already been computed by the first write instead of\nthose two that have never been computed? And then how the fourth\nwrite computes filters that have already been computed by the second\nwrite? And ultimately we'll need a write without '--max-new-filters' (or\nwith '--max-new-filters=') to compute all remaining\nfilters.\n\nWith '--split=replace' it appears to work as expected.\n\n"},{"id":"404939","messageId":"20200902202323.GA5281@nand.local","threadId":"53975","inReplyTo":"20200819133526.GE29528@szeder.dev","subject":"Re: [PATCH v3 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-02T20:23:23Z","receivedAt":"2020-09-02T20:23:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"(Finally getting back to this review after working on some other topics\nfor a while..., sorry for the late response).\n\nOn Wed, Aug 19, 2020 at 03:35:26PM +0200, SZEDER Gábor wrote:\n> On Tue, Aug 11, 2020 at 04:52:07PM -0400, Taylor Blau wrote:\n> > When a commit has more than a certain number of changed paths (commonly\n> > 512), the commit-graph machinery represents it as a zero-length filter.\n> > This is done since having many entries in the Bloom filter has\n> > undesirable effects on the false positivity rate.\n>\n> This is not the case, the false positive probability depends on the\n> ratio of the Bloom filter's size and the number of elements it\n> contains, and we size the filters proportional to the number of\n> elements they contain, so the number of elements shouldn't affect the\n> false positive rate.\n\nI'm not sure that I understand. I agree that the FPR depends on the\nratio between the number of elements in the filter and the filter's\n\"size\". But, consider a Bloom filter that is too small to faithfully\nrepresent all its elements. Such a filter would likely have all its bits\nset high, in which case every query would return \"maybe\", and the FPR\nwould go up.\n\n> On the contrary, it's the small filters, up to around 30-35 bytes,\n> that tend to have larger than expected false positive rate when using\n> double hashing.\n\nI agree that small filters suffer from the same, but I think this is an\n\"in addition\" not an \"on the contrary\".\n\nIn either case, I don't think that this is an important detail for the\ncommit message. What matters is the representation (that we truncate >=\n512 elements to a length-zero filter), not why (that can be found in\nanother commit). I'd have expected to find the rationale in ed591febb4\n(bloom.c: core Bloom filter implementation for changed paths.,\n2020-03-30), but I couldn't find anything there.\n\nSo, I'll drop this sentence entirely to avoid an unimportant detail.\n\nThanks,\nTaylor\n"},{"id":"404941","messageId":"20200902204042.GB5281@nand.local","threadId":"53975","inReplyTo":"20200901143546.GA5678@szeder.dev","subject":"Re: [PATCH v3 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-02T20:40:42Z","receivedAt":"2020-09-02T20:40:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 01, 2020 at 04:35:46PM +0200, SZEDER Gábor wrote:\n> On Tue, Aug 11, 2020 at 04:52:07PM -0400, Taylor Blau wrote:\n> > This is already undesirable, since it means that we are wasting\n> > considerable effort to discover that a commit with too many changed\n> > paths, only to throw that effort away (and then repeat the process the\n> > next time a roll-up is performed).\n>\n> Is this really the case?\n>\n> If a commit has a corresponding entry in the Bloom filters chunks,\n> then the commit-graph machinery does know the Bloom filter associated\n> with that commit. The size of that filter should not matter, i.e.\n> even if it is a zero-length filter, the commit-graph machinery should\n> know about it all the same. And as far as I can tell it does indeed,\n> because load_bloom_filter_from_graph() sets a non-NULL 'filter->data'\n> pointer even if 'filter->len' is zero, which get_bloom_filter() treats\n> as \"we know about this\", and returns early without (re)computing the\n> filter. Even the test 'Bloom generation does not recompute too-large\n> filters' added in this patch is expected to succeed, and my\n> superficial and makeshift testing seems to corroborate this; at least\n> I couldn't find a combination of commands and options that would\n> recompute any existing zero-length Bloom filters.\n>\n> Am I missing something?\n\nI'm not sure that this would work, or at least that it creates another\npair of cases that we have to disambiguate. Here's what I'm thinking\nafter reading what you wrote:\n\n - By this point in the series, we expect every commit in a\n commit-graph layer to have a corresponding entry in the 'BIDX' chunk\n (if one exists, obviously all of this is meaningless without\n specifying '--changed-paths').\n\n - In a future commit (specifically, \"builtin/commit-graph.c: introduce\n '--max-new-filters='\"), this will no longer be the case.\n\n - So, by that point in the series, we would have two possible reasons\n why a commit did not show up in the BIDX. Either:\n\n * The commit has too many changed paths, in which case we don't\n want to write it down, or\n\n * We had already computed too many new filters, and so didn't have\n the budget to compute the filter for some commit(s).\n\nIn the first of those two cases, we want to avoid recomputing the\ntoo-large filter. But, in the latter case, we *do* want to compute the\nfilter from scratch, since we want to \"backfill\" commits with missing\nfilters by letting them trickle in over time.\n\nI might be missing something, too, but I think that those two cases are\njust as indistinguishable as what's in the commit-graph today (without\nthe 'BFXL' chunk).\n\n> > To address this, add a new chunk which encodes a bitmap where the ith\n> > bit is on iff the ith commit has zero or at least 512 changed paths.\n> > Likewise store the maximum number of changed paths we are willing to\n> > store in order to prepare for eventually making this value more easily\n> > customizable.\n>\n> I don't understand how storing this value would make it any easier to\n> customize.\n\nIt doesn't, but it's also not meant to. The idea is that if the value\never were easy to customize, then we could keep track of which layers\nwere written with which threshold. This would allow you to invent fancy\nrules like dropping filters when the threshold lowers, or recomputing\nones that you wouldn't have otherwise when it goes up.\n\n> > Another approach would be to introduce a new BIDX chunk (say, one\n> > identified by 'BID2') which is identical to the existing BIDX chunk,\n> > except the most-significant bit of each offset is interpreted as \"this\n> > filter is too big\" iff looking at a BID2 chunk. This avoids having to\n> > write a bitmap, but forces older clients to rewrite their commit-graphs\n> > (as well as reduces the theoretical largest Bloom filters we couldl\n>\n> And it reduces the max possible size of the BDAT chunk, and thus the\n> max number of commits with Bloom filters as well.\n\nFair point.\n\n> s/couldl/could/\n\nOops, thanks.\n\n> > write, and forces us to maintain the code necessary to translate BIDX\n> > chunks to BID2 ones). Separately from this patch, I implemented this\n> > alternate approach and did not find it to be advantageous.\n>\n> Let's take a step back to reconsider what should be stored in this\n> bitmap for a moment. Sure, setting a bit for each commit that doesn't\n> modify any paths or modifies too many makes it possible to repliably\n> identify commits that don't have Bloom filters yet. But isn't it a\n> bit roundabout way? I think it would be better to directly track\n> which commits don't have Bloom filters yet. IOW what you really want\n> is a, say, BNCY \"Bloom filter Not Computed Yet\" chunk, where we set\n> the corresponding bit for each commit which has an entry in the BIDX\n> chunk but for which a Bloom filter hasn't been computed yet.\n>\n> - It's simpler and easier to explain (IMO).\n>\n> - This bitmap chunk can easily be made optional: if all Bloom\n> filters have been computed, then the bitmap will contain all\n> zeros. So why bother writing it, when we can save a bit of space\n> instead?\n\nI don't think that this is true. Omitting the chunk (because you have\ncomputed--or tried to compute--filters for every commit in the graph)\nisn't distinguishable from what exists today, so we are back at square\none there.\n\n> - It avoids the unpleasentness of setting a bit in the _Large_ Bloom\n> Filters chunks for commits _not_ modifying any paths.\n\nI agree it's unpleasant, but I also don't think it's a show-stopper.\n\n> - Less incentive to spill implementation details to the format\n> specification (e.g. 512 modified paths).\n>\n> Now, let's take another step back: is such a bitmap really necessary?\n> We could write a single-byte Bloom filter with no bits set for commits\n> not modifying any paths, and a single-byte Bloom filter with all bits\n> set for commits modifying too many paths. This is compatible with the\n> specs and any existing implementation should do the right thing when\n> reading such filters, this would allow us to interpret zero-length\n> filters as \"not computed yet\", and if that bitmap chunk won't be\n> optional, then this would save space as long as less than 1/8 of\n> commits modify no or too many paths. Unfortunately, however, all\n> existing zero-length Bloom filters have to be recomputed.\n\nI think this is a semantic difference: either you store a bitmap, or a\nBloom filter containing the same data. To me, I don't think there's a\nhuge difference, since we're talking about 1 bit per commit. If we were\nreally worried, we could store them as EWAH-compressed bitmaps, but I\ndon't get a sense that such a concern exists.\n\nI do feel strongly about using a non-probabilistic data structure,\nthough, since the point of this feature is to be able to make tighter\nguarentees about the runtime of 'git commit-graph write'.\n\n>\n> > Signed-off-by: Taylor Blau \n> > ---\n> > .../technical/commit-graph-format.txt | 12 +++\n> > bloom.h | 2 +-\n> > commit-graph.c | 96 ++++++++++++++++---\n> > commit-graph.h | 4 +\n> > t/t4216-log-bloom.sh | 25 ++++-\n> > 5 files changed, 124 insertions(+), 15 deletions(-)\n> >\n> > diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n> > index 440541045d..5f2d9ab4d7 100644\n> > --- a/Documentation/technical/commit-graph-format.txt\n> > +++ b/Documentation/technical/commit-graph-format.txt\n> > @@ -123,6 +123,18 @@ CHUNK DATA:\n> > of length zero.\n> > * The BDAT chunk is present if and only if BIDX is present.\n> >\n> > + Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n> > + * It starts with a 32-bit unsigned integer specifying the maximum number of\n> > + changed-paths that can be stored in a single Bloom filter.\n>\n> Should this value be the same in all elements of a commit-graph chain?\n> Note that in this case having different values won't affect revision\n> walks using modified path Bloom filters.\n\nI don't think it needs to be the same necessarily.\n\n> > + * It then contains a list of 64-bit words (the length of this list is\n> > + determined by the width of the chunk) which is a bitmap. The 'i'th bit is\n> > + set exactly when the 'i'th commit in the graph has a changed-path Bloom\n> > + filter with zero entries (either because the commit is empty, or because\n> > + it contains more than 512 changed paths).\n>\n> Please make clear the byte order of these 64 bit words in the specs as\n> well.\n>\n> Furthermore, that 512 path limit is an implementation detail, so it\n> would be better if it didn't leak into the specification of this new\n> chunk.\n\nAddressed both, thanks.\n\nThanks,\nTaylor\n"},{"id":"404943","messageId":"20200902210212.GC5281@nand.local","threadId":"53975","inReplyTo":"20200819095656.GD29528@szeder.dev","subject":"Re: [PATCH v3 13/14] commit-graph: rename 'split_commit_graph_opts'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-02T21:02:12Z","receivedAt":"2020-09-02T21:02:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 19, 2020 at 11:56:56AM +0200, SZEDER Gábor wrote:\n> On Tue, Aug 11, 2020 at 04:52:11PM -0400, Taylor Blau wrote:\n> > In the subsequent commit, additional options will be added to the\n> > commit-graph API which have nothing to do with splitting.\n> >\n> > Rename the 'split_commit_graph_opts' structure to the more-generic\n> > 'commit_graph_opts' to encompass both.\n>\n> Good. Note, however, that write_commit_graph() has a 'flags'\n> parameter as well, and when a feature is enabled via this 'flags',\n> then first a corresponding 'ctx->foo' field is set, and that\n> 'ctx->foo' is checked while computing and writing the commit-graph.\n> With the generic options struct some other feature will be enabled via\n> the 'opts->bar' field, so simply 'ctx->opts->bar' is checked while\n> writing the commit-graph.\n>\n> With the generic options struct there really is no need for a separate\n> flags parameter, the values in the flags can be stored in the options\n> struct, and we can eliminate this inconsistency instead of adding even\n> more.\n\nI like the direction that you're headed in, but I'm not entirely sure\nwhat you're suggesting. Do you want to make the\n'enum commit_graph_write_flags flags' part of the new options struct?\nBreak out the fields into individual bits on that struct?\n\nI'm not opposed to either, but note that there is also already a 'flags'\nfield on the options structure related to splitting, so that would have\nto be untangled, too.\n\nWhat I'm trying to say is that I think there's more complexity here than\nyou're giving it credit for. I'd rather press on with what we have here,\nand devote adequate time to unraveling the complexity appropriately than\ntry to shove in another patch that takes a half-step in the right\ndirection.\n\n>\n> > diff --git a/commit-graph.h b/commit-graph.h\n> > index ddbca1b59d..af08c4505d 100644\n> > --- a/commit-graph.h\n> > +++ b/commit-graph.h\n> > @@ -109,7 +109,7 @@ enum commit_graph_split_flags {\n> > \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n> > };\n> >\n> > -struct split_commit_graph_opts {\n> > +struct commit_graph_opts {\n> > \tint size_multiple;\n> > \tint max_commits;\n> > \ttimestamp_t expire_time;\n> enum commit_graph_split_flags flags;\n>\n> While this was 'struct split_commit_graph_opts *split_opts' it was\n> clear what kind of flags were in this 'flags' field. Now that the\n> struct is generic it's not clear anymore, so perhaps it should be\n> renamed as well (e.g. 'split_flags'), or even turned into a couple of\n> bit fields.\n\nThis I can definitely vouch for, so I'll 's/flags/split_flags' in the\nnext revision.\n\nThanks,\nTaylor\n"},{"id":"404944","messageId":"20200902210322.GD5281@nand.local","threadId":"53975","inReplyTo":"20200817225004.GB29528@szeder.dev","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-02T21:03:22Z","receivedAt":"2020-09-02T21:03:29Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Aug 18, 2020 at 12:50:04AM +0200, SZEDER Gábor wrote:\n> On Fri, Aug 14, 2020 at 04:20:21PM -0400, Taylor Blau wrote:\n> > > > @@ -1486,10 +1499,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> > > > \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n> > > > \t\t&ctx->commits);\n> > > >\n> > > > +\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n> > > > +\t\tctx->opts->max_new_filters : ctx->commits.nr;\n> > >\n> > > git_test_write_commit_graph_or_die() invokes\n> > > write_commit_graph_reachable() with opts=0x0, so 'ctx->opts' is NULL,\n> > > and we get segfault. This breaks a lot of tests when run with\n> > > GIT_TEST_COMMIT_GRAPH=1 GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1.\n> >\n> > Great catch, thanks. Fixing this is as simple as adding 'ctx->opts &&'\n> > right before we dereference 'ctx->opts', since setting this variable\n> > equal to 'ctx->commits.nr' is the right thing to do in that case.\n>\n> That would avoid the segfault, sure, but I would rather see all\n> callers of write_commit_graph{_reachable}() passing a valid opts\n> instance. Just like we don't call the diff machinery with a NULL\n> diff_options, or the revision walking machinery with a NULL rev_info.\n\nI wouldn't mind that either, but this is definitely a common pattern\nthroughout the commit-graph machinery. So, if/when we do get away from\nit, I'd rather do so uniformly than in some spots.\n\n> > Unrelated to this comment, I am hoping to send out a final version of\n> > this series sometime next week so that we can keep moving forward with\n> > Bloom filter improvements.\n> >\n> > Have you had a chance to review the rest of the patches? I'll happily\n> > wait until you have had a chance to do so before sending v5 so that we\n>\n> v5? This is v3, and I'm unable to a find a v4.\n\nSorry, I clearly had too much on my mind when I was writing this ;). I'm\nhopeful that with your careful review that v4 will be the last of this\ntopic.\n\nThanks,\nTaylor\n"},{"id":"404970","messageId":"20200903163513.GA6521@nand.local","threadId":"53975","inReplyTo":"20200818222329.GC29528@szeder.dev","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T16:35:44Z","receivedAt":"2020-09-03T16:35:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 19, 2020 at 12:23:29AM +0200, SZEDER Gábor wrote:\n> On Tue, Aug 11, 2020 at 04:52:14PM -0400, Taylor Blau wrote:\n> > Introduce a command-line flag and configuration variable to fill in the\n> > 'max_new_filters' variable introduced by the previous patch.\n>\n> 'max_new_filters' is introduced in this patch.\n>\n> > The command-line option '--max-new-filters' takes precedence over\n> > 'commitGraph.maxNewFilters', which is the default value.\n>\n> What \"filters\"? While misnamed, the '--changed-paths' options did a\n> good job at hiding the implementation detail that we use Bloom filters\n> to speed up pathspec-limited revision walks; as far as I remember this\n> was a conscious design decision. Including \"filter\" in the name of\n> the option and corresponding config variable goes against this\n> decision. Furthermore, by not being specific we might end up in abad\n> situation when adding some other filters to the commit-graph format.\n> Unfortunately, I can't offhand propose a better option name, all my\n> ideas were horrible.\n\nI don't disagree, but I can't come up with a better name either.\nmaxNewChangedPaths? maxNewSomething? I don't know. Usually I think\nexposing this sort of detail is smelly, but I'm not so bothered by it\nhere. Similarly, this thread has been around for a while, and nobody\nelse has suggested a better name, so, I'm inclined to keep it.\n\n> > +With the `--max-new-filters=` option, generate at most `n` new Bloom\n> > +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n> > +enforced. Overrides the `commitGraph.maxNewFilters` configuration.\n>\n> I think this description should also detail what happens with those\n> commits for which no modified path Bloom filters are calculated, and\n> which commands will calculate them (even implicitly).\n\nOK, sure.\n\n> > static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> > -\t\t\t\t\t const struct commit *c)\n> > +\t\t\t\t\t const struct commit *c,\n> > +\t\t\t\t\t uint32_t max_changed_paths)\n> > {\n> > \tuint32_t graph_pos = commit_graph_position(c);\n> > \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n> > @@ -965,6 +966,17 @@ static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n> >\n> > \tif (!(g && g->bloom_large))\n> > \t\treturn 0;\n> > +\tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n> > +\t\t/*\n> > +\t\t * Force all commits which are subject to a different\n> > +\t\t * 'max_changed_paths' limit to be recomputed from scratch.\n> > +\t\t *\n> > +\t\t * Note that this could likely be improved, but is ignored since\n> > +\t\t * all real-world graphs set the maximum number of changed paths\n> > +\t\t * at 512.\n>\n> I don't understand what the second part of this comment is trying to\n> say; and real-world graphs might very well contain Bloom filters with\n> more than 512 modified paths, because the applying that limit was\n> buggy.\n\nI'm trying to say that there is room for us to improve when we do and\ndon't recompute filters when the limit on the number of changed paths\ndeviates between layers, but that such deviations don't currently exist\nin the wild.\n\nBugs are another thing, but we can't tell that they exist without\nrecomputing every filter.\n\nThanks,\nTaylor\n"},{"id":"404971","messageId":"20200903164225.GA7905@nand.local","threadId":"53975","inReplyTo":"20200819082021.GA5238@szeder.dev","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T16:42:25Z","receivedAt":"2020-09-03T16:42:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Aug 19, 2020 at 10:20:21AM +0200, SZEDER Gábor wrote:\n> On Tue, Aug 11, 2020 at 04:52:14PM -0400, Taylor Blau wrote:\n> > Introduce a command-line flag and configuration variable to fill in the\n> > 'max_new_filters' variable introduced by the previous patch.\n> >\n> > The command-line option '--max-new-filters' takes precedence over\n> > 'commitGraph.maxNewFilters', which is the default value.\n> > '--no-max-new-filters' can also be provided, which sets the value back\n> > to '-1', indicating that an unlimited number of new Bloom filters may be\n> > generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n> > '0', hence a custom callback was used instead).\n>\n> Forgot the most important thing: Why? Please explain in the commit\n> message why this option is necesary, what problems does it solve,\n> how it is supposed to interact with other options and why so.\n\nThis is already explained in detail in the patch 'commit-graph: add\nlarge-filters bitmap chunk', although there is an error in the quoted\npart of your email (which I wrote) which refers the reader to the\nprevious patch. The patch I'm actually referring two is the\ntwice-previous patch.\n\nI'll fix that locally before re-sending.\n\nThanks,\nTaylor\n"},{"id":"404975","messageId":"20200903184920.GA8946@nand.local","threadId":"53975","inReplyTo":"20200901143650.GA22631@szeder.dev","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T18:49:20Z","receivedAt":"2020-09-03T18:49:28Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 01, 2020 at 04:36:50PM +0200, SZEDER Gábor wrote:\n> Something seems to be wrong in this patch, though I haven't looked\n> closer. Consider this test with a bit of makeshift tracing:\n>\n> --- >8 ---\n>\n> diff --git a/bloom.c b/bloom.c\n> index 8d07209c6b..1a0dec35cd 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -222,6 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n> \tif (!compute_if_not_present)\n> \t\treturn NULL;\n>\n> +\tprintf(\"get_or_compute_bloom_filter(): diff: %s\\n\", oid_to_hex(&c->object.oid));\n> \trepo_diff_setup(r, &diffopt);\n> \tdiffopt.flags.recursive = 1;\n> \tdiffopt.detect_rename = 0;\n> diff --git a/t/t9999-test.sh b/t/t9999-test.sh\n> new file mode 100755\n> index 0000000000..0833e6ff7e\n> --- /dev/null\n> +++ b/t/t9999-test.sh\n> @@ -0,0 +1,25 @@\n> +#!/bin/sh\n> +\n> +test_description='test'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'test' '\n> +\tfor i in 1 2 3 4 5 6\n> +\tdo\n> +\t\tgit commit -q --allow-empty -m $i || return 1\n> +\tdone &&\n> +\tgit log --oneline &&\n> +\n> +\t# We have 6 commits and compute 2 Bloom filters per execution,\n> +\t# so 3 executions should be enough... but, alas, it isnt.\n> +\tfor i in 1 2 3 4 5\n> +\tdo\n> +\t\t# No --split=replace!\n> +\t\tgit commit-graph write --reachable --changed-paths --max-new-filters=2 || return 1\n> +\tdone &&\n> +\n> +\tgit commit-graph write --reachable --changed-paths --max-new-filters=4\n> +'\n> +\n> +test_done\n>\n> --- >8 ---\n>\n> It's output looks like:\n>\n> [...]\n> + git log --oneline\n> 13fcefa (HEAD -> master) 6\n> 0c71516 5\n> a82a61c 4\n> 54c6b2c 3\n> fc99def 2\n> a3a8cd3 1\n> + git commit-graph write --reachable --changed-paths --max-new-filters=2\n> get_or_compute_bloom_filter(): diff: 0c71516945bf0a23813e80205961d29ebc1020e0\n> get_or_compute_bloom_filter(): diff: 13fcefa4bb859a15c4edc6bb01b8b6c91b4f32b6\n> + git commit-graph write --reachable --changed-paths --max-new-filters=2\n> get_or_compute_bloom_filter(): diff: 54c6b2cd4fb50066683a197cc6d677689618505a\n> get_or_compute_bloom_filter(): diff: a3a8cd3c82028671bf51502d77277baf14a2f528\n> + git commit-graph write --reachable --changed-paths --max-new-filters=2\n> get_or_compute_bloom_filter(): diff: 0c71516945bf0a23813e80205961d29ebc1020e0\n> get_or_compute_bloom_filter(): diff: 13fcefa4bb859a15c4edc6bb01b8b6c91b4f32b6\n> + git commit-graph write --reachable --changed-paths --max-new-filters=2\n> get_or_compute_bloom_filter(): diff: 54c6b2cd4fb50066683a197cc6d677689618505a\n> get_or_compute_bloom_filter(): diff: a3a8cd3c82028671bf51502d77277baf14a2f528\n> + git commit-graph write --reachable --changed-paths --max-new-filters=2\n> get_or_compute_bloom_filter(): diff: 0c71516945bf0a23813e80205961d29ebc1020e0\n> get_or_compute_bloom_filter(): diff: 13fcefa4bb859a15c4edc6bb01b8b6c91b4f32b6\n> + git commit-graph write --reachable --changed-paths\n> get_or_compute_bloom_filter(): diff: 54c6b2cd4fb50066683a197cc6d677689618505a\n> get_or_compute_bloom_filter(): diff: a3a8cd3c82028671bf51502d77277baf14a2f528\n> get_or_compute_bloom_filter(): diff: a82a61c79b2b07c4440e292613e11a69e33ef7a2\n> get_or_compute_bloom_filter(): diff: fc99def8b1df27bcab7d1f4b7ced73239f9bd7ec\n>\n> See how the third write with '--max-new-filters=2' computes the\n> filters that have already been computed by the first write instead of\n> those two that have never been computed? And then how the fourth\n> write computes filters that have already been computed by the second\n> write? And ultimately we'll need a write without '--max-new-filters' (or\n> with '--max-new-filters=') to compute all remaining\n> filters.\n\nOuch. This definitely has to do with the empty commits, since swapping\nout your 'git commit ... --allow-empty' for a 'test_commit' produces the\noutput that you'd expect.\n\n> With '--split=replace' it appears to work as expected.\n\nThis is definitely the critical bit. The crux of the issue is that\n'copy_oids_to_commits()' handles split and non-split graphs differently.\nThe critical bits here are:\n\n * 43d3561805 (commit-graph write: don't die if the existing graph is\n corrupt, 2019-03-25) which forces the relevant data to *not* be\n loaded from an existing commit-graph, and\n\n * 8a6ac287b2 (builtin/commit-graph.c: introduce split strategy\n 'replace', 2020-04-13), which does load data from an existing\n commit-graph with '--split=replace'.\n\nWhen writing a graph with '--split=replace', commits are loaded from the\ngraph, which includes setting their '->graph_pos' (or rather setting\nthis data in a commit slab, which is I guess how it's done these days).\nWithout '--split=replace', the graph position will never be set.\n\nSo, by the time we get to 'get_bloom_filter_large_in_graph', the graph\nposition is 'COMMIT_NOT_FROM_GRAPH', which in turn forces us to\nrecompute the filter from scratch, since we assume that being\n'NOT_FROM_GRAPH' implies that we won't find it in any 'BFXL' chunk.\n\nRegardless of whether or not we should be trusting the parentage\ninformation on-disk, recomputing the Bloom filters from scratch is\nsimply too expensive (and the opposite of the point of this series). So,\ndoing the following to force 'get_bloom_filter_large_in_graph' to lookup\nthe Bloom and BFXL data in a commit graph by forcibly loading its graph\nposition is the right thing to do.\n\nThis is sufficient to get us unstuck:\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex bec4e5b725..243c7253ff 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -956,8 +956,8 @@ int get_bloom_filter_large_in_graph(struct commit_graph *g,\n const struct commit *c,\n uint32_t max_changed_paths)\n {\n- uint32_t graph_pos = commit_graph_position(c);\n- if (graph_pos == COMMIT_NOT_FROM_GRAPH)\n+ uint32_t graph_pos;\n+ if (!find_commit_in_graph(c, g, &graph_pos))\n return 0;\n\n while (g && graph_pos < g->num_commits_in_base)\n\n...but adding something like:\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex a7a3b41919..571676cef2 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -310,4 +310,29 @@ test_expect_success 'Bloom generation backfills previously-skipped filters' '\n )\n '\n\n+test_expect_success 'Bloom generation backfills empty commits' '\n+ git init empty &&\n+ test_when_finished \"rm -fr empty\" &&\n+ (\n+ cd empty &&\n+ for i in $(test_seq 1 6)\n+ do\n+ git commit --allow-empty -m \"$i\"\n+ done &&\n+\n+ # Generate Bloom filters for empty commits 1-6, two at a time.\n+ test_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+ 0 2 2 &&\n+ test_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+ 2 2 2 &&\n+ test_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+ 4 2 2 &&\n+\n+ # Finally, make sure that once all commits have filters, that\n+ # none are subsequently recomputed.\n+ test_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+ 6 0 0\n+ )\n+'\n+\n test_done\n\nis a good idea to harden what you wrote in your t9999 into an actual\ntest to prevent against regression. I'll fold both of those into this\npatch.\n\nThanks for the bug report. It led to an interesting investigation as a\nresult :).\n\nThanks,\nTaylor\n"},{"id":"404993","messageId":"xmqq8sdq8rg1.fsf@gitster.c.googlers.com","threadId":"53975","inReplyTo":"cover.1597178914.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/14] more miscellaneous Bloom filter improvements","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-03T21:45:50Z","receivedAt":"2020-09-03T21:45:59Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> Here's a(nother) re-roll of mine and Stolee's series to introduce the\n> new BFXL commit-graph chunk, along with the '--max-new-filters' option\n> to 'git commit-graph write'.\n>\n> Not really much has changed since v2, other than a rebase onto the\n> latest from master (the fifth 2.29 batch, at the time of writing), and\n> to squash in a few fixups that I sent in response to my v2 series.\n\nIt seems we've seen more than enough comments and enthusiasm on this\nround to make another round of update worthwhile, but it seems it\nmay take a bit more time (e.g. <20200903184920.GA8946@nand.local>)\nto get issues resolved? Just pinging, no rush.\n\n\n \n\n \n"},{"id":"404999","messageId":"20200903223325.GA25635@nand.local","threadId":"53975","inReplyTo":"xmqq8sdq8rg1.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v3 00/14] more miscellaneous Bloom filter improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:33:25Z","receivedAt":"2020-09-03T22:33:33Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 03, 2020 at 02:45:50PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n> > Not really much has changed since v2, other than a rebase onto the\n> > latest from master (the fifth 2.29 batch, at the time of writing), and\n> > to squash in a few fixups that I sent in response to my v2 series.\n>\n> It seems we've seen more than enough comments and enthusiasm on this\n> round to make another round of update worthwhile, but it seems it\n> may take a bit more time (e.g. <20200903184920.GA8946@nand.local>)\n> to get issues resolved? Just pinging, no rush.\n\nYeah. I am definitely partially to blame, since I have been distracted\nfor the past ~month or so working on some bitmaps-related topics that I\nam hoping to send soon.\n\nThe issues in <20200903184920.GA8946@nand.local> are already resolved,\nand I have v4 ready to send. But, I need to give it a final proofreading\npass before letting it hit the list.\n\nPatches shortly.\n\n\nThanks,\nTaylor\n"},{"id":"405001","messageId":"cover.1599172907.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1596480582.git.me@ttaylorr.com","subject":"[PATCH v4 00/14] more miscellaneous Bloom filter improvements","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:45:25Z","receivedAt":"2020-09-03T22:45:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi,\n\nHere is another reroll of my series to introduce a '--max-new-filters'\noption to the 'git commit-graph write' sub-command to limit the number\nof changed-path Bloom filters a single process is willing to compute\nfrom scratch.\n\nAs a reminder (since it's been a while since v3), this is done by adding\na new 'BFXL' chunk, which specifies which Bloom filters (a) have been\ncomputed and (b) were too large to store, and thus are encoded as\nzero-length filters.\n\nThings seem to have settled down since the review in v3, so I'm hoping\nthat this is what will end up being queued.\n\nThanks again for all of your review, and sorry for the prolonged radio\nsilence on this topic. I've had my hands full working on something else\nthat I'm hoping to start testing and sending to the list soon.\n\nDerrick Stolee (1):\n bloom/diff: properly short-circuit on max_changes\n\nTaylor Blau (13):\n commit-graph: introduce 'get_bloom_filter_settings()'\n t4216: use an '&&'-chain\n commit-graph: pass a 'struct repository *' in more places\n t/helper/test-read-graph.c: prepare repo settings\n commit-graph: respect 'commitGraph.readChangedPaths'\n commit-graph.c: store maximum changed paths\n bloom: split 'get_bloom_filter()' in two\n bloom: use provided 'struct bloom_filter_settings'\n commit-graph.c: sort index into commits list\n csum-file.h: introduce 'hashwrite_be64()'\n commit-graph: add large-filters bitmap chunk\n commit-graph: rename 'split_commit_graph_opts'\n builtin/commit-graph.c: introduce '--max-new-filters='\n\n Documentation/config.txt | 2 +\n Documentation/config/commitgraph.txt | 8 +\n Documentation/git-commit-graph.txt | 6 +\n .../technical/commit-graph-format.txt | 13 +\n blame.c | 8 +-\n bloom.c | 43 ++-\n bloom.h | 22 +-\n builtin/commit-graph.c | 61 +++-\n commit-graph.c | 278 +++++++++++++-----\n commit-graph.h | 28 +-\n csum-file.h | 6 +\n diff.h | 2 -\n fuzz-commit-graph.c | 5 +-\n line-log.c | 2 +-\n make: *** [Makefile | 0\n midx.c | 3 +-\n repo-settings.c | 3 +\n repository.h | 1 +\n revision.c | 7 +-\n t/helper/test-bloom.c | 4 +-\n t/helper/test-read-graph.c | 3 +-\n t/t4216-log-bloom.sh | 173 +++++++++--\n t/t5324-split-commit-graph.sh | 13 +\n tree-diff.c | 5 +-\n 24 files changed, 553 insertions(+), 143 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n create mode 100644 make: *** [Makefile\n\nRange-diff against v3:\n[rebased onto master]\n 1: e714e54240 = 232: 97d80f109f commit-graph: introduce 'get_bloom_filter_settings()'\n 2: 9fc8b17d6f = 233: d60698d2f8 t4216: use an '&&'-chain\n 3: 8dbe4838b7 = 234: 639a962a49 commit-graph: pass a 'struct repository *' in more places\n 4: f59db1e30d = 235: ccab59dfe4 t/helper/test-read-graph.c: prepare repo settings\n 5: daae6788c0 = 236: 8aff54d83e commit-graph: respect 'commitGraph.readChangedPaths'\n 6: bf498844ef = 237: 965489d361 commit-graph.c: store maximum changed paths\n 7: eba2794873 = 238: ba89a0cb83 bloom: split 'get_bloom_filter()' in two\n 8: 4f08177dbe = 239: 89bedba089 bloom: use provided 'struct bloom_filter_settings'\n 9: cc1dc8b121 = 240: 427f129656 bloom/diff: properly short-circuit on max_changes\n 10: 23fd52c3b8 = 241: 08b5f185f6 commit-graph.c: sort index into commits list\n 11: 4800cd373e = 242: d7cbd4ca1a csum-file.h: introduce 'hashwrite_be64()'\n 12: 619e0c619d ! 243: 3063beb588 commit-graph: add large-filters bitmap chunk\n @@ Commit message\n\n When a commit has more than a certain number of changed paths (commonly\n 512), the commit-graph machinery represents it as a zero-length filter.\n - This is done since having many entries in the Bloom filter has\n - undesirable effects on the false positivity rate.\n -\n In addition to these too-large filters, the commit-graph machinery also\n represents commits with no filter and commits with no changed paths in\n the same way.\n @@ Commit message\n data in network byte order from the 64-bit words. This means we also\n need to read the array from the commit-graph file by translating each\n word from network byte order using get_be64() when loading the commit\n - graph. (Note that this *could* be delayed until first-use, but a later\n - patch will rely on this being initialized early, so we assume the\n - up-front cost when parsing instead of delaying initialization).\n + graph. Initialize this bitmap lazily to avoid paying a linear-time cost\n + upon each commit-graph load even if we do not need the bitmaps\n + themselves.\n\n By avoiding the need to move to new versions of the BDAT and BIDX chunk,\n we can give ourselves more time to consider whether or not other\n @@ Commit message\n except the most-significant bit of each offset is interpreted as \"this\n filter is too big\" iff looking at a BID2 chunk. This avoids having to\n write a bitmap, but forces older clients to rewrite their commit-graphs\n - (as well as reduces the theoretical largest Bloom filters we couldl\n + (as well as reduces the theoretical largest Bloom filters we could\n write, and forces us to maintain the code necessary to translate BIDX\n chunks to BID2 ones). Separately from this patch, I implemented this\n alternate approach and did not find it to be advantageous.\n @@ Documentation/technical/commit-graph-format.txt: CHUNK DATA:\n + Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n + * It starts with a 32-bit unsigned integer specifying the maximum number of\n + changed-paths that can be stored in a single Bloom filter.\n -+ * It then contains a list of 64-bit words (the length of this list is\n -+ determined by the width of the chunk) which is a bitmap. The 'i'th bit is\n -+ set exactly when the 'i'th commit in the graph has a changed-path Bloom\n -+ filter with zero entries (either because the commit is empty, or because\n -+ it contains more than 512 changed paths).\n ++ * It then contains a list of 64-bit words in network order (the length of\n ++ this list is determined by the width of the chunk) which is a bitmap. The\n ++ 'i'th bit is set exactly when the 'i'th commit in the graph has a\n ++ changed-path Bloom filter with zero entries (either because the commit is\n ++ empty, or because it contains more entries than is allowed per filter by\n ++ the layer that contains it).\n + * The BFXL chunk is present only when the BIDX and BDAT chunks are\n + also present.\n +\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n +\t\t\tif (graph->chunk_bloom_large_filters)\n +\t\t\t\tchunk_repeated = 1;\n +\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n -+\t\t\t\tsize_t alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n +\t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n ++\t\t\t\tgraph->bloom_large_alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n +\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n -+\t\t\t\tif (alloc) {\n -+\t\t\t\t\tsize_t j;\n -+\t\t\t\t\tgraph->bloom_large = bitmap_word_alloc(alloc);\n -+\n -+\t\t\t\t\tfor (j = 0; j < graph->bloom_large->word_alloc; j++)\n -+\t\t\t\t\t\tgraph->bloom_large->words[j] = get_be64(\n -+\t\t\t\t\t\t\tgraph->chunk_bloom_large_filters + j * sizeof(eword_t));\n -+\t\t\t\t}\n +\t\t\t}\n +\t\t\tbreak;\n \t\t}\n @@ commit-graph.c: struct commit_graph *parse_commit_graph(struct repository *r,\n \t\tgraph->chunk_bloom_indexes = NULL;\n \t\tgraph->chunk_bloom_data = NULL;\n +\t\tgraph->chunk_bloom_large_filters = NULL;\n ++\t\tgraph->bloom_large_alloc = 0;\n \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n -+\t\tbitmap_free(graph->bloom_large);\n \t}\n\n - \thashcpy(graph->oid.hash, graph->data + graph->data_len - graph->hash_len);\n @@ commit-graph.c: struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n\n -+static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n -+\t\t\t\t\t const struct commit *c)\n ++int get_bloom_filter_large_in_graph(struct commit_graph *g,\n ++\t\t\t\t const struct commit *c)\n +{\n -+\tuint32_t graph_pos = commit_graph_position(c);\n -+\tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n ++\tuint32_t graph_pos;\n ++\tif (!find_commit_in_graph(c, g, &graph_pos))\n +\t\treturn 0;\n +\n +\twhile (g && graph_pos < g->num_commits_in_base)\n +\t\tg = g->base_graph;\n +\n -+\tif (!(g && g->bloom_large))\n ++\tif (!g)\n ++\t\treturn 0;\n ++\n ++\tif (!g->bloom_large && g->bloom_large_alloc) {\n ++\t\tsize_t i;\n ++\t\tg->bloom_large = bitmap_word_alloc(g->bloom_large_alloc);\n ++\n ++\t\tfor (i = 0; i < g->bloom_large->word_alloc; i++)\n ++\t\t\tg->bloom_large->words[i] = get_be64(\n ++\t\t\t\tg->chunk_bloom_large_filters + i * sizeof(eword_t));\n ++\t}\n ++\n ++\tif (!g->bloom_large)\n +\t\treturn 0;\n +\treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n +}\n @@ commit-graph.h\n\n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n #define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n +@@ commit-graph.h: void load_commit_graph_info(struct repository *r, struct commit *item);\n + struct tree *get_commit_tree_in_graph(struct repository *r,\n + \t\t\t\t const struct commit *c);\n +\n ++int get_bloom_filter_large_in_graph(struct commit_graph *g,\n ++\t\t\t\t const struct commit *c);\n ++\n + struct commit_graph {\n + \tconst unsigned char *data;\n + \tsize_t data_len;\n @@ commit-graph.h: struct commit_graph {\n \tconst unsigned char *chunk_base_graphs;\n \tconst unsigned char *chunk_bloom_indexes;\n @@ commit-graph.h: struct commit_graph {\n +\tconst unsigned char *chunk_bloom_large_filters;\n +\n +\tstruct bitmap *bloom_large;\n ++\tsize_t bloom_large_alloc;\n\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n +@@ commit-graph.h: struct commit_graph *read_commit_graph_one(struct repository *r,\n + struct commit_graph *parse_commit_graph(struct repository *r,\n + \t\t\t\t\tvoid *graph_map, size_t graph_size);\n +\n ++void prepare_commit_graph_bloom_large(struct commit_graph *g);\n ++\n + /*\n + * Return 1 if and only if the repository has a commit-graph\n + * file and generation numbers are computed in that file.\n\n ## t/t4216-log-bloom.sh ##\n @@ t/t4216-log-bloom.sh: test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n - \tgit commit-graph write --reachable --changed-paths\n + \tEOF\n '\n graph_read_expect () {\n -\tNUM_CHUNKS=5\n +\tNUM_CHUNKS=6\n \tcat >expect <<- EOF\n - \theader: 43475048 1 1 $NUM_CHUNKS 0\n + \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n @@ t/t4216-log-bloom.sh: test_expect_success 'correctly report changes over limit' '\n \t\tdone\n 13: b2e33ecba8 ! 244: ee0bc109f3 commit-graph: rename 'split_commit_graph_opts'\n @@ Commit message\n commit-graph API which have nothing to do with splitting.\n\n Rename the 'split_commit_graph_opts' structure to the more-generic\n - 'commit_graph_opts' to encompass both.\n + 'commit_graph_opts' to encompass both. Likewise, rename the 'flags'\n + member to instead be 'split_flags' to clarify that it only has to do\n + with the behavior implied by '--split'.\n\n - Suggsted-by: Derrick Stolee \n + Suggested-by: Derrick Stolee \n Signed-off-by: Taylor Blau \n\n ## builtin/commit-graph.c ##\n @@ builtin/commit-graph.c: static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"enable computation for changed paths\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n -\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n -+\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.flags, NULL,\n ++\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.split_flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\twrite_option_parse_split),\n @@ commit-graph.c: static void close_reachable(struct write_commit_graph_context *c\n -\tenum commit_graph_split_flags flags = ctx->split_opts ?\n -\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n +\tenum commit_graph_split_flags flags = ctx->opts ?\n -+\t\tctx->opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n ++\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n\n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n @@ commit-graph.c: static uint32_t count_distinct_commits(struct write_commit_graph\n -\tenum commit_graph_split_flags flags = ctx->split_opts ?\n -\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n +\tenum commit_graph_split_flags flags = ctx->opts ?\n -+\t\tctx->opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n ++\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n\n \tctx->num_extra_edges = 0;\n \tif (ctx->report_progress)\n @@ commit-graph.c: static void split_graph_merge_strategy(struct write_commit_graph\n +\t\t\tsize_mult = ctx->opts->size_multiple;\n\n -\t\tflags = ctx->split_opts->flags;\n -+\t\tflags = ctx->opts->flags;\n ++\t\tflags = ctx->opts->split_flags;\n \t}\n\n \tg = ctx->r->objects->commit_graph;\n @@ commit-graph.c: int write_commit_graph(struct object_directory *odb,\n -\t\tif (ctx->split_opts)\n -\t\t\treplace = ctx->split_opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n +\t\tif (ctx->opts)\n -+\t\t\treplace = ctx->opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n ++\t\t\treplace = ctx->opts->split_flags & COMMIT_GRAPH_SPLIT_REPLACE;\n \t}\n\n \tctx->approx_nr_objects = approximate_object_count();\n @@ commit-graph.h: enum commit_graph_split_flags {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n +-\tenum commit_graph_split_flags flags;\n ++\tenum commit_graph_split_flags split_flags;\n + };\n +\n + /*\n @@ commit-graph.h: struct split_commit_graph_opts {\n */\n int write_commit_graph_reachable(struct object_directory *odb,\n @@ commit-graph.h: struct split_commit_graph_opts {\n\n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n\n +\n + ## make: *** [Makefile (new) ##\n 14: 09f6871f66 ! 245: cd0a9da639 builtin/commit-graph.c: introduce '--max-new-filters='\n @@ Commit message\n builtin/commit-graph.c: introduce '--max-new-filters='\n\n Introduce a command-line flag and configuration variable to fill in the\n - 'max_new_filters' variable introduced by the previous patch.\n + 'max_new_filters' variable introduced two patches ago.\n\n The command-line option '--max-new-filters' takes precedence over\n 'commitGraph.maxNewFilters', which is the default value.\n @@ Documentation/git-commit-graph.txt: this option is given, future commit-graph wr\n +\n +With the `--max-new-filters=` option, generate at most `n` new Bloom\n +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n -+enforced. Overrides the `commitGraph.maxNewFilters` configuration.\n ++enforced. Commits whose filters are not calculated are stored as a\n ++length zero Bloom filter, and their bit is marked in the `BFXL` chunk.\n ++Overrides the `commitGraph.maxNewFilters` configuration.\n ++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\n\n ## bloom.c ##\n -@@ bloom.c: static int load_bloom_filter_from_graph(struct commit_graph *g,\n - \telse\n - \t\tstart_index = 0;\n +@@ bloom.c: struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n\n -+\tif ((start_index == end_index) &&\n -+\t (g->bloom_large && !bitmap_get(g->bloom_large, lex_pos))) {\n -+\t\t/*\n -+\t\t * If the filter is zero-length, either (1) the filter has no\n -+\t\t * changes, (2) the filter has too many changes, or (3) it\n -+\t\t * wasn't computed (eg., due to '--max-new-filters').\n -+\t\t *\n -+\t\t * If either (1) or (2) is the case, the 'large' bit will be set\n -+\t\t * for this Bloom filter. If it is unset, then it wasn't\n -+\t\t * computed. In that case, return nothing, since we don't have\n -+\t\t * that filter in the graph.\n -+\t\t */\n -+\t\treturn 0;\n -+\t}\n + \tif (!filter->data) {\n + \t\tload_commit_graph_info(r, c);\n +-\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n +-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n +-\t\t\t\treturn filter;\n ++\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n ++\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n + \t}\n +\n +-\tif (filter->data)\n ++\tif (filter->data && filter->len)\n + \t\treturn filter;\n + \tif (!compute_if_not_present)\n + \t\treturn NULL;\n +\n ++\tif (filter && !filter->len &&\n ++\t get_bloom_filter_large_in_graph(r->objects->commit_graph, c,\n ++\t\t\t\t\t settings->max_changed_paths))\n ++\t\treturn filter;\n ++\n +\n - \tfilter->len = end_index - start_index;\n - \tfilter->data = (unsigned char *)(g->chunk_bloom_data +\n - \t\t\t\t\tsizeof(unsigned char) * start_index +\n + \trepo_diff_setup(r, &diffopt);\n + \tdiffopt.flags.recursive = 1;\n + \tdiffopt.detect_rename = 0;\n\n ## builtin/commit-graph.c ##\n @@ builtin/commit-graph.c: static char const * const builtin_commit_graph_usage[] = {\n @@ commit-graph.c\n @@ commit-graph.c: struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n }\n\n - static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n --\t\t\t\t\t const struct commit *c)\n -+\t\t\t\t\t const struct commit *c,\n -+\t\t\t\t\t uint32_t max_changed_paths)\n + int get_bloom_filter_large_in_graph(struct commit_graph *g,\n +-\t\t\t\t const struct commit *c)\n ++\t\t\t\t const struct commit *c,\n ++\t\t\t\t uint32_t max_changed_paths)\n {\n - \tuint32_t graph_pos = commit_graph_position(c);\n - \tif (graph_pos == COMMIT_NOT_FROM_GRAPH)\n -@@ commit-graph.c: static int get_bloom_filter_large_in_graph(struct commit_graph *g,\n + \tuint32_t graph_pos;\n + \tif (!find_commit_in_graph(c, g, &graph_pos))\n +@@ commit-graph.c: int get_bloom_filter_large_in_graph(struct commit_graph *g,\n\n - \tif (!(g && g->bloom_large))\n + \tif (!g->bloom_large)\n \t\treturn 0;\n +\tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n +\t\t/*\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_cont\n \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n \t\t&ctx->commits);\n\n -+\tmax_new_filters = ctx->opts->max_new_filters >= 0 ?\n ++\tmax_new_filters = ctx->opts && ctx->opts->max_new_filters >= 0 ?\n +\t\tctx->opts->max_new_filters : ctx->commits.nr;\n +\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_cont\n \t}\n\n ## commit-graph.h ##\n +@@ commit-graph.h: struct tree *get_commit_tree_in_graph(struct repository *r,\n + \t\t\t\t const struct commit *c);\n +\n + int get_bloom_filter_large_in_graph(struct commit_graph *g,\n +-\t\t\t\t const struct commit *c);\n ++\t\t\t\t const struct commit *c,\n ++\t\t\t\t uint32_t max_changed_paths);\n +\n + struct commit_graph {\n + \tconst unsigned char *data;\n @@ commit-graph.h: struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n - \tenum commit_graph_split_flags flags;\n + \tenum commit_graph_split_flags split_flags;\n +\tint max_new_filters;\n };\n\n @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation does not recompute t\n +\t\t\t2 0 1\n +\t)\n +'\n ++\n ++test_expect_success 'Bloom generation backfills empty commits' '\n ++\tgit init empty &&\n ++\ttest_when_finished \"rm -fr empty\" &&\n ++\t(\n ++\t\tcd empty &&\n ++\t\tfor i in $(test_seq 1 6)\n ++\t\tdo\n ++\t\t\tgit commit --allow-empty -m \"$i\"\n ++\t\tdone &&\n ++\n ++\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n ++\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n ++\t\t\t0 2 2 &&\n ++\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n ++\t\t\t2 2 2 &&\n ++\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n ++\t\t\t4 2 2 &&\n ++\n ++\t\t# Finally, make sure that once all commits have filters, that\n ++\t\t# none are subsequently recomputed.\n ++\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n ++\t\t\t6 0 0\n ++\t)\n ++'\n +\n test_done\n--\n2.27.0.2918.gc99a27ff8f\n"},{"id":"405002","messageId":"97d80f109f5c470a8ca3d17ef9d451787b0b609e.1599172907.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 01/14] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:00Z","receivedAt":"2020-09-03T22:46:07Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Many places in the code often need a pointer to the commit-graph's\n'struct bloom_filter_settings', in which case they often take the value\nfrom the top-most commit-graph.\n\nIn the non-split case, this works as expected. In the split case,\nhowever, things get a little tricky. Not all layers in a chain of\nincremental commit-graphs are required to themselves have Bloom data,\nand so whether or not some part of the code uses Bloom filters depends\nentirely on whether or not the top-most level of the commit-graph chain\nhas Bloom filters.\n\nThis has been the behavior since Bloom filters were introduced, and has\nbeen codified into the tests since a759bfa9ee (t4216: add end to end\ntests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\nrequires that Bloom filters are not used in exactly the case described\nearlier.\n\nThere is no reason that this needs to be the case, since it is perfectly\nvalid for commits in an earlier layer to have Bloom filters when commits\nin a newer layer do not.\n\nSince Bloom settings are guaranteed to be the same for any layer in a\nchain that has Bloom data, it is sufficient to traverse the\n'->base_graph' pointer until either (1) a non-null 'struct\nbloom_filter_settings *' is found, or (2) until we are at the root of\nthe commit-graph chain.\n\nIntroduce a 'get_bloom_filter_settings()' function that does just this,\nand use it instead of purely dereferencing the top-most graph's\n'->bloom_filter_settings' pointer.\n\nWhile we're at it, add an additional test in t5324 to guard against code\nin the commit-graph writing machinery that doesn't correctly handle a\nNULL 'struct bloom_filter *'.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n blame.c | 6 ++++--\n bloom.c | 6 +++---\n commit-graph.c | 11 +++++++++++\n commit-graph.h | 2 ++\n revision.c | 5 +----\n t/t4216-log-bloom.sh | 9 ++++++---\n t/t5324-split-commit-graph.sh | 13 +++++++++++++\n 7 files changed, 40 insertions(+), 12 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 1be1cd82a2..903e23af23 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -2892,16 +2892,18 @@ void setup_blame_bloom_data(struct blame_scoreboard *sb,\n \t\t\t const char *path)\n {\n \tstruct blame_bloom_data *bd;\n+\tstruct bloom_filter_settings *bs;\n \n \tif (!sb->repo->objects->commit_graph)\n \t\treturn;\n \n-\tif (!sb->repo->objects->commit_graph->bloom_filter_settings)\n+\tbs = get_bloom_filter_settings(sb->repo);\n+\tif (!bs)\n \t\treturn;\n \n \tbd = xmalloc(sizeof(struct blame_bloom_data));\n \n-\tbd->settings = sb->repo->objects->commit_graph->bloom_filter_settings;\n+\tbd->settings = bs;\n \n \tbd->alloc = 4;\n \tbd->nr = 0;\ndiff --git a/bloom.c b/bloom.c\nindex 1a573226e7..cd9380ac62 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -38,7 +38,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \twhile (graph_pos < g->num_commits_in_base)\n \t\tg = g->base_graph;\n \n-\t/* The commit graph commit 'c' lives in doesn't carry bloom filters. */\n+\t/* The commit graph commit 'c' lives in doesn't carry Bloom filters. */\n \tif (!g->chunk_bloom_indexes)\n \t\treturn 0;\n \n@@ -195,8 +195,8 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes)\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n+\t\t\t\treturn filter;\n \t}\n \n \tif (filter->data)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0ed003e218..6a36ed0b06 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -667,6 +667,17 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r)\n+{\n+\tstruct commit_graph *g = r->objects->commit_graph;\n+\twhile (g) {\n+\t\tif (g->bloom_filter_settings)\n+\t\t\treturn g->bloom_filter_settings;\n+\t\tg = g->base_graph;\n+\t}\n+\treturn NULL;\n+}\n+\n static void close_commit_graph_one(struct commit_graph *g)\n {\n \tif (!g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 09a97030dc..0677dd1031 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -87,6 +87,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n */\n int generation_numbers_enabled(struct repository *r);\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r);\n+\n enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_APPEND = (1 << 0),\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\ndiff --git a/revision.c b/revision.c\nindex 08c2ad23af..857274408c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -680,10 +680,7 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \trepo_parse_commit(revs->repo, revs->commits->item);\n \n-\tif (!revs->repo->objects->commit_graph)\n-\t\treturn;\n-\n-\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n+\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n \tif (!revs->bloom_filter_settings)\n \t\treturn;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 4bb9e9dbe2..715912ad0f 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -65,7 +65,7 @@ setup () {\n \n test_bloom_filters_used () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":${2:-0},\\\"maybe\\\"\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom &&\n@@ -139,8 +139,11 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n-test_expect_success 'Do not use Bloom filters if the latest graph does not have Bloom filters.' '\n-\ttest_bloom_filters_not_used \"-- A/B\"\n+test_expect_success 'use Bloom filters even if the latest graph does not have Bloom filters' '\n+\t# Ensure that the number of empty filters is equal to the number of\n+\t# filters in the latest graph layer to prove that they are loaded (and\n+\t# ignored).\n+\ttest_bloom_filters_used \"-- A/B\" 3\n '\n \n test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 18216463c7..c334ee9155 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -427,4 +427,17 @@ done <<\\EOF\n 0600 -r--------\n EOF\n \n+test_expect_success '--split=replace with partial Bloom data' '\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/3 &&\n+\tgit rev-list -1 HEAD~2 >a &&\n+\tgit rev-list -1 HEAD~1 >b &&\n+\tgit commit-graph write --split=no-merge --stdin-commits --changed-paths graph-files &&\n+\ttest_line_count = 1 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n test_done\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405003","messageId":"d60698d2f8f2888b15729f1ec1fea4a1128f90c4.1599172907.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 02/14] t4216: use an '&&'-chain","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:05Z","receivedAt":"2020-09-03T22:46:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a759bfa9ee (t4216: add end to end tests for git log with Bloom\nfilters, 2020-04-06), a 'rm' invocation was added without a\ncorresponding '&&' chain.\n\nWhen 'trace.perf' already exists, everything works fine. However, the\nfunction can be executed without 'trace.perf' on disk (eg., when the\nsubset of tests run is altered with '--run'), and so the bare 'rm'\ncomplains about a missing file.\n\nTo remove some noise from the test log, invoke 'rm' with '-f', at which\npoint it is sensible to place the 'rm -f' in an '&&'-chain, which is\nboth (1) our usual style, and (2) avoids a broken chain in the future if\nmore commands are added at the beginning of the function.\n\nHelped-by: Eric Sunshine \nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 715912ad0f..cd89c75002 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -58,7 +58,7 @@ sane_unset GIT_TRACE2_PERF_BRIEF\n sane_unset GIT_TRACE2_CONFIG_PARAMS\n \n setup () {\n-\trm \"$TRASH_DIRECTORY/trace.perf\"\n+\trm -f \"$TRASH_DIRECTORY/trace.perf\" &&\n \tgit -c core.commitGraph=false log --pretty=\"format:%s\" $1 >log_wo_bloom &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.perf\" git -c core.commitGraph=true log --pretty=\"format:%s\" $1 >log_w_bloom\n }\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405004","messageId":"ccab59dfe4b2c1102c9f48f8205b8c01329ce1ac.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 04/14] t/helper/test-read-graph.c: prepare repo settings","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:14Z","receivedAt":"2020-09-03T22:46:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The read-graph test-tool is used by a number of the commit-graph test to\nassert various properties about a commit-graph. Previously, this program\nnever ran 'prepare_repo_settings()'. There was no need to do so, since\nnone of the commit-graph machinery is affected by the repo settings.\n\nIn the next patch, the commit-graph machinery's behavior will become\ndependent on the repo settings, and so loading them before running the\nrest of the test tool is critical.\n\nAs such, teach the test tool to call 'prepare_repo_settings()'.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 6d0c962438..5f585a1725 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -12,11 +12,12 @@ int cmd__read_graph(int argc, const char **argv)\n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n \n+\tprepare_repo_settings(the_repository);\n+\n \tgraph = read_commit_graph_one(the_repository, odb);\n \tif (!graph)\n \t\treturn 1;\n \n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405005","messageId":"639a962a4920d35002aa5111d1d983b96277e882.1599172907.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 03/14] commit-graph: pass a 'struct repository *' in more places","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:10Z","receivedAt":"2020-09-03T22:46:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a future commit, some commit-graph internals will want access to\n'r->settings', but we only have the 'struct object_directory *'\ncorresponding to that repository.\n\nAdd an additional parameter to pass the repository around in more\nplaces.\n\nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 2 +-\n commit-graph.c | 17 ++++++++++-------\n commit-graph.h | 6 ++++--\n fuzz-commit-graph.c | 5 +++--\n 4 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 523501f217..ba5584463f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -106,7 +106,7 @@ static int graph_verify(int argc, const char **argv)\n \tFREE_AND_NULL(graph_name);\n \n \tif (open_ok)\n-\t\tgraph = load_commit_graph_one_fd_st(fd, &st, odb);\n+\t\tgraph = load_commit_graph_one_fd_st(the_repository, fd, &st, odb);\n \telse\n \t\tgraph = read_commit_graph_one(the_repository, odb);\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6a36ed0b06..72a838bd00 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -231,7 +231,8 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st)\n \treturn 1;\n }\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tvoid *graph_map;\n@@ -247,7 +248,7 @@ struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n \t}\n \tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n-\tret = parse_commit_graph(graph_map, graph_size);\n+\tret = parse_commit_graph(r, graph_map, graph_size);\n \n \tif (ret)\n \t\tret->odb = odb;\n@@ -287,7 +288,8 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n \tconst unsigned char *data, *chunk_lookup;\n \tuint32_t i;\n@@ -452,7 +454,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n \treturn NULL;\n }\n \n-static struct commit_graph *load_commit_graph_one(const char *graph_file,\n+static struct commit_graph *load_commit_graph_one(struct repository *r,\n+\t\t\t\t\t\t const char *graph_file,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \n@@ -464,7 +467,7 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file,\n \tif (!open_ok)\n \t\treturn NULL;\n \n-\tg = load_commit_graph_one_fd_st(fd, &st, odb);\n+\tg = load_commit_graph_one_fd_st(r, fd, &st, odb);\n \n \tif (g)\n \t\tg->filename = xstrdup(graph_file);\n@@ -476,7 +479,7 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tchar *graph_name = get_commit_graph_filename(odb);\n-\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \tfree(graph_name);\n \n \treturn g;\n@@ -557,7 +560,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \t\tvalid = 0;\n \t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\t\tchar *graph_name = get_split_graph_filename(odb, line.buf);\n-\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \n \t\t\tfree(graph_name);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 0677dd1031..d9acb22bac 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,13 @@ struct commit_graph {\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb);\n struct commit_graph *read_commit_graph_one(struct repository *r,\n \t\t\t\t\t struct object_directory *odb);\n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n /*\n * Return 1 if and only if the repository has a commit-graph\ndiff --git a/fuzz-commit-graph.c b/fuzz-commit-graph.c\nindex 430817214d..e7cf6d5b0f 100644\n--- a/fuzz-commit-graph.c\n+++ b/fuzz-commit-graph.c\n@@ -1,7 +1,8 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n \n@@ -10,7 +11,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \tstruct commit_graph *g;\n \n \tinitialize_the_repository();\n-\tg = parse_commit_graph((void *)data, size);\n+\tg = parse_commit_graph(the_repository, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\n \n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405006","messageId":"8aff54d83e26953476da7ca9a1676bd945c00aa2.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 05/14] commit-graph: respect 'commitGraph.readChangedPaths'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:19Z","receivedAt":"2020-09-03T22:46:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Git uses the 'core.commitGraph' configuration value to control whether\nor not the commit graph is used when parsing commits or performing a\ntraversal.\n\nNow that commit-graphs can also contain a section for changed-path Bloom\nfilters, administrators that already have commit-graphs may find it\nconvenient to use those graphs without relying on their changed-path\nBloom filters. This can happen, for example, during a staged roll-out,\nor in the event of an incident.\n\nIntroduce 'commitGraph.readChangedPaths' to control whether or not Bloom\nfilters are read. Note that this configuration is independent from both:\n\n - 'core.commitGraph', to allow flexibility in using all parts of a\n commit-graph _except_ for its Bloom filters.\n\n - The '--changed-paths' option for 'git commit-graph write', to allow\n reading and writing Bloom filters to be controlled independently.\n\nWhen the variable is set, pretend as if no Bloom data was specified at\nall. This avoids adding additional special-casing outside of the\ncommit-graph internals.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 ++\n Documentation/config/commitgraph.txt | 4 ++++\n commit-graph.c | 6 ++++--\n repo-settings.c | 3 +++\n repository.h | 1 +\n t/t4216-log-bloom.sh | 4 +++-\n 6 files changed, 17 insertions(+), 3 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 3042d80978..770ae79b82 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -340,6 +340,8 @@ include::config/column.txt[]\n \n include::config/commit.txt[]\n \n+include::config/commitgraph.txt[]\n+\n include::config/credential.txt[]\n \n include::config/completion.txt[]\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nnew file mode 100644\nindex 0000000000..cff0797b54\n--- /dev/null\n+++ b/Documentation/config/commitgraph.txt\n@@ -0,0 +1,4 @@\n+commitGraph.readChangedPaths::\n+\tIf true, then git will use the changed-path Bloom filters in the\n+\tcommit-graph file (if it exists, and they are present). Defaults to\n+\ttrue. See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 72a838bd00..ea54d108b9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -327,6 +327,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n+\tprepare_repo_settings(r);\n+\n \tgraph = alloc_commit_graph();\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -403,14 +405,14 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n \t\t\tif (graph->chunk_bloom_indexes)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n \t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n \t\t\tbreak;\n \n \t\tcase GRAPH_CHUNKID_BLOOMDATA:\n \t\t\tif (graph->chunk_bloom_data)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n \t\t\t\tuint32_t hash_version;\n \t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n \t\t\t\thash_version = get_be32(data + chunk_offset);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex aa61a35338..88ccce2036 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -17,9 +17,12 @@ void prepare_repo_settings(struct repository *r)\n \n \tif (!repo_config_get_bool(r, \"core.commitgraph\", &value))\n \t\tr->settings.core_commit_graph = value;\n+\tif (!repo_config_get_bool(r, \"commitgraph.readchangedpaths\", &value))\n+\t\tr->settings.commit_graph_read_changed_paths = value;\n \tif (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n \t\tr->settings.gc_write_commit_graph = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_commit_graph, 1);\n+\tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n \n \tif (!repo_config_get_int(r, \"index.version\", &value))\ndiff --git a/repository.h b/repository.h\nindex 628c834367..bacf843d46 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -30,6 +30,7 @@ struct repo_settings {\n \tint initialized;\n \n \tint core_commit_graph;\n+\tint commit_graph_read_changed_paths;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex cd89c75002..fc7693806c 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -95,7 +95,9 @@ do\n \t\t \"--ancestry-path side..master\"\n \tdo\n \t\ttest_expect_success \"git log option: $option for path: $path\" '\n-\t\t\ttest_bloom_filters_used \"$option -- $path\"\n+\t\t\ttest_bloom_filters_used \"$option -- $path\" &&\n+\t\t\ttest_config commitgraph.readChangedPaths false &&\n+\t\t\ttest_bloom_filters_not_used \"$option -- $path\"\n \t\t'\n \tdone\n done\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405007","messageId":"965489d3617ec03be90d12402fda29fc2b1ead63.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 06/14] commit-graph.c: store maximum changed paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:24Z","receivedAt":"2020-09-03T22:46:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For now, we assume that there is a fixed constant describing the\nmaximum number of changed paths we are willing to store in a Bloom\nfilter.\n\nPrepare for that to (at least partially) not be the case by making it a\nmember of the 'struct bloom_filter_settings'. This will be helpful in\nthe subsequent patches by reducing the size of test cases that exercise\nstoring too many changed paths, as well as preparing for an eventual\nfuture in which this value might change.\n\nThis patch alone does not cause newly generated Bloom filters to use\na custom upper-bound on the maximum number of changed paths a single\nBloom filter can hold, that will occur in a later patch.\n\nSigned-off-by: Taylor Blau \n---\n bloom.h | 11 ++++++++++-\n commit-graph.c | 3 +++\n t/t4216-log-bloom.sh | 4 ++--\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.h b/bloom.h\nindex d8fbb0fbf1..0b9b59a6fe 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -28,9 +28,18 @@ struct bloom_filter_settings {\n \t * that contain n*b bits.\n \t */\n \tuint32_t bits_per_entry;\n+\n+\t/*\n+\t * The maximum number of changed paths per commit\n+\t * before declaring a Bloom filter to be too-large.\n+\t *\n+\t * Not written to the commit-graph file.\n+\t */\n+\tuint32_t max_changed_paths;\n };\n \n-#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10 }\n+#define DEFAULT_BLOOM_MAX_CHANGES 512\n+#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10, DEFAULT_BLOOM_MAX_CHANGES }\n #define BITS_PER_WORD 8\n #define BLOOMDATA_CHUNK_HEADER_SIZE 3 * sizeof(uint32_t)\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex ea54d108b9..55af498aa0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1201,6 +1201,7 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n \tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n \tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_object_intmax(&jw, \"max_changed_paths\", ctx->bloom_settings->max_changed_paths);\n \tjw_end(&jw);\n \n \ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n@@ -1669,6 +1670,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n \t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n \t\t\t\t\t\t\t bloom_settings.num_hashes);\n+\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n \t\tctx->bloom_settings = &bloom_settings;\n \t}\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fc7693806c..47ddf2641f 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -174,11 +174,11 @@ test_expect_success 'persist filter settings' '\n \t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n \t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n \t\tGIT_TRACE2_EVENT_NESTING=5 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2-auto.txt\n '\n \n test_expect_success 'correctly report changes over limit' '\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405008","messageId":"ba89a0cb837abc5fadbaa9514169636d85ee50cf.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:28Z","receivedAt":"2020-09-03T22:46:42Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'get_bloom_filter' takes a flag to control whether it will compute a\nBloom filter if the requested one is missing. In the next patch, we'll\nadd yet another parameter to this method, which would force all but one\ncaller to specify an extra 'NULL' parameter at the end.\n\nInstead of doing this, split 'get_bloom_filter' into two functions:\n'get_bloom_filter' and 'get_or_compute_bloom_filter'. The former only\nlooks up a Bloom filter (and does not compute one if it's missing,\nthus dropping the 'compute_if_not_present' flag). The latter does\ncompute missing Bloom filters, with an additional parameter to store\nwhether or not it needed to do so.\n\nThis simplifies many call-sites, since the majority of existing callers\nto 'get_bloom_filter' do not want missing Bloom filters to be computed\n(so they can drop the parameter entirely and use the simpler version of\nthe function).\n\nWhile we're at it, instrument the new 'get_or_compute_bloom_filter()'\nwith two counters in the 'write_commit_graph_context' struct which store\nthe number of filters that we computed, and the number of those which\nwere too large to store.\n\nIt would be nice to drop the 'compute_if_not_present' flag entirely,\nsince all remaining callers of 'get_or_compute_bloom_filter' pass it as\n'1', but this will change in a future patch and hence cannot be removed.\n\nSigned-off-by: Taylor Blau \n---\n blame.c | 2 +-\n bloom.c | 13 ++++++++++---\n bloom.h | 10 +++++++---\n commit-graph.c | 38 +++++++++++++++++++++++++++++++++++---\n line-log.c | 2 +-\n revision.c | 2 +-\n t/helper/test-bloom.c | 3 ++-\n 7 files changed, 57 insertions(+), 13 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 903e23af23..e5ba35dbd1 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -1276,7 +1276,7 @@ static int maybe_changed_path(struct repository *r,\n \tif (commit_graph_generation(origin->commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn 1;\n \n-\tfilter = get_bloom_filter(r, origin->commit, 0);\n+\tfilter = get_bloom_filter(r, origin->commit);\n \n \tif (!filter)\n \t\treturn 1;\ndiff --git a/bloom.c b/bloom.c\nindex cd9380ac62..a8a21762f4 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,9 +177,10 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present)\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t int *computed)\n {\n \tstruct bloom_filter *filter;\n \tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n@@ -187,6 +188,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n+\tif (computed)\n+\t\t*computed = 0;\n+\n \tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n@@ -273,6 +277,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \t\tfilter->len = 0;\n \t}\n \n+\tif (computed)\n+\t\t*computed = 1;\n+\n \tfree(diff_queued_diff.queue);\n \tDIFF_QUEUE_CLEAR(&diff_queued_diff);\n \ndiff --git a/bloom.h b/bloom.h\nindex 0b9b59a6fe..baa91926db 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -89,9 +89,13 @@ void add_key_to_filter(const struct bloom_key *key,\n \n void init_bloom_filters(void);\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present);\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t int *computed);\n+\n+#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n+\t(r), (c), 0, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 55af498aa0..cabac7f45b 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -971,6 +971,9 @@ struct write_commit_graph_context {\n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n+\n+\tint count_bloom_filter_found_large;\n+\tint count_bloom_filter_computed;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1182,7 +1185,7 @@ static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tuint32_t cur_pos = 0;\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \t\tcur_pos += len;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1222,7 +1225,7 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1392,6 +1395,22 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n+{\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"filter_found_large\",\n+\t\t\t ctx->count_bloom_filter_found_large);\n+\tjw_object_intmax(&jw, \"filter_computed\",\n+\t\t\t ctx->count_bloom_filter_computed);\n+\tjw_end(&jw);\n+\n+\ttrace2_data_json(\"commit-graph\", the_repository, \"bloom_statistics\", &jw);\n+\n+\tjw_release(&jw);\n+}\n+\n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n@@ -1414,12 +1433,25 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tint computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n+\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\tctx->r,\n+\t\t\tc,\n+\t\t\t1,\n+\t\t\t&computed);\n+\t\tif (computed) {\n+\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (filter && !filter->len)\n+\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t}\n \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n+\tif (trace2_is_enabled())\n+\t\ttrace2_bloom_filter_write_statistics(ctx);\n+\n \tfree(sorted_commits);\n \tstop_progress(&progress);\n }\ndiff --git a/line-log.c b/line-log.c\nindex bf73ea95ac..68eeb425f8 100644\n--- a/line-log.c\n+++ b/line-log.c\n@@ -1159,7 +1159,7 @@ static int bloom_filter_check(struct rev_info *rev,\n \t\treturn 1;\n \n \tif (!rev->bloom_filter_settings ||\n-\t !(filter = get_bloom_filter(rev->repo, commit, 0)))\n+\t !(filter = get_bloom_filter(rev->repo, commit)))\n \t\treturn 1;\n \n \tif (!range)\ndiff --git a/revision.c b/revision.c\nindex 857274408c..f4be5d1650 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -751,7 +751,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \tif (commit_graph_generation(commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn -1;\n \n-\tfilter = get_bloom_filter(revs->repo, commit, 0);\n+\tfilter = get_bloom_filter(revs->repo, commit);\n \n \tif (!filter) {\n \t\tcount_bloom_filter_not_present++;\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 5e77d56f59..9f7bb729fc 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -39,7 +39,8 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tstruct bloom_filter *filter;\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n-\tfilter = get_bloom_filter(the_repository, c, 1);\n+\tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n \n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405009","messageId":"89bedba0899bfd1f9904c1122c2872ef39f69564.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 08/14] bloom: use provided 'struct bloom_filter_settings'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:33Z","receivedAt":"2020-09-03T22:46:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When 'get_or_compute_bloom_filter()' needs to compute a Bloom filter\nfrom scratch, it looks to the default 'struct bloom_filter_settings' in\norder to determine the maximum number of changed paths, number of bits\nper entry, and so on.\n\nAll of these values have so far been constant, and so there was no need\nto pass in a pointer from the caller (eg., the one that is stored in the\n'struct write_commit_graph_context').\n\nStart passing in a 'struct bloom_filter_settings *' instead of using the\ndefault values to respect graph-specific settings (eg., in the case of\nsetting 'GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS').\n\nIn order to have an initialized value for these settings, move its\ninitialization to earlier in the commit-graph write.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 13 ++++++-------\n bloom.h | 3 ++-\n commit-graph.c | 21 ++++++++++-----------\n t/helper/test-bloom.c | 1 +\n 4 files changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex a8a21762f4..0cf1962dc5 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -180,13 +180,12 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t int *computed)\n {\n \tstruct bloom_filter *filter;\n-\tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \tint i;\n \tstruct diff_options diffopt;\n-\tint max_changes = 512;\n \n \tif (computed)\n \t\t*computed = 0;\n@@ -211,7 +210,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n \tdiffopt.detect_rename = 0;\n-\tdiffopt.max_changes = max_changes;\n+\tdiffopt.max_changes = settings->max_changed_paths;\n \tdiff_setup_done(&diffopt);\n \n \t/* ensure commit is parsed so we have parent information */\n@@ -223,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= max_changes) {\n+\tif (diffopt.num_changes <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -260,13 +259,13 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n-\t\tfilter->len = (hashmap_get_size(&pathmap) * settings.bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n \t\t\tstruct bloom_key key;\n-\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, &settings);\n-\t\t\tadd_key_to_filter(&key, filter, &settings);\n+\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, settings);\n+\t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\ndiff --git a/bloom.h b/bloom.h\nindex baa91926db..3f19e3fca4 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -92,10 +92,11 @@ void init_bloom_filters(void);\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t int *computed);\n \n #define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n-\t(r), (c), 0, NULL)\n+\t(r), (c), 0, NULL, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex cabac7f45b..7ba9ae26e1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1439,6 +1439,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->r,\n \t\t\tc,\n \t\t\t1,\n+\t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed) {\n \t\t\tctx->count_bloom_filter_computed++;\n@@ -1695,17 +1696,6 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tint num_chunks = 3;\n \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n-\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n-\n-\tif (!ctx->bloom_settings) {\n-\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n-\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n-\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n-\t\t\t\t\t\t\t bloom_settings.num_hashes);\n-\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n-\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n-\t\tctx->bloom_settings = &bloom_settings;\n-\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -2151,6 +2141,7 @@ int write_commit_graph(struct object_directory *odb,\n \tuint32_t i, count_distinct = 0;\n \tint res = 0;\n \tint replace = 0;\n+\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n@@ -2164,6 +2155,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split_opts = split_opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n+\t\t\t\t\t\t bloom_settings.bits_per_entry);\n+\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n+\t\t\t\t\t\t bloom_settings.num_hashes);\n+\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n+\tctx->bloom_settings = &bloom_settings;\n+\n \tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n \t\tctx->changed_paths = 1;\n \tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 9f7bb729fc..46e97b04eb 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -40,6 +40,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n \tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t &settings,\n \t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405010","messageId":"427f129656c0b8112c33e75cd2b49959be42481e.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 09/14] bloom/diff: properly short-circuit on max_changes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:38Z","receivedAt":"2020-09-03T22:46:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nCommit e3696980 (diff: halt tree-diff early after max_changes,\n2020-03-30) intended to create a mechanism to short-circuit a diff\ncalculation after a certain number of paths were modified. By\nincrementing a \"num_changes\" counter throughout the recursive\nll_diff_tree_paths(), this was supposed to match the number of changes\nthat would be written into the changed-path Bloom filters.\nUnfortunately, this was not implemented correctly and instead misses\nsimple cases like file modifications. This then does not stop very\nlarge changed-path filters from being written (unless they add or remove\nmany files).\n\nTo start, change the implementation in ll_diff_tree_paths() to instead\nuse the global diff_queue_diff struct's 'nr' member as the count. This\nis a way to simplify the logic instead of making more mistakes in the\ncomplicated diff code.\n\nThis has a drawback: the diff_queue_diff struct only lists the paths\ncorresponding to blob changes, not their leading directories. Thus,\nget_or_compute_bloom_filter() needs an additional check to see if the\nhashmap with the leading directories becomes too large.\n\nOne reason why this was not caught by test cases was that the test in\nt4216-log-bloom.sh that was supposed to check this \"too many changes\"\ncondition only checked this on the initial commit of a repository. The\nold logic counted these values correctly. Update this test in a few\nways:\n\n1. Use GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS to reduce the limit,\n allowing smaller commits to engage with this logic.\n\n2. Create several interesting cases of edits, adds, removes, and mode\n changes (in the second commit). By testing both sides of the\n inequality with the *_MAX_CHANGED_PATHS variable, we can see that\n the count is exactly correct, so none of these changes are missed\n or over-counted.\n\n3. Use the trace2 data value filter_found_large to verify that these\n commits are on the correct side of the limit.\n\nAnother way to verify the behavior is correct is through performance\ntests. By testing on my local copies of the Git repository and the Linux\nkernel repository, I could measure the effect of these short-circuits\nwhen computing a fresh commit-graph file with changed-path Bloom filters\nusing the command\n\n GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=N time \\\n git commit-graph write --reachable --changed-paths\n\nand reporting the wall time and resulting commit-graph size.\n\nFor Git, the results are\n\n| | N=1 | N=10 | N=512 |\n|--------|----------------|----------------|----------------|\n| HEAD~1 | 10.90s 9.18MB | 11.11s 9.34MB | 11.31s 9.35MB |\n| HEAD | 9.21s 8.62MB | 11.11s 9.29MB | 11.29s 9.34MB |\n\nFor Linux, the results are\n\n| | N=1 | N=20 | N=512 |\n|--------|----------------|---------------|---------------|\n| HEAD~1 | 61.28s 64.3MB | 76.9s 72.6MB | 77.6s 72.6MB |\n| HEAD | 49.44s 56.3MB | 68.7s 65.9MB | 69.2s 65.9MB |\n\nNaturally, the improvement becomes much less as the limit grows, as\nfewer commits satisfy the short-circuit.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n bloom.c | 6 +++-\n diff.h | 2 --\n t/t4216-log-bloom.sh | 85 +++++++++++++++++++++++++++++++++++++++-----\n tree-diff.c | 5 +--\n 4 files changed, 82 insertions(+), 16 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 0cf1962dc5..ed54e96e57 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -222,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= settings->max_changed_paths) {\n+\tif (diff_queued_diff.nr <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -259,6 +259,9 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n+\t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths)\n+\t\t\tgoto cleanup;\n+\n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n@@ -268,6 +271,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n+\tcleanup:\n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\ndiff --git a/diff.h b/diff.h\nindex e0c0af6286..1d32b71885 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -287,8 +287,6 @@ struct diff_options {\n \n \t/* If non-zero, then stop computing after this many changes. */\n \tint max_changes;\n-\t/* For internal use only. */\n-\tint num_changes;\n \n \tint ita_invisible_in_index;\n /* white-space error highlighting */\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 47ddf2641f..e8788749bf 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -182,20 +182,87 @@ test_expect_success 'persist filter settings' '\n '\n \n test_expect_success 'correctly report changes over limit' '\n-\tgit init 513changes &&\n+\tgit init limits &&\n \t(\n-\t\tcd 513changes &&\n-\t\tfor i in $(test_seq 1 513)\n+\t\tcd limits &&\n+\t\tmkdir d &&\n+\t\tmkdir d/e &&\n+\n+\t\tfor i in $(test_seq 1 2)\n \t\tdo\n-\t\t\techo $i >file$i.txt || return 1\n+\t\t\tprintf $i >d/file$i.txt &&\n+\t\t\tprintf $i >d/e/file$i.txt || return 1\n \t\tdone &&\n-\t\tgit add . &&\n+\n+\t\tmkdir mode &&\n+\t\tprintf bash >mode/script.sh &&\n+\n+\t\tmkdir foo &&\n+\t\ttouch foo/bar &&\n+\t\ttouch foo.txt &&\n+\n+\t\tgit add d foo foo.txt mode &&\n \t\tgit commit -m \"files\" &&\n-\t\tgit commit-graph write --reachable --changed-paths &&\n-\t\tfor i in $(test_seq 1 513)\n+\n+\t\t# Commit has 7 file and 4 directory adds\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":10\" trace &&\n+\t\tgrep \"\\\"filter_found_large\\\":1\" trace &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n \t\tdo\n-\t\t\tgit -c core.commitGraph=false log -- file$i.txt >expect &&\n-\t\t\tgit log -- file$i.txt >actual &&\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# Make a variety of path changes\n+\t\tprintf new1 >d/e/file1.txt &&\n+\t\tprintf new2 >d/file2.txt &&\n+\t\trm d/e/file2.txt &&\n+\t\trm -r foo &&\n+\t\tprintf text >foo &&\n+\t\tmkdir f &&\n+\t\tprintf new1 >f/file1.txt &&\n+\n+\t\t# including a mode-only change (counts as modified)\n+\t\tgit update-index --chmod=+x mode/script.sh &&\n+\n+\t\tgit add foo d f &&\n+\t\tgit commit -m \"complicated\" &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-edit\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":10\" trace-edit &&\n+\t\tgrep \"\\\"filter_found_large\\\":2\" trace-edit &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=11 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-update\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":11\" trace-update &&\n+\t\tgrep \"\\\"filter_found_large\\\":0\" trace-update &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n \t\t\ttest_cmp expect actual || return 1\n \t\tdone\n \t)\ndiff --git a/tree-diff.c b/tree-diff.c\nindex 6ebad1a46f..7cebbb327e 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -434,7 +434,7 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tif (diff_can_quit_early(opt))\n \t\t\tbreak;\n \n-\t\tif (opt->max_changes && opt->num_changes > opt->max_changes)\n+\t\tif (opt->max_changes && diff_queued_diff.nr > opt->max_changes)\n \t\t\tbreak;\n \n \t\tif (opt->pathspec.nr) {\n@@ -521,7 +521,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \n \t\t\t/* t↓ */\n \t\t\tupdate_tree_entry(&t);\n-\t\t\topt->num_changes++;\n \t\t}\n \n \t\t/* t > p[imin] */\n@@ -539,7 +538,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tskip_emit_tp:\n \t\t\t/* ∀ pi=p[imin] pi↓ */\n \t\t\tupdate_tp_entries(tp, nparent);\n-\t\t\topt->num_changes++;\n \t\t}\n \t}\n \n@@ -557,7 +555,6 @@ struct combine_diff_path *diff_tree_paths(\n \tconst struct object_id **parents_oid, int nparent,\n \tstruct strbuf *base, struct diff_options *opt)\n {\n-\topt->num_changes = 0;\n \tp = ll_diff_tree_paths(p, oid, parents_oid, nparent, base, opt);\n \n \t/*\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405011","messageId":"08b5f185f615de5d43e2397ae58730e82b18b7dd.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 10/14] commit-graph.c: sort index into commits list","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:42Z","receivedAt":"2020-09-03T22:46:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For locality, 'compute_bloom_filters()' sorts the commits for which it\nwants to compute Bloom filters in a preferred order (cf., 3d11275505\n(commit-graph: examine commits by generation number, 2020-03-30) for\ndetails).\n\nA future patch will want to recover the new graph position of each\ncommit. Since the 'packed_commit_list' already stores a double-pointer,\navoid a 'COPY_ARRAY' and instead keep track of an index into the\noriginal list. (Use an integer index instead of a memory address, since\nthis involves a needlessly confusing triple-pointer).\n\nAlter the two sorting routines 'commit_pos_cmp' and 'commit_gen_cmp' to\ntake into account the packed_commit_list they are sorting with respect\nto. Since 'compute_bloom_filters()' is the only caller for each of those\ncomparison functions, no other call-sites need updating.\n\nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 43 ++++++++++++++++++++++++-------------------\n 1 file changed, 24 insertions(+), 19 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 7ba9ae26e1..35535f4192 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -79,10 +79,18 @@ static void set_commit_pos(struct repository *r, const struct object_id *oid)\n \t*commit_pos_at(&commit_pos, commit) = max_pos++;\n }\n \n-static int commit_pos_cmp(const void *va, const void *vb)\n+struct packed_commit_list {\n+\tstruct commit **list;\n+\tint nr;\n+\tint alloc;\n+};\n+\n+static int commit_pos_cmp(const void *va, const void *vb, void *ctx)\n {\n-\tconst struct commit *a = *(const struct commit **)va;\n-\tconst struct commit *b = *(const struct commit **)vb;\n+\tstruct packed_commit_list *commits = ctx;\n+\n+\tconst struct commit *a = commits->list[*(int *)va];\n+\tconst struct commit *b = commits->list[*(int *)vb];\n \treturn commit_pos_at(&commit_pos, a) -\n \t commit_pos_at(&commit_pos, b);\n }\n@@ -139,10 +147,12 @@ static struct commit_graph_data *commit_graph_data_at(const struct commit *c)\n \treturn data;\n }\n \n-static int commit_gen_cmp(const void *va, const void *vb)\n+static int commit_gen_cmp(const void *va, const void *vb, void *ctx)\n {\n-\tconst struct commit *a = *(const struct commit **)va;\n-\tconst struct commit *b = *(const struct commit **)vb;\n+\tstruct packed_commit_list *commits = ctx;\n+\n+\tconst struct commit *a = commits->list[*(int *)va];\n+\tconst struct commit *b = commits->list[*(int *)vb];\n \n \tuint32_t generation_a = commit_graph_generation(a);\n \tuint32_t generation_b = commit_graph_generation(b);\n@@ -929,11 +939,6 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n-struct packed_commit_list {\n-\tstruct commit **list;\n-\tint nr;\n-\tint alloc;\n-};\n \n struct packed_oid_list {\n \tstruct object_id *list;\n@@ -1415,7 +1420,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct progress *progress = NULL;\n-\tstruct commit **sorted_commits;\n+\tint *sorted_commits;\n \n \tinit_bloom_filters();\n \n@@ -1425,16 +1430,16 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->commits.nr);\n \n \tALLOC_ARRAY(sorted_commits, ctx->commits.nr);\n-\tCOPY_ARRAY(sorted_commits, ctx->commits.list, ctx->commits.nr);\n-\n-\tif (ctx->order_by_pack)\n-\t\tQSORT(sorted_commits, ctx->commits.nr, commit_pos_cmp);\n-\telse\n-\t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n+\tfor (i = 0; i < ctx->commits.nr; i++)\n+\t\tsorted_commits[i] = i;\n+\tQSORT_S(sorted_commits, ctx->commits.nr,\n+\t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n+\t\t&ctx->commits);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tint computed = 0;\n-\t\tstruct commit *c = sorted_commits[i];\n+\t\tint pos = sorted_commits[i];\n+\t\tstruct commit *c = ctx->commits.list[pos];\n \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\tctx->r,\n \t\t\tc,\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405012","messageId":"d7cbd4ca1a5dd63fb1d19ef97fac9765daa5ae03.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 11/14] csum-file.h: introduce 'hashwrite_be64()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:48Z","receivedAt":"2020-09-03T22:46:54Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"A small handful of writers who wish to encode 64-bit values in network\norder have worked around the lack of such a helper by calling the 32-bit\nvariant twice.\n\nThe subsequent commit will add another caller who wants to write a\n64-bit value. To ease their (and the existing caller's) pain, introduce\na helper to do just that, and convert existing call-sites.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n commit-graph.c | 8 ++------\n csum-file.h | 6 ++++++\n midx.c | 3 +--\n 3 files changed, 9 insertions(+), 8 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 35535f4192..01d791343a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1791,12 +1791,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \n \tchunk_offset = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tfor (i = 0; i <= num_chunks; i++) {\n-\t\tuint32_t chunk_write[3];\n-\n-\t\tchunk_write[0] = htonl(chunks[i].id);\n-\t\tchunk_write[1] = htonl(chunk_offset >> 32);\n-\t\tchunk_write[2] = htonl(chunk_offset & 0xffffffff);\n-\t\thashwrite(f, chunk_write, 12);\n+\t\thashwrite_be32(f, chunks[i].id);\n+\t\thashwrite_be64(f, chunk_offset);\n \n \t\tchunk_offset += chunks[i].size;\n \t}\ndiff --git a/csum-file.h b/csum-file.h\nindex f9cbd317fb..b026ec7766 100644\n--- a/csum-file.h\n+++ b/csum-file.h\n@@ -62,4 +62,10 @@ static inline void hashwrite_be32(struct hashfile *f, uint32_t data)\n \thashwrite(f, &data, sizeof(data));\n }\n \n+static inline void hashwrite_be64(struct hashfile *f, uint64_t data)\n+{\n+\thashwrite_be32(f, data >> 32);\n+\thashwrite_be32(f, data & 0xffffffffUL);\n+}\n+\n #endif\ndiff --git a/midx.c b/midx.c\nindex e9b2e1253a..32cc5fdc22 100644\n--- a/midx.c\n+++ b/midx.c\n@@ -789,8 +789,7 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n \t\tif (!(offset >> 31))\n \t\t\tcontinue;\n \n-\t\thashwrite_be32(f, offset >> 32);\n-\t\thashwrite_be32(f, offset & 0xffffffffUL);\n+\t\thashwrite_be64(f, offset);\n \t\twritten += 2 * sizeof(uint32_t);\n \n \t\tnr_large_offset--;\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405013","messageId":"3063beb58859f997188ae0cca046aaa93eec3d9d.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 12/14] commit-graph: add large-filters bitmap chunk","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:52Z","receivedAt":"2020-09-03T22:46:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a commit has more than a certain number of changed paths (commonly\n512), the commit-graph machinery represents it as a zero-length filter.\nIn addition to these too-large filters, the commit-graph machinery also\nrepresents commits with no filter and commits with no changed paths in\nthe same way.\n\nWhen writing a commit-graph that aggregates several incremental\ncommit-graph layers (eg., with '--split=replace'), the commit-graph\nmachinery first computes all of the Bloom filters that it wants to write\nbut does not already know about from existing graph layers. Because we\noverload the zero-length filter in the above fashion, this leads to\nrecomputing large filters over and over again.\n\nThis is already undesirable, since it means that we are wasting\nconsiderable effort to discover that a commit with too many changed\npaths, only to throw that effort away (and then repeat the process the\nnext time a roll-up is performed).\n\nIn a subsequent patch, we will add a '--max-new-filters=' option,\nwhich specifies an upper-bound on the number of new filters we are\nwilling to compute from scratch. Suppose that there are 'N' too-large\nfilters, and we specify '--max-new-filters=M'. If 'N >= M', it is\nunlikely that any filters will be generated, since we'll spend most of\nour effort on filters that we ultimately throw away. If 'N < M', filters\nwill trickle in over time, but only at most 'M - N' per-write.\n\nTo address this, add a new chunk which encodes a bitmap where the ith\nbit is on iff the ith commit has zero or at least 512 changed paths.\nLikewise store the maximum number of changed paths we are willing to\nstore in order to prepare for eventually making this value more easily\ncustomizable. When computing Bloom filters, first consult the relevant\nbitmap (in the case that we are rolling up existing layers) to see if\ncomputing the Bloom filter from scratch would be a waste of time.\n\nThis patch implements a new chunk instead of extending the existing BIDX\nand BDAT chunks because modifying these chunks would confuse old\nclients. (Eg., setting the most-significant bit in the BIDX chunk would\nconfuse old clients and require a version bump).\n\nTo allow using the existing bitmap code with 64-bit words, we write the\ndata in network byte order from the 64-bit words. This means we also\nneed to read the array from the commit-graph file by translating each\nword from network byte order using get_be64() when loading the commit\ngraph. Initialize this bitmap lazily to avoid paying a linear-time cost\nupon each commit-graph load even if we do not need the bitmaps\nthemselves.\n\nBy avoiding the need to move to new versions of the BDAT and BIDX chunk,\nwe can give ourselves more time to consider whether or not other\nmodifications to these chunks are worthwhile without holding up this\nchange.\n\nAnother approach would be to introduce a new BIDX chunk (say, one\nidentified by 'BID2') which is identical to the existing BIDX chunk,\nexcept the most-significant bit of each offset is interpreted as \"this\nfilter is too big\" iff looking at a BID2 chunk. This avoids having to\nwrite a bitmap, but forces older clients to rewrite their commit-graphs\n(as well as reduces the theoretical largest Bloom filters we could\nwrite, and forces us to maintain the code necessary to translate BIDX\nchunks to BID2 ones). Separately from this patch, I implemented this\nalternate approach and did not find it to be advantageous.\n\nSigned-off-by: Taylor Blau \n---\n .../technical/commit-graph-format.txt | 13 +++\n bloom.h | 2 +-\n commit-graph.c | 100 +++++++++++++++---\n commit-graph.h | 10 ++\n t/t4216-log-bloom.sh | 25 ++++-\n 5 files changed, 135 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 6ddbceba15..db212b8cc5 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -128,6 +128,19 @@ CHUNK DATA:\n of length zero.\n * The BDAT chunk is present if and only if BIDX is present.\n \n+ Large Bloom Filters (ID: {'B', 'F', 'X', 'L'}) [Optional]\n+ * It starts with a 32-bit unsigned integer specifying the maximum number of\n+ changed-paths that can be stored in a single Bloom filter.\n+ * It then contains a list of 64-bit words in network order (the length of\n+ this list is determined by the width of the chunk) which is a bitmap. The\n+ 'i'th bit is set exactly when the 'i'th commit in the graph has a\n+ changed-path Bloom filter with zero entries (either because the commit is\n+ empty, or because it contains more entries than is allowed per filter by\n+ the layer that contains it).\n+ * The BFXL chunk is present only when the BIDX and BDAT chunks are\n+ also present.\n+\n+\n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n This list of H-byte hashes describe a set of B commit-graph files that\n form a commit-graph chain. The graph position for the ith commit in this\ndiff --git a/bloom.h b/bloom.h\nindex 3f19e3fca4..464d9b57de 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -33,7 +33,7 @@ struct bloom_filter_settings {\n \t * The maximum number of changed paths per commit\n \t * before declaring a Bloom filter to be too-large.\n \t *\n-\t * Not written to the commit-graph file.\n+\t * Written to the 'BFXL' chunk (instead of 'BDAT').\n \t */\n \tuint32_t max_changed_paths;\n };\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 01d791343a..68ffa6ec35 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -41,8 +41,9 @@ void git_test_write_commit_graph_or_die(void)\n #define GRAPH_CHUNKID_EXTRAEDGES 0x45444745 /* \"EDGE\" */\n #define GRAPH_CHUNKID_BLOOMINDEXES 0x42494458 /* \"BIDX\" */\n #define GRAPH_CHUNKID_BLOOMDATA 0x42444154 /* \"BDAT\" */\n+#define GRAPH_CHUNKID_BLOOMLARGE 0x4246584c /* \"BFXL\" */\n #define GRAPH_CHUNKID_BASE 0x42415345 /* \"BASE\" */\n-#define MAX_NUM_CHUNKS 7\n+#define MAX_NUM_CHUNKS 8\n \n #define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n@@ -436,6 +437,16 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n \t\t\t}\n \t\t\tbreak;\n+\n+\t\tcase GRAPH_CHUNKID_BLOOMLARGE:\n+\t\t\tif (graph->chunk_bloom_large_filters)\n+\t\t\t\tchunk_repeated = 1;\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n+\t\t\t\tgraph->chunk_bloom_large_filters = data + chunk_offset + sizeof(uint32_t);\n+\t\t\t\tgraph->bloom_large_alloc = get_be64(chunk_lookup + 4) - chunk_offset - sizeof(uint32_t);\n+\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = get_be32(data + chunk_offset);\n+\t\t\t}\n+\t\t\tbreak;\n \t\t}\n \n \t\tif (chunk_repeated) {\n@@ -450,6 +461,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t/* We need both the bloom chunks to exist together. Else ignore the data */\n \t\tgraph->chunk_bloom_indexes = NULL;\n \t\tgraph->chunk_bloom_data = NULL;\n+\t\tgraph->chunk_bloom_large_filters = NULL;\n+\t\tgraph->bloom_large_alloc = 0;\n \t\tFREE_AND_NULL(graph->bloom_filter_settings);\n \t}\n \n@@ -939,6 +952,32 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n \treturn get_commit_tree_in_graph_one(r, r->objects->commit_graph, c);\n }\n \n+int get_bloom_filter_large_in_graph(struct commit_graph *g,\n+\t\t\t\t const struct commit *c)\n+{\n+\tuint32_t graph_pos;\n+\tif (!find_commit_in_graph(c, g, &graph_pos))\n+\t\treturn 0;\n+\n+\twhile (g && graph_pos < g->num_commits_in_base)\n+\t\tg = g->base_graph;\n+\n+\tif (!g)\n+\t\treturn 0;\n+\n+\tif (!g->bloom_large && g->bloom_large_alloc) {\n+\t\tsize_t i;\n+\t\tg->bloom_large = bitmap_word_alloc(g->bloom_large_alloc);\n+\n+\t\tfor (i = 0; i < g->bloom_large->word_alloc; i++)\n+\t\t\tg->bloom_large->words[i] = get_be64(\n+\t\t\t\tg->chunk_bloom_large_filters + i * sizeof(eword_t));\n+\t}\n+\n+\tif (!g->bloom_large)\n+\t\treturn 0;\n+\treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n+}\n \n struct packed_oid_list {\n \tstruct object_id *list;\n@@ -977,8 +1016,10 @@ struct write_commit_graph_context {\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \n+\tint count_bloom_filter_known_large;\n \tint count_bloom_filter_found_large;\n \tint count_bloom_filter_computed;\n+\tstruct bitmap *bloom_large;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1242,6 +1283,23 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \treturn 0;\n }\n \n+static int write_graph_chunk_bloom_large(struct hashfile *f,\n+\t\t\t\t\t struct write_commit_graph_context *ctx)\n+{\n+\tsize_t i, alloc = ctx->commits.nr / BITS_IN_EWORD;\n+\tif (ctx->commits.nr % BITS_IN_EWORD)\n+\t\talloc++;\n+\tif (alloc > ctx->bloom_large->word_alloc)\n+\t\tBUG(\"write_graph_chunk_bloom_large: bitmap not large enough\");\n+\n+\ttrace2_region_enter(\"commit-graph\", \"bloom_large\", ctx->r);\n+\thashwrite_be32(f, ctx->bloom_settings->max_changed_paths);\n+\tfor (i = 0; i < ctx->bloom_large->word_alloc; i++)\n+\t\thashwrite_be64(f, ctx->bloom_large->words[i]);\n+\ttrace2_region_leave(\"commit-graph\", \"bloom_large\", ctx->r);\n+\treturn 0;\n+}\n+\n static int oid_compare(const void *_a, const void *_b)\n {\n \tconst struct object_id *a = (const struct object_id *)_a;\n@@ -1405,6 +1463,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \tstruct json_writer jw = JSON_WRITER_INIT;\n \n \tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"filter_known_large\",\n+\t\t\t ctx->count_bloom_filter_known_large);\n \tjw_object_intmax(&jw, \"filter_found_large\",\n \t\t\t ctx->count_bloom_filter_found_large);\n \tjw_object_intmax(&jw, \"filter_computed\",\n@@ -1423,6 +1483,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint *sorted_commits;\n \n \tinit_bloom_filters();\n+\tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n \n \tif (ctx->report_progress)\n \t\tprogress = start_delayed_progress(\n@@ -1437,21 +1498,28 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t&ctx->commits);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n-\t\tint computed = 0;\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n-\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n-\t\t\tctx->r,\n-\t\t\tc,\n-\t\t\t1,\n-\t\t\tctx->bloom_settings,\n-\t\t\t&computed);\n-\t\tif (computed) {\n-\t\t\tctx->count_bloom_filter_computed++;\n-\t\t\tif (filter && !filter->len)\n-\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n+\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\tctx->count_bloom_filter_known_large++;\n+\t\t} else {\n+\t\t\tint computed = 0;\n+\t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\t\tctx->r,\n+\t\t\t\tc,\n+\t\t\t\t1,\n+\t\t\t\tctx->bloom_settings,\n+\t\t\t\t&computed);\n+\t\t\tif (computed) {\n+\t\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\t\tif (filter && !filter->len) {\n+\t\t\t\t\tbitmap_set(ctx->bloom_large, pos);\n+\t\t\t\t\tctx->count_bloom_filter_found_large++;\n+\t\t\t\t}\n+\t\t\t}\n+\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\t}\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n@@ -1771,6 +1839,11 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t + ctx->total_bloom_filter_data_size;\n \t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_data;\n \t\tnum_chunks++;\n+\t\tchunks[num_chunks].id = GRAPH_CHUNKID_BLOOMLARGE;\n+\t\tchunks[num_chunks].size = sizeof(eword_t) * ctx->bloom_large->word_alloc\n+\t\t\t\t\t+ sizeof(uint32_t);\n+\t\tchunks[num_chunks].write_fn = write_graph_chunk_bloom_large;\n+\t\tnum_chunks++;\n \t}\n \tif (ctx->num_commit_graphs_after > 1) {\n \t\tchunks[num_chunks].id = GRAPH_CHUNKID_BASE;\n@@ -2510,6 +2583,7 @@ void free_commit_graph(struct commit_graph *g)\n \t}\n \tfree(g->filename);\n \tfree(g->bloom_filter_settings);\n+\tbitmap_free(g->bloom_large);\n \tfree(g);\n }\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex d9acb22bac..9afb1477d5 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -4,6 +4,7 @@\n #include \"git-compat-util.h\"\n #include \"object-store.h\"\n #include \"oidset.h\"\n+#include \"ewah/ewok.h\"\n \n #define GIT_TEST_COMMIT_GRAPH \"GIT_TEST_COMMIT_GRAPH\"\n #define GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE \"GIT_TEST_COMMIT_GRAPH_DIE_ON_PARSE\"\n@@ -50,6 +51,9 @@ void load_commit_graph_info(struct repository *r, struct commit *item);\n struct tree *get_commit_tree_in_graph(struct repository *r,\n \t\t\t\t const struct commit *c);\n \n+int get_bloom_filter_large_in_graph(struct commit_graph *g,\n+\t\t\t\t const struct commit *c);\n+\n struct commit_graph {\n \tconst unsigned char *data;\n \tsize_t data_len;\n@@ -71,6 +75,10 @@ struct commit_graph {\n \tconst unsigned char *chunk_base_graphs;\n \tconst unsigned char *chunk_bloom_indexes;\n \tconst unsigned char *chunk_bloom_data;\n+\tconst unsigned char *chunk_bloom_large_filters;\n+\n+\tstruct bitmap *bloom_large;\n+\tsize_t bloom_large_alloc;\n \n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n@@ -83,6 +91,8 @@ struct commit_graph *read_commit_graph_one(struct repository *r,\n struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n+void prepare_commit_graph_bloom_large(struct commit_graph *g);\n+\n /*\n * Return 1 if and only if the repository has a commit-graph\n * file and generation numbers are computed in that file.\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex e8788749bf..fed4929af3 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -38,7 +38,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \tEOF\n '\n graph_read_expect () {\n-\tNUM_CHUNKS=5\n+\tNUM_CHUNKS=6\n \tcat >expect <<- EOF\n \theader: 43475048 1 $(test_oid oid_version) $NUM_CHUNKS 0\n \tnum_commits: $1\n@@ -267,5 +267,28 @@ test_expect_success 'correctly report changes over limit' '\n \t\tdone\n \t)\n '\n+test_bloom_filters_computed () {\n+\tcommit_graph_args=$1\n+\tbloom_trace_prefix=\"{\\\"filter_known_large\\\":$2,\\\"filter_found_large\\\":$3,\\\"filter_computed\\\":$4\"\n+\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n+\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write $commit_graph_args &&\n+\tgrep \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.event\"\n+}\n+\n+test_expect_success 'Bloom generation does not recompute too-large filters' '\n+\t(\n+\t\tcd limits &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t--split=replace &&\n+\t\ttest_commit c1 filter &&\n+\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n+\t\t\t2 0 1\n+\t)\n+'\n \n test_done\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405014","messageId":"ee0bc109f3db7caf59f55736744646a9ee047b01.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 13/14] commit-graph: rename 'split_commit_graph_opts'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:46:57Z","receivedAt":"2020-09-03T22:47:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the subsequent commit, additional options will be added to the\ncommit-graph API which have nothing to do with splitting.\n\nRename the 'split_commit_graph_opts' structure to the more-generic\n'commit_graph_opts' to encompass both. Likewise, rename the 'flags'\nmember to instead be 'split_flags' to clarify that it only has to do\nwith the behavior implied by '--split'.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 20 ++++++++++----------\n commit-graph.c | 40 ++++++++++++++++++++--------------------\n commit-graph.h | 8 ++++----\n make: *** [Makefile | 0\n 4 files changed, 34 insertions(+), 34 deletions(-)\n create mode 100644 make: *** [Makefile\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ba5584463f..f3243bd982 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,7 +119,7 @@ static int graph_verify(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n-static struct split_commit_graph_opts split_opts;\n+static struct commit_graph_opts write_opts;\n \n static int write_option_parse_split(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n@@ -187,24 +187,24 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n \t\t\tN_(\"enable computation for changed paths\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.split_flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\twrite_option_parse_split),\n-\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\tOPT_INTEGER(0, \"max-commits\", &write_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n-\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\tOPT_INTEGER(0, \"size-multiple\", &write_opts.size_multiple,\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n-\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n \t\tOPT_END(),\n \t};\n \n \topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n-\tsplit_opts.size_multiple = 2;\n-\tsplit_opts.max_commits = 0;\n-\tsplit_opts.expire_time = 0;\n+\twrite_opts.size_multiple = 2;\n+\twrite_opts.max_commits = 0;\n+\twrite_opts.expire_time = 0;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -232,7 +232,7 @@ static int graph_write(int argc, const char **argv)\n \todb = find_odb(the_repository, opts.obj_dir);\n \n \tif (opts.reachable) {\n-\t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n+\t\tif (write_commit_graph_reachable(odb, flags, &write_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n@@ -261,7 +261,7 @@ static int graph_write(int argc, const char **argv)\n \t\t\t opts.stdin_packs ? &pack_indexes : NULL,\n \t\t\t opts.stdin_commits ? &commits : NULL,\n \t\t\t flags,\n-\t\t\t &split_opts))\n+\t\t\t &write_opts))\n \t\tresult = 1;\n \n cleanup:\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 68ffa6ec35..33fcf01a7a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1012,7 +1012,7 @@ struct write_commit_graph_context {\n \t\t changed_paths:1,\n \t\t order_by_pack:1;\n \n-\tconst struct split_commit_graph_opts *split_opts;\n+\tconst struct commit_graph_opts *opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \n@@ -1353,8 +1353,8 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1554,7 +1554,7 @@ static int add_ref_to_set(const char *refname,\n \n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t\t\t const struct commit_graph_opts *opts)\n {\n \tstruct oidset commits = OIDSET_INIT;\n \tstruct refs_cb_data data;\n@@ -1571,7 +1571,7 @@ int write_commit_graph_reachable(struct object_directory *odb,\n \tstop_progress(&data.progress);\n \n \tresult = write_commit_graph(odb, NULL, &commits,\n-\t\t\t\t flags, split_opts);\n+\t\t\t\t flags, opts);\n \n \toidset_clear(&commits);\n \treturn result;\n@@ -1686,8 +1686,8 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tctx->num_extra_edges = 0;\n \tif (ctx->report_progress)\n@@ -1973,13 +1973,13 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tint max_commits = 0;\n \tint size_mult = 2;\n \n-\tif (ctx->split_opts) {\n-\t\tmax_commits = ctx->split_opts->max_commits;\n+\tif (ctx->opts) {\n+\t\tmax_commits = ctx->opts->max_commits;\n \n-\t\tif (ctx->split_opts->size_multiple)\n-\t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t\tif (ctx->opts->size_multiple)\n+\t\t\tsize_mult = ctx->opts->size_multiple;\n \n-\t\tflags = ctx->split_opts->flags;\n+\t\tflags = ctx->opts->split_flags;\n \t}\n \n \tg = ctx->r->objects->commit_graph;\n@@ -2157,8 +2157,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n \n-\tif (ctx->split_opts && ctx->split_opts->expire_time)\n-\t\texpire_time = ctx->split_opts->expire_time;\n+\tif (ctx->opts && ctx->opts->expire_time)\n+\t\texpire_time = ctx->opts->expire_time;\n \tif (!ctx->split) {\n \t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n@@ -2209,7 +2209,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t const struct commit_graph_opts *opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -2226,7 +2226,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n-\tctx->split_opts = split_opts;\n+\tctx->opts = opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n@@ -2274,15 +2274,15 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\t}\n \t\t}\n \n-\t\tif (ctx->split_opts)\n-\t\t\treplace = ctx->split_opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n+\t\tif (ctx->opts)\n+\t\t\treplace = ctx->opts->split_flags & COMMIT_GRAPH_SPLIT_REPLACE;\n \t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n-\t\tctx->oids.alloc = split_opts->max_commits;\n+\tif (ctx->split && opts && ctx->oids.alloc > opts->max_commits)\n+\t\tctx->oids.alloc = opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->odb);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 9afb1477d5..fe798a4047 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -115,11 +115,11 @@ enum commit_graph_split_flags {\n \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n };\n \n-struct split_commit_graph_opts {\n+struct commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n-\tenum commit_graph_split_flags flags;\n+\tenum commit_graph_split_flags split_flags;\n };\n \n /*\n@@ -130,12 +130,12 @@ struct split_commit_graph_opts {\n */\n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t\t\t const struct commit_graph_opts *opts);\n int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t const struct commit_graph_opts *opts);\n \n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n \ndiff --git a/make: *** [Makefile b/make: *** [Makefile\nnew file mode 100644\nindex 0000000000..e69de29bb2\n-- \n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405015","messageId":"cd0a9da6390bb110ac787135e9e5cf343ffef550.1599172908.git.me@ttaylorr.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"[PATCH v4 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-03T22:47:01Z","receivedAt":"2020-09-03T22:47:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a command-line flag and configuration variable to fill in the\n'max_new_filters' variable introduced two patches ago.\n\nThe command-line option '--max-new-filters' takes precedence over\n'commitGraph.maxNewFilters', which is the default value.\n'--no-max-new-filters' can also be provided, which sets the value back\nto '-1', indicating that an unlimited number of new Bloom filters may be\ngenerated. (OPT_INTEGER only allows setting the '--no-' variant back to\n'0', hence a custom callback was used instead).\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 4 +++\n Documentation/git-commit-graph.txt | 6 ++++\n bloom.c | 13 +++++---\n builtin/commit-graph.c | 39 ++++++++++++++++++++++--\n commit-graph.c | 27 ++++++++++++++---\n commit-graph.h | 4 ++-\n t/t4216-log-bloom.sh | 44 ++++++++++++++++++++++++++++\n 7 files changed, 125 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex cff0797b54..4582c39fc4 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitGraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..81a2e65903 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,12 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Commits whose filters are not calculated are stored as a\n+length zero Bloom filter, and their bit is marked in the `BFXL` chunk.\n+Overrides the `commitGraph.maxNewFilters` configuration.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex ed54e96e57..34503898ac 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -197,16 +197,21 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n-\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n+\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n \n-\tif (filter->data)\n+\tif (filter->data && filter->len)\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\n \n+\tif (filter && !filter->len &&\n+\t get_bloom_filter_large_in_graph(r->objects->commit_graph, c,\n+\t\t\t\t\t settings->max_changed_paths))\n+\t\treturn filter;\n+\n+\n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n \tdiffopt.detect_rename = 0;\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f3243bd982..e7a1539b08 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n \n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n \n@@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -270,6 +293,16 @@ static int graph_write(int argc, const char **argv)\n \treturn result;\n }\n \n+static int git_commit_graph_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, \"commitgraph.maxnewfilters\")) {\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_commit_graph_options[] = {\n@@ -283,7 +316,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_commit_graph_usage,\n \t\t\t\t builtin_commit_graph_options);\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(git_commit_graph_config, &opts);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_commit_graph_options,\n \t\t\t builtin_commit_graph_usage,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 33fcf01a7a..243c7253ff 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -953,7 +953,8 @@ struct tree *get_commit_tree_in_graph(struct repository *r, const struct commit\n }\n \n int get_bloom_filter_large_in_graph(struct commit_graph *g,\n-\t\t\t\t const struct commit *c)\n+\t\t\t\t const struct commit *c,\n+\t\t\t\t uint32_t max_changed_paths)\n {\n \tuint32_t graph_pos;\n \tif (!find_commit_in_graph(c, g, &graph_pos))\n@@ -976,6 +977,17 @@ int get_bloom_filter_large_in_graph(struct commit_graph *g,\n \n \tif (!g->bloom_large)\n \t\treturn 0;\n+\tif (g->bloom_filter_settings->max_changed_paths != max_changed_paths) {\n+\t\t/*\n+\t\t * Force all commits which are subject to a different\n+\t\t * 'max_changed_paths' limit to be recomputed from scratch.\n+\t\t *\n+\t\t * Note that this could likely be improved, but is ignored since\n+\t\t * all real-world graphs set the maximum number of changed paths\n+\t\t * at 512.\n+\t\t */\n+\t\treturn 0;\n+\t}\n \treturn bitmap_get(g->bloom_large, graph_pos - g->num_commits_in_base);\n }\n \n@@ -1481,6 +1493,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tint *sorted_commits;\n+\tint max_new_filters;\n \n \tinit_bloom_filters();\n \tctx->bloom_large = bitmap_word_alloc(ctx->commits.nr / BITS_IN_EWORD + 1);\n@@ -1497,10 +1510,15 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tctx->order_by_pack ? commit_pos_cmp : commit_gen_cmp,\n \t\t&ctx->commits);\n \n+\tmax_new_filters = ctx->opts && ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tint pos = sorted_commits[i];\n \t\tstruct commit *c = ctx->commits.list[pos];\n-\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph, c)) {\n+\t\tif (get_bloom_filter_large_in_graph(ctx->r->objects->commit_graph,\n+\t\t\t\t\t\t c,\n+\t\t\t\t\t\t ctx->bloom_settings->max_changed_paths)) {\n \t\t\tbitmap_set(ctx->bloom_large, pos);\n \t\t\tctx->count_bloom_filter_known_large++;\n \t\t} else {\n@@ -1508,7 +1526,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\t\tctx->r,\n \t\t\t\tc,\n-\t\t\t\t1,\n+\t\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n \t\t\t\tctx->bloom_settings,\n \t\t\t\t&computed);\n \t\t\tif (computed) {\n@@ -1518,7 +1536,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\t\tctx->count_bloom_filter_found_large++;\n \t\t\t\t}\n \t\t\t}\n-\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\t\tif (filter)\n+\t\t\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\t}\n \t\tdisplay_progress(progress, i + 1);\n \t}\ndiff --git a/commit-graph.h b/commit-graph.h\nindex fe798a4047..eac4efc7a6 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -52,7 +52,8 @@ struct tree *get_commit_tree_in_graph(struct repository *r,\n \t\t\t\t const struct commit *c);\n \n int get_bloom_filter_large_in_graph(struct commit_graph *g,\n-\t\t\t\t const struct commit *c);\n+\t\t\t\t const struct commit *c,\n+\t\t\t\t uint32_t max_changed_paths);\n \n struct commit_graph {\n \tconst unsigned char *data;\n@@ -120,6 +121,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags split_flags;\n+\tint max_new_filters;\n };\n \n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fed4929af3..571676cef2 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -291,4 +291,48 @@ test_expect_success 'Bloom generation does not recompute too-large filters' '\n \t)\n '\n \n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n+\t\t\t2 0 2\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=1\" \\\n+\t\t\t2 0 1\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills empty commits' '\n+\tgit init empty &&\n+\ttest_when_finished \"rm -fr empty\" &&\n+\t(\n+\t\tcd empty &&\n+\t\tfor i in $(test_seq 1 6)\n+\t\tdo\n+\t\t\tgit commit --allow-empty -m \"$i\"\n+\t\tdone &&\n+\n+\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t0 2 2 &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t2 2 2 &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 2 2 &&\n+\n+\t\t# Finally, make sure that once all commits have filters, that\n+\t\t# none are subsequently recomputed.\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t6 0 0\n+\t)\n+'\n+\n test_done\n-- \n2.27.0.2918.gc99a27ff8f\n"},{"id":"405020","messageId":"20200904085036.GA6209@szeder.dev","threadId":"53975","inReplyTo":"20200903164225.GA7905@nand.local","subject":"Re: [PATCH v3 14/14] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-09-04T08:50:36Z","receivedAt":"2020-09-04T08:50:48Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Sep 03, 2020 at 12:42:25PM -0400, Taylor Blau wrote:\n> On Wed, Aug 19, 2020 at 10:20:21AM +0200, SZEDER Gábor wrote:\n> > On Tue, Aug 11, 2020 at 04:52:14PM -0400, Taylor Blau wrote:\n> > > Introduce a command-line flag and configuration variable to fill in the\n> > > 'max_new_filters' variable introduced by the previous patch.\n> > >\n> > > The command-line option '--max-new-filters' takes precedence over\n> > > 'commitGraph.maxNewFilters', which is the default value.\n> > > '--no-max-new-filters' can also be provided, which sets the value back\n> > > to '-1', indicating that an unlimited number of new Bloom filters may be\n> > > generated. (OPT_INTEGER only allows setting the '--no-' variant back to\n> > > '0', hence a custom callback was used instead).\n> >\n> > Forgot the most important thing: Why? Please explain in the commit\n> > message why this option is necesary, what problems does it solve,\n> > how it is supposed to interact with other options and why so.\n> \n> This is already explained in detail in the patch 'commit-graph: add\n> large-filters bitmap chunk', although there is an error in the quoted\n> part of your email (which I wrote) which refers the reader to the\n> previous patch. The patch I'm actually referring two is the\n> twice-previous patch.\n\nThe proposed log message of that patch only briefly mentions what this\noption will do, and goes into detail about what problems it _causes_\nand how that new chunk is supposed to solve _those_ problems. It does\nnot explain what problems this new option is supposed to solve, and\nwhy would anyone want to use this option.\n\n> I'll fix that locally before re-sending.\n> \n> Thanks,\n> Taylor\n"},{"id":"405042","messageId":"30fbea4f-651e-11c9-c782-ab5fe645ec1c@gmail.com","threadId":"53975","inReplyTo":"cover.1599172907.git.me@ttaylorr.com","subject":"Re: [PATCH v4 00/14] more miscellaneous Bloom filter improvements","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-09-04T14:39:50Z","receivedAt":"2020-09-04T14:40:06Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/3/2020 6:45 PM, Taylor Blau wrote:\n> Things seem to have settled down since the review in v3, so I'm hoping\n> that this is what will end up being queued.\nI haven't paid close attention to this topic, but I took a close\nlook at v4 and found nothing to improve. I'm happy with this\nversion.\n\nThanks,\n-Stolee\n"},{"id":"405044","messageId":"20200904152021.GA15674@nand.local","threadId":"53975","inReplyTo":"ee0bc109f3db7caf59f55736744646a9ee047b01.1599172908.git.me@ttaylorr.com","subject":"Re: [PATCH v4 13/14] commit-graph: rename 'split_commit_graph_opts'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-04T15:20:21Z","receivedAt":"2020-09-04T15:20:30Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 03, 2020 at 06:46:57PM -0400, Taylor Blau wrote:\n> In the subsequent commit, additional options will be added to the\n> commit-graph API which have nothing to do with splitting.\n\nI have no idea how 'make *** [Makefile' snuck into this, but it\ndefinitely shouldn't be there. Let's use the below version instead.\n\n--- >8 ---\n\nSubject: [PATCH] commit-graph: rename 'split_commit_graph_opts'\n\nIn the subsequent commit, additional options will be added to the\ncommit-graph API which have nothing to do with splitting.\n\nRename the 'split_commit_graph_opts' structure to the more-generic\n'commit_graph_opts' to encompass both. Likewise, rename the 'flags'\nmember to instead be 'split_flags' to clarify that it only has to do\nwith the behavior implied by '--split'.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 20 ++++++++++----------\n commit-graph.c | 40 ++++++++++++++++++++--------------------\n commit-graph.h | 8 ++++----\n 3 files changed, 34 insertions(+), 34 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ba5584463f..f3243bd982 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,7 +119,7 @@ static int graph_verify(int argc, const char **argv)\n }\n\n extern int read_replace_refs;\n-static struct split_commit_graph_opts split_opts;\n+static struct commit_graph_opts write_opts;\n\n static int write_option_parse_split(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n@@ -187,24 +187,24 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n \t\t\tN_(\"enable computation for changed paths\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.split_flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\twrite_option_parse_split),\n-\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\tOPT_INTEGER(0, \"max-commits\", &write_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n-\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\tOPT_INTEGER(0, \"size-multiple\", &write_opts.size_multiple,\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n-\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n \t\tOPT_END(),\n \t};\n\n \topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n-\tsplit_opts.size_multiple = 2;\n-\tsplit_opts.max_commits = 0;\n-\tsplit_opts.expire_time = 0;\n+\twrite_opts.size_multiple = 2;\n+\twrite_opts.max_commits = 0;\n+\twrite_opts.expire_time = 0;\n\n \ttrace2_cmd_mode(\"write\");\n\n@@ -232,7 +232,7 @@ static int graph_write(int argc, const char **argv)\n \todb = find_odb(the_repository, opts.obj_dir);\n\n \tif (opts.reachable) {\n-\t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n+\t\tif (write_commit_graph_reachable(odb, flags, &write_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n@@ -261,7 +261,7 @@ static int graph_write(int argc, const char **argv)\n \t\t\t opts.stdin_packs ? &pack_indexes : NULL,\n \t\t\t opts.stdin_commits ? &commits : NULL,\n \t\t\t flags,\n-\t\t\t &split_opts))\n+\t\t\t &write_opts))\n \t\tresult = 1;\n\n cleanup:\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 68ffa6ec35..33fcf01a7a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1012,7 +1012,7 @@ struct write_commit_graph_context {\n \t\t changed_paths:1,\n \t\t order_by_pack:1;\n\n-\tconst struct split_commit_graph_opts *split_opts;\n+\tconst struct commit_graph_opts *opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n\n@@ -1353,8 +1353,8 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n\n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1554,7 +1554,7 @@ static int add_ref_to_set(const char *refname,\n\n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t\t\t const struct commit_graph_opts *opts)\n {\n \tstruct oidset commits = OIDSET_INIT;\n \tstruct refs_cb_data data;\n@@ -1571,7 +1571,7 @@ int write_commit_graph_reachable(struct object_directory *odb,\n \tstop_progress(&data.progress);\n\n \tresult = write_commit_graph(odb, NULL, &commits,\n-\t\t\t\t flags, split_opts);\n+\t\t\t\t flags, opts);\n\n \toidset_clear(&commits);\n \treturn result;\n@@ -1686,8 +1686,8 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n\n \tctx->num_extra_edges = 0;\n \tif (ctx->report_progress)\n@@ -1973,13 +1973,13 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tint max_commits = 0;\n \tint size_mult = 2;\n\n-\tif (ctx->split_opts) {\n-\t\tmax_commits = ctx->split_opts->max_commits;\n+\tif (ctx->opts) {\n+\t\tmax_commits = ctx->opts->max_commits;\n\n-\t\tif (ctx->split_opts->size_multiple)\n-\t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t\tif (ctx->opts->size_multiple)\n+\t\t\tsize_mult = ctx->opts->size_multiple;\n\n-\t\tflags = ctx->split_opts->flags;\n+\t\tflags = ctx->opts->split_flags;\n \t}\n\n \tg = ctx->r->objects->commit_graph;\n@@ -2157,8 +2157,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n\n-\tif (ctx->split_opts && ctx->split_opts->expire_time)\n-\t\texpire_time = ctx->split_opts->expire_time;\n+\tif (ctx->opts && ctx->opts->expire_time)\n+\t\texpire_time = ctx->opts->expire_time;\n \tif (!ctx->split) {\n \t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n@@ -2209,7 +2209,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t const struct commit_graph_opts *opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -2226,7 +2226,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n-\tctx->split_opts = split_opts;\n+\tctx->opts = opts;\n \tctx->total_bloom_filter_data_size = 0;\n\n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n@@ -2274,15 +2274,15 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\t}\n \t\t}\n\n-\t\tif (ctx->split_opts)\n-\t\t\treplace = ctx->split_opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n+\t\tif (ctx->opts)\n+\t\t\treplace = ctx->opts->split_flags & COMMIT_GRAPH_SPLIT_REPLACE;\n \t}\n\n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n\n-\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n-\t\tctx->oids.alloc = split_opts->max_commits;\n+\tif (ctx->split && opts && ctx->oids.alloc > opts->max_commits)\n+\t\tctx->oids.alloc = opts->max_commits;\n\n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->odb);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 9afb1477d5..fe798a4047 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -115,11 +115,11 @@ enum commit_graph_split_flags {\n \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n };\n\n-struct split_commit_graph_opts {\n+struct commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n-\tenum commit_graph_split_flags flags;\n+\tenum commit_graph_split_flags split_flags;\n };\n\n /*\n@@ -130,12 +130,12 @@ struct split_commit_graph_opts {\n */\n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t\t\t const struct commit_graph_opts *opts);\n int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t const struct commit_graph_opts *opts);\n\n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n\n--\n2.27.0.2918.gc99a27ff8f\n\n"},{"id":"405057","messageId":"bca1af61-e664-b89c-03f8-9481c670baf8@web.de","threadId":"53975","inReplyTo":"d7cbd4ca1a5dd63fb1d19ef97fac9765daa5ae03.1599172908.git.me@ttaylorr.com","subject":"Re: [PATCH v4 11/14] csum-file.h: introduce 'hashwrite_be64()'","fromName":"René Scharfe","fromEmail":"l.s.r@web.de","sentAt":"2020-09-04T20:18:38Z","receivedAt":"2020-09-04T20:19:24Z","isPatch":true,"sender":{"key":"l.s.r@web.de","avatar":"https://avatars.githubusercontent.com/u/26122331?v=4"},"body":"Am 04.09.20 um 00:46 schrieb Taylor Blau:\n> A small handful of writers who wish to encode 64-bit values in network\n> order have worked around the lack of such a helper by calling the 32-bit\n> variant twice.\n>\n> The subsequent commit will add another caller who wants to write a\n> 64-bit value. To ease their (and the existing caller's) pain, introduce\n> a helper to do just that, and convert existing call-sites.\n>\n> Suggested-by: Derrick Stolee \n> Signed-off-by: Taylor Blau \n> ---\n> commit-graph.c | 8 ++------\n> csum-file.h | 6 ++++++\n> midx.c | 3 +--\n> 3 files changed, 9 insertions(+), 8 deletions(-)\n>\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 35535f4192..01d791343a 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -1791,12 +1791,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n>\n> \tchunk_offset = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n> \tfor (i = 0; i <= num_chunks; i++) {\n> -\t\tuint32_t chunk_write[3];\n> -\n> -\t\tchunk_write[0] = htonl(chunks[i].id);\n> -\t\tchunk_write[1] = htonl(chunk_offset >> 32);\n> -\t\tchunk_write[2] = htonl(chunk_offset & 0xffffffff);\n> -\t\thashwrite(f, chunk_write, 12);\n> +\t\thashwrite_be32(f, chunks[i].id);\n> +\t\thashwrite_be64(f, chunk_offset);\n>\n> \t\tchunk_offset += chunks[i].size;\n> \t}\n> diff --git a/csum-file.h b/csum-file.h\n> index f9cbd317fb..b026ec7766 100644\n> --- a/csum-file.h\n> +++ b/csum-file.h\n> @@ -62,4 +62,10 @@ static inline void hashwrite_be32(struct hashfile *f, uint32_t data)\n> \thashwrite(f, &data, sizeof(data));\n> }\n>\n> +static inline void hashwrite_be64(struct hashfile *f, uint64_t data)\n> +{\n> +\thashwrite_be32(f, data >> 32);\n> +\thashwrite_be32(f, data & 0xffffffffUL);\n> +}\n> +\n> #endif\n> diff --git a/midx.c b/midx.c\n> index e9b2e1253a..32cc5fdc22 100644\n> --- a/midx.c\n> +++ b/midx.c\n> @@ -789,8 +789,7 @@ static size_t write_midx_large_offsets(struct hashfile *f, uint32_t nr_large_off\n> \t\tif (!(offset >> 31))\n> \t\t\tcontinue;\n>\n> -\t\thashwrite_be32(f, offset >> 32);\n> -\t\thashwrite_be32(f, offset & 0xffffffffUL);\n> +\t\thashwrite_be64(f, offset);\n> \t\twritten += 2 * sizeof(uint32_t);\n\n\"2 * sizeof(uint32_t)\" looks slightly out of sync with the hashwrite_be64()\ncall now; \"sizeof(uint64_t)\" would be more fitting.\n\n>\n> \t\tnr_large_offset--;\n>\n\nThere's also this potential caller:\n\nmidx.c=802=static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\nmidx.c:981: hashwrite_be32(f, chunk_ids[i]);\nmidx.c:982: hashwrite_be32(f, chunk_offsets[i] >> 32);\nmidx.c:983: hashwrite_be32(f, chunk_offsets[i]);\n\nNot sure it's worth a reroll, though.\n\n(I'd probably leave those conversions for a later series.)\n\nRené\n"},{"id":"405058","messageId":"20200904202226.GA21837@nand.local","threadId":"53975","inReplyTo":"bca1af61-e664-b89c-03f8-9481c670baf8@web.de","subject":"Re: [PATCH v4 11/14] csum-file.h: introduce 'hashwrite_be64()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-04T20:22:26Z","receivedAt":"2020-09-04T20:22:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 04, 2020 at 10:18:38PM +0200, René Scharfe wrote:\n> Am 04.09.20 um 00:46 schrieb Taylor Blau:\n> \"2 * sizeof(uint32_t)\" looks slightly out of sync with the hashwrite_be64()\n> call now; \"sizeof(uint64_t)\" would be more fitting.\n\nYeah, agreed.\n\n> >\n> > \t\tnr_large_offset--;\n> >\n>\n> There's also this potential caller:\n>\n> midx.c=802=static int write_midx_internal(const char *object_dir, struct multi_pack_index *m,\n> midx.c:981: hashwrite_be32(f, chunk_ids[i]);\n> midx.c:982: hashwrite_be32(f, chunk_offsets[i] >> 32);\n> midx.c:983: hashwrite_be32(f, chunk_offsets[i]);\n>\n> Not sure it's worth a reroll, though.\n>\n> (I'd probably leave those conversions for a later series.)\n\nAgreed. If we were earlier on, or there wasn't already a patch that I\nhad swapped out for a manual fixup after sending this v4, I'd certainly\nfold these in, but I think at this point it's easier to apply this\nseparately on top.\n\nThanks for pointing them out.\n\n> René\n\nThanks,\nTaylor\n"},{"id":"405064","messageId":"85h7sc9m0v.fsf@gmail.com","threadId":"53975","inReplyTo":"ba89a0cb837abc5fadbaa9514169636d85ee50cf.1599172908.git.me@ttaylorr.com","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2020-09-05T17:22:08Z","receivedAt":"2020-09-05T17:22:18Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Taylor Blau writes:\n\n[...]\n> While we're at it, instrument the new 'get_or_compute_bloom_filter()'\n> with two counters in the 'write_commit_graph_context' struct which store\n> the number of filters that we computed, and the number of those which\n> were too large to store.\n\n[...]\n> @@ -1414,12 +1433,25 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n> \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n> \n> \tfor (i = 0; i < ctx->commits.nr; i++) {\n> +\t\tint computed = 0;\n> \t\tstruct commit *c = sorted_commits[i];\n> -\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n> +\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n> +\t\t\tctx->r,\n> +\t\t\tc,\n> +\t\t\t1,\n> +\t\t\t&computed);\n> +\t\tif (computed) {\n> +\t\t\tctx->count_bloom_filter_computed++;\n> +\t\t\tif (filter && !filter->len)\n> +\t\t\t\tctx->count_bloom_filter_found_large++;\n\nHow do you distinguish between no changed paths stored because there\nwere no changes (which should not count as *_found_large), and no\nchanged paths stored because there were too many changes? If I remember\nit correctly in current implemetation both are represented as\nzero-length filter (no changed paths could have been represented as all\nzeros filter, too many changed paths could have been represented as all\nones filter).\n\nNo changes to store in filter can happen not only with `--allow-empty`\n(e.g. via interactive rebase), but also with merge where all changes\ncame from the second parent -- we are storing only changes to first\nparent, if I remember it correctly.\n\nThis is a minor issue, though.\n\n> +\t\t}\n> \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n> \t\tdisplay_progress(progress, i + 1);\n> \t}\n> \n> +\tif (trace2_is_enabled())\n> +\t\ttrace2_bloom_filter_write_statistics(ctx);\n> +\n> \tfree(sorted_commits);\n> \tstop_progress(&progress);\n> }\n\nBest,\n-- \nJakub Narębski\n"},{"id":"405066","messageId":"20200905173825.GA6820@nand.local","threadId":"53975","inReplyTo":"85h7sc9m0v.fsf@gmail.com","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-05T17:38:25Z","receivedAt":"2020-09-05T17:38:36Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 05, 2020 at 07:22:08PM +0200, Jakub Narębski wrote:\n> Taylor Blau writes:\n> How do you distinguish between no changed paths stored because there\n> were no changes (which should not count as *_found_large), and no\n> changed paths stored because there were too many changes? If I remember\n> it correctly in current implemetation both are represented as\n> zero-length filter (no changed paths could have been represented as all\n> zeros filter, too many changed paths could have been represented as all\n> ones filter).\n\nRight, once we get handed back a filter from\n'get_or_compute_bloom_filter()', we can't distinguish between (a) a\ncommit with too many changes to store in a single Bloom filter, and (b)\na commit with no changes at all.\n\nIt's unfortunate that callers can't pick between the two, but this\nimplementation is actually an improvement on the status-quo! Why?\nBecause right now we'll see an \"empty\" Bloom filter and recompute it\nbecause it's \"missing\", only to discover that it has no changes.\n\nWith this patch, we'll say \"this filter looks too large\", and stop\ncomputing it, because we have already gone through the effort to compute\nit once (and marked it in the BFXL chunk).\n\nNow, you could certainly argue that 'BFXL' could be called 'BFWC'\n(\"Bloom filter was computed\"), and then the \"was this filter too large\"\ncheck means that the commit was (a) marked in a BFWC chunk, and (b) has\nlength zero. I'm not necessarily opposed, but I think that this is\nprobably not worth it, since we're trying to disambiguate something that\nis inherently ambiguous. (That is, even with this new check, a\nlength-zero would still be thought to be \"too large\", since it was\ncomputed, and has length 0).\n\n> No changes to store in filter can happen not only with `--allow-empty`\n> (e.g. via interactive rebase), but also with merge where all changes\n> came from the second parent -- we are storing only changes to first\n> parent, if I remember it correctly.\n\nAgreed. And yes, Bloom filters store changes only to their commit's\nfirst parent.\n\n> This is a minor issue, though.\n\nThanks for raising it. I don't think that this is a show-stopper for\nthis series.\n\nThanks,\nTaylor\n"},{"id":"405067","messageId":"CANQwDwcWveN03h=Ex0GFo-Mm89UT8_HXFMQcZWdDx9MPZYYodQ@mail.gmail.com","threadId":"53975","inReplyTo":"20200905173825.GA6820@nand.local","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2020-09-05T17:50:01Z","receivedAt":"2020-09-05T17:50:45Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Hello,\n\nOn Sat, 5 Sep 2020 at 19:38, Taylor Blau wrote:\n> On Sat, Sep 05, 2020 at 07:22:08PM +0200, Jakub Narębski wrote:\n> > Taylor Blau writes:\n> >\n> > How do you distinguish between no changed paths stored because there\n> > were no changes (which should not count as *_found_large), and no\n> > changed paths stored because there were too many changes? If I remember\n> > it correctly in current implementation both are represented as\n> > zero-length filter (no changed paths could have been represented as all\n> > zeros filter, too many changed paths could have been represented as all\n> > ones filter).\n>\n> Right, once we get handed back a filter from\n> 'get_or_compute_bloom_filter()', we can't distinguish between (a) a\n> commit with too many changes to store in a single Bloom filter, and (b)\n> a commit with no changes at all.\n\nWe could change how we store either no-changes Bloom filter (as all\nzeros minimal size filter), or too-many-changes Bloom filter (as all\nones, i.e. max unsigned value, minimal size filter). This change would\nnot require to change any user of Bloom filter.\n\n> It's unfortunate that callers can't pick between the two, but this\n> implementation is actually an improvement on the status-quo! Why?\n> Because right now we'll see an \"empty\" Bloom filter and recompute it\n> because it's \"missing\", only to discover that it has no changes.\n>\n> With this patch, we'll say \"this filter looks too large\", and stop\n> computing it, because we have already gone through the effort to compute\n> it once (and marked it in the BFXL chunk).\n\nCan we use this when computing trace2 values?\n\n[...]\n> > This is a minor issue, though.\n>\n> Thanks for raising it. I don't think that this is a show-stopper for\n> this series.\n\nI agree.\n\nBest,\n-- \nJakub Narębski\n"},{"id":"405068","messageId":"20200905180120.GB6820@nand.local","threadId":"53975","inReplyTo":"CANQwDwcWveN03h=Ex0GFo-Mm89UT8_HXFMQcZWdDx9MPZYYodQ@mail.gmail.com","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-05T18:01:20Z","receivedAt":"2020-09-05T18:01:31Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 05, 2020 at 07:50:01PM +0200, Jakub Narębski wrote:\n> Hello,\n>\n> On Sat, 5 Sep 2020 at 19:38, Taylor Blau wrote:\n> > Right, once we get handed back a filter from\n> > 'get_or_compute_bloom_filter()', we can't distinguish between (a) a\n> > commit with too many changes to store in a single Bloom filter, and (b)\n> > a commit with no changes at all.\n>\n> We could change how we store either no-changes Bloom filter (as all\n> zeros minimal size filter), or too-many-changes Bloom filter (as all\n> ones, i.e. max unsigned value, minimal size filter). This change would\n> not require to change any user of Bloom filter.\n\nI don't think that's true. Say that we changed the empty Bloom filter to\nbe encoded as only having the most-significant bit set. First, we'd have\nto write a Bloom filter where we didn't have to before. But the real\nissue is that commit-graph files generated with new clients would\nsuddenly be unreadable by old clients.\n>\n> > It's unfortunate that callers can't pick between the two, but this\n> > implementation is actually an improvement on the status-quo! Why?\n> > Because right now we'll see an \"empty\" Bloom filter and recompute it\n> > because it's \"missing\", only to discover that it has no changes.\n> >\n> > With this patch, we'll say \"this filter looks too large\", and stop\n> > computing it, because we have already gone through the effort to compute\n> > it once (and marked it in the BFXL chunk).\n>\n> Can we use this when computing trace2 values?\n\nWe could, but I don't think it's absolutely necessary. The test coverage\nin t4216 gives us enough confidence already.\n\n> [...]\n> > > This is a minor issue, though.\n> >\n> > Thanks for raising it. I don't think that this is a show-stopper for\n> > this series.\n>\n> I agree.\n\nThanks for your input!\n\n> Best,\n> --\n> Jakub Narębski\n\nThanks,\nTaylor\n"},{"id":"405069","messageId":"CANQwDwcD5XD3e7ar=8HPupobOZGDN4bgQJBS1Ad1m2NDx8eurw@mail.gmail.com","threadId":"53975","inReplyTo":"20200905180120.GB6820@nand.local","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Jakub Narębski","fromEmail":"jnareb@gmail.com","sentAt":"2020-09-05T18:18:40Z","receivedAt":"2020-09-05T18:19:20Z","isPatch":true,"sender":{"key":"jnareb@gmail.com","avatar":"https://avatars.githubusercontent.com/u/2706?v=4"},"body":"Hello,\n\nOn Sat, 5 Sep 2020 at 20:01, Taylor Blau wrote:\n> On Sat, Sep 05, 2020 at 07:50:01PM +0200, Jakub Narębski wrote:\n> > On Sat, 5 Sep 2020 at 19:38, Taylor Blau wrote:\n> > >\n> > > Right, once we get handed back a filter from\n> > > 'get_or_compute_bloom_filter()', we can't distinguish between (a) a\n> > > commit with too many changes to store in a single Bloom filter, and (b)\n> > > a commit with no changes at all.\n> >\n> > We could change how we store either no-changes Bloom filter (as all\n> > zeros minimal size filter), or too-many-changes Bloom filter (as all\n> > ones, i.e. max unsigned value, minimal size filter). This change would\n> > not require to change any user of Bloom filter.\n>\n> I don't think that's true. Say that we changed the empty Bloom filter to\n> be encoded as only having the most-significant bit set. First, we'd have\n> to write a Bloom filter where we didn't have to before.\n\nThat's true.\n\n> But the real\n> issue is that commit-graph files generated with new clients would\n> suddenly be unreadable by old clients.\n\nActually it is, at least in the form that I have proposed. The Bloom filter\nwhich has all bits set to zero would for every possible path reply that\nthe path is not in set. Old clients would therefore work without changes.\nTherefore this is good representation of no-changes Bloom filter.\n\nThe Bloom filter which has all bits set to one would for every possible\npath reply that the path is maybe in set. This is a good alternative\nrepresentation of too-many-changes Bloom filter. Again, old clients\nwould work without changes.\n\n[...]\n\nBest,\n-- \nJakub Narębski\n"},{"id":"405071","messageId":"20200905183854.GC6820@nand.local","threadId":"53975","inReplyTo":"CANQwDwcD5XD3e7ar=8HPupobOZGDN4bgQJBS1Ad1m2NDx8eurw@mail.gmail.com","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-05T18:38:54Z","receivedAt":"2020-09-05T18:39:02Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 05, 2020 at 08:18:40PM +0200, Jakub Narębski wrote:\n> Hello,\n>\n> On Sat, 5 Sep 2020 at 20:01, Taylor Blau wrote:\n> > But the real issue is that commit-graph files generated with new\n> > clients would suddenly be unreadable by old clients.\n>\n> Actually it is, at least in the form that I have proposed. The Bloom filter\n> which has all bits set to zero would for every possible path reply that\n> the path is not in set. Old clients would therefore work without changes.\n> Therefore this is good representation of no-changes Bloom filter.\n>\n> The Bloom filter which has all bits set to one would for every possible\n> path reply that the path is maybe in set. This is a good alternative\n> representation of too-many-changes Bloom filter. Again, old clients\n> would work without changes.\n\nThat's a very interesting thought. To be honest, I'm not crazy about\ngenerating our own Bloom filters that have special meaning (i.e., that\neven though they are interpreted the same way as any other filter, they\nare specially-crafted to carry a certain meaning). Your proposal also\nmeans that commit-graph files are going to get bigger, which is\nsomething that you may or may not care about.\n\nOn the other hand, it does get rid of the BFXL chunk, which certainly\nisn't the most elegant thing.\n\nI don't know. I think my biggest objection is the size: we use the BIDX\nchunk today to avoid having to write the length-zero Bloom filters; your\nscheme would force us to write every filter. On the other hand, we could\ncontinue to avoid writing length-zero filters, so long as the\ncommit-graph indicates that it knows this optimization.\n\nStolee: what do you think?\n\n> Best,\n> --\n> Jakub Narębski\n\nThanks,\nTaylor\n"},{"id":"405072","messageId":"20200905185534.GA7416@nand.local","threadId":"53975","inReplyTo":"20200905183854.GC6820@nand.local","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-05T18:55:34Z","receivedAt":"2020-09-05T18:55:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 05, 2020 at 02:38:54PM -0400, Taylor Blau wrote:\n> I don't know. I think my biggest objection is the size: we use the BIDX\n> chunk today to avoid having to write the length-zero Bloom filters; your\n> scheme would force us to write every filter. On the other hand, we could\n> continue to avoid writing length-zero filters, so long as the\n> commit-graph indicates that it knows this optimization.\n\nThinking about it a little bit more, I'm pretty sure that this isn't as\neasy as it sounds. Say that we:\n\n - continued to encode length-zero Bloom filters as equal adjacent\n entries in the BIDX, but reserve the length-zero filter for commits\n with no changed-paths, _or_ commits whose Bloom filters have not yet\n been computed\n\n - write \"too large\" Bloom filters (i.e., commits with >= 512 changed\n paths in a diff to their first parent) as a non-empty Bloom filter\n with all bits set high.\n\nI think we're still no better off today than before, because of the\noverloading in the length-zero Bloom filter. Because we would treat\nempty filters the same as ones that haven't been computed, we would\nrecompute empty filters, and that would count against our\n'--max-new-filters' budget.\n\nI don't see a non-convoluted way to split the overloaded length-zero\ncase into something that is distinguishable without a format extension.\nBy the way, I think that your idea is good, and that it would be\nworkable without the existing structure of the BIDX chunk (which itself\nmade sense at the time that it was written).\n\nSo, I really want your idea to work. But, I think that ultimately the\nBFXL chunk is a more straightforward path forward.\n\n\nThanks,\nTaylor\n"},{"id":"405073","messageId":"20200905190449.GB6209@szeder.dev","threadId":"53975","inReplyTo":"20200905185534.GA7416@nand.local","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-09-05T19:04:50Z","receivedAt":"2020-09-05T19:05:43Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Sat, Sep 05, 2020 at 02:55:34PM -0400, Taylor Blau wrote:\n> On Sat, Sep 05, 2020 at 02:38:54PM -0400, Taylor Blau wrote:\n> > I don't know. I think my biggest objection is the size: we use the BIDX\n> > chunk today to avoid having to write the length-zero Bloom filters; your\n> > scheme would force us to write every filter. On the other hand, we could\n> > continue to avoid writing length-zero filters, so long as the\n> > commit-graph indicates that it knows this optimization.\n> \n> Thinking about it a little bit more, I'm pretty sure that this isn't as\n> easy as it sounds. Say that we:\n> \n> - continued to encode length-zero Bloom filters as equal adjacent\n> entries in the BIDX, but reserve the length-zero filter for commits\n> with no changed-paths, _or_ commits whose Bloom filters have not yet\n> been computed\n\nNo, use zero-length filters for commits whose Bloom filters have not\nyet been computed, and use a one-byte all zero bits Bloom filter for\ncommits with no modified paths.\n\nAnd this is exactly what I proposed earlier.\n\n> - write \"too large\" Bloom filters (i.e., commits with >= 512 changed\n> paths in a diff to their first parent) as a non-empty Bloom filter\n> with all bits set high.\n> \n> I think we're still no better off today than before, because of the\n> overloading in the length-zero Bloom filter. Because we would treat\n> empty filters the same as ones that haven't been computed, we would\n> recompute empty filters, and that would count against our\n> '--max-new-filters' budget.\n> \n> I don't see a non-convoluted way to split the overloaded length-zero\n> case into something that is distinguishable without a format extension.\n\nSee above, no format extension needed.\n\n> By the way, I think that your idea is good, and that it would be\n> workable without the existing structure of the BIDX chunk (which itself\n> made sense at the time that it was written).\n> \n> So, I really want your idea to work. But, I think that ultimately the\n> BFXL chunk is a more straightforward path forward.\n> \n> \n> Thanks,\n> Taylor\n"},{"id":"405083","messageId":"20200905194926.GA7964@nand.local","threadId":"53975","inReplyTo":"20200905190449.GB6209@szeder.dev","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-05T19:49:26Z","receivedAt":"2020-09-05T19:49:34Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Sat, Sep 05, 2020 at 09:04:50PM +0200, SZEDER Gábor wrote:\n> On Sat, Sep 05, 2020 at 02:55:34PM -0400, Taylor Blau wrote:\n> > On Sat, Sep 05, 2020 at 02:38:54PM -0400, Taylor Blau wrote:\n> > > I don't know. I think my biggest objection is the size: we use the BIDX\n> > > chunk today to avoid having to write the length-zero Bloom filters; your\n> > > scheme would force us to write every filter. On the other hand, we could\n> > > continue to avoid writing length-zero filters, so long as the\n> > > commit-graph indicates that it knows this optimization.\n> >\n> > Thinking about it a little bit more, I'm pretty sure that this isn't as\n> > easy as it sounds. Say that we:\n> >\n> > - continued to encode length-zero Bloom filters as equal adjacent\n> > entries in the BIDX, but reserve the length-zero filter for commits\n> > with no changed-paths, _or_ commits whose Bloom filters have not yet\n> > been computed\n>\n> No, use zero-length filters for commits whose Bloom filters have not\n> yet been computed, and use a one-byte all zero bits Bloom filter for\n> commits with no modified paths.\n>\n> And this is exactly what I proposed earlier.\n\nFair enough, I bet that would work.\n\nJunio, let's eject this series while I try to see if SZEDER's idea is\nworkable.\n\n> > I don't see a non-convoluted way to split the overloaded length-zero\n> > case into something that is distinguishable without a format extension.\n>\n> See above, no format extension needed.\n\nWhere?\n\nThanks,\nTaylor\n"},{"id":"405101","messageId":"xmqq8sdm4lpa.fsf@gitster.c.googlers.com","threadId":"53975","inReplyTo":"20200905194926.GA7964@nand.local","subject":"Re: [PATCH v4 07/14] bloom: split 'get_bloom_filter()' in two","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-06T21:52:33Z","receivedAt":"2020-09-06T21:52:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n>> And this is exactly what I proposed earlier.\n>\n> Fair enough, I bet that would work.\n>\n> Junio, let's eject this series while I try to see if SZEDER's idea is\n> workable.\n\nThanks, I'll mark the topic as such for now.\n"}]}