{"thread":{"id":"54216","subject":"[PATCH 03/12] commit-graph: pass a 'struct repository *' in more places","startedAt":"2020-09-09T15:29:37Z","lastAt":"2020-09-18T17:43:57Z","messageCount":75,"participants":["Taylor Blau","Jeff King","Derrick Stolee","Junio C Hamano","SZEDER Gábor"],"isPatch":true,"patchVersion":1,"patchTotal":12},"messages":[{"id":"405248","messageId":"0817b0f618700ced34af6c597eee7312993b0d1f.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 03/12] commit-graph: pass a 'struct repository *' in more places","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:22:56Z","receivedAt":"2020-09-09T15:29:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a future commit, some commit-graph internals will want access to\n'r->settings', but we only have the 'struct object_directory *'\ncorresponding to that repository.\n\nAdd an additional parameter to pass the repository around in more\nplaces.\n\nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 2 +-\n commit-graph.c | 17 ++++++++++-------\n commit-graph.h | 6 ++++--\n fuzz-commit-graph.c | 5 +++--\n 4 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 523501f217..ba5584463f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -106,7 +106,7 @@ static int graph_verify(int argc, const char **argv)\n \tFREE_AND_NULL(graph_name);\n \n \tif (open_ok)\n-\t\tgraph = load_commit_graph_one_fd_st(fd, &st, odb);\n+\t\tgraph = load_commit_graph_one_fd_st(the_repository, fd, &st, odb);\n \telse\n \t\tgraph = read_commit_graph_one(the_repository, odb);\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6a36ed0b06..72a838bd00 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -231,7 +231,8 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st)\n \treturn 1;\n }\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tvoid *graph_map;\n@@ -247,7 +248,7 @@ struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n \t}\n \tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n-\tret = parse_commit_graph(graph_map, graph_size);\n+\tret = parse_commit_graph(r, graph_map, graph_size);\n \n \tif (ret)\n \t\tret->odb = odb;\n@@ -287,7 +288,8 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n \tconst unsigned char *data, *chunk_lookup;\n \tuint32_t i;\n@@ -452,7 +454,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n \treturn NULL;\n }\n \n-static struct commit_graph *load_commit_graph_one(const char *graph_file,\n+static struct commit_graph *load_commit_graph_one(struct repository *r,\n+\t\t\t\t\t\t const char *graph_file,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \n@@ -464,7 +467,7 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file,\n \tif (!open_ok)\n \t\treturn NULL;\n \n-\tg = load_commit_graph_one_fd_st(fd, &st, odb);\n+\tg = load_commit_graph_one_fd_st(r, fd, &st, odb);\n \n \tif (g)\n \t\tg->filename = xstrdup(graph_file);\n@@ -476,7 +479,7 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tchar *graph_name = get_commit_graph_filename(odb);\n-\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \tfree(graph_name);\n \n \treturn g;\n@@ -557,7 +560,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \t\tvalid = 0;\n \t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\t\tchar *graph_name = get_split_graph_filename(odb, line.buf);\n-\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \n \t\t\tfree(graph_name);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 0677dd1031..d9acb22bac 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,13 @@ struct commit_graph {\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb);\n struct commit_graph *read_commit_graph_one(struct repository *r,\n \t\t\t\t\t struct object_directory *odb);\n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n /*\n * Return 1 if and only if the repository has a commit-graph\ndiff --git a/fuzz-commit-graph.c b/fuzz-commit-graph.c\nindex 430817214d..e7cf6d5b0f 100644\n--- a/fuzz-commit-graph.c\n+++ b/fuzz-commit-graph.c\n@@ -1,7 +1,8 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n \n@@ -10,7 +11,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \tstruct commit_graph *g;\n \n \tinitialize_the_repository();\n-\tg = parse_commit_graph((void *)data, size);\n+\tg = parse_commit_graph(the_repository, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\n \n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405249","messageId":"2816028f44e9d9dde8f360355e9a61b8f2d26dd9.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 11/12] commit-graph: rename 'split_commit_graph_opts'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:23:55Z","receivedAt":"2020-09-09T15:32:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the subsequent commit, additional options will be added to the\ncommit-graph API which have nothing to do with splitting.\n\nRename the 'split_commit_graph_opts' structure to the more-generic\n'commit_graph_opts' to encompass both. Likewise, rename the 'flags'\nmember to instead be 'split_flags' to clarify that it only has to do\nwith the behavior implied by '--split'.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 20 ++++++++++----------\n commit-graph.c | 40 ++++++++++++++++++++--------------------\n commit-graph.h | 8 ++++----\n 3 files changed, 34 insertions(+), 34 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ba5584463f..f3243bd982 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,7 +119,7 @@ static int graph_verify(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n-static struct split_commit_graph_opts split_opts;\n+static struct commit_graph_opts write_opts;\n \n static int write_option_parse_split(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n@@ -187,24 +187,24 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n \t\t\tN_(\"enable computation for changed paths\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.split_flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\twrite_option_parse_split),\n-\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\tOPT_INTEGER(0, \"max-commits\", &write_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n-\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\tOPT_INTEGER(0, \"size-multiple\", &write_opts.size_multiple,\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n-\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n \t\tOPT_END(),\n \t};\n \n \topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n-\tsplit_opts.size_multiple = 2;\n-\tsplit_opts.max_commits = 0;\n-\tsplit_opts.expire_time = 0;\n+\twrite_opts.size_multiple = 2;\n+\twrite_opts.max_commits = 0;\n+\twrite_opts.expire_time = 0;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -232,7 +232,7 @@ static int graph_write(int argc, const char **argv)\n \todb = find_odb(the_repository, opts.obj_dir);\n \n \tif (opts.reachable) {\n-\t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n+\t\tif (write_commit_graph_reachable(odb, flags, &write_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n@@ -261,7 +261,7 @@ static int graph_write(int argc, const char **argv)\n \t\t\t opts.stdin_packs ? &pack_indexes : NULL,\n \t\t\t opts.stdin_commits ? &commits : NULL,\n \t\t\t flags,\n-\t\t\t &split_opts))\n+\t\t\t &write_opts))\n \t\tresult = 1;\n \n cleanup:\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d402743e61..dcc27b74e3 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -968,7 +968,7 @@ struct write_commit_graph_context {\n \t\t changed_paths:1,\n \t\t order_by_pack:1;\n \n-\tconst struct split_commit_graph_opts *split_opts;\n+\tconst struct commit_graph_opts *opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \n@@ -1292,8 +1292,8 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1490,7 +1490,7 @@ static int add_ref_to_set(const char *refname,\n \n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t\t\t const struct commit_graph_opts *opts)\n {\n \tstruct oidset commits = OIDSET_INIT;\n \tstruct refs_cb_data data;\n@@ -1507,7 +1507,7 @@ int write_commit_graph_reachable(struct object_directory *odb,\n \tstop_progress(&data.progress);\n \n \tresult = write_commit_graph(odb, NULL, &commits,\n-\t\t\t\t flags, split_opts);\n+\t\t\t\t flags, opts);\n \n \toidset_clear(&commits);\n \treturn result;\n@@ -1622,8 +1622,8 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tctx->num_extra_edges = 0;\n \tif (ctx->report_progress)\n@@ -1908,13 +1908,13 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tint max_commits = 0;\n \tint size_mult = 2;\n \n-\tif (ctx->split_opts) {\n-\t\tmax_commits = ctx->split_opts->max_commits;\n+\tif (ctx->opts) {\n+\t\tmax_commits = ctx->opts->max_commits;\n \n-\t\tif (ctx->split_opts->size_multiple)\n-\t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t\tif (ctx->opts->size_multiple)\n+\t\t\tsize_mult = ctx->opts->size_multiple;\n \n-\t\tflags = ctx->split_opts->flags;\n+\t\tflags = ctx->opts->split_flags;\n \t}\n \n \tg = ctx->r->objects->commit_graph;\n@@ -2092,8 +2092,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n \n-\tif (ctx->split_opts && ctx->split_opts->expire_time)\n-\t\texpire_time = ctx->split_opts->expire_time;\n+\tif (ctx->opts && ctx->opts->expire_time)\n+\t\texpire_time = ctx->opts->expire_time;\n \tif (!ctx->split) {\n \t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n@@ -2144,7 +2144,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t const struct commit_graph_opts *opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -2161,7 +2161,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n-\tctx->split_opts = split_opts;\n+\tctx->opts = opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n@@ -2209,15 +2209,15 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\t}\n \t\t}\n \n-\t\tif (ctx->split_opts)\n-\t\t\treplace = ctx->split_opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n+\t\tif (ctx->opts)\n+\t\t\treplace = ctx->opts->split_flags & COMMIT_GRAPH_SPLIT_REPLACE;\n \t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n-\t\tctx->oids.alloc = split_opts->max_commits;\n+\tif (ctx->split && opts && ctx->oids.alloc > opts->max_commits)\n+\t\tctx->oids.alloc = opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->odb);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d9acb22bac..b7914b0a7a 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -105,11 +105,11 @@ enum commit_graph_split_flags {\n \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n };\n \n-struct split_commit_graph_opts {\n+struct commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n-\tenum commit_graph_split_flags flags;\n+\tenum commit_graph_split_flags split_flags;\n };\n \n /*\n@@ -120,12 +120,12 @@ struct split_commit_graph_opts {\n */\n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t\t\t const struct commit_graph_opts *opts);\n int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t const struct commit_graph_opts *opts);\n \n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n \n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405254","messageId":"4ff11cec37d17d788a3ee076b7c3de1c873a5fbd.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:24:00Z","receivedAt":"2020-09-09T17:17:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a command-line flag and configuration variable to specify the\nmaximum number of new Bloom filters that a 'git commit-graph write' is\nwilling to compute from scratch.\n\nPrior to this patch, a commit-graph write with '--changed-paths' would\ncompute Bloom filters for all selected commits which haven't already\nbeen computed (i.e., by a previous commit-graph write with '--split'\nsuch that a roll-up or replacement is performed).\n\nThis behavior can cause prohibitively-long commit-graph writes for a\nvariety of reasons:\n\n * There may be lots of filters whose diffs take a long time to\n generate (for example, they have close to the maximum number of\n changes, diffing itself takes a long time, etc).\n\n * Old-style commit-graphs (which encode filters with too many entries\n as not having been computed at all) cause us to waste time\n recomputing filters that appear to have not been computed only to\n discover that they are too-large.\n\nThis can make the upper-bound of the time it takes for 'git commit-graph\nwrite --changed-paths' to be rather unpredictable.\n\nTo make this command behave more predictably, introduce\n'--max-new-filters=' to allow computing at most '' Bloom filters\nfrom scratch. This lets \"computing\" already-known filters proceed\nquickly, while bounding the number of slow tasks that Git is willing to\ndo.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 4 +++\n Documentation/git-commit-graph.txt | 6 ++++\n bloom.c | 7 ++--\n builtin/commit-graph.c | 39 ++++++++++++++++++--\n commit-graph.c | 9 +++--\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 53 ++++++++++++++++++++++++++++\n 7 files changed, 110 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex cff0797b54..4582c39fc4 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitGraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..81a2e65903 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,12 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Commits whose filters are not calculated are stored as a\n+length zero Bloom filter, and their bit is marked in the `BFXL` chunk.\n+Overrides the `commitGraph.maxNewFilters` configuration.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex 194b6ab8ad..022dd6e0f9 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -197,12 +197,11 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n-\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n+\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n \n-\tif (filter->data)\n+\tif (filter->data && filter->len)\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f3243bd982..e7a1539b08 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n \n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n \n@@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -270,6 +293,16 @@ static int graph_write(int argc, const char **argv)\n \treturn result;\n }\n \n+static int git_commit_graph_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, \"commitgraph.maxnewfilters\")) {\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_commit_graph_options[] = {\n@@ -283,7 +316,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_commit_graph_usage,\n \t\t\t\t builtin_commit_graph_options);\n \n-\tgit_config(git_default_config, NULL);\n+\tgit_config(git_commit_graph_config, &opts);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_commit_graph_options,\n \t\t\t builtin_commit_graph_usage,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex dcc27b74e3..1d9f8cc7e9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1422,6 +1422,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tstruct commit **sorted_commits;\n+\tint max_new_filters;\n \n \tinit_bloom_filters();\n \n@@ -1438,13 +1439,16 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \telse\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n \n+\tmax_new_filters = ctx->opts && ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tenum bloom_filter_computed computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\tctx->r,\n \t\t\tc,\n-\t\t\t1,\n+\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n \t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n@@ -1455,7 +1459,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n \t\t\tctx->count_bloom_filter_not_computed++;\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\tctx->total_bloom_filter_data_size += filter\n+\t\t\t? sizeof(unsigned char) * filter->len : 0;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex b7914b0a7a..a22bd86701 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -110,6 +110,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags split_flags;\n+\tint max_new_filters;\n };\n \n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex a56327ffd4..24deb8104a 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -287,5 +287,58 @@ test_expect_success 'correctly report commits with no changed paths' '\n \t\tgrep \"\\\"filter_trunc_large\\\":0\" trace\n \t)\n '\n+test_bloom_filters_computed () {\n+\tcommit_graph_args=$1\n+\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n+\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write \\\n+\t\t$commit_graph_args &&\n+\tgrep \"\\\"filter_not_computed\\\":$2\" \"$TRASH_DIRECTORY/trace.event\" &&\n+\tgrep \"\\\"filter_trunc_large\\\":$3\" \"$TRASH_DIRECTORY/trace.event\" &&\n+\tgrep \"\\\"filter_computed\\\":$4\" \"$TRASH_DIRECTORY/trace.event\"\n+}\n+\n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n+\t\t\t3 0 2\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=1\" \\\n+\t\t\t4 0 1\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills empty commits' '\n+\tgit init empty &&\n+\ttest_when_finished \"rm -fr empty\" &&\n+\t(\n+\t\tcd empty &&\n+\t\tfor i in $(test_seq 1 6)\n+\t\tdo\n+\t\t\tgit commit --allow-empty -m \"$i\"\n+\t\tdone &&\n+\n+\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 0 2 &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 0 2 &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 0 2 &&\n+\n+\t\t# Finally, make sure that once all commits have filters, that\n+\t\t# none are subsequently recomputed.\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t6 0 0\n+\t)\n+'\n \n test_done\n-- \n2.28.0.462.g4ff11cec37\n"},{"id":"405255","messageId":"809972ba5f261c72146efa0461c19cb34daaa513.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 04/12] t/helper/test-read-graph.c: prepare repo settings","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:23:03Z","receivedAt":"2020-09-09T17:18:40Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The read-graph test-tool is used by a number of the commit-graph test to\nassert various properties about a commit-graph. Previously, this program\nnever ran 'prepare_repo_settings()'. There was no need to do so, since\nnone of the commit-graph machinery is affected by the repo settings.\n\nIn the next patch, the commit-graph machinery's behavior will become\ndependent on the repo settings, and so loading them before running the\nrest of the test tool is critical.\n\nAs such, teach the test tool to call 'prepare_repo_settings()'.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 6d0c962438..5f585a1725 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -12,11 +12,12 @@ int cmd__read_graph(int argc, const char **argv)\n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n \n+\tprepare_repo_settings(the_repository);\n+\n \tgraph = read_commit_graph_one(the_repository, odb);\n \tif (!graph)\n \t\treturn 1;\n \n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405256","messageId":"63c4a4901b6d695a29a44d0c734eea04087e71ca.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 05/12] commit-graph: respect 'commitGraph.readChangedPaths'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:23:10Z","receivedAt":"2020-09-09T17:18:44Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Git uses the 'core.commitGraph' configuration value to control whether\nor not the commit graph is used when parsing commits or performing a\ntraversal.\n\nNow that commit-graphs can also contain a section for changed-path Bloom\nfilters, administrators that already have commit-graphs may find it\nconvenient to use those graphs without relying on their changed-path\nBloom filters. This can happen, for example, during a staged roll-out,\nor in the event of an incident.\n\nIntroduce 'commitGraph.readChangedPaths' to control whether or not Bloom\nfilters are read. Note that this configuration is independent from both:\n\n - 'core.commitGraph', to allow flexibility in using all parts of a\n commit-graph _except_ for its Bloom filters.\n\n - The '--changed-paths' option for 'git commit-graph write', to allow\n reading and writing Bloom filters to be controlled independently.\n\nWhen the variable is set, pretend as if no Bloom data was specified at\nall. This avoids adding additional special-casing outside of the\ncommit-graph internals.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 ++\n Documentation/config/commitgraph.txt | 4 ++++\n commit-graph.c | 6 ++++--\n repo-settings.c | 3 +++\n repository.h | 1 +\n t/t4216-log-bloom.sh | 4 +++-\n 6 files changed, 17 insertions(+), 3 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 3042d80978..770ae79b82 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -340,6 +340,8 @@ include::config/column.txt[]\n \n include::config/commit.txt[]\n \n+include::config/commitgraph.txt[]\n+\n include::config/credential.txt[]\n \n include::config/completion.txt[]\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nnew file mode 100644\nindex 0000000000..cff0797b54\n--- /dev/null\n+++ b/Documentation/config/commitgraph.txt\n@@ -0,0 +1,4 @@\n+commitGraph.readChangedPaths::\n+\tIf true, then git will use the changed-path Bloom filters in the\n+\tcommit-graph file (if it exists, and they are present). Defaults to\n+\ttrue. See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 72a838bd00..ea54d108b9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -327,6 +327,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n+\tprepare_repo_settings(r);\n+\n \tgraph = alloc_commit_graph();\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -403,14 +405,14 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n \t\t\tif (graph->chunk_bloom_indexes)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n \t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n \t\t\tbreak;\n \n \t\tcase GRAPH_CHUNKID_BLOOMDATA:\n \t\t\tif (graph->chunk_bloom_data)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n \t\t\t\tuint32_t hash_version;\n \t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n \t\t\t\thash_version = get_be32(data + chunk_offset);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex aa61a35338..88ccce2036 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -17,9 +17,12 @@ void prepare_repo_settings(struct repository *r)\n \n \tif (!repo_config_get_bool(r, \"core.commitgraph\", &value))\n \t\tr->settings.core_commit_graph = value;\n+\tif (!repo_config_get_bool(r, \"commitgraph.readchangedpaths\", &value))\n+\t\tr->settings.commit_graph_read_changed_paths = value;\n \tif (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n \t\tr->settings.gc_write_commit_graph = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_commit_graph, 1);\n+\tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n \n \tif (!repo_config_get_int(r, \"index.version\", &value))\ndiff --git a/repository.h b/repository.h\nindex 628c834367..bacf843d46 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -30,6 +30,7 @@ struct repo_settings {\n \tint initialized;\n \n \tint core_commit_graph;\n+\tint commit_graph_read_changed_paths;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex cd89c75002..fc7693806c 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -95,7 +95,9 @@ do\n \t\t \"--ancestry-path side..master\"\n \tdo\n \t\ttest_expect_success \"git log option: $option for path: $path\" '\n-\t\t\ttest_bloom_filters_used \"$option -- $path\"\n+\t\t\ttest_bloom_filters_used \"$option -- $path\" &&\n+\t\t\ttest_config commitgraph.readChangedPaths false &&\n+\t\t\ttest_bloom_filters_not_used \"$option -- $path\"\n \t\t'\n \tdone\n done\n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405257","messageId":"2841ecf152fc42a9955e7609c174959d525f0830.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 08/12] bloom: use provided 'struct bloom_filter_settings'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:23:32Z","receivedAt":"2020-09-09T17:18:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When 'get_or_compute_bloom_filter()' needs to compute a Bloom filter\nfrom scratch, it looks to the default 'struct bloom_filter_settings' in\norder to determine the maximum number of changed paths, number of bits\nper entry, and so on.\n\nAll of these values have so far been constant, and so there was no need\nto pass in a pointer from the caller (eg., the one that is stored in the\n'struct write_commit_graph_context').\n\nStart passing in a 'struct bloom_filter_settings *' instead of using the\ndefault values to respect graph-specific settings (eg., in the case of\nsetting 'GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS').\n\nIn order to have an initialized value for these settings, move its\ninitialization to earlier in the commit-graph write.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 13 ++++++-------\n bloom.h | 3 ++-\n commit-graph.c | 21 ++++++++++-----------\n t/helper/test-bloom.c | 1 +\n 4 files changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 393c61b4bc..2d6aef9098 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -180,13 +180,12 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t enum bloom_filter_computed *computed)\n {\n \tstruct bloom_filter *filter;\n-\tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \tint i;\n \tstruct diff_options diffopt;\n-\tint max_changes = 512;\n \n \tif (computed)\n \t\t*computed = BLOOM_NOT_COMPUTED;\n@@ -211,7 +210,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n \tdiffopt.detect_rename = 0;\n-\tdiffopt.max_changes = max_changes;\n+\tdiffopt.max_changes = settings->max_changed_paths;\n \tdiff_setup_done(&diffopt);\n \n \t/* ensure commit is parsed so we have parent information */\n@@ -223,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= max_changes) {\n+\tif (diffopt.num_changes <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -260,13 +259,13 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n-\t\tfilter->len = (hashmap_get_size(&pathmap) * settings.bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n \t\t\tstruct bloom_key key;\n-\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, &settings);\n-\t\t\tadd_key_to_filter(&key, filter, &settings);\n+\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, settings);\n+\t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\ndiff --git a/bloom.h b/bloom.h\nindex e2e035ad14..c6d77e8393 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -98,10 +98,11 @@ enum bloom_filter_computed {\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t enum bloom_filter_computed *computed);\n \n #define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n-\t(r), (c), 0, NULL)\n+\t(r), (c), 0, NULL, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex b8b2c7ca65..500f29525a 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1442,6 +1442,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->r,\n \t\t\tc,\n \t\t\t1,\n+\t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n \t\t\tctx->count_bloom_filter_computed++;\n@@ -1699,17 +1700,6 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tint num_chunks = 3;\n \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n-\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n-\n-\tif (!ctx->bloom_settings) {\n-\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n-\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n-\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n-\t\t\t\t\t\t\t bloom_settings.num_hashes);\n-\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n-\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n-\t\tctx->bloom_settings = &bloom_settings;\n-\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -2155,6 +2145,7 @@ int write_commit_graph(struct object_directory *odb,\n \tuint32_t i, count_distinct = 0;\n \tint res = 0;\n \tint replace = 0;\n+\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n@@ -2168,6 +2159,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split_opts = split_opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n+\t\t\t\t\t\t bloom_settings.bits_per_entry);\n+\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n+\t\t\t\t\t\t bloom_settings.num_hashes);\n+\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n+\tctx->bloom_settings = &bloom_settings;\n+\n \tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n \t\tctx->changed_paths = 1;\n \tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 9f7bb729fc..46e97b04eb 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -40,6 +40,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n \tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t &settings,\n \t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405258","messageId":"9bd699560b5489d64ebc830f0b3cdbba42c373f1.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 06/12] commit-graph.c: store maximum changed paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:23:18Z","receivedAt":"2020-09-09T17:19:03Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For now, we assume that there is a fixed constant describing the\nmaximum number of changed paths we are willing to store in a Bloom\nfilter.\n\nPrepare for that to (at least partially) not be the case by making it a\nmember of the 'struct bloom_filter_settings'. This will be helpful in\nthe subsequent patches by reducing the size of test cases that exercise\nstoring too many changed paths, as well as preparing for an eventual\nfuture in which this value might change.\n\nThis patch alone does not cause newly generated Bloom filters to use\na custom upper-bound on the maximum number of changed paths a single\nBloom filter can hold, that will occur in a later patch.\n\nSigned-off-by: Taylor Blau \n---\n bloom.h | 11 ++++++++++-\n commit-graph.c | 3 +++\n t/t4216-log-bloom.sh | 4 ++--\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.h b/bloom.h\nindex d8fbb0fbf1..0b9b59a6fe 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -28,9 +28,18 @@ struct bloom_filter_settings {\n \t * that contain n*b bits.\n \t */\n \tuint32_t bits_per_entry;\n+\n+\t/*\n+\t * The maximum number of changed paths per commit\n+\t * before declaring a Bloom filter to be too-large.\n+\t *\n+\t * Not written to the commit-graph file.\n+\t */\n+\tuint32_t max_changed_paths;\n };\n \n-#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10 }\n+#define DEFAULT_BLOOM_MAX_CHANGES 512\n+#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10, DEFAULT_BLOOM_MAX_CHANGES }\n #define BITS_PER_WORD 8\n #define BLOOMDATA_CHUNK_HEADER_SIZE 3 * sizeof(uint32_t)\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex ea54d108b9..55af498aa0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1201,6 +1201,7 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n \tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n \tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_object_intmax(&jw, \"max_changed_paths\", ctx->bloom_settings->max_changed_paths);\n \tjw_end(&jw);\n \n \ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n@@ -1669,6 +1670,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n \t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n \t\t\t\t\t\t\t bloom_settings.num_hashes);\n+\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n \t\tctx->bloom_settings = &bloom_settings;\n \t}\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fc7693806c..47ddf2641f 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -174,11 +174,11 @@ test_expect_success 'persist filter settings' '\n \t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n \t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n \t\tGIT_TRACE2_EVENT_NESTING=5 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2-auto.txt\n '\n \n test_expect_success 'correctly report changes over limit' '\n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405259","messageId":"e5bdae44567e72dfc0d6cc9010cadf5d20903642.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 09/12] bloom/diff: properly short-circuit on max_changes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:23:42Z","receivedAt":"2020-09-09T17:19:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nCommit e3696980 (diff: halt tree-diff early after max_changes,\n2020-03-30) intended to create a mechanism to short-circuit a diff\ncalculation after a certain number of paths were modified. By\nincrementing a \"num_changes\" counter throughout the recursive\nll_diff_tree_paths(), this was supposed to match the number of changes\nthat would be written into the changed-path Bloom filters.\nUnfortunately, this was not implemented correctly and instead misses\nsimple cases like file modifications. This then does not stop very\nlarge changed-path filters from being written (unless they add or remove\nmany files).\n\nTo start, change the implementation in ll_diff_tree_paths() to instead\nuse the global diff_queue_diff struct's 'nr' member as the count. This\nis a way to simplify the logic instead of making more mistakes in the\ncomplicated diff code.\n\nThis has a drawback: the diff_queue_diff struct only lists the paths\ncorresponding to blob changes, not their leading directories. Thus,\nget_or_compute_bloom_filter() needs an additional check to see if the\nhashmap with the leading directories becomes too large.\n\nOne reason why this was not caught by test cases was that the test in\nt4216-log-bloom.sh that was supposed to check this \"too many changes\"\ncondition only checked this on the initial commit of a repository. The\nold logic counted these values correctly. Update this test in a few\nways:\n\n1. Use GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS to reduce the limit,\n allowing smaller commits to engage with this logic.\n\n2. Create several interesting cases of edits, adds, removes, and mode\n changes (in the second commit). By testing both sides of the\n inequality with the *_MAX_CHANGED_PATHS variable, we can see that\n the count is exactly correct, so none of these changes are missed\n or over-counted.\n\n3. Use the trace2 data value filter_found_large to verify that these\n commits are on the correct side of the limit.\n\nAnother way to verify the behavior is correct is through performance\ntests. By testing on my local copies of the Git repository and the Linux\nkernel repository, I could measure the effect of these short-circuits\nwhen computing a fresh commit-graph file with changed-path Bloom filters\nusing the command\n\n GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=N time \\\n git commit-graph write --reachable --changed-paths\n\nand reporting the wall time and resulting commit-graph size.\n\nFor Git, the results are\n\n| | N=1 | N=10 | N=512 |\n|--------|----------------|----------------|----------------|\n| HEAD~1 | 10.90s 9.18MB | 11.11s 9.34MB | 11.31s 9.35MB |\n| HEAD | 9.21s 8.62MB | 11.11s 9.29MB | 11.29s 9.34MB |\n\nFor Linux, the results are\n\n| | N=1 | N=20 | N=512 |\n|--------|----------------|---------------|---------------|\n| HEAD~1 | 61.28s 64.3MB | 76.9s 72.6MB | 77.6s 72.6MB |\n| HEAD | 49.44s 56.3MB | 68.7s 65.9MB | 69.2s 65.9MB |\n\nNaturally, the improvement becomes much less as the limit grows, as\nfewer commits satisfy the short-circuit.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n bloom.c | 9 ++++-\n diff.h | 2 -\n t/t4216-log-bloom.sh | 88 +++++++++++++++++++++++++++++++++++++++-----\n tree-diff.c | 5 +--\n 4 files changed, 88 insertions(+), 16 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 2d6aef9098..db9fb82437 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -222,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= settings->max_changed_paths) {\n+\tif (diff_queued_diff.nr <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -259,6 +259,12 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n+\t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths) {\n+\t\t\tif (computed)\n+\t\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n@@ -268,6 +274,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n+\tcleanup:\n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\ndiff --git a/diff.h b/diff.h\nindex e0c0af6286..1d32b71885 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -287,8 +287,6 @@ struct diff_options {\n \n \t/* If non-zero, then stop computing after this many changes. */\n \tint max_changes;\n-\t/* For internal use only. */\n-\tint num_changes;\n \n \tint ita_invisible_in_index;\n /* white-space error highlighting */\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 47ddf2641f..f375e752cd 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -182,20 +182,90 @@ test_expect_success 'persist filter settings' '\n '\n \n test_expect_success 'correctly report changes over limit' '\n-\tgit init 513changes &&\n+\tgit init limits &&\n \t(\n-\t\tcd 513changes &&\n-\t\tfor i in $(test_seq 1 513)\n+\t\tcd limits &&\n+\t\tmkdir d &&\n+\t\tmkdir d/e &&\n+\n+\t\tfor i in $(test_seq 1 2)\n \t\tdo\n-\t\t\techo $i >file$i.txt || return 1\n+\t\t\tprintf $i >d/file$i.txt &&\n+\t\t\tprintf $i >d/e/file$i.txt || return 1\n \t\tdone &&\n-\t\tgit add . &&\n+\n+\t\tmkdir mode &&\n+\t\tprintf bash >mode/script.sh &&\n+\n+\t\tmkdir foo &&\n+\t\ttouch foo/bar &&\n+\t\ttouch foo.txt &&\n+\n+\t\tgit add d foo foo.txt mode &&\n \t\tgit commit -m \"files\" &&\n-\t\tgit commit-graph write --reachable --changed-paths &&\n-\t\tfor i in $(test_seq 1 513)\n+\n+\t\t# Commit has 7 file and 4 directory adds\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":10\" trace &&\n+\t\tgrep \"\\\"filter_computed\\\":1\" trace &&\n+\t\tgrep \"\\\"filter_trunc_large\\\":1\" trace &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n \t\tdo\n-\t\t\tgit -c core.commitGraph=false log -- file$i.txt >expect &&\n-\t\t\tgit log -- file$i.txt >actual &&\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# Make a variety of path changes\n+\t\tprintf new1 >d/e/file1.txt &&\n+\t\tprintf new2 >d/file2.txt &&\n+\t\trm d/e/file2.txt &&\n+\t\trm -r foo &&\n+\t\tprintf text >foo &&\n+\t\tmkdir f &&\n+\t\tprintf new1 >f/file1.txt &&\n+\n+\t\t# including a mode-only change (counts as modified)\n+\t\tgit update-index --chmod=+x mode/script.sh &&\n+\n+\t\tgit add foo d f &&\n+\t\tgit commit -m \"complicated\" &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-edit\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":10\" trace-edit &&\n+\t\tgrep \"\\\"filter_computed\\\":2\" trace-edit &&\n+\t\tgrep \"\\\"filter_trunc_large\\\":2\" trace-edit &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=11 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-update\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"max_changed_paths\\\":11\" trace-update &&\n+\t\tgrep \"\\\"filter_computed\\\":2\" trace-update &&\n+\t\tgrep \"\\\"filter_trunc_large\\\":0\" trace-update &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n \t\t\ttest_cmp expect actual || return 1\n \t\tdone\n \t)\ndiff --git a/tree-diff.c b/tree-diff.c\nindex 6ebad1a46f..7cebbb327e 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -434,7 +434,7 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tif (diff_can_quit_early(opt))\n \t\t\tbreak;\n \n-\t\tif (opt->max_changes && opt->num_changes > opt->max_changes)\n+\t\tif (opt->max_changes && diff_queued_diff.nr > opt->max_changes)\n \t\t\tbreak;\n \n \t\tif (opt->pathspec.nr) {\n@@ -521,7 +521,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \n \t\t\t/* t↓ */\n \t\t\tupdate_tree_entry(&t);\n-\t\t\topt->num_changes++;\n \t\t}\n \n \t\t/* t > p[imin] */\n@@ -539,7 +538,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tskip_emit_tp:\n \t\t\t/* ∀ pi=p[imin] pi↓ */\n \t\t\tupdate_tp_entries(tp, nparent);\n-\t\t\topt->num_changes++;\n \t\t}\n \t}\n \n@@ -557,7 +555,6 @@ struct combine_diff_path *diff_tree_paths(\n \tconst struct object_id **parents_oid, int nparent,\n \tstruct strbuf *base, struct diff_options *opt)\n {\n-\topt->num_changes = 0;\n \tp = ll_diff_tree_paths(p, oid, parents_oid, nparent, base, opt);\n \n \t/*\n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405260","messageId":"1c993b83980e77594b0de2b1884cbc4cf484f9ab.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 10/12] bloom: encode out-of-bounds filters as non-empty","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:23:48Z","receivedAt":"2020-09-09T17:19:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a changed-path Bloom filter has either zero, or more than a\ncertain number (commonly 512) of entries, the commit-graph machinery\nencodes it as \"missing\". More specifically, it sets the indices adjacent\nin the BIDX chunk as equal to each other to indicate a \"length 0\"\nfilter; that is, that the filter occupies zero bytes on disk.\n\nThis has heretofore been fine, since the commit-graph machinery has no\nneed to care about these filters with too few or too many changed paths.\nBoth cases act like no filter has been generated at all, and so there is\nno need to store them.\n\nIn a subsequent commit, however, the commit-graph machinery will learn\nto only compute Bloom filters for some commits in the current\ncommit-graph layer. This is a change from the current implementation\nwhich computes Bloom filters for all commits that are in the layer being\nwritten. Critically for this patch, only computing some of the Bloom\nfilters means adding a third state for length 0 Bloom filters: zero\nentries, too many entries, or \"hasn't been computed\".\n\nIt will be important for that future patch to distinguish between \"not\nrepresentable\" (i.e., zero or too-many changed paths), and \"hasn't been\ncomputed\". In particular, we don't want to waste time recomputing\nfilters that have already been computed.\n\nTo that end, change how we store Bloom filters in the \"computed but not\nrepresentable\" category:\n\n - Bloom filters with no entries are stored as a single byte with all\n bits low (i.e., all queries to that Bloom filter will return\n \"definitely not\")\n\n - Bloom filters with too many entries are stored as a single byte with\n all bits set high (i.e., all queries to that Bloom filter will\n return \"maybe\").\n\nThese rules are sufficient to not incur a behavior change by changing\nthe on-disk representation of these two classes. Likewise, no\nspecification changes are necessary for the commit-graph format, either:\n\n - Filters that were previously empty will be recomputed and stored\n according to the new rules, and\n\n - old clients reading filters generated by new clients will interpret\n the filters correctly and be none the wiser to how they were\n generated.\n\nClients will invoke the Bloom machinery in more cases than before, but\nthis can be addressed by returning a NULL filter when all bits are set\nhigh. This can be addressed in a future patch.\n\nFinally, note that this does increase the size of on-disk commit-graphs,\nbut far less than other proposals. In particular, this is generally more\nefficient than storing a bitmap for which commits haven't computed their\nBloom filters. Storing a bitmap incurs a penalty of one bit per commit,\nwhereas storing explicit filters as above incurs a penalty of one byte\nper too-large or too-small commit.\n\nIn practice, these boundary commits likely occupy a small proportion of\nthe overall number of commits, and so the size penalty is likely smaller\nthan storing a bitmap for all commits.\n\nA test to exercise filters which contain too many changed path entries\nwill be introduced in a subsequent patch.\n\nSuggested-by: SZEDER Gábor \nSuggested-by: Jakub Narębski \nHelped-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n .../technical/commit-graph-format.txt | 2 +-\n bloom.c | 10 +++++++--\n bloom.h | 1 +\n commit-graph.c | 5 +++++\n t/t0095-bloom.sh | 4 ++--\n t/t4216-log-bloom.sh | 21 +++++++++++++++++--\n 6 files changed, 36 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 6ddbceba15..6585f1948a 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -125,7 +125,7 @@ CHUNK DATA:\n * The rest of the chunk is the concatenation of all the computed Bloom\n filters for the commits in lexicographic order.\n * Note: Commits with no changes or more than 512 changes have Bloom filters\n- of length zero.\n+ of length one, with either all bits set to zero or one respectively.\n * The BDAT chunk is present if and only if BIDX is present.\n \n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\ndiff --git a/bloom.c b/bloom.c\nindex db9fb82437..194b6ab8ad 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -266,6 +266,11 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t}\n \n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tif (!filter->len) {\n+\t\t\tif (computed)\n+\t\t\t\t*computed |= BLOOM_TRUNC_SMALL;\n+\t\t\tfilter->len = 1;\n+\t\t}\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n@@ -279,8 +284,9 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n-\t\tfilter->data = NULL;\n-\t\tfilter->len = 0;\n+\t\tfilter->data = xmalloc(1);\n+\t\tfilter->data[0] = 0xFF;\n+\t\tfilter->len = 1;\n \n \t\tif (computed)\n \t\t\t*computed |= BLOOM_TRUNC_LARGE;\ndiff --git a/bloom.h b/bloom.h\nindex c6d77e8393..70a8840896 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -93,6 +93,7 @@ enum bloom_filter_computed {\n \tBLOOM_NOT_COMPUTED = (1 << 0),\n \tBLOOM_COMPUTED = (1 << 1),\n \tBLOOM_TRUNC_LARGE = (1 << 2),\n+\tBLOOM_TRUNC_SMALL = (1 << 3),\n };\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 500f29525a..d402743e61 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -974,6 +974,7 @@ struct write_commit_graph_context {\n \n \tint count_bloom_filter_computed;\n \tint count_bloom_filter_not_computed;\n+\tint count_bloom_filter_trunc_small;\n \tint count_bloom_filter_trunc_large;\n };\n \n@@ -1405,6 +1406,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \t\t\t ctx->count_bloom_filter_computed);\n \tjw_object_intmax(&jw, \"filter_not_computed\",\n \t\t\t ctx->count_bloom_filter_not_computed);\n+\tjw_object_intmax(&jw, \"filter_trunc_small\",\n+\t\t\t ctx->count_bloom_filter_trunc_small);\n \tjw_object_intmax(&jw, \"filter_trunc_large\",\n \t\t\t ctx->count_bloom_filter_trunc_large);\n \tjw_end(&jw);\n@@ -1446,6 +1449,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n \t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (computed & BLOOM_TRUNC_SMALL)\n+\t\t\t\tctx->count_bloom_filter_trunc_small++;\n \t\t\tif (computed & BLOOM_TRUNC_LARGE)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex 232ba2c485..4d0e512bcb 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -71,8 +71,8 @@ test_expect_success 'get bloom filters for commit with no changes' '\n \tgit init &&\n \tgit commit --allow-empty -m \"c0\" &&\n \tcat >expect <<-\\EOF &&\n-\tFilter_Length:0\n-\tFilter_Data:\n+\tFilter_Length:1\n+\tFilter_Data:00|\n \tEOF\n \ttest-tool bloom get_filter_for_commit \"$(git rev-parse HEAD)\" >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex f375e752cd..a56327ffd4 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -30,6 +30,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \trm file_to_be_deleted &&\n \tgit add . &&\n \tgit commit -m \"file removed\" &&\n+\tgit commit --allow-empty -m \"empty\" &&\n \tgit commit-graph write --reachable --changed-paths &&\n \n \ttest_oid_cache <<-EOF\n@@ -49,7 +50,7 @@ graph_read_expect () {\n }\n \n test_expect_success 'commit-graph write wrote out the bloom chunks' '\n-\tgraph_read_expect 15\n+\tgraph_read_expect 16\n '\n \n # Turn off any inherited trace2 settings for this test.\n@@ -156,7 +157,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":9\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n@@ -271,4 +272,20 @@ test_expect_success 'correctly report changes over limit' '\n \t)\n '\n \n+test_expect_success 'correctly report commits with no changed paths' '\n+\tgit init small &&\n+\ttest_when_finished \"rm -fr small\" &&\n+\t(\n+\t\tcd small &&\n+\n+\t\tgit commit --allow-empty -m \"initial commit\" &&\n+\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"filter_computed\\\":1\" trace &&\n+\t\tgrep \"\\\"filter_trunc_small\\\":1\" trace &&\n+\t\tgrep \"\\\"filter_trunc_large\\\":0\" trace\n+\t)\n+'\n+\n test_done\n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405261","messageId":"7475ce47ebcdb9c666e658ab8727b2d1df384c8d.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 07/12] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:23:25Z","receivedAt":"2020-09-09T17:19:17Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'get_bloom_filter' takes a flag to control whether it will compute a\nBloom filter if the requested one is missing. In the next patch, we'll\nadd yet another parameter to this method, which would force all but one\ncaller to specify an extra 'NULL' parameter at the end.\n\nInstead of doing this, split 'get_bloom_filter' into two functions:\n'get_bloom_filter' and 'get_or_compute_bloom_filter'. The former only\nlooks up a Bloom filter (and does not compute one if it's missing,\nthus dropping the 'compute_if_not_present' flag). The latter does\ncompute missing Bloom filters, with an additional parameter to store\nwhether or not it needed to do so.\n\nThis simplifies many call-sites, since the majority of existing callers\nto 'get_bloom_filter' do not want missing Bloom filters to be computed\n(so they can drop the parameter entirely and use the simpler version of\nthe function).\n\nWhile we're at it, instrument the new 'get_or_compute_bloom_filter()'\nwith counters in the 'write_commit_graph_context' struct which store\nthe number of filters that we did and didn't compute, as well as filters\nthat were truncated.\n\nIt would be nice to drop the 'compute_if_not_present' flag entirely,\nsince all remaining callers of 'get_or_compute_bloom_filter' pass it as\n'1', but this will change in a future patch and hence cannot be removed.\n\nSigned-off-by: Taylor Blau \n---\n blame.c | 2 +-\n bloom.c | 16 +++++++++++++---\n bloom.h | 16 +++++++++++++---\n commit-graph.c | 42 +++++++++++++++++++++++++++++++++++++++---\n line-log.c | 2 +-\n revision.c | 2 +-\n t/helper/test-bloom.c | 3 ++-\n 7 files changed, 70 insertions(+), 13 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 903e23af23..e5ba35dbd1 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -1276,7 +1276,7 @@ static int maybe_changed_path(struct repository *r,\n \tif (commit_graph_generation(origin->commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn 1;\n \n-\tfilter = get_bloom_filter(r, origin->commit, 0);\n+\tfilter = get_bloom_filter(r, origin->commit);\n \n \tif (!filter)\n \t\treturn 1;\ndiff --git a/bloom.c b/bloom.c\nindex cd9380ac62..393c61b4bc 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,9 +177,10 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present)\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t enum bloom_filter_computed *computed)\n {\n \tstruct bloom_filter *filter;\n \tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n@@ -187,6 +188,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n+\tif (computed)\n+\t\t*computed = BLOOM_NOT_COMPUTED;\n+\n \tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n@@ -271,8 +275,14 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\tfilter->data = NULL;\n \t\tfilter->len = 0;\n+\n+\t\tif (computed)\n+\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n \t}\n \n+\tif (computed)\n+\t\t*computed |= BLOOM_COMPUTED;\n+\n \tfree(diff_queued_diff.queue);\n \tDIFF_QUEUE_CLEAR(&diff_queued_diff);\n \ndiff --git a/bloom.h b/bloom.h\nindex 0b9b59a6fe..e2e035ad14 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -89,9 +89,19 @@ void add_key_to_filter(const struct bloom_key *key,\n \n void init_bloom_filters(void);\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present);\n+enum bloom_filter_computed {\n+\tBLOOM_NOT_COMPUTED = (1 << 0),\n+\tBLOOM_COMPUTED = (1 << 1),\n+\tBLOOM_TRUNC_LARGE = (1 << 2),\n+};\n+\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t enum bloom_filter_computed *computed);\n+\n+#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n+\t(r), (c), 0, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 55af498aa0..b8b2c7ca65 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -971,6 +971,10 @@ struct write_commit_graph_context {\n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n+\n+\tint count_bloom_filter_computed;\n+\tint count_bloom_filter_not_computed;\n+\tint count_bloom_filter_trunc_large;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1182,7 +1186,7 @@ static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tuint32_t cur_pos = 0;\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \t\tcur_pos += len;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1222,7 +1226,7 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1392,6 +1396,24 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n+{\n+\tstruct json_writer jw = JSON_WRITER_INIT;\n+\n+\tjw_object_begin(&jw, 0);\n+\tjw_object_intmax(&jw, \"filter_computed\",\n+\t\t\t ctx->count_bloom_filter_computed);\n+\tjw_object_intmax(&jw, \"filter_not_computed\",\n+\t\t\t ctx->count_bloom_filter_not_computed);\n+\tjw_object_intmax(&jw, \"filter_trunc_large\",\n+\t\t\t ctx->count_bloom_filter_trunc_large);\n+\tjw_end(&jw);\n+\n+\ttrace2_data_json(\"commit-graph\", the_repository, \"bloom_statistics\", &jw);\n+\n+\tjw_release(&jw);\n+}\n+\n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n@@ -1414,12 +1436,26 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tenum bloom_filter_computed computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n+\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\tctx->r,\n+\t\t\tc,\n+\t\t\t1,\n+\t\t\t&computed);\n+\t\tif (computed & BLOOM_COMPUTED) {\n+\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (computed & BLOOM_TRUNC_LARGE)\n+\t\t\t\tctx->count_bloom_filter_trunc_large++;\n+\t\t} else if (computed & BLOOM_NOT_COMPUTED)\n+\t\t\tctx->count_bloom_filter_not_computed++;\n \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n+\tif (trace2_is_enabled())\n+\t\ttrace2_bloom_filter_write_statistics(ctx);\n+\n \tfree(sorted_commits);\n \tstop_progress(&progress);\n }\ndiff --git a/line-log.c b/line-log.c\nindex bf73ea95ac..68eeb425f8 100644\n--- a/line-log.c\n+++ b/line-log.c\n@@ -1159,7 +1159,7 @@ static int bloom_filter_check(struct rev_info *rev,\n \t\treturn 1;\n \n \tif (!rev->bloom_filter_settings ||\n-\t !(filter = get_bloom_filter(rev->repo, commit, 0)))\n+\t !(filter = get_bloom_filter(rev->repo, commit)))\n \t\treturn 1;\n \n \tif (!range)\ndiff --git a/revision.c b/revision.c\nindex 857274408c..f4be5d1650 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -751,7 +751,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \tif (commit_graph_generation(commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn -1;\n \n-\tfilter = get_bloom_filter(revs->repo, commit, 0);\n+\tfilter = get_bloom_filter(revs->repo, commit);\n \n \tif (!filter) {\n \t\tcount_bloom_filter_not_present++;\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 5e77d56f59..9f7bb729fc 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -39,7 +39,8 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tstruct bloom_filter *filter;\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n-\tfilter = get_bloom_filter(the_repository, c, 1);\n+\tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n \n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405262","messageId":"44df936e51e88e41d5ed9aa02d0bd3ca86696f09.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 02/12] t4216: use an '&&'-chain","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:22:50Z","receivedAt":"2020-09-09T17:19:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a759bfa9ee (t4216: add end to end tests for git log with Bloom\nfilters, 2020-04-06), a 'rm' invocation was added without a\ncorresponding '&&' chain.\n\nWhen 'trace.perf' already exists, everything works fine. However, the\nfunction can be executed without 'trace.perf' on disk (eg., when the\nsubset of tests run is altered with '--run'), and so the bare 'rm'\ncomplains about a missing file.\n\nTo remove some noise from the test log, invoke 'rm' with '-f', at which\npoint it is sensible to place the 'rm -f' in an '&&'-chain, which is\nboth (1) our usual style, and (2) avoids a broken chain in the future if\nmore commands are added at the beginning of the function.\n\nHelped-by: Eric Sunshine \nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 715912ad0f..cd89c75002 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -58,7 +58,7 @@ sane_unset GIT_TRACE2_PERF_BRIEF\n sane_unset GIT_TRACE2_CONFIG_PARAMS\n \n setup () {\n-\trm \"$TRASH_DIRECTORY/trace.perf\"\n+\trm -f \"$TRASH_DIRECTORY/trace.perf\" &&\n \tgit -c core.commitGraph=false log --pretty=\"format:%s\" $1 >log_wo_bloom &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.perf\" git -c core.commitGraph=true log --pretty=\"format:%s\" $1 >log_w_bloom\n }\n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405263","messageId":"cover.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":null,"subject":"[PATCH 00/12] more miscellaneous Bloom filter improvements, redux","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:22:33Z","receivedAt":"2020-09-09T17:19:49Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here is a rejiggered version of my series in [1], which accomplishes the\nsame without changing any of the on-disk commit-graph format.\n\nAs a reminder, the main goal of this series is to introduce a\n'--max-new-filters' flag to 'git commit-graph write' to place a limit on\nthe number of new Bloom filters a writer is willing to compute from\nscratch. The main difficulty is disambiguating between empty/too-large\nfilters and ones that haven't been computed yet. See \"bloom: encode\nout-of-bounds filters as non-empty\" for the details.\n\nThe series is organized as follows:\n\n * Patches 1-4 are uninteresting preparatory steps.,\n * Patch 5 introduces the 'commitGraph.readChangedPaths' configuration.\n * Patches 6-8 are more preparation.\n * Patch 9 is from Stolee and fixes a bug where computing Bloom filters\n from scratch wouldn't stop at the limit of 512.\n * Patches 10-12 prepares for and then introduces '--max-new-filters'.\n\nThe first nine patches are basically unchanged from [1] where they were\nthoroughly reviewed. The tenth patch is new, and the final two patches\nare only touched up and simplified to work with this new approach, but\nthey have otherwise been reviewed.\n\nSince the old thread was getting long, and this is a substantially new\napproach, I'm sending this as \"v1\" of a new series, which hopefully\nnobody minds.\n\n[1]: https://lore.kernel.org/git/cover.1596480582.git.me@ttaylorr.com/\n\nDerrick Stolee (1):\n bloom/diff: properly short-circuit on max_changes\n\nTaylor Blau (11):\n commit-graph: introduce 'get_bloom_filter_settings()'\n t4216: use an '&&'-chain\n commit-graph: pass a 'struct repository *' in more places\n t/helper/test-read-graph.c: prepare repo settings\n commit-graph: respect 'commitGraph.readChangedPaths'\n commit-graph.c: store maximum changed paths\n bloom: split 'get_bloom_filter()' in two\n bloom: use provided 'struct bloom_filter_settings'\n bloom: encode out-of-bounds filters as non-empty\n commit-graph: rename 'split_commit_graph_opts'\n builtin/commit-graph.c: introduce '--max-new-filters='\n\n Documentation/config.txt | 2 +\n Documentation/config/commitgraph.txt | 8 +\n Documentation/git-commit-graph.txt | 6 +\n .../technical/commit-graph-format.txt | 2 +-\n blame.c | 8 +-\n bloom.c | 53 +++--\n bloom.h | 29 ++-\n builtin/commit-graph.c | 61 ++++--\n commit-graph.c | 148 ++++++++++----\n commit-graph.h | 17 +-\n diff.h | 2 -\n fuzz-commit-graph.c | 5 +-\n line-log.c | 2 +-\n repo-settings.c | 3 +\n repository.h | 1 +\n revision.c | 7 +-\n t/helper/test-bloom.c | 4 +-\n t/helper/test-read-graph.c | 3 +-\n t/t0095-bloom.sh | 4 +-\n t/t4216-log-bloom.sh | 181 ++++++++++++++++--\n t/t5324-split-commit-graph.sh | 13 ++\n tree-diff.c | 5 +-\n 22 files changed, 442 insertions(+), 122 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\n--\n2.28.0.462.g4ff11cec37\n"},{"id":"405264","messageId":"c33976713be2ee8fa42fc13357c49d5b3d8baebf.1599664389.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH 01/12] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-09T15:22:44Z","receivedAt":"2020-09-09T17:19:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Many places in the code often need a pointer to the commit-graph's\n'struct bloom_filter_settings', in which case they often take the value\nfrom the top-most commit-graph.\n\nIn the non-split case, this works as expected. In the split case,\nhowever, things get a little tricky. Not all layers in a chain of\nincremental commit-graphs are required to themselves have Bloom data,\nand so whether or not some part of the code uses Bloom filters depends\nentirely on whether or not the top-most level of the commit-graph chain\nhas Bloom filters.\n\nThis has been the behavior since Bloom filters were introduced, and has\nbeen codified into the tests since a759bfa9ee (t4216: add end to end\ntests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\nrequires that Bloom filters are not used in exactly the case described\nearlier.\n\nThere is no reason that this needs to be the case, since it is perfectly\nvalid for commits in an earlier layer to have Bloom filters when commits\nin a newer layer do not.\n\nSince Bloom settings are guaranteed in practice to be the same for any\nlayer in a chain that has Bloom data, it is sufficient to traverse the\n'->base_graph' pointer until either (1) a non-null 'struct\nbloom_filter_settings *' is found, or (2) until we are at the root of\nthe commit-graph chain.\n\nIntroduce a 'get_bloom_filter_settings()' function that does just this,\nand use it instead of purely dereferencing the top-most graph's\n'->bloom_filter_settings' pointer.\n\nWhile we're at it, add an additional test in t5324 to guard against code\nin the commit-graph writing machinery that doesn't correctly handle a\nNULL 'struct bloom_filter *'.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n blame.c | 6 ++++--\n bloom.c | 6 +++---\n commit-graph.c | 11 +++++++++++\n commit-graph.h | 2 ++\n revision.c | 5 +----\n t/t4216-log-bloom.sh | 9 ++++++---\n t/t5324-split-commit-graph.sh | 13 +++++++++++++\n 7 files changed, 40 insertions(+), 12 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 1be1cd82a2..903e23af23 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -2892,16 +2892,18 @@ void setup_blame_bloom_data(struct blame_scoreboard *sb,\n \t\t\t const char *path)\n {\n \tstruct blame_bloom_data *bd;\n+\tstruct bloom_filter_settings *bs;\n \n \tif (!sb->repo->objects->commit_graph)\n \t\treturn;\n \n-\tif (!sb->repo->objects->commit_graph->bloom_filter_settings)\n+\tbs = get_bloom_filter_settings(sb->repo);\n+\tif (!bs)\n \t\treturn;\n \n \tbd = xmalloc(sizeof(struct blame_bloom_data));\n \n-\tbd->settings = sb->repo->objects->commit_graph->bloom_filter_settings;\n+\tbd->settings = bs;\n \n \tbd->alloc = 4;\n \tbd->nr = 0;\ndiff --git a/bloom.c b/bloom.c\nindex 1a573226e7..cd9380ac62 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -38,7 +38,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \twhile (graph_pos < g->num_commits_in_base)\n \t\tg = g->base_graph;\n \n-\t/* The commit graph commit 'c' lives in doesn't carry bloom filters. */\n+\t/* The commit graph commit 'c' lives in doesn't carry Bloom filters. */\n \tif (!g->chunk_bloom_indexes)\n \t\treturn 0;\n \n@@ -195,8 +195,8 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes)\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n+\t\t\t\treturn filter;\n \t}\n \n \tif (filter->data)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0ed003e218..6a36ed0b06 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -667,6 +667,17 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r)\n+{\n+\tstruct commit_graph *g = r->objects->commit_graph;\n+\twhile (g) {\n+\t\tif (g->bloom_filter_settings)\n+\t\t\treturn g->bloom_filter_settings;\n+\t\tg = g->base_graph;\n+\t}\n+\treturn NULL;\n+}\n+\n static void close_commit_graph_one(struct commit_graph *g)\n {\n \tif (!g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 09a97030dc..0677dd1031 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -87,6 +87,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n */\n int generation_numbers_enabled(struct repository *r);\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r);\n+\n enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_APPEND = (1 << 0),\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\ndiff --git a/revision.c b/revision.c\nindex 08c2ad23af..857274408c 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -680,10 +680,7 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \trepo_parse_commit(revs->repo, revs->commits->item);\n \n-\tif (!revs->repo->objects->commit_graph)\n-\t\treturn;\n-\n-\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n+\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n \tif (!revs->bloom_filter_settings)\n \t\treturn;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 4bb9e9dbe2..715912ad0f 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -65,7 +65,7 @@ setup () {\n \n test_bloom_filters_used () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":${2:-0},\\\"maybe\\\"\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom &&\n@@ -139,8 +139,11 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n-test_expect_success 'Do not use Bloom filters if the latest graph does not have Bloom filters.' '\n-\ttest_bloom_filters_not_used \"-- A/B\"\n+test_expect_success 'use Bloom filters even if the latest graph does not have Bloom filters' '\n+\t# Ensure that the number of empty filters is equal to the number of\n+\t# filters in the latest graph layer to prove that they are loaded (and\n+\t# ignored).\n+\ttest_bloom_filters_used \"-- A/B\" 3\n '\n \n test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 18216463c7..c334ee9155 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -427,4 +427,17 @@ done <<\\EOF\n 0600 -r--------\n EOF\n \n+test_expect_success '--split=replace with partial Bloom data' '\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/3 &&\n+\tgit rev-list -1 HEAD~2 >a &&\n+\tgit rev-list -1 HEAD~1 >b &&\n+\tgit commit-graph write --split=no-merge --stdin-commits --changed-paths graph-files &&\n+\ttest_line_count = 1 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n test_done\n-- \n2.28.0.462.g4ff11cec37\n\n"},{"id":"405308","messageId":"20200910033557.GA2178@nand.local","threadId":"54216","inReplyTo":"1c993b83980e77594b0de2b1884cbc4cf484f9ab.1599664389.git.me@ttaylorr.com","subject":"Re: [PATCH 10/12] bloom: encode out-of-bounds filters as non-empty","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-10T03:35:57Z","receivedAt":"2020-09-10T03:36:13Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 09, 2020 at 11:23:48AM -0400, Taylor Blau wrote:\n> -\t\tfilter->data = NULL;\n> -\t\tfilter->len = 0;\n> +\t\tfilter->data = xmalloc(1);\n> +\t\tfilter->data[0] = 0xFF;\n> +\t\tfilter->len = 1;\n>\n> \t\tif (computed)\n> \t\t\t*computed |= BLOOM_TRUNC_LARGE;\n\nOops, I missed the case that added by the previous patch where the\nnumber of diff entries is smaller than the limit, but the hashmap\nentries (after directories are added and such) crosses the threshold.\n\nSpecifically, this patch doesn't write the 0xFF filter like it should.\nI'll send a different version of this patch tomorrow.\n\nThanks,\nTaylor\n"},{"id":"405365","messageId":"20200910154516.GA32117@nand.local","threadId":"54216","inReplyTo":"20200910033557.GA2178@nand.local","subject":"Re: [PATCH 10/12] bloom: encode out-of-bounds filters as non-empty","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-10T15:45:16Z","receivedAt":"2020-09-10T19:21:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 09, 2020 at 11:35:57PM -0400, Taylor Blau wrote:\n> On Wed, Sep 09, 2020 at 11:23:48AM -0400, Taylor Blau wrote:\n> > -\t\tfilter->data = NULL;\n> > -\t\tfilter->len = 0;\n> > +\t\tfilter->data = xmalloc(1);\n> > +\t\tfilter->data[0] = 0xFF;\n> > +\t\tfilter->len = 1;\n> >\n> > \t\tif (computed)\n> > \t\t\t*computed |= BLOOM_TRUNC_LARGE;\n>\n> Oops, I missed the case that added by the previous patch where the\n> number of diff entries is smaller than the limit, but the hashmap\n> entries (after directories are added and such) crosses the threshold.\n>\n> Specifically, this patch doesn't write the 0xFF filter like it should.\n> I'll send a different version of this patch tomorrow.\n\nThis one should do the trick. Let's use it instead.\n\n--- >8 ---\n\nSubject: [PATCH] bloom: encode out-of-bounds filters as non-empty\n\nWhen a changed-path Bloom filter has either zero, or more than a\ncertain number (commonly 512) of entries, the commit-graph machinery\nencodes it as \"missing\". More specifically, it sets the indices adjacent\nin the BIDX chunk as equal to each other to indicate a \"length 0\"\nfilter; that is, that the filter occupies zero bytes on disk.\n\nThis has heretofore been fine, since the commit-graph machinery has no\nneed to care about these filters with too few or too many changed paths.\nBoth cases act like no filter has been generated at all, and so there is\nno need to store them.\n\nIn a subsequent commit, however, the commit-graph machinery will learn\nto only compute Bloom filters for some commits in the current\ncommit-graph layer. This is a change from the current implementation\nwhich computes Bloom filters for all commits that are in the layer being\nwritten. Critically for this patch, only computing some of the Bloom\nfilters means adding a third state for length 0 Bloom filters: zero\nentries, too many entries, or \"hasn't been computed\".\n\nIt will be important for that future patch to distinguish between \"not\nrepresentable\" (i.e., zero or too-many changed paths), and \"hasn't been\ncomputed\". In particular, we don't want to waste time recomputing\nfilters that have already been computed.\n\nTo that end, change how we store Bloom filters in the \"computed but not\nrepresentable\" category:\n\n - Bloom filters with no entries are stored as a single byte with all\n bits low (i.e., all queries to that Bloom filter will return\n \"definitely not\")\n\n - Bloom filters with too many entries are stored as a single byte with\n all bits set high (i.e., all queries to that Bloom filter will\n return \"maybe\").\n\nThese rules are sufficient to not incur a behavior change by changing\nthe on-disk representation of these two classes. Likewise, no\nspecification changes are necessary for the commit-graph format, either:\n\n - Filters that were previously empty will be recomputed and stored\n according to the new rules, and\n\n - old clients reading filters generated by new clients will interpret\n the filters correctly and be none the wiser to how they were\n generated.\n\nClients will invoke the Bloom machinery in more cases than before, but\nthis can be addressed by returning a NULL filter when all bits are set\nhigh. This can be addressed in a future patch.\n\nFinally, note that this does increase the size of on-disk commit-graphs,\nbut far less than other proposals. In particular, this is generally more\nefficient than storing a bitmap for which commits haven't computed their\nBloom filters. Storing a bitmap incurs a penalty of one bit per commit,\nwhereas storing explicit filters as above incurs a penalty of one byte\nper too-large or too-small commit.\n\nIn practice, these boundary commits likely occupy a small proportion of\nthe overall number of commits, and so the size penalty is likely smaller\nthan storing a bitmap for all commits.\n\nA test to exercise filters which contain too many changed path entries\nwill be introduced in a subsequent patch.\n\nSuggested-by: SZEDER Gábor \nSuggested-by: Jakub Narębski \nHelped-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n .../technical/commit-graph-format.txt | 2 +-\n bloom.c | 16 ++++++++++++--\n bloom.h | 1 +\n commit-graph.c | 5 +++++\n t/t0095-bloom.sh | 8 +++----\n t/t4216-log-bloom.sh | 21 +++++++++++++++++--\n 6 files changed, 44 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 6ddbceba15..6585f1948a 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -125,7 +125,7 @@ CHUNK DATA:\n * The rest of the chunk is the concatenation of all the computed Bloom\n filters for the commits in lexicographic order.\n * Note: Commits with no changes or more than 512 changes have Bloom filters\n- of length zero.\n+ of length one, with either all bits set to zero or one respectively.\n * The BDAT chunk is present if and only if BIDX is present.\n\n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\ndiff --git a/bloom.c b/bloom.c\nindex db9fb82437..d24747a1d5 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,6 +177,13 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n\n+static void init_truncated_large_filter(struct bloom_filter *filter)\n+{\n+\tfilter->data = xmalloc(1);\n+\tfilter->data[0] = 0xFF;\n+\tfilter->len = 1;\n+}\n+\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n@@ -260,12 +267,18 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t}\n\n \t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths) {\n+\t\t\tinit_truncated_large_filter(filter);\n \t\t\tif (computed)\n \t\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n \t\t\tgoto cleanup;\n \t\t}\n\n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tif (!filter->len) {\n+\t\t\tif (computed)\n+\t\t\t\t*computed |= BLOOM_TRUNC_SMALL;\n+\t\t\tfilter->len = 1;\n+\t\t}\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n\n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n@@ -279,8 +292,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n-\t\tfilter->data = NULL;\n-\t\tfilter->len = 0;\n+\t\tinit_truncated_large_filter(filter);\n\n \t\tif (computed)\n \t\t\t*computed |= BLOOM_TRUNC_LARGE;\ndiff --git a/bloom.h b/bloom.h\nindex c6d77e8393..70a8840896 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -93,6 +93,7 @@ enum bloom_filter_computed {\n \tBLOOM_NOT_COMPUTED = (1 << 0),\n \tBLOOM_COMPUTED = (1 << 1),\n \tBLOOM_TRUNC_LARGE = (1 << 2),\n+\tBLOOM_TRUNC_SMALL = (1 << 3),\n };\n\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 500f29525a..d402743e61 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -974,6 +974,7 @@ struct write_commit_graph_context {\n\n \tint count_bloom_filter_computed;\n \tint count_bloom_filter_not_computed;\n+\tint count_bloom_filter_trunc_small;\n \tint count_bloom_filter_trunc_large;\n };\n\n@@ -1405,6 +1406,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \t\t\t ctx->count_bloom_filter_computed);\n \tjw_object_intmax(&jw, \"filter_not_computed\",\n \t\t\t ctx->count_bloom_filter_not_computed);\n+\tjw_object_intmax(&jw, \"filter_trunc_small\",\n+\t\t\t ctx->count_bloom_filter_trunc_small);\n \tjw_object_intmax(&jw, \"filter_trunc_large\",\n \t\t\t ctx->count_bloom_filter_trunc_large);\n \tjw_end(&jw);\n@@ -1446,6 +1449,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n \t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (computed & BLOOM_TRUNC_SMALL)\n+\t\t\t\tctx->count_bloom_filter_trunc_small++;\n \t\t\tif (computed & BLOOM_TRUNC_LARGE)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex 232ba2c485..7e4ab1795f 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -71,8 +71,8 @@ test_expect_success 'get bloom filters for commit with no changes' '\n \tgit init &&\n \tgit commit --allow-empty -m \"c0\" &&\n \tcat >expect <<-\\EOF &&\n-\tFilter_Length:0\n-\tFilter_Data:\n+\tFilter_Length:1\n+\tFilter_Data:00|\n \tEOF\n \ttest-tool bloom get_filter_for_commit \"$(git rev-parse HEAD)\" >actual &&\n \ttest_cmp expect actual\n@@ -107,8 +107,8 @@ test_expect_success EXPENSIVE 'get bloom filter for commit with 513 changes' '\n \tgit add bigDir &&\n \tgit commit -m \"commit with 513 changes\" &&\n \tcat >expect <<-\\EOF &&\n-\tFilter_Length:0\n-\tFilter_Data:\n+\tFilter_Length:1\n+\tFilter_Data:ff|\n \tEOF\n \ttest-tool bloom get_filter_for_commit \"$(git rev-parse HEAD)\" >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex f375e752cd..a56327ffd4 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -30,6 +30,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \trm file_to_be_deleted &&\n \tgit add . &&\n \tgit commit -m \"file removed\" &&\n+\tgit commit --allow-empty -m \"empty\" &&\n \tgit commit-graph write --reachable --changed-paths &&\n\n \ttest_oid_cache <<-EOF\n@@ -49,7 +50,7 @@ graph_read_expect () {\n }\n\n test_expect_success 'commit-graph write wrote out the bloom chunks' '\n-\tgraph_read_expect 15\n+\tgraph_read_expect 16\n '\n\n # Turn off any inherited trace2 settings for this test.\n@@ -156,7 +157,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n\n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":9\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n@@ -271,4 +272,20 @@ test_expect_success 'correctly report changes over limit' '\n \t)\n '\n\n+test_expect_success 'correctly report commits with no changed paths' '\n+\tgit init small &&\n+\ttest_when_finished \"rm -fr small\" &&\n+\t(\n+\t\tcd small &&\n+\n+\t\tgit commit --allow-empty -m \"initial commit\" &&\n+\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\tgrep \"\\\"filter_computed\\\":1\" trace &&\n+\t\tgrep \"\\\"filter_trunc_small\\\":1\" trace &&\n+\t\tgrep \"\\\"filter_trunc_large\\\":0\" trace\n+\t)\n+'\n+\n test_done\n--\n2.27.0.2918.g22f69e1f31\n\n"},{"id":"405427","messageId":"20200911175216.GA2693949@coredump.intra.peff.net","threadId":"54216","inReplyTo":"4ff11cec37d17d788a3ee076b7c3de1c873a5fbd.1599664389.git.me@ttaylorr.com","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-09-11T17:52:16Z","receivedAt":"2020-09-11T17:52:20Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 09, 2020 at 11:24:00AM -0400, Taylor Blau wrote:\n\n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 17405c73a9..81a2e65903 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -67,6 +67,12 @@ this option is given, future commit-graph writes will automatically assume\n> that this option was intended. Use `--no-changed-paths` to stop storing this\n> data.\n> +\n> +With the `--max-new-filters=` option, generate at most `n` new Bloom\n> +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n> +enforced. Commits whose filters are not calculated are stored as a\n> +length zero Bloom filter, and their bit is marked in the `BFXL` chunk.\n> +Overrides the `commitGraph.maxNewFilters` configuration.\n\nThe BFXL chunk doesn't exist anymore in this iteration, right?\n\nI wondered about having a user-facing \"-1\" here. My gut feeling is that\nwe usually use \"0\" to mean \"no limit\" in other places, and it probably\nmake sense to be consistent. It does look like we use both, though, and\nI'm having trouble formulating a grep pattern to find examples that\ndoesn't produce a lot of noise.\n\nThese are \"0 is no limit\":\n\n pack.windowMemory\n pack.deltaCacheSize\n git-daemon --max-connections\n\nThese are \"-1 is no limit\":\n\n git-grep --max-depth\n rev-list --max-parents (I think?)\n\nSo I dunno. It's a pretty minor thing, but I think it's good to aim for\nconsistency, and since this is user-facing we won't be able to change it\nlater.\n\n-Peff\n"},{"id":"405428","messageId":"ee881087-76b7-901d-d75c-814ad508f90d@gmail.com","threadId":"54216","inReplyTo":"20200910154516.GA32117@nand.local","subject":"Re: [PATCH 10/12] bloom: encode out-of-bounds filters as non-empty","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-09-11T18:15:18Z","receivedAt":"2020-09-11T18:15:33Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/10/2020 11:45 AM, Taylor Blau wrote:\n> On Wed, Sep 09, 2020 at 11:35:57PM -0400, Taylor Blau wrote:\n>> On Wed, Sep 09, 2020 at 11:23:48AM -0400, Taylor Blau wrote:\n>>> -\t\tfilter->data = NULL;\n>>> -\t\tfilter->len = 0;\n>>> +\t\tfilter->data = xmalloc(1);\n>>> +\t\tfilter->data[0] = 0xFF;\n>>> +\t\tfilter->len = 1;\n>>>\n>>> \t\tif (computed)\n>>> \t\t\t*computed |= BLOOM_TRUNC_LARGE;\n>>\n>> Oops, I missed the case that added by the previous patch where the\n>> number of diff entries is smaller than the limit, but the hashmap\n>> entries (after directories are added and such) crosses the threshold.\n>>\n>> Specifically, this patch doesn't write the 0xFF filter like it should.\n>> I'll send a different version of this patch tomorrow.\n> \n> This one should do the trick. Let's use it instead.\n> \n> --- >8 ---\n> \n> Subject: [PATCH] bloom: encode out-of-bounds filters as non-empty\n> \n> When a changed-path Bloom filter has either zero, or more than a\n> certain number (commonly 512) of entries, the commit-graph machinery\n> encodes it as \"missing\". More specifically, it sets the indices adjacent\n> in the BIDX chunk as equal to each other to indicate a \"length 0\"\n> filter; that is, that the filter occupies zero bytes on disk.\n> \n> This has heretofore been fine, since the commit-graph machinery has no\n> need to care about these filters with too few or too many changed paths.\n> Both cases act like no filter has been generated at all, and so there is\n> no need to store them.\n> \n> In a subsequent commit, however, the commit-graph machinery will learn\n> to only compute Bloom filters for some commits in the current\n> commit-graph layer. This is a change from the current implementation\n> which computes Bloom filters for all commits that are in the layer being\n> written. Critically for this patch, only computing some of the Bloom\n> filters means adding a third state for length 0 Bloom filters: zero\n> entries, too many entries, or \"hasn't been computed\".\n> \n> It will be important for that future patch to distinguish between \"not\n> representable\" (i.e., zero or too-many changed paths), and \"hasn't been\n> computed\". In particular, we don't want to waste time recomputing\n> filters that have already been computed.\n> \n> To that end, change how we store Bloom filters in the \"computed but not\n> representable\" category:\n> \n> - Bloom filters with no entries are stored as a single byte with all\n> bits low (i.e., all queries to that Bloom filter will return\n> \"definitely not\")\n> \n> - Bloom filters with too many entries are stored as a single byte with\n> all bits set high (i.e., all queries to that Bloom filter will\n> return \"maybe\").\n> \n> These rules are sufficient to not incur a behavior change by changing\n> the on-disk representation of these two classes. Likewise, no\n> specification changes are necessary for the commit-graph format, either:\n> \n> - Filters that were previously empty will be recomputed and stored\n> according to the new rules, and\n> \n> - old clients reading filters generated by new clients will interpret\n> the filters correctly and be none the wiser to how they were\n> generated.\n> \n> Clients will invoke the Bloom machinery in more cases than before, but\n> this can be addressed by returning a NULL filter when all bits are set\n> high. This can be addressed in a future patch.\n> \n> Finally, note that this does increase the size of on-disk commit-graphs,\n> but far less than other proposals. In particular, this is generally more\n> efficient than storing a bitmap for which commits haven't computed their\n> Bloom filters. Storing a bitmap incurs a penalty of one bit per commit,\n> whereas storing explicit filters as above incurs a penalty of one byte\n> per too-large or too-small commit.\n> \n> In practice, these boundary commits likely occupy a small proportion of\n> the overall number of commits, and so the size penalty is likely smaller\n> than storing a bitmap for all commits.\n> \n> A test to exercise filters which contain too many changed path entries\n> will be introduced in a subsequent patch.\n> \n> Suggested-by: SZEDER Gábor \n> Suggested-by: Jakub Narębski \n> Helped-by: Derrick Stolee \n> Signed-off-by: Taylor Blau \n> ---\n> .../technical/commit-graph-format.txt | 2 +-\n> bloom.c | 16 ++++++++++++--\n> bloom.h | 1 +\n> commit-graph.c | 5 +++++\n> t/t0095-bloom.sh | 8 +++----\n> t/t4216-log-bloom.sh | 21 +++++++++++++++++--\n> 6 files changed, 44 insertions(+), 9 deletions(-)\n> \n> diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n> index 6ddbceba15..6585f1948a 100644\n> --- a/Documentation/technical/commit-graph-format.txt\n> +++ b/Documentation/technical/commit-graph-format.txt\n> @@ -125,7 +125,7 @@ CHUNK DATA:\n> * The rest of the chunk is the concatenation of all the computed Bloom\n> filters for the commits in lexicographic order.\n> * Note: Commits with no changes or more than 512 changes have Bloom filters\n> - of length zero.\n> + of length one, with either all bits set to zero or one respectively.\n> * The BDAT chunk is present if and only if BIDX is present.\n> \n> Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\n> diff --git a/bloom.c b/bloom.c\n> index db9fb82437..d24747a1d5 100644\n> --- a/bloom.c\n> +++ b/bloom.c\n> @@ -177,6 +177,13 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n> \treturn strcmp(e1->path, e2->path);\n> }\n> \n> +static void init_truncated_large_filter(struct bloom_filter *filter)\n> +{\n> +\tfilter->data = xmalloc(1);\n> +\tfilter->data[0] = 0xFF;\n> +\tfilter->len = 1;\n> +}\n> +\n\nSo this hunk is essentially the new bit in this version.\n\n> diff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\n> index 232ba2c485..7e4ab1795f 100755\n> --- a/t/t0095-bloom.sh\n> +++ b/t/t0095-bloom.sh\n> @@ -71,8 +71,8 @@ test_expect_success 'get bloom filters for commit with no changes' '\n> \tgit init &&\n> \tgit commit --allow-empty -m \"c0\" &&\n> \tcat >expect <<-\\EOF &&\n> -\tFilter_Length:0\n> -\tFilter_Data:\n> +\tFilter_Length:1\n> +\tFilter_Data:00|\n> \tEOF\n> \ttest-tool bloom get_filter_for_commit \"$(git rev-parse HEAD)\" >actual &&\n> \ttest_cmp expect actual\n> @@ -107,8 +107,8 @@ test_expect_success EXPENSIVE 'get bloom filter for commit with 513 changes' '\n> \tgit add bigDir &&\n> \tgit commit -m \"commit with 513 changes\" &&\n> \tcat >expect <<-\\EOF &&\n> -\tFilter_Length:0\n> -\tFilter_Data:\n> +\tFilter_Length:1\n> +\tFilter_Data:ff|\n> \tEOF\n> \ttest-tool bloom get_filter_for_commit \"$(git rev-parse HEAD)\" >actual &&\n> \ttest_cmp expect actual\n\nAnd these two tests now show the new behavior. Nice.\n\nThis version matches my expectations. Thanks.\n\n-Stolee\n"},{"id":"405438","messageId":"20200911185934.GA2871@xor.lan","threadId":"54216","inReplyTo":"20200911175216.GA2693949@coredump.intra.peff.net","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-11T18:59:34Z","receivedAt":"2020-09-11T18:59:47Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 11, 2020 at 01:52:16PM -0400, Jeff King wrote:\n> On Wed, Sep 09, 2020 at 11:24:00AM -0400, Taylor Blau wrote:\n> > +With the `--max-new-filters=` option, generate at most `n` new Bloom\n> > +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n> > +enforced. Commits whose filters are not calculated are stored as a\n> > +length zero Bloom filter, and their bit is marked in the `BFXL` chunk.\n> > +Overrides the `commitGraph.maxNewFilters` configuration.\n>\n> The BFXL chunk doesn't exist anymore in this iteration, right?\n\nAck; I'll have to drop that.\n\n> I wondered about having a user-facing \"-1\" here. My gut feeling is that\n> we usually use \"0\" to mean \"no limit\" in other places, and it probably\n> make sense to be consistent. It does look like we use both, though, and\n> I'm having trouble formulating a grep pattern to find examples that\n> doesn't produce a lot of noise.\n>\n> These are \"0 is no limit\":\n>\n> pack.windowMemory\n> pack.deltaCacheSize\n> git-daemon --max-connections\n>\n> These are \"-1 is no limit\":\n>\n> git-grep --max-depth\n> rev-list --max-parents (I think?)\n>\n> So I dunno. It's a pretty minor thing, but I think it's good to aim for\n> consistency, and since this is user-facing we won't be able to change it\n> later.\n\nI think that we have to treat \"-1\" as the no-limit indicator, or\notherwise we'd have to specify some other way to say we don't want to\ngenerate any filters. With this patch, users can write:\n\n $ git commit-graph write --changed-paths .. --max-new-filters=0\n\nto generate a commit-graph without writing any new filters. This is\nimportant to be able to do since we also have a\n'commitGraph.maxNewFilters' configuration, which callers may want to\noverride.\n\nYou may wonder why you wouldn't just write '--no-changed-paths' instead.\nDoing so would indeed generate no new filters, but it also wouldn't\nwrite any already existing filters into a new graph which is important\nwhen rolling up graph layers that already have incrementals, for example\nwith '--split'.\n\nI'm happy to include all or none of this in a re-rolled commit message\nif you think it's relevant, too.\n\n> -Peff\n\nThanks,\nTaylor\n"},{"id":"405446","messageId":"20200911192555.GA3612@nand.local","threadId":"54216","inReplyTo":"20200911185934.GA2871@xor.lan","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-11T19:25:55Z","receivedAt":"2020-09-11T19:26:10Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 11, 2020 at 02:59:34PM -0400, Taylor Blau wrote:\n> On Fri, Sep 11, 2020 at 01:52:16PM -0400, Jeff King wrote:\n> > On Wed, Sep 09, 2020 at 11:24:00AM -0400, Taylor Blau wrote:\n> > > +With the `--max-new-filters=` option, generate at most `n` new Bloom\n> > > +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n> > > +enforced. Commits whose filters are not calculated are stored as a\n> > > +length zero Bloom filter, and their bit is marked in the `BFXL` chunk.\n> > > +Overrides the `commitGraph.maxNewFilters` configuration.\n> >\n> > The BFXL chunk doesn't exist anymore in this iteration, right?\n>\n> Ack; I'll have to drop that.\n\nJunio, I know that I've already sent one replacement patch. If you don't\nmind, here's another (and if you do mind, I'm happy to re-roll the\nseries).\n\nThanks.\n\n--- >8 ---\n\nSubject: [PATCH] builtin/commit-graph.c: introduce '--max-new-filters='\n\nIntroduce a command-line flag and configuration variable to specify the\nmaximum number of new Bloom filters that a 'git commit-graph write' is\nwilling to compute from scratch.\n\nPrior to this patch, a commit-graph write with '--changed-paths' would\ncompute Bloom filters for all selected commits which haven't already\nbeen computed (i.e., by a previous commit-graph write with '--split'\nsuch that a roll-up or replacement is performed).\n\nThis behavior can cause prohibitively-long commit-graph writes for a\nvariety of reasons:\n\n * There may be lots of filters whose diffs take a long time to\n generate (for example, they have close to the maximum number of\n changes, diffing itself takes a long time, etc).\n\n * Old-style commit-graphs (which encode filters with too many entries\n as not having been computed at all) cause us to waste time\n recomputing filters that appear to have not been computed only to\n discover that they are too-large.\n\nThis can make the upper-bound of the time it takes for 'git commit-graph\nwrite --changed-paths' to be rather unpredictable.\n\nTo make this command behave more predictably, introduce\n'--max-new-filters=' to allow computing at most '' Bloom filters\nfrom scratch. This lets \"computing\" already-known filters proceed\nquickly, while bounding the number of slow tasks that Git is willing to\ndo.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 4 +++\n Documentation/git-commit-graph.txt | 6 ++++\n bloom.c | 7 ++--\n builtin/commit-graph.c | 39 ++++++++++++++++++--\n commit-graph.c | 9 +++--\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 53 ++++++++++++++++++++++++++++\n 7 files changed, 110 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex cff0797b54..4582c39fc4 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitGraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..60df4e4bfa 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,12 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Commits whose filters are not calculated are stored as a\n+length zero Bloom filter. Overrides the `commitGraph.maxNewFilters`\n+configuration.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex d24747a1d5..230a515831 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -204,12 +204,11 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n-\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n+\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n\n-\tif (filter->data)\n+\tif (filter->data && filter->len)\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f3243bd982..e7a1539b08 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n\n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n\n@@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n\n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n\n@@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n\n \ttrace2_cmd_mode(\"write\");\n\n@@ -270,6 +293,16 @@ static int graph_write(int argc, const char **argv)\n \treturn result;\n }\n\n+static int git_commit_graph_config(const char *var, const char *value, void *cb)\n+{\n+\tif (!strcmp(var, \"commitgraph.maxnewfilters\")) {\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t\treturn 0;\n+\t}\n+\n+\treturn git_default_config(var, value, cb);\n+}\n+\n int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n {\n \tstatic struct option builtin_commit_graph_options[] = {\n@@ -283,7 +316,7 @@ int cmd_commit_graph(int argc, const char **argv, const char *prefix)\n \t\tusage_with_options(builtin_commit_graph_usage,\n \t\t\t\t builtin_commit_graph_options);\n\n-\tgit_config(git_default_config, NULL);\n+\tgit_config(git_commit_graph_config, &opts);\n \targc = parse_options(argc, argv, prefix,\n \t\t\t builtin_commit_graph_options,\n \t\t\t builtin_commit_graph_usage,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex dcc27b74e3..1d9f8cc7e9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1422,6 +1422,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tstruct commit **sorted_commits;\n+\tint max_new_filters;\n\n \tinit_bloom_filters();\n\n@@ -1438,13 +1439,16 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \telse\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n\n+\tmax_new_filters = ctx->opts && ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tenum bloom_filter_computed computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\tctx->r,\n \t\t\tc,\n-\t\t\t1,\n+\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n \t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n@@ -1455,7 +1459,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n \t\t\tctx->count_bloom_filter_not_computed++;\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\tctx->total_bloom_filter_data_size += filter\n+\t\t\t? sizeof(unsigned char) * filter->len : 0;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n\ndiff --git a/commit-graph.h b/commit-graph.h\nindex b7914b0a7a..a22bd86701 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -110,6 +110,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags split_flags;\n+\tint max_new_filters;\n };\n\n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex a56327ffd4..24deb8104a 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -287,5 +287,58 @@ test_expect_success 'correctly report commits with no changed paths' '\n \t\tgrep \"\\\"filter_trunc_large\\\":0\" trace\n \t)\n '\n+test_bloom_filters_computed () {\n+\tcommit_graph_args=$1\n+\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n+\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write \\\n+\t\t$commit_graph_args &&\n+\tgrep \"\\\"filter_not_computed\\\":$2\" \"$TRASH_DIRECTORY/trace.event\" &&\n+\tgrep \"\\\"filter_trunc_large\\\":$3\" \"$TRASH_DIRECTORY/trace.event\" &&\n+\tgrep \"\\\"filter_computed\\\":$4\" \"$TRASH_DIRECTORY/trace.event\"\n+}\n+\n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n+\t\t\t3 0 2\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=1\" \\\n+\t\t\t4 0 1\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills empty commits' '\n+\tgit init empty &&\n+\ttest_when_finished \"rm -fr empty\" &&\n+\t(\n+\t\tcd empty &&\n+\t\tfor i in $(test_seq 1 6)\n+\t\tdo\n+\t\t\tgit commit --allow-empty -m \"$i\"\n+\t\tdone &&\n+\n+\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 0 2 &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 0 2 &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 0 2 &&\n+\n+\t\t# Finally, make sure that once all commits have filters, that\n+\t\t# none are subsequently recomputed.\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t6 0 0\n+\t)\n+'\n\n test_done\n--\n2.27.0.2924.ga64bac9092.dirty\n\n"},{"id":"405447","messageId":"xmqqft7onm9s.fsf@gitster.c.googlers.com","threadId":"54216","inReplyTo":"20200911175216.GA2693949@coredump.intra.peff.net","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-11T19:31:11Z","receivedAt":"2020-09-11T19:31:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff King writes:\n\n> I wondered about having a user-facing \"-1\" here. My gut feeling is that\n> we usually use \"0\" to mean \"no limit\" in other places, and it probably\n> make sense to be consistent. It does look like we use both, though, and\n> I'm having trouble formulating a grep pattern to find examples that\n> doesn't produce a lot of noise.\n>\n> These are \"0 is no limit\":\n>\n> pack.windowMemory\n> pack.deltaCacheSize\n> git-daemon --max-connections\n>\n> These are \"-1 is no limit\":\n>\n> git-grep --max-depth\n> rev-list --max-parents (I think?)\n\nI am unsure if \"limiting to the top-level\" is depth 0 or depth 1,\nbut if it is depth 0, --max-depth=0 that does not recurse is\nsensible and cannot be used as a signal for \"unlimited\".\n\nSame for --max-parents=0 that would be a legit way to ask for \"root\ncommits only\".\n\nI do not think the system fundamentally would not work with 0 bytes\nof window memory or 0 connections, so \"0 is unlimited\" for them\nsounds appropriate. I would not be surprised if the reason why \"0\nis unlimited\" fields did not choose to use \"-1\" as the \"unlimited\"\nsignal was because the internal type for these fields is unsigned.\n\n"},{"id":"405449","messageId":"20200911194730.GA2697651@coredump.intra.peff.net","threadId":"54216","inReplyTo":"20200911185934.GA2871@xor.lan","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2020-09-11T19:47:30Z","receivedAt":"2020-09-11T19:47:36Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Fri, Sep 11, 2020 at 02:59:34PM -0400, Taylor Blau wrote:\n\n> I think that we have to treat \"-1\" as the no-limit indicator, or\n> otherwise we'd have to specify some other way to say we don't want to\n> generate any filters. With this patch, users can write:\n> \n> $ git commit-graph write --changed-paths .. --max-new-filters=0\n> \n> to generate a commit-graph without writing any new filters. This is\n> important to be able to do since we also have a\n> 'commitGraph.maxNewFilters' configuration, which callers may want to\n> override.\n\nOK, that makes sense. Consistency would be nice, but I agree it just\nwouldn't work here (and we're not entirely consistent anyway, so it's\nnot that big a loss).\n\n-Peff\n"},{"id":"405528","messageId":"20200914201258.GA12431@nand.local","threadId":"54216","inReplyTo":"20200911192555.GA3612@nand.local","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-14T20:12:58Z","receivedAt":"2020-09-14T20:13:06Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 11, 2020 at 03:25:55PM -0400, Taylor Blau wrote:\n> On Fri, Sep 11, 2020 at 02:59:34PM -0400, Taylor Blau wrote:\n> > On Fri, Sep 11, 2020 at 01:52:16PM -0400, Jeff King wrote:\n> > > On Wed, Sep 09, 2020 at 11:24:00AM -0400, Taylor Blau wrote:\n> > > > +With the `--max-new-filters=` option, generate at most `n` new Bloom\n> > > > +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n> > > > +enforced. Commits whose filters are not calculated are stored as a\n> > > > +length zero Bloom filter, and their bit is marked in the `BFXL` chunk.\n> > > > +Overrides the `commitGraph.maxNewFilters` configuration.\n> > >\n> > > The BFXL chunk doesn't exist anymore in this iteration, right?\n> >\n> > Ack; I'll have to drop that.\n>\n> Junio, I know that I've already sent one replacement patch. If you don't\n> mind, here's another (and if you do mind, I'm happy to re-roll the\n> series).\n\nJust kidding. Let's use *this* version which fixes a bug reading the\ncommitGraph.maxNewFilters configuration. At this point, the\nfix-ups are:\n\n - This patch (attached below the scisors) instead of 12/12, and\n\n - This [1] patch instead of 10/12.\n\n[1]: https://lore.kernel.org/git/20200910154516.GA32117@nand.local/\n\nLet me know if you'd rather have a full re-roll.\n\n--- 8< ---\n\nSubject: [PATCH] builtin/commit-graph.c: introduce '--max-new-filters='\n\nIntroduce a command-line flag and configuration variable to specify the\nmaximum number of new Bloom filters that a 'git commit-graph write' is\nwilling to compute from scratch.\n\nPrior to this patch, a commit-graph write with '--changed-paths' would\ncompute Bloom filters for all selected commits which haven't already\nbeen computed (i.e., by a previous commit-graph write with '--split'\nsuch that a roll-up or replacement is performed).\n\nThis behavior can cause prohibitively-long commit-graph writes for a\nvariety of reasons:\n\n * There may be lots of filters whose diffs take a long time to\n generate (for example, they have close to the maximum number of\n changes, diffing itself takes a long time, etc).\n\n * Old-style commit-graphs (which encode filters with too many entries\n as not having been computed at all) cause us to waste time\n recomputing filters that appear to have not been computed only to\n discover that they are too-large.\n\nThis can make the upper-bound of the time it takes for 'git commit-graph\nwrite --changed-paths' to be rather unpredictable.\n\nTo make this command behave more predictably, introduce\n'--max-new-filters=' to allow computing at most '' Bloom filters\nfrom scratch. This lets \"computing\" already-known filters proceed\nquickly, while bounding the number of slow tasks that Git is willing to\ndo.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 4 ++\n Documentation/git-commit-graph.txt | 6 +++\n bloom.c | 7 ++--\n builtin/commit-graph.c | 41 ++++++++++++++++++++-\n commit-graph.c | 9 ++++-\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 55 ++++++++++++++++++++++++++++\n 7 files changed, 115 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex cff0797b54..4582c39fc4 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitGraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..60df4e4bfa 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,12 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Commits whose filters are not calculated are stored as a\n+length zero Bloom filter. Overrides the `commitGraph.maxNewFilters`\n+configuration.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex d24747a1d5..230a515831 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -204,12 +204,11 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n-\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n+\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n\n-\tif (filter->data)\n+\tif (filter->data && filter->len)\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f3243bd982..988445abdf 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n\n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n\n@@ -162,6 +164,35 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n\n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n+static int git_commit_graph_write_config(const char *var, const char *value,\n+\t\t\t\t\t void *cb)\n+{\n+\tif (!strcmp(var, \"commitgraph.maxnewfilters\"))\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t/*\n+\t * No need to fall-back to 'git_default_config', since this was already\n+\t * called in 'cmd_commit_graph()'.\n+\t */\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +228,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n\n@@ -205,9 +239,12 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n\n \ttrace2_cmd_mode(\"write\");\n\n+\tgit_config(git_commit_graph_write_config, &opts);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\ndiff --git a/commit-graph.c b/commit-graph.c\nindex dcc27b74e3..1d9f8cc7e9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1422,6 +1422,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tstruct commit **sorted_commits;\n+\tint max_new_filters;\n\n \tinit_bloom_filters();\n\n@@ -1438,13 +1439,16 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \telse\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n\n+\tmax_new_filters = ctx->opts && ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tenum bloom_filter_computed computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\tctx->r,\n \t\t\tc,\n-\t\t\t1,\n+\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n \t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n@@ -1455,7 +1459,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n \t\t\tctx->count_bloom_filter_not_computed++;\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\tctx->total_bloom_filter_data_size += filter\n+\t\t\t? sizeof(unsigned char) * filter->len : 0;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n\ndiff --git a/commit-graph.h b/commit-graph.h\nindex b7914b0a7a..a22bd86701 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -110,6 +110,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags split_flags;\n+\tint max_new_filters;\n };\n\n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex a56327ffd4..3cb766301d 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -287,5 +287,60 @@ test_expect_success 'correctly report commits with no changed paths' '\n \t\tgrep \"\\\"filter_trunc_large\\\":0\" trace\n \t)\n '\n+test_bloom_filters_computed () {\n+\tcommit_graph_args=$1\n+\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n+\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write \\\n+\t\t$commit_graph_args &&\n+\tgrep \"\\\"filter_not_computed\\\":$2\" \"$TRASH_DIRECTORY/trace.event\" &&\n+\tgrep \"\\\"filter_trunc_large\\\":$3\" \"$TRASH_DIRECTORY/trace.event\" &&\n+\tgrep \"\\\"filter_computed\\\":$4\" \"$TRASH_DIRECTORY/trace.event\"\n+}\n+\n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace --max-new-filters=2\" \\\n+\t\t\t3 0 2\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t# Check specifying commitGraph.maxNewFilters over \"git config\" works.\n+\ttest_config -C limits commitGraph.maxNewFilters 1 &&\n+\t(\n+\t\tcd limits &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n+\t\t\t4 0 1\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills empty commits' '\n+\tgit init empty &&\n+\ttest_when_finished \"rm -fr empty\" &&\n+\t(\n+\t\tcd empty &&\n+\t\tfor i in $(test_seq 1 6)\n+\t\tdo\n+\t\t\tgit commit --allow-empty -m \"$i\"\n+\t\tdone &&\n+\n+\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 0 2 &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 0 2 &&\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t4 0 2 &&\n+\n+\t\t# Finally, make sure that once all commits have filters, that\n+\t\t# none are subsequently recomputed.\n+\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n+\t\t\t6 0 0\n+\t)\n+'\n\n test_done\n--\n2.27.0.2924.ga64bac9092.dirty\n\n"},{"id":"405530","messageId":"134d64a0-abb6-bdc9-2c05-7aded01a906a@gmail.com","threadId":"54216","inReplyTo":"20200914201258.GA12431@nand.local","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-09-14T20:31:03Z","receivedAt":"2020-09-14T20:32:03Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/14/2020 4:12 PM, Taylor Blau wrote:\n> On Fri, Sep 11, 2020 at 03:25:55PM -0400, Taylor Blau wrote:\n>> On Fri, Sep 11, 2020 at 02:59:34PM -0400, Taylor Blau wrote:\n>>> On Fri, Sep 11, 2020 at 01:52:16PM -0400, Jeff King wrote:\n>>>> On Wed, Sep 09, 2020 at 11:24:00AM -0400, Taylor Blau wrote:\n>>>>> +With the `--max-new-filters=` option, generate at most `n` new Bloom\n>>>>> +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n>>>>> +enforced. Commits whose filters are not calculated are stored as a\n>>>>> +length zero Bloom filter, and their bit is marked in the `BFXL` chunk.\n>>>>> +Overrides the `commitGraph.maxNewFilters` configuration.\n>>>>\n>>>> The BFXL chunk doesn't exist anymore in this iteration, right?\n>>>\n>>> Ack; I'll have to drop that.\n>>\n>> Junio, I know that I've already sent one replacement patch. If you don't\n>> mind, here's another (and if you do mind, I'm happy to re-roll the\n>> series).\n> \n> Just kidding. Let's use *this* version which fixes a bug reading the\n> commitGraph.maxNewFilters configuration. At this point, the\n> fix-ups are:\n> \n> - This patch (attached below the scisors) instead of 12/12, and\n> \n> - This [1] patch instead of 10/12.\n> \n> [1]: https://lore.kernel.org/git/20200910154516.GA32117@nand.local/\n> \n> Let me know if you'd rather have a full re-roll.\n\nIt's getting a bit difficult to track all of these \"use this instead\"\npatches. But, I'm not the one applying them, so maybe that's not actually\na problem.\n\nYou might need a re-roll, anyway, as I have a few comments here:\n\n> --- 8< ---\n> \n> Subject: [PATCH] builtin/commit-graph.c: introduce '--max-new-filters='\n\nYou also introduce commitGraph.maxNewFitlers here, which is not\nmentioned in the commit message anywhere. In fact, it might be\ngood to include it as a separate patch so its implementation and\ntests can be isolated from the command-line functionality.\n\n> +length zero Bloom filter. Overrides the `commitGraph.maxNewFilters`\n> +configuration.\n\nWe have found it valuable to demonstrate these overrides in tests.\nLet's inspect your tests for this.\n\n> +test_bloom_filters_computed () {\n> +\tcommit_graph_args=$1\n> +\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n> +\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write \\\n> +\t\t$commit_graph_args &&\n> +\tgrep \"\\\"filter_not_computed\\\":$2\" \"$TRASH_DIRECTORY/trace.event\" &&\n> +\tgrep \"\\\"filter_trunc_large\\\":$3\" \"$TRASH_DIRECTORY/trace.event\" &&\n> +\tgrep \"\\\"filter_computed\\\":$4\" \"$TRASH_DIRECTORY/trace.event\"\n> +}\n\nIf the arguments were moved to the last parameter, then we could do a few\ninteresting things here.\n\ntest_bloom_filters_computed () {\n\tNOT_COMPUTED=\"\\\"filter_not_computed\\\":$1\" &&\n\tshift &&\n\tTRUNCATED=\"\\\"filter_trunc_large\\\":$1\" &&\n\tshift &&\n\tCOMPUTED=\"\\\"filter_computed\\\":$1\" &&\n\tshift &&\n\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write $@ &&\n\tgrep \"$NOT_COMPUTED\" \"$TRASH_DIRECTORY/trace.event\" &&\n\tgrep \"$TRUNCATED\" \"$TRASH_DIRECTORY/trace.event\" &&\n\tgrep \"$COMPUTED\" \"$TRASH_DIRECTORY/trace.event\"\n}\n\n\n(I have not tested this script. It might need some work.)\nThis would make your callers a bit cleaner-looking, for example:\n\ntest_expect_success 'Bloom generation is limited by --max-new-filters' '\n\t(\n\t\tcd limits &&\n\t\ttest_commit c2 filter &&\n\t\ttest_commit c3 filter &&\n\t\ttest_commit c4 no-filter &&\n\t\ttest_bloom_filters_computed 3 0 2 \\\n\t\t\t--reachable --changed-paths --split=replace --max-new-filters=2\n\t)\n'\n\nAt least, this looks nicer to me.\n\n> +test_expect_success 'Bloom generation backfills previously-skipped filters' '\n> +\t# Check specifying commitGraph.maxNewFilters over \"git config\" works.\n> +\ttest_config -C limits commitGraph.maxNewFilters 1 &&\n> +\t(\n> +\t\tcd limits &&\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n> +\t\t\t4 0 1\n> +\t)\n> +'\n\nAdding a case for `commitGraph.maxNewFilters=1` and `--max-new-filters=2` might\nbe interesting for the override rules.\n\n> +\n> +test_expect_success 'Bloom generation backfills empty commits' '\n> +\tgit init empty &&\n> +\ttest_when_finished \"rm -fr empty\" &&\n> +\t(\n> +\t\tcd empty &&\n> +\t\tfor i in $(test_seq 1 6)\n> +\t\tdo\n> +\t\t\tgit commit --allow-empty -m \"$i\"\n> +\t\tdone &&\n> +\n> +\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n> +\t\t\t4 0 2 &&\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n> +\t\t\t4 0 2 &&\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n> +\t\t\t4 0 2 &&\n\nI'm concerned that the max-new-filters limit (2) is a divisor\nof the full number of commits (6). It might be good to add one\nmore commit here and test again with a limit of 2. That would\nhandle both \"equal to limit\" and \"less than limit\" cases.\n\n> +\t\t# Finally, make sure that once all commits have filters, that\n> +\t\t# none are subsequently recomputed.\n> +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n> +\t\t\t6 0 0\n> +\t)\n> +'\n\nThanks,\n-Stolee\n\n"},{"id":"405532","messageId":"20200914203659.GA12855@nand.local","threadId":"54216","inReplyTo":"134d64a0-abb6-bdc9-2c05-7aded01a906a@gmail.com","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-14T20:36:59Z","receivedAt":"2020-09-14T20:37:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Sep 14, 2020 at 04:31:03PM -0400, Derrick Stolee wrote:\n> On 9/14/2020 4:12 PM, Taylor Blau wrote:\n> > - This patch (attached below the scisors) instead of 12/12, and\n> >\n> > - This [1] patch instead of 10/12.\n> >\n> > [1]: https://lore.kernel.org/git/20200910154516.GA32117@nand.local/\n> >\n> > Let me know if you'd rather have a full re-roll.\n>\n> It's getting a bit difficult to track all of these \"use this instead\"\n> patches. But, I'm not the one applying them, so maybe that's not actually\n> a problem.\n\nThe above list is the only changes that I've made, so I'm happy if Junio\nwants to follow what's written there, but I'm equally happy to send a\nnew reroll.\n\n> You might need a re-roll, anyway, as I have a few comments here:\n\nLet's take a look...\n\n> You also introduce commitGraph.maxNewFitlers here, which is not\n> mentioned in the commit message anywhere. In fact, it might be\n> good to include it as a separate patch so its implementation and\n> tests can be isolated from the command-line functionality.\n\nI could go either way on both of these, to be honest. I don't think\nthere's anything interesting that isn't said in the documentation\nchanges introduced by that commit that is worth convering there, so I'm\nnot sue 'commitGraph.maxNewFilters' needs the additional call-out.\n\n> > +length zero Bloom filter. Overrides the `commitGraph.maxNewFilters`\n> > +configuration.\n>\n> We have found it valuable to demonstrate these overrides in tests.\n> Let's inspect your tests for this.\n>\n> > +test_bloom_filters_computed () {\n> > +\tcommit_graph_args=$1\n> > +\trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n> > +\tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write \\\n> > +\t\t$commit_graph_args &&\n> > +\tgrep \"\\\"filter_not_computed\\\":$2\" \"$TRASH_DIRECTORY/trace.event\" &&\n> > +\tgrep \"\\\"filter_trunc_large\\\":$3\" \"$TRASH_DIRECTORY/trace.event\" &&\n> > +\tgrep \"\\\"filter_computed\\\":$4\" \"$TRASH_DIRECTORY/trace.event\"\n> > +}\n>\n> If the arguments were moved to the last parameter, then we could do a few\n> interesting things here.\n>\n> test_bloom_filters_computed () {\n> \tNOT_COMPUTED=\"\\\"filter_not_computed\\\":$1\" &&\n> \tshift &&\n> \tTRUNCATED=\"\\\"filter_trunc_large\\\":$1\" &&\n> \tshift &&\n> \tCOMPUTED=\"\\\"filter_computed\\\":$1\" &&\n> \tshift &&\n> \trm -f \"$TRASH_DIRECTORY/trace.event\" &&\n> \tGIT_TRACE2_EVENT=\"$TRASH_DIRECTORY/trace.event\" git commit-graph write $@ &&\n> \tgrep \"$NOT_COMPUTED\" \"$TRASH_DIRECTORY/trace.event\" &&\n> \tgrep \"$TRUNCATED\" \"$TRASH_DIRECTORY/trace.event\" &&\n> \tgrep \"$COMPUTED\" \"$TRASH_DIRECTORY/trace.event\"\n> }\n>\n>\n> (I have not tested this script. It might need some work.)\n> This would make your callers a bit cleaner-looking, for example:\n>\n> test_expect_success 'Bloom generation is limited by --max-new-filters' '\n> \t(\n> \t\tcd limits &&\n> \t\ttest_commit c2 filter &&\n> \t\ttest_commit c3 filter &&\n> \t\ttest_commit c4 no-filter &&\n> \t\ttest_bloom_filters_computed 3 0 2 \\\n> \t\t\t--reachable --changed-paths --split=replace --max-new-filters=2\n> \t)\n> '\n>\n> At least, this looks nicer to me.\n\nYeah, but I think we're still stuck with the test_config below unless\nyou write \"git $@\" instead of \"git commit-graph write $@\". I don't think\nthat I have strong feelings about this unless you do.\n\n> > +test_expect_success 'Bloom generation backfills previously-skipped filters' '\n> > +\t# Check specifying commitGraph.maxNewFilters over \"git config\" works.\n> > +\ttest_config -C limits commitGraph.maxNewFilters 1 &&\n> > +\t(\n> > +\t\tcd limits &&\n> > +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --split=replace\" \\\n> > +\t\t\t4 0 1\n> > +\t)\n> > +'\n>\n> Adding a case for `commitGraph.maxNewFilters=1` and `--max-new-filters=2` might\n> be interesting for the override rules.\n\nPotentially. I'm equally happy to do it in a follow-up series. I worry\nslightly about adding too many test-cases for somewhat trivial behavior.\n\n> > +\n> > +test_expect_success 'Bloom generation backfills empty commits' '\n> > +\tgit init empty &&\n> > +\ttest_when_finished \"rm -fr empty\" &&\n> > +\t(\n> > +\t\tcd empty &&\n> > +\t\tfor i in $(test_seq 1 6)\n> > +\t\tdo\n> > +\t\t\tgit commit --allow-empty -m \"$i\"\n> > +\t\tdone &&\n> > +\n> > +\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n> > +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n> > +\t\t\t4 0 2 &&\n> > +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n> > +\t\t\t4 0 2 &&\n> > +\t\ttest_bloom_filters_computed \"--reachable --changed-paths --max-new-filters=2\" \\\n> > +\t\t\t4 0 2 &&\n>\n> I'm concerned that the max-new-filters limit (2) is a divisor\n> of the full number of commits (6). It might be good to add one\n> more commit here and test again with a limit of 2. That would\n> handle both \"equal to limit\" and \"less than limit\" cases.\n\nThat case is already covered in the test two above this one (\"Bloom\ngeneration is limited by --max-new-filters\").\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"405543","messageId":"aa53f17a-6e48-f4e7-821c-1db35cf48652@gmail.com","threadId":"54216","inReplyTo":"20200914203659.GA12855@nand.local","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-09-15T00:59:28Z","receivedAt":"2020-09-15T00:59:35Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/14/2020 4:36 PM, Taylor Blau wrote:\n> On Mon, Sep 14, 2020 at 04:31:03PM -0400, Derrick Stolee wrote:\n>> On 9/14/2020 4:12 PM, Taylor Blau wrote:\n>>> - This patch (attached below the scisors) instead of 12/12, and\n>>>\n>>> - This [1] patch instead of 10/12.\n>>>\n>>> [1]: https://lore.kernel.org/git/20200910154516.GA32117@nand.local/\n>>>\n>>> Let me know if you'd rather have a full re-roll.\n>>\n>> It's getting a bit difficult to track all of these \"use this instead\"\n>> patches. But, I'm not the one applying them, so maybe that's not actually\n>> a problem.\n> \n> The above list is the only changes that I've made, so I'm happy if Junio\n> wants to follow what's written there, but I'm equally happy to send a\n> new reroll.\n> \n>> You might need a re-roll, anyway, as I have a few comments here:\n> \n> Let's take a look...\n> \n>> You also introduce commitGraph.maxNewFitlers here, which is not\n>> mentioned in the commit message anywhere. In fact, it might be\n>> good to include it as a separate patch so its implementation and\n>> tests can be isolated from the command-line functionality.\n> \n> I could go either way on both of these, to be honest. I don't think\n> there's anything interesting that isn't said in the documentation\n> changes introduced by that commit that is worth convering there, so I'm\n> not sue 'commitGraph.maxNewFilters' needs the additional call-out.\n\nThis is fine. Adding an option along with the config version of it\nis easy enough. Just a thought for future series.\n\nI'm fine with the series as-is. My nits are just that.\n\nThanks,\n-Stolee\n"},{"id":"405550","messageId":"20200915043154.GA19585@nand.local","threadId":"54216","inReplyTo":"aa53f17a-6e48-f4e7-821c-1db35cf48652@gmail.com","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-15T04:31:54Z","receivedAt":"2020-09-15T04:32:00Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Mon, Sep 14, 2020 at 08:59:28PM -0400, Derrick Stolee wrote:\n> This is fine. Adding an option along with the config version of it\n> is easy enough. Just a thought for future series.\n>\n> I'm fine with the series as-is. My nits are just that.\n\nThanks for your review, and for all of your thoughts and help on this\nseries in general. I'm sorry if I seemed dismissive; I just wanted to\navoid holding up some important fixes besides the '--max-new-filters'\nfeature.\n\nI'm not quite sure how to handle series like these. On the one hand, I'd\nlike to send a couple of small series to fix the important bugs quickly.\nOn the other hand, the small series can generate a lot of noise and\nburden reviewers and the maintainer when the dependencies between those\nseries aren't straightforward.\n\nSo, I dunno. That's at least shedding a little bit of light on how this\nseries came to be / got so large, which is part of the reason that it\ntook so long.\n\nAnyway, thank you again. I'm looking forward to seeing this merged\nhopefully soon.\n\n> Thanks,\n> -Stolee\n\nThanks,\nTaylor\n"},{"id":"405576","messageId":"xmqqv9geyaku.fsf@gitster.c.googlers.com","threadId":"54216","inReplyTo":"20200914203659.GA12855@nand.local","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-15T21:49:37Z","receivedAt":"2020-09-15T21:50:02Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n>> It's getting a bit difficult to track all of these \"use this instead\"\n>> patches. But, I'm not the one applying them, so maybe that's not actually\n>> a problem.\n>\n> The above list is the only changes that I've made, so I'm happy if Junio\n> wants to follow what's written there, but I'm equally happy to send a\n> new reroll.\n\nIt's getting so unorganized to follow from sidelines. Even\nresending just the few steps that needs replacement, indicating\nwhich ones are replaced with them, would be easier to manage (and\nfull replacement would be the easiest to handle).\n\nThanks.\n"},{"id":"405577","messageId":"20200915215359.GA2726@xor.lan","threadId":"54216","inReplyTo":"xmqqv9geyaku.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 12/12] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-15T21:53:59Z","receivedAt":"2020-09-15T21:54:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Tue, Sep 15, 2020 at 02:49:37PM -0700, Junio C Hamano wrote:\n> It's getting so unorganized to follow from sidelines. Even\n> resending just the few steps that needs replacement, indicating\n> which ones are replaced with them, would be easier to manage (and\n> full replacement would be the easiest to handle).\n>\n> Thanks.\n\nI'll send you a full re-roll, no problem.\n\nThanks,\nTaylor\n"},{"id":"405627","messageId":"cover.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH v2 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:06:45Z","receivedAt":"2020-09-16T18:07:37Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Here's another re-roll of mine and Stolee's series to add a series of\nimprovements to the Bloom filter machinery, culminating in the\ncommand-line flag '--max-new-filters'.\n\nMuch is left the same from [1], with two notable exceptions:\n\n - I took much of Stolee's feedback from [2], since the suggestions\n were good and we are rerolling, anyway.\n\n - I dropped the trace2 calls that write JSON from within the\n commit-graph machinery in favor of pure \"data\" writes, which are\n easier to grep for. This caused some test refactoring and allowed us\n to ultimately drop the 'test_bloom_filters_computed' function.\n\nHopefully this is it for this series ;). I think that it's in good shape\nnow, and I couldn't find anything in my own inspection that I wanted to\nchange. So, if others feel good, too, I think we should focus on\nincremental fixes on top of this.\n\n[1]: https://lore.kernel.org/git/cover.1599664389.git.me@ttaylorr.com/\n[2]: https://lore.kernel.org/git/134d64a0-abb6-bdc9-2c05-7aded01a906a@gmail.com/\n\nDerrick Stolee (1):\n bloom/diff: properly short-circuit on max_changes\n\nTaylor Blau (12):\n commit-graph: introduce 'get_bloom_filter_settings()'\n t4216: use an '&&'-chain\n commit-graph: pass a 'struct repository *' in more places\n t/helper/test-read-graph.c: prepare repo settings\n commit-graph: respect 'commitGraph.readChangedPaths'\n commit-graph.c: store maximum changed paths\n bloom: split 'get_bloom_filter()' in two\n bloom: use provided 'struct bloom_filter_settings'\n bloom: encode out-of-bounds filters as non-empty\n commit-graph: rename 'split_commit_graph_opts'\n builtin/commit-graph.c: introduce '--max-new-filters='\n commit-graph: introduce 'commitGraph.maxNewFilters'\n\n Documentation/config.txt | 2 +\n Documentation/config/commitgraph.txt | 8 +\n Documentation/git-commit-graph.txt | 6 +\n .../technical/commit-graph-format.txt | 2 +-\n blame.c | 8 +-\n bloom.c | 59 +++--\n bloom.h | 29 ++-\n builtin/commit-graph.c | 63 ++++-\n commit-graph.c | 140 +++++++---\n commit-graph.h | 17 +-\n diff.h | 2 -\n fuzz-commit-graph.c | 5 +-\n line-log.c | 2 +-\n repo-settings.c | 3 +\n repository.h | 1 +\n revision.c | 7 +-\n t/helper/test-bloom.c | 4 +-\n t/helper/test-read-graph.c | 3 +-\n t/t0095-bloom.sh | 8 +-\n t/t4216-log-bloom.sh | 241 ++++++++++++++++--\n t/t5324-split-commit-graph.sh | 13 +\n tree-diff.c | 5 +-\n 22 files changed, 505 insertions(+), 123 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\n--\n2.28.0.510.g86fdc5f89a\n"},{"id":"405628","messageId":"9416c467394e46e0a04c2c32737229a88998dcf7.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 04/13] t/helper/test-read-graph.c: prepare repo settings","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:07:12Z","receivedAt":"2020-09-16T18:07:38Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The read-graph test-tool is used by a number of the commit-graph test to\nassert various properties about a commit-graph. Previously, this program\nnever ran 'prepare_repo_settings()'. There was no need to do so, since\nnone of the commit-graph machinery is affected by the repo settings.\n\nIn the next patch, the commit-graph machinery's behavior will become\ndependent on the repo settings, and so loading them before running the\nrest of the test tool is critical.\n\nAs such, teach the test tool to call 'prepare_repo_settings()'.\n\nSigned-off-by: Taylor Blau \n---\n t/helper/test-read-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 6d0c962438..5f585a1725 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -12,11 +12,12 @@ int cmd__read_graph(int argc, const char **argv)\n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n \n+\tprepare_repo_settings(the_repository);\n+\n \tgraph = read_commit_graph_one(the_repository, odb);\n \tif (!graph)\n \t\treturn 1;\n \n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405629","messageId":"f9c7109daf44b2d99f21a42843a5d5591abb4087.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 03/13] commit-graph: pass a 'struct repository *' in more places","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:07:05Z","receivedAt":"2020-09-16T18:07:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a future commit, some commit-graph internals will want access to\n'r->settings', but we only have the 'struct object_directory *'\ncorresponding to that repository.\n\nAdd an additional parameter to pass the repository around in more\nplaces.\n\nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 2 +-\n commit-graph.c | 17 ++++++++++-------\n commit-graph.h | 6 ++++--\n fuzz-commit-graph.c | 5 +++--\n 4 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 523501f217..ba5584463f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -106,7 +106,7 @@ static int graph_verify(int argc, const char **argv)\n \tFREE_AND_NULL(graph_name);\n \n \tif (open_ok)\n-\t\tgraph = load_commit_graph_one_fd_st(fd, &st, odb);\n+\t\tgraph = load_commit_graph_one_fd_st(the_repository, fd, &st, odb);\n \telse\n \t\tgraph = read_commit_graph_one(the_repository, odb);\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6a36ed0b06..72a838bd00 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -231,7 +231,8 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st)\n \treturn 1;\n }\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tvoid *graph_map;\n@@ -247,7 +248,7 @@ struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n \t}\n \tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n-\tret = parse_commit_graph(graph_map, graph_size);\n+\tret = parse_commit_graph(r, graph_map, graph_size);\n \n \tif (ret)\n \t\tret->odb = odb;\n@@ -287,7 +288,8 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n \tconst unsigned char *data, *chunk_lookup;\n \tuint32_t i;\n@@ -452,7 +454,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n \treturn NULL;\n }\n \n-static struct commit_graph *load_commit_graph_one(const char *graph_file,\n+static struct commit_graph *load_commit_graph_one(struct repository *r,\n+\t\t\t\t\t\t const char *graph_file,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \n@@ -464,7 +467,7 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file,\n \tif (!open_ok)\n \t\treturn NULL;\n \n-\tg = load_commit_graph_one_fd_st(fd, &st, odb);\n+\tg = load_commit_graph_one_fd_st(r, fd, &st, odb);\n \n \tif (g)\n \t\tg->filename = xstrdup(graph_file);\n@@ -476,7 +479,7 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tchar *graph_name = get_commit_graph_filename(odb);\n-\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \tfree(graph_name);\n \n \treturn g;\n@@ -557,7 +560,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \t\tvalid = 0;\n \t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\t\tchar *graph_name = get_split_graph_filename(odb, line.buf);\n-\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \n \t\t\tfree(graph_name);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 0677dd1031..d9acb22bac 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,13 @@ struct commit_graph {\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb);\n struct commit_graph *read_commit_graph_one(struct repository *r,\n \t\t\t\t\t struct object_directory *odb);\n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n /*\n * Return 1 if and only if the repository has a commit-graph\ndiff --git a/fuzz-commit-graph.c b/fuzz-commit-graph.c\nindex 430817214d..e7cf6d5b0f 100644\n--- a/fuzz-commit-graph.c\n+++ b/fuzz-commit-graph.c\n@@ -1,7 +1,8 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n \n@@ -10,7 +11,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \tstruct commit_graph *g;\n \n \tinitialize_the_repository();\n-\tg = parse_commit_graph((void *)data, size);\n+\tg = parse_commit_graph(the_repository, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\n \n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405630","messageId":"29bc0d07c9422d58ae19750390bb08b8c739a03e.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 05/13] commit-graph: respect 'commitGraph.readChangedPaths'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:07:19Z","receivedAt":"2020-09-16T18:07:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Git uses the 'core.commitGraph' configuration value to control whether\nor not the commit graph is used when parsing commits or performing a\ntraversal.\n\nNow that commit-graphs can also contain a section for changed-path Bloom\nfilters, administrators that already have commit-graphs may find it\nconvenient to use those graphs without relying on their changed-path\nBloom filters. This can happen, for example, during a staged roll-out,\nor in the event of an incident.\n\nIntroduce 'commitGraph.readChangedPaths' to control whether or not Bloom\nfilters are read. Note that this configuration is independent from both:\n\n - 'core.commitGraph', to allow flexibility in using all parts of a\n commit-graph _except_ for its Bloom filters.\n\n - The '--changed-paths' option for 'git commit-graph write', to allow\n reading and writing Bloom filters to be controlled independently.\n\nWhen the variable is set, pretend as if no Bloom data was specified at\nall. This avoids adding additional special-casing outside of the\ncommit-graph internals.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n Documentation/config.txt | 2 ++\n Documentation/config/commitgraph.txt | 4 ++++\n commit-graph.c | 6 ++++--\n repo-settings.c | 3 +++\n repository.h | 1 +\n t/t4216-log-bloom.sh | 4 +++-\n 6 files changed, 17 insertions(+), 3 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 3042d80978..770ae79b82 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -340,6 +340,8 @@ include::config/column.txt[]\n \n include::config/commit.txt[]\n \n+include::config/commitgraph.txt[]\n+\n include::config/credential.txt[]\n \n include::config/completion.txt[]\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nnew file mode 100644\nindex 0000000000..cff0797b54\n--- /dev/null\n+++ b/Documentation/config/commitgraph.txt\n@@ -0,0 +1,4 @@\n+commitGraph.readChangedPaths::\n+\tIf true, then git will use the changed-path Bloom filters in the\n+\tcommit-graph file (if it exists, and they are present). Defaults to\n+\ttrue. See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 72a838bd00..ea54d108b9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -327,6 +327,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n+\tprepare_repo_settings(r);\n+\n \tgraph = alloc_commit_graph();\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -403,14 +405,14 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n \t\t\tif (graph->chunk_bloom_indexes)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n \t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n \t\t\tbreak;\n \n \t\tcase GRAPH_CHUNKID_BLOOMDATA:\n \t\t\tif (graph->chunk_bloom_data)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n \t\t\t\tuint32_t hash_version;\n \t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n \t\t\t\thash_version = get_be32(data + chunk_offset);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex aa61a35338..88ccce2036 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -17,9 +17,12 @@ void prepare_repo_settings(struct repository *r)\n \n \tif (!repo_config_get_bool(r, \"core.commitgraph\", &value))\n \t\tr->settings.core_commit_graph = value;\n+\tif (!repo_config_get_bool(r, \"commitgraph.readchangedpaths\", &value))\n+\t\tr->settings.commit_graph_read_changed_paths = value;\n \tif (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n \t\tr->settings.gc_write_commit_graph = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_commit_graph, 1);\n+\tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n \n \tif (!repo_config_get_int(r, \"index.version\", &value))\ndiff --git a/repository.h b/repository.h\nindex 628c834367..bacf843d46 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -30,6 +30,7 @@ struct repo_settings {\n \tint initialized;\n \n \tint core_commit_graph;\n+\tint commit_graph_read_changed_paths;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex cd89c75002..fc7693806c 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -95,7 +95,9 @@ do\n \t\t \"--ancestry-path side..master\"\n \tdo\n \t\ttest_expect_success \"git log option: $option for path: $path\" '\n-\t\t\ttest_bloom_filters_used \"$option -- $path\"\n+\t\t\ttest_bloom_filters_used \"$option -- $path\" &&\n+\t\t\ttest_config commitgraph.readChangedPaths false &&\n+\t\t\ttest_bloom_filters_not_used \"$option -- $path\"\n \t\t'\n \tdone\n done\n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405631","messageId":"dcc6769ca9fcf3496f266c9131ac355ad422642f.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 06/13] commit-graph.c: store maximum changed paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:07:26Z","receivedAt":"2020-09-16T18:07:55Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For now, we assume that there is a fixed constant describing the\nmaximum number of changed paths we are willing to store in a Bloom\nfilter.\n\nPrepare for that to (at least partially) not be the case by making it a\nmember of the 'struct bloom_filter_settings'. This will be helpful in\nthe subsequent patches by reducing the size of test cases that exercise\nstoring too many changed paths, as well as preparing for an eventual\nfuture in which this value might change.\n\nThis patch alone does not cause newly generated Bloom filters to use\na custom upper-bound on the maximum number of changed paths a single\nBloom filter can hold, that will occur in a later patch.\n\nSigned-off-by: Taylor Blau \n---\n bloom.h | 11 ++++++++++-\n commit-graph.c | 3 +++\n t/t4216-log-bloom.sh | 4 ++--\n 3 files changed, 15 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.h b/bloom.h\nindex d8fbb0fbf1..0b9b59a6fe 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -28,9 +28,18 @@ struct bloom_filter_settings {\n \t * that contain n*b bits.\n \t */\n \tuint32_t bits_per_entry;\n+\n+\t/*\n+\t * The maximum number of changed paths per commit\n+\t * before declaring a Bloom filter to be too-large.\n+\t *\n+\t * Not written to the commit-graph file.\n+\t */\n+\tuint32_t max_changed_paths;\n };\n \n-#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10 }\n+#define DEFAULT_BLOOM_MAX_CHANGES 512\n+#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10, DEFAULT_BLOOM_MAX_CHANGES }\n #define BITS_PER_WORD 8\n #define BLOOMDATA_CHUNK_HEADER_SIZE 3 * sizeof(uint32_t)\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex ea54d108b9..55af498aa0 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1201,6 +1201,7 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n \tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n \tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_object_intmax(&jw, \"max_changed_paths\", ctx->bloom_settings->max_changed_paths);\n \tjw_end(&jw);\n \n \ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n@@ -1669,6 +1670,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n \t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n \t\t\t\t\t\t\t bloom_settings.num_hashes);\n+\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n \t\tctx->bloom_settings = &bloom_settings;\n \t}\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fc7693806c..47ddf2641f 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -174,11 +174,11 @@ test_expect_success 'persist filter settings' '\n \t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n \t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n \t\tGIT_TRACE2_EVENT_NESTING=5 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2-auto.txt\n '\n \n test_expect_success 'correctly report changes over limit' '\n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405632","messageId":"677b2f1692b3b6dca8695e415fdb11b320b0a432.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 02/13] t4216: use an '&&'-chain","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:07:00Z","receivedAt":"2020-09-16T18:07:59Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a759bfa9ee (t4216: add end to end tests for git log with Bloom\nfilters, 2020-04-06), a 'rm' invocation was added without a\ncorresponding '&&' chain.\n\nWhen 'trace.perf' already exists, everything works fine. However, the\nfunction can be executed without 'trace.perf' on disk (eg., when the\nsubset of tests run is altered with '--run'), and so the bare 'rm'\ncomplains about a missing file.\n\nTo remove some noise from the test log, invoke 'rm' with '-f', at which\npoint it is sensible to place the 'rm -f' in an '&&'-chain, which is\nboth (1) our usual style, and (2) avoids a broken chain in the future if\nmore commands are added at the beginning of the function.\n\nHelped-by: Eric Sunshine \nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \n---\n t/t4216-log-bloom.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 715912ad0f..cd89c75002 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -58,7 +58,7 @@ sane_unset GIT_TRACE2_PERF_BRIEF\n sane_unset GIT_TRACE2_CONFIG_PARAMS\n \n setup () {\n-\trm \"$TRASH_DIRECTORY/trace.perf\"\n+\trm -f \"$TRASH_DIRECTORY/trace.perf\" &&\n \tgit -c core.commitGraph=false log --pretty=\"format:%s\" $1 >log_wo_bloom &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.perf\" git -c core.commitGraph=true log --pretty=\"format:%s\" $1 >log_w_bloom\n }\n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405633","messageId":"db305fe3d698f297f1d3359516f5e5d18627adbe.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 01/13] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:06:53Z","receivedAt":"2020-09-16T18:08:26Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Many places in the code often need a pointer to the commit-graph's\n'struct bloom_filter_settings', in which case they often take the value\nfrom the top-most commit-graph.\n\nIn the non-split case, this works as expected. In the split case,\nhowever, things get a little tricky. Not all layers in a chain of\nincremental commit-graphs are required to themselves have Bloom data,\nand so whether or not some part of the code uses Bloom filters depends\nentirely on whether or not the top-most level of the commit-graph chain\nhas Bloom filters.\n\nThis has been the behavior since Bloom filters were introduced, and has\nbeen codified into the tests since a759bfa9ee (t4216: add end to end\ntests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\nrequires that Bloom filters are not used in exactly the case described\nearlier.\n\nThere is no reason that this needs to be the case, since it is perfectly\nvalid for commits in an earlier layer to have Bloom filters when commits\nin a newer layer do not.\n\nSince Bloom settings are guaranteed in practice to be the same for any\nlayer in a chain that has Bloom data, it is sufficient to traverse the\n'->base_graph' pointer until either (1) a non-null 'struct\nbloom_filter_settings *' is found, or (2) until we are at the root of\nthe commit-graph chain.\n\nIntroduce a 'get_bloom_filter_settings()' function that does just this,\nand use it instead of purely dereferencing the top-most graph's\n'->bloom_filter_settings' pointer.\n\nWhile we're at it, add an additional test in t5324 to guard against code\nin the commit-graph writing machinery that doesn't correctly handle a\nNULL 'struct bloom_filter *'.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n blame.c | 6 ++++--\n bloom.c | 6 +++---\n commit-graph.c | 11 +++++++++++\n commit-graph.h | 2 ++\n revision.c | 5 +----\n t/t4216-log-bloom.sh | 9 ++++++---\n t/t5324-split-commit-graph.sh | 13 +++++++++++++\n 7 files changed, 40 insertions(+), 12 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 1be1cd82a2..903e23af23 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -2892,16 +2892,18 @@ void setup_blame_bloom_data(struct blame_scoreboard *sb,\n \t\t\t const char *path)\n {\n \tstruct blame_bloom_data *bd;\n+\tstruct bloom_filter_settings *bs;\n \n \tif (!sb->repo->objects->commit_graph)\n \t\treturn;\n \n-\tif (!sb->repo->objects->commit_graph->bloom_filter_settings)\n+\tbs = get_bloom_filter_settings(sb->repo);\n+\tif (!bs)\n \t\treturn;\n \n \tbd = xmalloc(sizeof(struct blame_bloom_data));\n \n-\tbd->settings = sb->repo->objects->commit_graph->bloom_filter_settings;\n+\tbd->settings = bs;\n \n \tbd->alloc = 4;\n \tbd->nr = 0;\ndiff --git a/bloom.c b/bloom.c\nindex 1a573226e7..cd9380ac62 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -38,7 +38,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \twhile (graph_pos < g->num_commits_in_base)\n \t\tg = g->base_graph;\n \n-\t/* The commit graph commit 'c' lives in doesn't carry bloom filters. */\n+\t/* The commit graph commit 'c' lives in doesn't carry Bloom filters. */\n \tif (!g->chunk_bloom_indexes)\n \t\treturn 0;\n \n@@ -195,8 +195,8 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes)\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n+\t\t\t\treturn filter;\n \t}\n \n \tif (filter->data)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0ed003e218..6a36ed0b06 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -667,6 +667,17 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r)\n+{\n+\tstruct commit_graph *g = r->objects->commit_graph;\n+\twhile (g) {\n+\t\tif (g->bloom_filter_settings)\n+\t\t\treturn g->bloom_filter_settings;\n+\t\tg = g->base_graph;\n+\t}\n+\treturn NULL;\n+}\n+\n static void close_commit_graph_one(struct commit_graph *g)\n {\n \tif (!g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 09a97030dc..0677dd1031 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -87,6 +87,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n */\n int generation_numbers_enabled(struct repository *r);\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r);\n+\n enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_APPEND = (1 << 0),\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\ndiff --git a/revision.c b/revision.c\nindex 1239023f93..6aeb764821 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -681,10 +681,7 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \trepo_parse_commit(revs->repo, revs->commits->item);\n \n-\tif (!revs->repo->objects->commit_graph)\n-\t\treturn;\n-\n-\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n+\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n \tif (!revs->bloom_filter_settings)\n \t\treturn;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 4bb9e9dbe2..715912ad0f 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -65,7 +65,7 @@ setup () {\n \n test_bloom_filters_used () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":${2:-0},\\\"maybe\\\"\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom &&\n@@ -139,8 +139,11 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n-test_expect_success 'Do not use Bloom filters if the latest graph does not have Bloom filters.' '\n-\ttest_bloom_filters_not_used \"-- A/B\"\n+test_expect_success 'use Bloom filters even if the latest graph does not have Bloom filters' '\n+\t# Ensure that the number of empty filters is equal to the number of\n+\t# filters in the latest graph layer to prove that they are loaded (and\n+\t# ignored).\n+\ttest_bloom_filters_used \"-- A/B\" 3\n '\n \n test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 18216463c7..c334ee9155 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -427,4 +427,17 @@ done <<\\EOF\n 0600 -r--------\n EOF\n \n+test_expect_success '--split=replace with partial Bloom data' '\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/3 &&\n+\tgit rev-list -1 HEAD~2 >a &&\n+\tgit rev-list -1 HEAD~1 >b &&\n+\tgit commit-graph write --split=no-merge --stdin-commits --changed-paths graph-files &&\n+\ttest_line_count = 1 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n test_done\n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405634","messageId":"ee8f527a0078161f0466a75e85eef611249a0a46.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 07/13] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:07:32Z","receivedAt":"2020-09-16T18:09:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'get_bloom_filter' takes a flag to control whether it will compute a\nBloom filter if the requested one is missing. In the next patch, we'll\nadd yet another parameter to this method, which would force all but one\ncaller to specify an extra 'NULL' parameter at the end.\n\nInstead of doing this, split 'get_bloom_filter' into two functions:\n'get_bloom_filter' and 'get_or_compute_bloom_filter'. The former only\nlooks up a Bloom filter (and does not compute one if it's missing,\nthus dropping the 'compute_if_not_present' flag). The latter does\ncompute missing Bloom filters, with an additional parameter to store\nwhether or not it needed to do so.\n\nThis simplifies many call-sites, since the majority of existing callers\nto 'get_bloom_filter' do not want missing Bloom filters to be computed\n(so they can drop the parameter entirely and use the simpler version of\nthe function).\n\nWhile we're at it, instrument the new 'get_or_compute_bloom_filter()'\nwith counters in the 'write_commit_graph_context' struct which store\nthe number of filters that we did and didn't compute, as well as filters\nthat were truncated.\n\nIt would be nice to drop the 'compute_if_not_present' flag entirely,\nsince all remaining callers of 'get_or_compute_bloom_filter' pass it as\n'1', but this will change in a future patch and hence cannot be removed.\n\nSigned-off-by: Taylor Blau \n---\n blame.c | 2 +-\n bloom.c | 16 +++++++++++++---\n bloom.h | 16 +++++++++++++---\n commit-graph.c | 34 +++++++++++++++++++++++++++++++---\n line-log.c | 2 +-\n revision.c | 2 +-\n t/helper/test-bloom.c | 3 ++-\n 7 files changed, 62 insertions(+), 13 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 903e23af23..e5ba35dbd1 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -1276,7 +1276,7 @@ static int maybe_changed_path(struct repository *r,\n \tif (commit_graph_generation(origin->commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn 1;\n \n-\tfilter = get_bloom_filter(r, origin->commit, 0);\n+\tfilter = get_bloom_filter(r, origin->commit);\n \n \tif (!filter)\n \t\treturn 1;\ndiff --git a/bloom.c b/bloom.c\nindex cd9380ac62..393c61b4bc 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,9 +177,10 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present)\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t enum bloom_filter_computed *computed)\n {\n \tstruct bloom_filter *filter;\n \tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n@@ -187,6 +188,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n+\tif (computed)\n+\t\t*computed = BLOOM_NOT_COMPUTED;\n+\n \tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n@@ -271,8 +275,14 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\tfilter->data = NULL;\n \t\tfilter->len = 0;\n+\n+\t\tif (computed)\n+\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n \t}\n \n+\tif (computed)\n+\t\t*computed |= BLOOM_COMPUTED;\n+\n \tfree(diff_queued_diff.queue);\n \tDIFF_QUEUE_CLEAR(&diff_queued_diff);\n \ndiff --git a/bloom.h b/bloom.h\nindex 0b9b59a6fe..e2e035ad14 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -89,9 +89,19 @@ void add_key_to_filter(const struct bloom_key *key,\n \n void init_bloom_filters(void);\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present);\n+enum bloom_filter_computed {\n+\tBLOOM_NOT_COMPUTED = (1 << 0),\n+\tBLOOM_COMPUTED = (1 << 1),\n+\tBLOOM_TRUNC_LARGE = (1 << 2),\n+};\n+\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t enum bloom_filter_computed *computed);\n+\n+#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n+\t(r), (c), 0, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 55af498aa0..45fcd62596 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -971,6 +971,10 @@ struct write_commit_graph_context {\n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n+\n+\tint count_bloom_filter_computed;\n+\tint count_bloom_filter_not_computed;\n+\tint count_bloom_filter_trunc_large;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1182,7 +1186,7 @@ static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tuint32_t cur_pos = 0;\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \t\tcur_pos += len;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1222,7 +1226,7 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1392,6 +1396,16 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n+{\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-computed\",\n+\t\t\t ctx->count_bloom_filter_computed);\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-not-computed\",\n+\t\t\t ctx->count_bloom_filter_not_computed);\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-large\",\n+\t\t\t ctx->count_bloom_filter_trunc_large);\n+}\n+\n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n@@ -1414,12 +1428,26 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tenum bloom_filter_computed computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n+\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\tctx->r,\n+\t\t\tc,\n+\t\t\t1,\n+\t\t\t&computed);\n+\t\tif (computed & BLOOM_COMPUTED) {\n+\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (computed & BLOOM_TRUNC_LARGE)\n+\t\t\t\tctx->count_bloom_filter_trunc_large++;\n+\t\t} else if (computed & BLOOM_NOT_COMPUTED)\n+\t\t\tctx->count_bloom_filter_not_computed++;\n \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n+\tif (trace2_is_enabled())\n+\t\ttrace2_bloom_filter_write_statistics(ctx);\n+\n \tfree(sorted_commits);\n \tstop_progress(&progress);\n }\ndiff --git a/line-log.c b/line-log.c\nindex bf73ea95ac..68eeb425f8 100644\n--- a/line-log.c\n+++ b/line-log.c\n@@ -1159,7 +1159,7 @@ static int bloom_filter_check(struct rev_info *rev,\n \t\treturn 1;\n \n \tif (!rev->bloom_filter_settings ||\n-\t !(filter = get_bloom_filter(rev->repo, commit, 0)))\n+\t !(filter = get_bloom_filter(rev->repo, commit)))\n \t\treturn 1;\n \n \tif (!range)\ndiff --git a/revision.c b/revision.c\nindex 6aeb764821..c53ab99e59 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -752,7 +752,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \tif (commit_graph_generation(commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn -1;\n \n-\tfilter = get_bloom_filter(revs->repo, commit, 0);\n+\tfilter = get_bloom_filter(revs->repo, commit);\n \n \tif (!filter) {\n \t\tcount_bloom_filter_not_present++;\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 5e77d56f59..9f7bb729fc 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -39,7 +39,8 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tstruct bloom_filter *filter;\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n-\tfilter = get_bloom_filter(the_repository, c, 1);\n+\tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n \n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405635","messageId":"b8e1e48bef3bbee631dbb88b832fae98a20c940b.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 08/13] bloom: use provided 'struct bloom_filter_settings'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:07:46Z","receivedAt":"2020-09-16T18:09:12Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When 'get_or_compute_bloom_filter()' needs to compute a Bloom filter\nfrom scratch, it looks to the default 'struct bloom_filter_settings' in\norder to determine the maximum number of changed paths, number of bits\nper entry, and so on.\n\nAll of these values have so far been constant, and so there was no need\nto pass in a pointer from the caller (eg., the one that is stored in the\n'struct write_commit_graph_context').\n\nStart passing in a 'struct bloom_filter_settings *' instead of using the\ndefault values to respect graph-specific settings (eg., in the case of\nsetting 'GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS').\n\nIn order to have an initialized value for these settings, move its\ninitialization to earlier in the commit-graph write.\n\nSigned-off-by: Taylor Blau \n---\n bloom.c | 13 ++++++-------\n bloom.h | 3 ++-\n commit-graph.c | 21 ++++++++++-----------\n t/helper/test-bloom.c | 1 +\n 4 files changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 393c61b4bc..2d6aef9098 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -180,13 +180,12 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t enum bloom_filter_computed *computed)\n {\n \tstruct bloom_filter *filter;\n-\tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \tint i;\n \tstruct diff_options diffopt;\n-\tint max_changes = 512;\n \n \tif (computed)\n \t\t*computed = BLOOM_NOT_COMPUTED;\n@@ -211,7 +210,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n \tdiffopt.detect_rename = 0;\n-\tdiffopt.max_changes = max_changes;\n+\tdiffopt.max_changes = settings->max_changed_paths;\n \tdiff_setup_done(&diffopt);\n \n \t/* ensure commit is parsed so we have parent information */\n@@ -223,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= max_changes) {\n+\tif (diffopt.num_changes <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -260,13 +259,13 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n-\t\tfilter->len = (hashmap_get_size(&pathmap) * settings.bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n \t\t\tstruct bloom_key key;\n-\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, &settings);\n-\t\t\tadd_key_to_filter(&key, filter, &settings);\n+\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, settings);\n+\t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\ndiff --git a/bloom.h b/bloom.h\nindex e2e035ad14..c6d77e8393 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -98,10 +98,11 @@ enum bloom_filter_computed {\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t enum bloom_filter_computed *computed);\n \n #define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n-\t(r), (c), 0, NULL)\n+\t(r), (c), 0, NULL, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 45fcd62596..1ca754f19c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1434,6 +1434,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->r,\n \t\t\tc,\n \t\t\t1,\n+\t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n \t\t\tctx->count_bloom_filter_computed++;\n@@ -1691,17 +1692,6 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tint num_chunks = 3;\n \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n-\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n-\n-\tif (!ctx->bloom_settings) {\n-\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n-\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n-\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n-\t\t\t\t\t\t\t bloom_settings.num_hashes);\n-\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n-\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n-\t\tctx->bloom_settings = &bloom_settings;\n-\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -2147,6 +2137,7 @@ int write_commit_graph(struct object_directory *odb,\n \tuint32_t i, count_distinct = 0;\n \tint res = 0;\n \tint replace = 0;\n+\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n@@ -2160,6 +2151,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split_opts = split_opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n+\t\t\t\t\t\t bloom_settings.bits_per_entry);\n+\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n+\t\t\t\t\t\t bloom_settings.num_hashes);\n+\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n+\tctx->bloom_settings = &bloom_settings;\n+\n \tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n \t\tctx->changed_paths = 1;\n \tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 9f7bb729fc..46e97b04eb 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -40,6 +40,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n \tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t &settings,\n \t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405636","messageId":"4653b5b4bcd254a3791797214b46722b4062dc18.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 10/13] bloom: encode out-of-bounds filters as non-empty","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:07:59Z","receivedAt":"2020-09-16T18:09:45Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a changed-path Bloom filter has either zero, or more than a\ncertain number (commonly 512) of entries, the commit-graph machinery\nencodes it as \"missing\". More specifically, it sets the indices adjacent\nin the BIDX chunk as equal to each other to indicate a \"length 0\"\nfilter; that is, that the filter occupies zero bytes on disk.\n\nThis has heretofore been fine, since the commit-graph machinery has no\nneed to care about these filters with too few or too many changed paths.\nBoth cases act like no filter has been generated at all, and so there is\nno need to store them.\n\nIn a subsequent commit, however, the commit-graph machinery will learn\nto only compute Bloom filters for some commits in the current\ncommit-graph layer. This is a change from the current implementation\nwhich computes Bloom filters for all commits that are in the layer being\nwritten. Critically for this patch, only computing some of the Bloom\nfilters means adding a third state for length 0 Bloom filters: zero\nentries, too many entries, or \"hasn't been computed\".\n\nIt will be important for that future patch to distinguish between \"not\nrepresentable\" (i.e., zero or too-many changed paths), and \"hasn't been\ncomputed\". In particular, we don't want to waste time recomputing\nfilters that have already been computed.\n\nTo that end, change how we store Bloom filters in the \"computed but not\nrepresentable\" category:\n\n - Bloom filters with no entries are stored as a single byte with all\n bits low (i.e., all queries to that Bloom filter will return\n \"definitely not\")\n\n - Bloom filters with too many entries are stored as a single byte with\n all bits set high (i.e., all queries to that Bloom filter will\n return \"maybe\").\n\nThese rules are sufficient to not incur a behavior change by changing\nthe on-disk representation of these two classes. Likewise, no\nspecification changes are necessary for the commit-graph format, either:\n\n - Filters that were previously empty will be recomputed and stored\n according to the new rules, and\n\n - old clients reading filters generated by new clients will interpret\n the filters correctly and be none the wiser to how they were\n generated.\n\nClients will invoke the Bloom machinery in more cases than before, but\nthis can be addressed by returning a NULL filter when all bits are set\nhigh. This can be addressed in a future patch.\n\nFinally, note that this does increase the size of on-disk commit-graphs,\nbut far less than other proposals. In particular, this is generally more\nefficient than storing a bitmap for which commits haven't computed their\nBloom filters. Storing a bitmap incurs a penalty of one bit per commit,\nwhereas storing explicit filters as above incurs a penalty of one byte\nper too-large or too-small commit.\n\nIn practice, these boundary commits likely occupy a small proportion of\nthe overall number of commits, and so the size penalty is likely smaller\nthan storing a bitmap for all commits.\n\nA test to exercise filters which contain too many changed path entries\nwill be introduced in a subsequent patch.\n\nSuggested-by: SZEDER Gábor \nSuggested-by: Jakub Narębski \nHelped-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n .../technical/commit-graph-format.txt | 2 +-\n bloom.c | 16 ++++++++--\n bloom.h | 1 +\n commit-graph.c | 5 ++++\n t/t0095-bloom.sh | 8 ++---\n t/t4216-log-bloom.sh | 30 +++++++++++++++++--\n 6 files changed, 53 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 6ddbceba15..6585f1948a 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -125,7 +125,7 @@ CHUNK DATA:\n * The rest of the chunk is the concatenation of all the computed Bloom\n filters for the commits in lexicographic order.\n * Note: Commits with no changes or more than 512 changes have Bloom filters\n- of length zero.\n+ of length one, with either all bits set to zero or one respectively.\n * The BDAT chunk is present if and only if BIDX is present.\n \n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\ndiff --git a/bloom.c b/bloom.c\nindex db9fb82437..d24747a1d5 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,6 +177,13 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n \n+static void init_truncated_large_filter(struct bloom_filter *filter)\n+{\n+\tfilter->data = xmalloc(1);\n+\tfilter->data[0] = 0xFF;\n+\tfilter->len = 1;\n+}\n+\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n@@ -260,12 +267,18 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t}\n \n \t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths) {\n+\t\t\tinit_truncated_large_filter(filter);\n \t\t\tif (computed)\n \t\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n \t\t\tgoto cleanup;\n \t\t}\n \n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tif (!filter->len) {\n+\t\t\tif (computed)\n+\t\t\t\t*computed |= BLOOM_TRUNC_SMALL;\n+\t\t\tfilter->len = 1;\n+\t\t}\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n@@ -279,8 +292,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n-\t\tfilter->data = NULL;\n-\t\tfilter->len = 0;\n+\t\tinit_truncated_large_filter(filter);\n \n \t\tif (computed)\n \t\t\t*computed |= BLOOM_TRUNC_LARGE;\ndiff --git a/bloom.h b/bloom.h\nindex c6d77e8393..70a8840896 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -93,6 +93,7 @@ enum bloom_filter_computed {\n \tBLOOM_NOT_COMPUTED = (1 << 0),\n \tBLOOM_COMPUTED = (1 << 1),\n \tBLOOM_TRUNC_LARGE = (1 << 2),\n+\tBLOOM_TRUNC_SMALL = (1 << 3),\n };\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 1ca754f19c..bd4247bca5 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -974,6 +974,7 @@ struct write_commit_graph_context {\n \n \tint count_bloom_filter_computed;\n \tint count_bloom_filter_not_computed;\n+\tint count_bloom_filter_trunc_small;\n \tint count_bloom_filter_trunc_large;\n };\n \n@@ -1402,6 +1403,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \t\t\t ctx->count_bloom_filter_computed);\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-not-computed\",\n \t\t\t ctx->count_bloom_filter_not_computed);\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-small\",\n+\t\t\t ctx->count_bloom_filter_trunc_small);\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-large\",\n \t\t\t ctx->count_bloom_filter_trunc_large);\n }\n@@ -1438,6 +1441,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n \t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (computed & BLOOM_TRUNC_SMALL)\n+\t\t\t\tctx->count_bloom_filter_trunc_small++;\n \t\t\tif (computed & BLOOM_TRUNC_LARGE)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex 232ba2c485..7e4ab1795f 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -71,8 +71,8 @@ test_expect_success 'get bloom filters for commit with no changes' '\n \tgit init &&\n \tgit commit --allow-empty -m \"c0\" &&\n \tcat >expect <<-\\EOF &&\n-\tFilter_Length:0\n-\tFilter_Data:\n+\tFilter_Length:1\n+\tFilter_Data:00|\n \tEOF\n \ttest-tool bloom get_filter_for_commit \"$(git rev-parse HEAD)\" >actual &&\n \ttest_cmp expect actual\n@@ -107,8 +107,8 @@ test_expect_success EXPENSIVE 'get bloom filter for commit with 513 changes' '\n \tgit add bigDir &&\n \tgit commit -m \"commit with 513 changes\" &&\n \tcat >expect <<-\\EOF &&\n-\tFilter_Length:0\n-\tFilter_Data:\n+\tFilter_Length:1\n+\tFilter_Data:ff|\n \tEOF\n \ttest-tool bloom get_filter_for_commit \"$(git rev-parse HEAD)\" >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 1ac8f4c4eb..a0c9c9ea23 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -30,6 +30,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \trm file_to_be_deleted &&\n \tgit add . &&\n \tgit commit -m \"file removed\" &&\n+\tgit commit --allow-empty -m \"empty\" &&\n \tgit commit-graph write --reachable --changed-paths &&\n \n \ttest_oid_cache <<-EOF\n@@ -49,7 +50,7 @@ graph_read_expect () {\n }\n \n test_expect_success 'commit-graph write wrote out the bloom chunks' '\n-\tgraph_read_expect 15\n+\tgraph_read_expect 16\n '\n \n # Turn off any inherited trace2 settings for this test.\n@@ -156,7 +157,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":9\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n@@ -185,10 +186,18 @@ test_max_changed_paths () {\n \tgrep \"\\\"max_changed_paths\\\":$1\" $2\n }\n \n+test_filter_not_computed () {\n+\tgrep \"\\\"key\\\":\\\"filter-not-computed\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n test_filter_computed () {\n \tgrep \"\\\"key\\\":\\\"filter-computed\\\",\\\"value\\\":\\\"$1\\\"\" $2\n }\n \n+test_filter_trunc_small () {\n+\tgrep \"\\\"key\\\":\\\"filter-trunc-small\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n test_filter_trunc_large () {\n \tgrep \"\\\"key\\\":\\\"filter-trunc-large\\\",\\\"value\\\":\\\"$1\\\"\" $2\n }\n@@ -283,4 +292,21 @@ test_expect_success 'correctly report changes over limit' '\n \t)\n '\n \n+test_expect_success 'correctly report commits with no changed paths' '\n+\tgit init small &&\n+\ttest_when_finished \"rm -fr small\" &&\n+\t(\n+\t\tcd small &&\n+\n+\t\tgit commit --allow-empty -m \"initial commit\" &&\n+\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\ttest_filter_computed 1 trace.event &&\n+\t\ttest_filter_not_computed 0 trace.event &&\n+\t\ttest_filter_trunc_small 1 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n test_done\n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405637","messageId":"51f9a398fc5f09d19bafbb48bd4218efbfa826b6.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 11/13] commit-graph: rename 'split_commit_graph_opts'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:08:06Z","receivedAt":"2020-09-16T18:09:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the subsequent commit, additional options will be added to the\ncommit-graph API which have nothing to do with splitting.\n\nRename the 'split_commit_graph_opts' structure to the more-generic\n'commit_graph_opts' to encompass both. Likewise, rename the 'flags'\nmember to instead be 'split_flags' to clarify that it only has to do\nwith the behavior implied by '--split'.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n builtin/commit-graph.c | 20 ++++++++++----------\n commit-graph.c | 40 ++++++++++++++++++++--------------------\n commit-graph.h | 8 ++++----\n 3 files changed, 34 insertions(+), 34 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ba5584463f..f3243bd982 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,7 +119,7 @@ static int graph_verify(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n-static struct split_commit_graph_opts split_opts;\n+static struct commit_graph_opts write_opts;\n \n static int write_option_parse_split(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n@@ -187,24 +187,24 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n \t\t\tN_(\"enable computation for changed paths\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.split_flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\twrite_option_parse_split),\n-\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\tOPT_INTEGER(0, \"max-commits\", &write_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n-\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\tOPT_INTEGER(0, \"size-multiple\", &write_opts.size_multiple,\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n-\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n \t\tOPT_END(),\n \t};\n \n \topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n-\tsplit_opts.size_multiple = 2;\n-\tsplit_opts.max_commits = 0;\n-\tsplit_opts.expire_time = 0;\n+\twrite_opts.size_multiple = 2;\n+\twrite_opts.max_commits = 0;\n+\twrite_opts.expire_time = 0;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -232,7 +232,7 @@ static int graph_write(int argc, const char **argv)\n \todb = find_odb(the_repository, opts.obj_dir);\n \n \tif (opts.reachable) {\n-\t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n+\t\tif (write_commit_graph_reachable(odb, flags, &write_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n@@ -261,7 +261,7 @@ static int graph_write(int argc, const char **argv)\n \t\t\t opts.stdin_packs ? &pack_indexes : NULL,\n \t\t\t opts.stdin_commits ? &commits : NULL,\n \t\t\t flags,\n-\t\t\t &split_opts))\n+\t\t\t &write_opts))\n \t\tresult = 1;\n \n cleanup:\ndiff --git a/commit-graph.c b/commit-graph.c\nindex bd4247bca5..913f78a9a1 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -968,7 +968,7 @@ struct write_commit_graph_context {\n \t\t changed_paths:1,\n \t\t order_by_pack:1;\n \n-\tconst struct split_commit_graph_opts *split_opts;\n+\tconst struct commit_graph_opts *opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \n@@ -1292,8 +1292,8 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1482,7 +1482,7 @@ static int add_ref_to_set(const char *refname,\n \n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t\t\t const struct commit_graph_opts *opts)\n {\n \tstruct oidset commits = OIDSET_INIT;\n \tstruct refs_cb_data data;\n@@ -1499,7 +1499,7 @@ int write_commit_graph_reachable(struct object_directory *odb,\n \tstop_progress(&data.progress);\n \n \tresult = write_commit_graph(odb, NULL, &commits,\n-\t\t\t\t flags, split_opts);\n+\t\t\t\t flags, opts);\n \n \toidset_clear(&commits);\n \treturn result;\n@@ -1614,8 +1614,8 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tctx->num_extra_edges = 0;\n \tif (ctx->report_progress)\n@@ -1900,13 +1900,13 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tint max_commits = 0;\n \tint size_mult = 2;\n \n-\tif (ctx->split_opts) {\n-\t\tmax_commits = ctx->split_opts->max_commits;\n+\tif (ctx->opts) {\n+\t\tmax_commits = ctx->opts->max_commits;\n \n-\t\tif (ctx->split_opts->size_multiple)\n-\t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t\tif (ctx->opts->size_multiple)\n+\t\t\tsize_mult = ctx->opts->size_multiple;\n \n-\t\tflags = ctx->split_opts->flags;\n+\t\tflags = ctx->opts->split_flags;\n \t}\n \n \tg = ctx->r->objects->commit_graph;\n@@ -2084,8 +2084,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n \n-\tif (ctx->split_opts && ctx->split_opts->expire_time)\n-\t\texpire_time = ctx->split_opts->expire_time;\n+\tif (ctx->opts && ctx->opts->expire_time)\n+\t\texpire_time = ctx->opts->expire_time;\n \tif (!ctx->split) {\n \t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n@@ -2136,7 +2136,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t const struct commit_graph_opts *opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -2153,7 +2153,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n-\tctx->split_opts = split_opts;\n+\tctx->opts = opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n@@ -2201,15 +2201,15 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\t}\n \t\t}\n \n-\t\tif (ctx->split_opts)\n-\t\t\treplace = ctx->split_opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n+\t\tif (ctx->opts)\n+\t\t\treplace = ctx->opts->split_flags & COMMIT_GRAPH_SPLIT_REPLACE;\n \t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n-\t\tctx->oids.alloc = split_opts->max_commits;\n+\tif (ctx->split && opts && ctx->oids.alloc > opts->max_commits)\n+\t\tctx->oids.alloc = opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->odb);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d9acb22bac..b7914b0a7a 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -105,11 +105,11 @@ enum commit_graph_split_flags {\n \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n };\n \n-struct split_commit_graph_opts {\n+struct commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n-\tenum commit_graph_split_flags flags;\n+\tenum commit_graph_split_flags split_flags;\n };\n \n /*\n@@ -120,12 +120,12 @@ struct split_commit_graph_opts {\n */\n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t\t\t const struct commit_graph_opts *opts);\n int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t const struct commit_graph_opts *opts);\n \n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n \n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405638","messageId":"ccb6482feb8d8606d82b5ab97e33184f26d6c5b6.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 12/13] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:08:10Z","receivedAt":"2020-09-16T18:09:57Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a command-line flag to specify the maximum number of new Bloom\nfilters that a 'git commit-graph write' is willing to compute from\nscratch.\n\nPrior to this patch, a commit-graph write with '--changed-paths' would\ncompute Bloom filters for all selected commits which haven't already\nbeen computed (i.e., by a previous commit-graph write with '--split'\nsuch that a roll-up or replacement is performed).\n\nThis behavior can cause prohibitively-long commit-graph writes for a\nvariety of reasons:\n\n * There may be lots of filters whose diffs take a long time to\n generate (for example, they have close to the maximum number of\n changes, diffing itself takes a long time, etc).\n\n * Old-style commit-graphs (which encode filters with too many entries\n as not having been computed at all) cause us to waste time\n recomputing filters that appear to have not been computed only to\n discover that they are too-large.\n\nThis can make the upper-bound of the time it takes for 'git commit-graph\nwrite --changed-paths' to be rather unpredictable.\n\nTo make this command behave more predictably, introduce\n'--max-new-filters=' to allow computing at most '' Bloom filters\nfrom scratch. This lets \"computing\" already-known filters proceed\nquickly, while bounding the number of slow tasks that Git is willing to\ndo.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/git-commit-graph.txt | 5 +++\n bloom.c | 7 ++-\n builtin/commit-graph.c | 27 +++++++++++-\n commit-graph.c | 9 +++-\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 70 ++++++++++++++++++++++++++++++\n 6 files changed, 111 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..8357846d30 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,11 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Commits whose filters are not calculated are stored as a\n+length zero Bloom filter.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex d24747a1d5..230a515831 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -204,12 +204,11 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n-\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n+\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n \n-\tif (filter->data)\n+\tif (filter->data && filter->len)\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f3243bd982..5df9b2ef80 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n \n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n \n@@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n \n \ttrace2_cmd_mode(\"write\");\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 913f78a9a1..33af6c2430 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1414,6 +1414,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tstruct commit **sorted_commits;\n+\tint max_new_filters;\n \n \tinit_bloom_filters();\n \n@@ -1430,13 +1431,16 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \telse\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n \n+\tmax_new_filters = ctx->opts && ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tenum bloom_filter_computed computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\tctx->r,\n \t\t\tc,\n-\t\t\t1,\n+\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n \t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n@@ -1447,7 +1451,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n \t\t\tctx->count_bloom_filter_not_computed++;\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\tctx->total_bloom_filter_data_size += filter\n+\t\t\t? sizeof(unsigned char) * filter->len : 0;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex b7914b0a7a..a22bd86701 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -110,6 +110,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags split_flags;\n+\tint max_new_filters;\n };\n \n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex a0c9c9ea23..9ce0c318e9 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -309,4 +309,74 @@ test_expect_success 'correctly report commits with no changed paths' '\n \t)\n '\n \n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --split=replace \\\n+\t\t\t\t--changed-paths --max-new-filters=2 &&\n+\n+\t\ttest_filter_computed 2 trace.event &&\n+\t\ttest_filter_not_computed 3 trace.event &&\n+\t\ttest_filter_trunc_small 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t(\n+\t\tcd limits &&\n+\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t\t--split=replace --max-new-filters=1 &&\n+\t\ttest_filter_computed 1 trace.event &&\n+\t\ttest_filter_not_computed 4 trace.event &&\n+\t\ttest_filter_trunc_small 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills empty commits' '\n+\tgit init empty &&\n+\ttest_when_finished \"rm -fr empty\" &&\n+\t(\n+\t\tcd empty &&\n+\t\tfor i in $(test_seq 1 6)\n+\t\tdo\n+\t\t\tgit commit --allow-empty -m \"$i\"\n+\t\tdone &&\n+\n+\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n+\t\tfor i in $(test_seq 1 3)\n+\t\tdo\n+\t\t\trm -f trace.event &&\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\t\tgit commit-graph write --reachable \\\n+\t\t\t\t\t--changed-paths --max-new-filters=2 &&\n+\t\t\ttest_filter_computed 2 trace.event &&\n+\t\t\ttest_filter_not_computed 4 trace.event &&\n+\t\t\ttest_filter_trunc_small 2 trace.event &&\n+\t\t\ttest_filter_trunc_large 0 trace.event\n+\t\tdone &&\n+\n+\t\t# Finally, make sure that once all commits have filters, that\n+\t\t# none are subsequently recomputed.\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable \\\n+\t\t\t\t--changed-paths --max-new-filters=2 &&\n+\t\ttest_filter_computed 0 trace.event &&\n+\t\ttest_filter_not_computed 6 trace.event &&\n+\t\ttest_filter_trunc_small 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n test_done\n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405639","messageId":"b6bb1cafe7b8af175dd2c550f2b2d87b04403682.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 09/13] bloom/diff: properly short-circuit on max_changes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:07:52Z","receivedAt":"2020-09-16T18:11:15Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nCommit e3696980 (diff: halt tree-diff early after max_changes,\n2020-03-30) intended to create a mechanism to short-circuit a diff\ncalculation after a certain number of paths were modified. By\nincrementing a \"num_changes\" counter throughout the recursive\nll_diff_tree_paths(), this was supposed to match the number of changes\nthat would be written into the changed-path Bloom filters.\nUnfortunately, this was not implemented correctly and instead misses\nsimple cases like file modifications. This then does not stop very\nlarge changed-path filters from being written (unless they add or remove\nmany files).\n\nTo start, change the implementation in ll_diff_tree_paths() to instead\nuse the global diff_queue_diff struct's 'nr' member as the count. This\nis a way to simplify the logic instead of making more mistakes in the\ncomplicated diff code.\n\nThis has a drawback: the diff_queue_diff struct only lists the paths\ncorresponding to blob changes, not their leading directories. Thus,\nget_or_compute_bloom_filter() needs an additional check to see if the\nhashmap with the leading directories becomes too large.\n\nOne reason why this was not caught by test cases was that the test in\nt4216-log-bloom.sh that was supposed to check this \"too many changes\"\ncondition only checked this on the initial commit of a repository. The\nold logic counted these values correctly. Update this test in a few\nways:\n\n1. Use GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS to reduce the limit,\n allowing smaller commits to engage with this logic.\n\n2. Create several interesting cases of edits, adds, removes, and mode\n changes (in the second commit). By testing both sides of the\n inequality with the *_MAX_CHANGED_PATHS variable, we can see that\n the count is exactly correct, so none of these changes are missed\n or over-counted.\n\n3. Use the trace2 data value filter_found_large to verify that these\n commits are on the correct side of the limit.\n\nAnother way to verify the behavior is correct is through performance\ntests. By testing on my local copies of the Git repository and the Linux\nkernel repository, I could measure the effect of these short-circuits\nwhen computing a fresh commit-graph file with changed-path Bloom filters\nusing the command\n\n GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=N time \\\n git commit-graph write --reachable --changed-paths\n\nand reporting the wall time and resulting commit-graph size.\n\nFor Git, the results are\n\n| | N=1 | N=10 | N=512 |\n|--------|----------------|----------------|----------------|\n| HEAD~1 | 10.90s 9.18MB | 11.11s 9.34MB | 11.31s 9.35MB |\n| HEAD | 9.21s 8.62MB | 11.11s 9.29MB | 11.29s 9.34MB |\n\nFor Linux, the results are\n\n| | N=1 | N=20 | N=512 |\n|--------|----------------|---------------|---------------|\n| HEAD~1 | 61.28s 64.3MB | 76.9s 72.6MB | 77.6s 72.6MB |\n| HEAD | 49.44s 56.3MB | 68.7s 65.9MB | 69.2s 65.9MB |\n\nNaturally, the improvement becomes much less as the limit grows, as\nfewer commits satisfy the short-circuit.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \n---\n bloom.c | 9 +++-\n diff.h | 2 -\n t/t4216-log-bloom.sh | 100 +++++++++++++++++++++++++++++++++++++++----\n tree-diff.c | 5 +--\n 4 files changed, 100 insertions(+), 16 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 2d6aef9098..db9fb82437 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -222,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= settings->max_changed_paths) {\n+\tif (diff_queued_diff.nr <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -259,6 +259,12 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n+\t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths) {\n+\t\t\tif (computed)\n+\t\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n@@ -268,6 +274,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n+\tcleanup:\n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\ndiff --git a/diff.h b/diff.h\nindex e0c0af6286..1d32b71885 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -287,8 +287,6 @@ struct diff_options {\n \n \t/* If non-zero, then stop computing after this many changes. */\n \tint max_changes;\n-\t/* For internal use only. */\n-\tint num_changes;\n \n \tint ita_invisible_in_index;\n /* white-space error highlighting */\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 47ddf2641f..1ac8f4c4eb 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -181,21 +181,103 @@ test_expect_success 'persist filter settings' '\n \tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2-auto.txt\n '\n \n+test_max_changed_paths () {\n+\tgrep \"\\\"max_changed_paths\\\":$1\" $2\n+}\n+\n+test_filter_computed () {\n+\tgrep \"\\\"key\\\":\\\"filter-computed\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n+test_filter_trunc_large () {\n+\tgrep \"\\\"key\\\":\\\"filter-trunc-large\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n test_expect_success 'correctly report changes over limit' '\n-\tgit init 513changes &&\n+\tgit init limits &&\n \t(\n-\t\tcd 513changes &&\n-\t\tfor i in $(test_seq 1 513)\n+\t\tcd limits &&\n+\t\tmkdir d &&\n+\t\tmkdir d/e &&\n+\n+\t\tfor i in $(test_seq 1 2)\n \t\tdo\n-\t\t\techo $i >file$i.txt || return 1\n+\t\t\tprintf $i >d/file$i.txt &&\n+\t\t\tprintf $i >d/e/file$i.txt || return 1\n \t\tdone &&\n-\t\tgit add . &&\n+\n+\t\tmkdir mode &&\n+\t\tprintf bash >mode/script.sh &&\n+\n+\t\tmkdir foo &&\n+\t\ttouch foo/bar &&\n+\t\ttouch foo.txt &&\n+\n+\t\tgit add d foo foo.txt mode &&\n \t\tgit commit -m \"files\" &&\n-\t\tgit commit-graph write --reachable --changed-paths &&\n-\t\tfor i in $(test_seq 1 513)\n+\n+\t\t# Commit has 7 file and 4 directory adds\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\ttest_max_changed_paths 10 trace &&\n+\t\ttest_filter_computed 1 trace &&\n+\t\ttest_filter_trunc_large 1 trace &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n \t\tdo\n-\t\t\tgit -c core.commitGraph=false log -- file$i.txt >expect &&\n-\t\t\tgit log -- file$i.txt >actual &&\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# Make a variety of path changes\n+\t\tprintf new1 >d/e/file1.txt &&\n+\t\tprintf new2 >d/file2.txt &&\n+\t\trm d/e/file2.txt &&\n+\t\trm -r foo &&\n+\t\tprintf text >foo &&\n+\t\tmkdir f &&\n+\t\tprintf new1 >f/file1.txt &&\n+\n+\t\t# including a mode-only change (counts as modified)\n+\t\tgit update-index --chmod=+x mode/script.sh &&\n+\n+\t\tgit add foo d f &&\n+\t\tgit commit -m \"complicated\" &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-edit\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\ttest_max_changed_paths 10 trace-edit &&\n+\t\ttest_filter_computed 2 trace-edit &&\n+\t\ttest_filter_trunc_large 2 trace-edit &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=11 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-update\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\ttest_max_changed_paths 11 trace-update &&\n+\t\ttest_filter_computed 2 trace-update &&\n+\t\ttest_filter_trunc_large 0 trace-update &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n \t\t\ttest_cmp expect actual || return 1\n \t\tdone\n \t)\ndiff --git a/tree-diff.c b/tree-diff.c\nindex 6ebad1a46f..7cebbb327e 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -434,7 +434,7 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tif (diff_can_quit_early(opt))\n \t\t\tbreak;\n \n-\t\tif (opt->max_changes && opt->num_changes > opt->max_changes)\n+\t\tif (opt->max_changes && diff_queued_diff.nr > opt->max_changes)\n \t\t\tbreak;\n \n \t\tif (opt->pathspec.nr) {\n@@ -521,7 +521,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \n \t\t\t/* t↓ */\n \t\t\tupdate_tree_entry(&t);\n-\t\t\topt->num_changes++;\n \t\t}\n \n \t\t/* t > p[imin] */\n@@ -539,7 +538,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tskip_emit_tp:\n \t\t\t/* ∀ pi=p[imin] pi↓ */\n \t\t\tupdate_tp_entries(tp, nparent);\n-\t\t\topt->num_changes++;\n \t\t}\n \t}\n \n@@ -557,7 +555,6 @@ struct combine_diff_path *diff_tree_paths(\n \tconst struct object_id **parents_oid, int nparent,\n \tstruct strbuf *base, struct diff_options *opt)\n {\n-\topt->num_changes = 0;\n \tp = ll_diff_tree_paths(p, oid, parents_oid, nparent, base, opt);\n \n \t/*\n-- \n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405645","messageId":"86fdc5f89a1637e889e686d60f75a904c0f4485c.1600279373.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"[PATCH v2 13/13] commit-graph: introduce 'commitGraph.maxNewFilters'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-16T18:08:15Z","receivedAt":"2020-09-16T18:12:14Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a configuration variable to specify a default value for the\nrecently-introduce '--max-new-filters' option of 'git commit-graph\nwrite'.\n\nSigned-off-by: Taylor Blau \n---\n Documentation/config/commitgraph.txt | 4 ++++\n Documentation/git-commit-graph.txt | 3 ++-\n builtin/commit-graph.c | 14 ++++++++++++++\n t/t4216-log-bloom.sh | 24 +++++++++++++++++++++++-\n 4 files changed, 43 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex cff0797b54..4582c39fc4 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitGraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 8357846d30..60df4e4bfa 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -70,7 +70,8 @@ data.\n With the `--max-new-filters=` option, generate at most `n` new Bloom\n filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n enforced. Commits whose filters are not calculated are stored as a\n-length zero Bloom filter.\n+length zero Bloom filter. Overrides the `commitGraph.maxNewFilters`\n+configuration.\n +\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 5df9b2ef80..988445abdf 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -181,6 +181,18 @@ static int write_option_max_new_filters(const struct option *opt,\n \treturn 0;\n }\n \n+static int git_commit_graph_write_config(const char *var, const char *value,\n+\t\t\t\t\t void *cb)\n+{\n+\tif (!strcmp(var, \"commitgraph.maxnewfilters\"))\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t/*\n+\t * No need to fall-back to 'git_default_config', since this was already\n+\t * called in 'cmd_commit_graph()'.\n+\t */\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -231,6 +243,8 @@ static int graph_write(int argc, const char **argv)\n \n \ttrace2_cmd_mode(\"write\");\n \n+\tgit_config(git_commit_graph_write_config, &opts);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 9ce0c318e9..dc7d62c778 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -329,13 +329,15 @@ test_expect_success 'Bloom generation is limited by --max-new-filters' '\n '\n \n test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t# Check specifying commitGraph.maxNewFilters over \"git config\" works.\n+\ttest_config -C limits commitGraph.maxNewFilters 1 &&\n \t(\n \t\tcd limits &&\n \n \t\trm -f trace.event &&\n \t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n \t\t\tgit commit-graph write --reachable --changed-paths \\\n-\t\t\t\t--split=replace --max-new-filters=1 &&\n+\t\t\t\t--split=replace &&\n \t\ttest_filter_computed 1 trace.event &&\n \t\ttest_filter_not_computed 4 trace.event &&\n \t\ttest_filter_trunc_small 0 trace.event &&\n@@ -343,6 +345,26 @@ test_expect_success 'Bloom generation backfills previously-skipped filters' '\n \t)\n '\n \n+test_expect_success '--max-new-filters overrides configuration' '\n+\tgit init override &&\n+\ttest_when_finished \"rm -fr override\" &&\n+\ttest_config -C override commitGraph.maxNewFilters 2 &&\n+\t(\n+\t\tcd override &&\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t\t--max-new-filters=1 &&\n+\t\ttest_filter_computed 1 trace.event &&\n+\t\ttest_filter_not_computed 1 trace.event &&\n+\t\ttest_filter_trunc_small 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n test_expect_success 'Bloom generation backfills empty commits' '\n \tgit init empty &&\n \ttest_when_finished \"rm -fr empty\" &&\n-- \n2.28.0.510.g86fdc5f89a\n"},{"id":"405706","messageId":"40abfff8-3b5f-fa63-73fd-280cd80cd25f@gmail.com","threadId":"54216","inReplyTo":"cover.1600279373.git.me@ttaylorr.com","subject":"Re: [PATCH v2 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-09-16T22:51:27Z","receivedAt":"2020-09-16T22:51:34Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/16/20 2:06 PM, Taylor Blau wrote:\n> Hopefully this is it for this series ;). I think that it's in good shape\n> now, and I couldn't find anything in my own inspection that I wanted to\n> change. So, if others feel good, too, I think we should focus on\n> incremental fixes on top of this.\n\nI am also happy with this version. The test updates are\nrather nice.\n\nIt is worth highlighging the tracing changes, in case\nanyone was planning to use that style of JSON data in\nproduction. I think it is better to include this simpler\ndata model of one variable per event instead of rolling\nour own JSON format.\n\nThanks,\n-Stolee\n"},{"id":"405708","messageId":"xmqqzh5ptj5k.fsf@gitster.c.googlers.com","threadId":"54216","inReplyTo":"40abfff8-3b5f-fa63-73fd-280cd80cd25f@gmail.com","subject":"Re: [PATCH v2 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-16T23:07:51Z","receivedAt":"2020-09-16T23:08:03Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> On 9/16/20 2:06 PM, Taylor Blau wrote:\n>> Hopefully this is it for this series ;). I think that it's in good shape\n>> now, and I couldn't find anything in my own inspection that I wanted to\n>> change. So, if others feel good, too, I think we should focus on\n>> incremental fixes on top of this.\n>\n> I am also happy with this version. The test updates are\n> rather nice.\n>\n> It is worth highlighging the tracing changes, in case\n> anyone was planning to use that style of JSON data in\n> production. I think it is better to include this simpler\n> data model of one variable per event instead of rolling\n> our own JSON format.\n\nYup, lets merge it down soonish.\n\nThanks.\n"},{"id":"405712","messageId":"20200917004518.GA6478@nand.local","threadId":"54216","inReplyTo":"xmqqzh5ptj5k.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-17T00:45:18Z","receivedAt":"2020-09-17T00:53:04Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 16, 2020 at 04:07:51PM -0700, Junio C Hamano wrote:\n> Yup, lets merge it down soonish.\n\nIt's me, the bearer of bad news. I noticed an uninitialized read when\nrunning the tests with SANITIZE=address,undefined. I *think* the fix is\nas simple as a single replacement, but let me double check before you\nmerge this.\n\nSorry this topic has been such a disaster. Assuming the fix is isolated\nto a single patch, do you want a new version of that patch, or the whole\nseries?\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"405715","messageId":"xmqqd02ltdzl.fsf@gitster.c.googlers.com","threadId":"54216","inReplyTo":"20200917004518.GA6478@nand.local","subject":"Re: [PATCH v2 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-17T00:59:26Z","receivedAt":"2020-09-17T00:59:40Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Wed, Sep 16, 2020 at 04:07:51PM -0700, Junio C Hamano wrote:\n>> Yup, lets merge it down soonish.\n>\n> It's me, the bearer of bad news. I noticed an uninitialized read when\n> running the tests with SANITIZE=address,undefined. I *think* the fix is\n> as simple as a single replacement, but let me double check before you\n> merge this.\n>\n> Sorry this topic has been such a disaster. Assuming the fix is isolated\n> to a single patch, do you want a new version of that patch, or the whole\n> series?\n\nDepends on the timing---if it takes less than 2 days, please expect\nthat the topic would still be in my cache and \"squash this into\npatch 07/13\" would be sufficient. More than 10 days, wholesale\nreplacement would be the easiest. A single patch replacement in\nbetween.\n\nThanks.\n\n"},{"id":"405716","messageId":"20200917011049.GB6478@nand.local","threadId":"54216","inReplyTo":"xmqqd02ltdzl.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-17T01:10:49Z","receivedAt":"2020-09-17T01:11:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Wed, Sep 16, 2020 at 05:59:26PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> > On Wed, Sep 16, 2020 at 04:07:51PM -0700, Junio C Hamano wrote:\n> >> Yup, lets merge it down soonish.\n> >\n> > It's me, the bearer of bad news. I noticed an uninitialized read when\n> > running the tests with SANITIZE=address,undefined. I *think* the fix is\n> > as simple as a single replacement, but let me double check before you\n> > merge this.\n> >\n> > Sorry this topic has been such a disaster. Assuming the fix is isolated\n> > to a single patch, do you want a new version of that patch, or the whole\n> > series?\n>\n> Depends on the timing---if it takes less than 2 days, please expect\n> that the topic would still be in my cache and \"squash this into\n> patch 07/13\" would be sufficient. More than 10 days, wholesale\n> replacement would be the easiest. A single patch replacement in\n> between.\n\nI should have the patch in your inbox by the end of tonight, depending\non how fast my workstation can run the ASan-enabled test suite 13 times\n;).\n\n> Thanks.\n\nThanks,\nTaylor\n"},{"id":"405748","messageId":"20200917133442.GA12879@nand.local","threadId":"54216","inReplyTo":"20200917011049.GB6478@nand.local","subject":"Re: [PATCH v2 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-17T13:34:42Z","receivedAt":"2020-09-17T13:43:08Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Junio,\n\nOn Wed, Sep 16, 2020 at 09:10:49PM -0400, Taylor Blau wrote:\n> I should have the patch in your inbox by the end of tonight, depending\n> on how fast my workstation can run the ASan-enabled test suite 13 times\n> ;).\n\nAll finished. This is sufficient to fix the ASan-enabled test suite,\nalong with fixing a bug where we wouldn't respect the limit on changed\npaths when loading an existing commit-graph. This has nothing to do with\nthe user-specified '--max-new-filters', nor does it mean that we're\nstoring the limit in the commit-graph file. Instead it's because we're\nloading the bloom_filter_settings struct from the graph and\ninitializing it ourselves, instead of using the default values (which is\nthe case when we don't load a graph at all).\n\nAnyway, let's use this instead of 6/13. Here's an inter-diff that shows\nthe fix and test change:\n\n diff --git a/commit-graph.c b/commit-graph.c\n index 33af6c2430..fc6c6fdc3e 100644\n --- a/commit-graph.c\n +++ b/commit-graph.c\n @@ -424,6 +424,7 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n graph->bloom_filter_settings->hash_version = hash_version;\n graph->bloom_filter_settings->num_hashes = get_be32(data + chunk_offset + 4);\n graph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n +\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n }\n break;\n }\n diff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\n index dc7d62c778..5bc1627568 100755\n --- a/t/t4216-log-bloom.sh\n +++ b/t/t4216-log-bloom.sh\n @@ -175,11 +175,11 @@ test_expect_success 'persist filter settings' '\n GIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n git commit-graph write --reachable --changed-paths &&\n -\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2.txt &&\n +\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2.txt &&\n GIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n GIT_TRACE2_EVENT_NESTING=5 \\\n git commit-graph write --reachable --changed-paths &&\n -\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15\" trace2-auto.txt\n +\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2-auto.txt\n '\n\n test_max_changed_paths () {\n\n...and here is the updated patch. Sorry again for all of the trouble.\n\n--- >8 ---\n\nSubject: [PATCH] commit-graph.c: store maximum changed paths\n\nFor now, we assume that there is a fixed constant describing the\nmaximum number of changed paths we are willing to store in a Bloom\nfilter.\n\nPrepare for that to (at least partially) not be the case by making it a\nmember of the 'struct bloom_filter_settings'. This will be helpful in\nthe subsequent patches by reducing the size of test cases that exercise\nstoring too many changed paths, as well as preparing for an eventual\nfuture in which this value might change.\n\nThis patch alone does not cause newly generated Bloom filters to use\na custom upper-bound on the maximum number of changed paths a single\nBloom filter can hold, that will occur in a later patch.\n\nSigned-off-by: Taylor Blau \n---\n bloom.h | 11 ++++++++++-\n commit-graph.c | 4 ++++\n t/t4216-log-bloom.sh | 4 ++--\n 3 files changed, 16 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.h b/bloom.h\nindex d8fbb0fbf1..0b9b59a6fe 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -28,9 +28,18 @@ struct bloom_filter_settings {\n \t * that contain n*b bits.\n \t */\n \tuint32_t bits_per_entry;\n+\n+\t/*\n+\t * The maximum number of changed paths per commit\n+\t * before declaring a Bloom filter to be too-large.\n+\t *\n+\t * Not written to the commit-graph file.\n+\t */\n+\tuint32_t max_changed_paths;\n };\n\n-#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10 }\n+#define DEFAULT_BLOOM_MAX_CHANGES 512\n+#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10, DEFAULT_BLOOM_MAX_CHANGES }\n #define BITS_PER_WORD 8\n #define BLOOMDATA_CHUNK_HEADER_SIZE 3 * sizeof(uint32_t)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ea54d108b9..ba6d4a4c6c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -424,6 +424,7 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\tgraph->bloom_filter_settings->hash_version = hash_version;\n \t\t\t\tgraph->bloom_filter_settings->num_hashes = get_be32(data + chunk_offset + 4);\n \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n+\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n \t\t\t}\n \t\t\tbreak;\n \t\t}\n@@ -1201,6 +1202,7 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n \tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n \tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_object_intmax(&jw, \"max_changed_paths\", ctx->bloom_settings->max_changed_paths);\n \tjw_end(&jw);\n\n \ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n@@ -1669,6 +1671,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n \t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n \t\t\t\t\t\t\t bloom_settings.num_hashes);\n+\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n \t\tctx->bloom_settings = &bloom_settings;\n \t}\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fc7693806c..593571358d 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -174,11 +174,11 @@ test_expect_success 'persist filter settings' '\n \t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n \t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n \t\tGIT_TRACE2_EVENT_NESTING=5 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2-auto.txt\n '\n\n test_expect_success 'correctly report changes over limit' '\n--\n2.28.0.510.g86fdc5f89a\n\n"},{"id":"405755","messageId":"5caab572-1ef0-4a68-76a1-828a2f69cb6d@gmail.com","threadId":"54216","inReplyTo":"20200917133442.GA12879@nand.local","subject":"Re: [PATCH v2 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-09-17T13:38:09Z","receivedAt":"2020-09-17T14:46:08Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 9/17/2020 9:34 AM, Taylor Blau wrote:\n> Junio,\n> \n> On Wed, Sep 16, 2020 at 09:10:49PM -0400, Taylor Blau wrote:\n>> I should have the patch in your inbox by the end of tonight, depending\n>> on how fast my workstation can run the ASan-enabled test suite 13 times\n>> ;).\n> \n> All finished. This is sufficient to fix the ASan-enabled test suite,\n> along with fixing a bug where we wouldn't respect the limit on changed\n> paths when loading an existing commit-graph. This has nothing to do with\n> the user-specified '--max-new-filters', nor does it mean that we're\n> storing the limit in the commit-graph file. Instead it's because we're\n> loading the bloom_filter_settings struct from the graph and\n> initializing it ourselves, instead of using the default values (which is\n> the case when we don't load a graph at all).\n> \n> Anyway, let's use this instead of 6/13. Here's an inter-diff that shows\n> the fix and test change:\n> \n> diff --git a/commit-graph.c b/commit-graph.c\n> index 33af6c2430..fc6c6fdc3e 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -424,6 +424,7 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n> graph->bloom_filter_settings->hash_version = hash_version;\n> graph->bloom_filter_settings->num_hashes = get_be32(data + chunk_offset + 4);\n> graph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n> +\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n\nThis whitespace looks strange in the inter-diff...\n\n> \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n> +\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n\n...but it is correct in the patch itself.\n\n> -\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n> +\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2.txt &&\n> \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n> \t\tGIT_TRACE2_EVENT_NESTING=5 \\\n> \t\tgit commit-graph write --reachable --changed-paths &&\n> -\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n> +\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2-auto.txt\n\nI appreciate the additional tests to guarantee this is set\ncorrectly.\n\nThanks,\n-Stolee\n\n"},{"id":"405806","messageId":"20200917221302.GC23146@szeder.dev","threadId":"54216","inReplyTo":"4653b5b4bcd254a3791797214b46722b4062dc18.1600279373.git.me@ttaylorr.com","subject":"Re: [PATCH v2 10/13] bloom: encode out-of-bounds filters as non-empty","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-09-17T22:13:02Z","receivedAt":"2020-09-17T22:13:09Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Sep 16, 2020 at 02:07:59PM -0400, Taylor Blau wrote:\n> When a changed-path Bloom filter has either zero, or more than a\n> certain number (commonly 512) of entries, the commit-graph machinery\n> encodes it as \"missing\". More specifically, it sets the indices adjacent\n> in the BIDX chunk as equal to each other to indicate a \"length 0\"\n> filter; that is, that the filter occupies zero bytes on disk.\n> \n> This has heretofore been fine, since the commit-graph machinery has no\n> need to care about these filters with too few or too many changed paths.\n> Both cases act like no filter has been generated at all, and so there is\n> no need to store them.\n> \n> In a subsequent commit, however, the commit-graph machinery will learn\n> to only compute Bloom filters for some commits in the current\n> commit-graph layer. This is a change from the current implementation\n> which computes Bloom filters for all commits that are in the layer being\n> written. Critically for this patch, only computing some of the Bloom\n> filters means adding a third state for length 0 Bloom filters: zero\n> entries, too many entries, or \"hasn't been computed\".\n> \n> It will be important for that future patch to distinguish between \"not\n> representable\" (i.e., zero or too-many changed paths), and \"hasn't been\n> computed\". In particular, we don't want to waste time recomputing\n> filters that have already been computed.\n> \n> To that end, change how we store Bloom filters in the \"computed but not\n> representable\" category:\n> \n> - Bloom filters with no entries are stored as a single byte with all\n> bits low (i.e., all queries to that Bloom filter will return\n> \"definitely not\")\n> \n> - Bloom filters with too many entries are stored as a single byte with\n> all bits set high (i.e., all queries to that Bloom filter will\n> return \"maybe\").\n> \n> These rules are sufficient to not incur a behavior change by changing\n> the on-disk representation of these two classes. Likewise, no\n> specification changes are necessary for the commit-graph format, either:\n> \n> - Filters that were previously empty will be recomputed and stored\n> according to the new rules, and\n> \n> - old clients reading filters generated by new clients will interpret\n> the filters correctly and be none the wiser to how they were\n> generated.\n> \n> Clients will invoke the Bloom machinery in more cases than before, but\n> this can be addressed by returning a NULL filter when all bits are set\n> high. This can be addressed in a future patch.\n\nOTOH, clients will invoke the tree-diff machinery in fewer cases than\nbefore, because querying the Bloom filter of commits not modifying any\nfiles will now return \"definitely not\".\n\n> Finally, note that this does increase the size of on-disk commit-graphs,\n> but far less than other proposals. In particular, this is generally more\n> efficient than storing a bitmap for which commits haven't computed their\n> Bloom filters. Storing a bitmap incurs a penalty of one bit per commit,\n> whereas storing explicit filters as above incurs a penalty of one byte\n> per too-large or too-small commit.\n\ns/too-small/empty/\n\n> In practice, these boundary commits likely occupy a small proportion of\n> the overall number of commits, and so the size penalty is likely smaller\n> than storing a bitmap for all commits.\n\n | Percentage of\n | commits modifying\n | 0 path | >= 512 paths\n ---------------+------------+----------------\n android-base | 13.20% | 0.13%\n cmssw | 0.15% | 0.23%\n cpython | 3.07% | 0.01%\n elasticsearch | 0.70% | 1.00%\n gcc | 0.00% | 0.08%\n gecko-dev | 0.14% | 0.64%\n git | 0.11% | 0.02%\n glibc | 0.02% | 0.10%\n go | 0.00% | 0.07%\n homebrew-cask | 0.40% | 0.02%\n homebrew-core | 0.01% | 0.01%\n jdk | 0.26% | 5.64%\n linux | 0.01% | 0.51%\n llvm-project | 0.12% | 0.03%\n rails | 0.10% | 0.10%\n rust | 0.07% | 0.17%\n tensorflow | 0.09% | 1.02%\n webkit | 0.05% | 0.31%\n\n\n> A test to exercise filters which contain too many changed path entries\n> will be introduced in a subsequent patch.\n\n\n> diff --git a/bloom.h b/bloom.h\n> index c6d77e8393..70a8840896 100644\n> --- a/bloom.h\n> +++ b/bloom.h\n> @@ -93,6 +93,7 @@ enum bloom_filter_computed {\n> \tBLOOM_NOT_COMPUTED = (1 << 0),\n> \tBLOOM_COMPUTED = (1 << 1),\n> \tBLOOM_TRUNC_LARGE = (1 << 2),\n> +\tBLOOM_TRUNC_SMALL = (1 << 3),\n\ns/SMALL/EMPTY/\n\nThis \"small\" suffix in the constant, variable, and trace2 key names is\nmisleading, because we only mean empty commits.\n\n> };\n> \n> struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n> diff --git a/commit-graph.c b/commit-graph.c\n> index 1ca754f19c..bd4247bca5 100644\n> --- a/commit-graph.c\n> +++ b/commit-graph.c\n> @@ -974,6 +974,7 @@ struct write_commit_graph_context {\n> \n> \tint count_bloom_filter_computed;\n> \tint count_bloom_filter_not_computed;\n> +\tint count_bloom_filter_trunc_small;\n> \tint count_bloom_filter_trunc_large;\n> };\n> \n"},{"id":"405810","messageId":"20200917231344.GA1591704@nand.local","threadId":"54216","inReplyTo":"20200917221302.GC23146@szeder.dev","subject":"Re: [PATCH v2 10/13] bloom: encode out-of-bounds filters as non-empty","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-17T23:13:44Z","receivedAt":"2020-09-17T23:13:52Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 18, 2020 at 12:13:02AM +0200, SZEDER Gábor wrote:\n> > Clients will invoke the Bloom machinery in more cases than before, but\n> > this can be addressed by returning a NULL filter when all bits are set\n> > high. This can be addressed in a future patch.\n>\n> OTOH, clients will invoke the tree-diff machinery in fewer cases than\n> before, because querying the Bloom filter of commits not modifying any\n> files will now return \"definitely not\".\n\nAbsolutely right.\n\n> > Finally, note that this does increase the size of on-disk commit-graphs,\n> > but far less than other proposals. In particular, this is generally more\n> > efficient than storing a bitmap for which commits haven't computed their\n> > Bloom filters. Storing a bitmap incurs a penalty of one bit per commit,\n> > whereas storing explicit filters as above incurs a penalty of one byte\n> > per too-large or too-small commit.\n>\n> s/too-small/empty/\n\nFair enough, although I'm not planning to alter this or any other patch\nnow that it's picked up unless there's a real show-stopper (this doesn't\nseem like one).\n\n> > In practice, these boundary commits likely occupy a small proportion of\n> > the overall number of commits, and so the size penalty is likely smaller\n> > than storing a bitmap for all commits.\n>\n> | Percentage of\n> | commits modifying\n> | 0 path | >= 512 paths\n> ---------------+------------+----------------\n> android-base | 13.20% | 0.13%\n> cmssw | 0.15% | 0.23%\n> cpython | 3.07% | 0.01%\n> elasticsearch | 0.70% | 1.00%\n> gcc | 0.00% | 0.08%\n> gecko-dev | 0.14% | 0.64%\n> git | 0.11% | 0.02%\n> glibc | 0.02% | 0.10%\n> go | 0.00% | 0.07%\n> homebrew-cask | 0.40% | 0.02%\n> homebrew-core | 0.01% | 0.01%\n> jdk | 0.26% | 5.64%\n> linux | 0.01% | 0.51%\n> llvm-project | 0.12% | 0.03%\n> rails | 0.10% | 0.10%\n> rust | 0.07% | 0.17%\n> tensorflow | 0.09% | 1.02%\n> webkit | 0.05% | 0.31%\n\nThis is very useful information to have! Without the total number of\ncommits, it's impossible to know whether or not this is a win over the\nBFXL chunk. But, since the number of commits is probably \"large\" versus\nthe percentage of boundary commits which is \"small\", it's almost\ncertainly an advantage.\n\n> > A test to exercise filters which contain too many changed path entries\n> > will be introduced in a subsequent patch.\n>\n>\n> > diff --git a/bloom.h b/bloom.h\n> > index c6d77e8393..70a8840896 100644\n> > --- a/bloom.h\n> > +++ b/bloom.h\n> > @@ -93,6 +93,7 @@ enum bloom_filter_computed {\n> > \tBLOOM_NOT_COMPUTED = (1 << 0),\n> > \tBLOOM_COMPUTED = (1 << 1),\n> > \tBLOOM_TRUNC_LARGE = (1 << 2),\n> > +\tBLOOM_TRUNC_SMALL = (1 << 3),\n>\n> s/SMALL/EMPTY/\n>\n> This \"small\" suffix in the constant, variable, and trace2 key names is\n> misleading, because we only mean empty commits.\n\nI could buy that it might be misleading; I only picked this since it was\nthe opposite of \"large\". You could imagine that BLOOM_TRUNC_X means\n\"truncated in the direction of 'x'\", but to be honest I don't think that\nthis matters.\n\nI understand the churn of coming back to this after the topic has\nalready been merged creates more hassle, but frankly this series has\nalready gone on for quite a while, and it has been holding up important\nbug fixes that are unrelated to the main feature.\n\nSo, I think that if it's truly misleading, we could revisit this after\nthe topic is merged. But, I'm not planning on changing anything at this\npoint.\n\nThanks,\nTaylor\n"},{"id":"405816","messageId":"xmqqo8m3oqis.fsf@gitster.c.googlers.com","threadId":"54216","inReplyTo":"20200917231344.GA1591704@nand.local","subject":"Re: [PATCH v2 10/13] bloom: encode out-of-bounds filters as non-empty","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-18T00:52:11Z","receivedAt":"2020-09-18T00:52:15Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n>> > In practice, these boundary commits likely occupy a small proportion of\n>> > the overall number of commits, and so the size penalty is likely smaller\n>> > than storing a bitmap for all commits.\n>>\n>> | Percentage of\n>> | commits modifying\n>> | 0 path | >= 512 paths\n>> ---------------+------------+----------------\n>> android-base | 13.20% | 0.13%\n>> cmssw | 0.15% | 0.23%\n>> cpython | 3.07% | 0.01%\n>> elasticsearch | 0.70% | 1.00%\n>> gcc | 0.00% | 0.08%\n>> gecko-dev | 0.14% | 0.64%\n>> git | 0.11% | 0.02%\n>> glibc | 0.02% | 0.10%\n>> go | 0.00% | 0.07%\n>> homebrew-cask | 0.40% | 0.02%\n>> homebrew-core | 0.01% | 0.01%\n>> jdk | 0.26% | 5.64%\n>> linux | 0.01% | 0.51%\n>> llvm-project | 0.12% | 0.03%\n>> rails | 0.10% | 0.10%\n>> rust | 0.07% | 0.17%\n>> tensorflow | 0.09% | 1.02%\n>> webkit | 0.05% | 0.31%\n>\n> This is very useful information to have! Without the total number of\n> commits, it's impossible to know whether or not this is a win over the\n> BFXL chunk. But, since the number of commits is probably \"large\" versus\n> the percentage of boundary commits which is \"small\", it's almost\n> certainly an advantage.\n\nDo you want to include it in either the log message in one of the\ncommits, in code comment, or a technical doc?\n\n> So, I think that if it's truly misleading, we could revisit this after\n> the topic is merged. But, I'm not planning on changing anything at this\n> point.\n\nIf you do not want to help us go the last-mile to completion, that\nis sad, but I do not want to see a basically good patch stall like\nthat, so let's find somebody else who can do the helping ;-)\n\nHere is what my trial rebase produced. I'll queue it to 'seen' (if\nyou prefer I can send a full v3 patch, but I expect that you know\nhow to fetch from 'seen' and review locally) after checking if the\nresult passes the tests locally; an extra set of eyeballs to verify\nthe result is pretty much appreciated.\n\nThanks.\n\n\n1: ca6c060171 ! 1: 9b294a0c66 bloom: encode out-of-bounds filters as non-empty\n @@ Commit message\n efficient than storing a bitmap for which commits haven't computed their\n Bloom filters. Storing a bitmap incurs a penalty of one bit per commit,\n whereas storing explicit filters as above incurs a penalty of one byte\n - per too-large or too-small commit.\n + per too-large or empty commit.\n \n In practice, these boundary commits likely occupy a small proportion of\n the overall number of commits, and so the size penalty is likely smaller\n @@ bloom.c: struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n +\t\tif (!filter->len) {\n +\t\t\tif (computed)\n -+\t\t\t\t*computed |= BLOOM_TRUNC_SMALL;\n ++\t\t\t\t*computed |= BLOOM_TRUNC_EMPTY;\n +\t\t\tfilter->len = 1;\n +\t\t}\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n @@ bloom.h: enum bloom_filter_computed {\n \tBLOOM_NOT_COMPUTED = (1 << 0),\n \tBLOOM_COMPUTED = (1 << 1),\n \tBLOOM_TRUNC_LARGE = (1 << 2),\n -+\tBLOOM_TRUNC_SMALL = (1 << 3),\n ++\tBLOOM_TRUNC_EMPTY = (1 << 3),\n };\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n @@ commit-graph.c: struct write_commit_graph_context {\n \n \tint count_bloom_filter_computed;\n \tint count_bloom_filter_not_computed;\n -+\tint count_bloom_filter_trunc_small;\n ++\tint count_bloom_filter_trunc_empty;\n \tint count_bloom_filter_trunc_large;\n };\n \n @@ commit-graph.c: static void trace2_bloom_filter_write_statistics(struct write_co\n \t\t\t ctx->count_bloom_filter_computed);\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-not-computed\",\n \t\t\t ctx->count_bloom_filter_not_computed);\n -+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-small\",\n -+\t\t\t ctx->count_bloom_filter_trunc_small);\n ++\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-empty\",\n ++\t\t\t ctx->count_bloom_filter_trunc_empty);\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-large\",\n \t\t\t ctx->count_bloom_filter_trunc_large);\n }\n @@ commit-graph.c: static void compute_bloom_filters(struct write_commit_graph_cont\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n \t\t\tctx->count_bloom_filter_computed++;\n -+\t\t\tif (computed & BLOOM_TRUNC_SMALL)\n -+\t\t\t\tctx->count_bloom_filter_trunc_small++;\n ++\t\t\tif (computed & BLOOM_TRUNC_EMPTY)\n ++\t\t\t\tctx->count_bloom_filter_trunc_empty++;\n \t\t\tif (computed & BLOOM_TRUNC_LARGE)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n @@ t/t4216-log-bloom.sh: test_max_changed_paths () {\n \tgrep \"\\\"key\\\":\\\"filter-computed\\\",\\\"value\\\":\\\"$1\\\"\" $2\n }\n \n -+test_filter_trunc_small () {\n -+\tgrep \"\\\"key\\\":\\\"filter-trunc-small\\\",\\\"value\\\":\\\"$1\\\"\" $2\n ++test_filter_trunc_empty () {\n ++\tgrep \"\\\"key\\\":\\\"filter-trunc-empty\\\",\\\"value\\\":\\\"$1\\\"\" $2\n +}\n +\n test_filter_trunc_large () {\n @@ t/t4216-log-bloom.sh: test_expect_success 'correctly report changes over limit'\n '\n \n +test_expect_success 'correctly report commits with no changed paths' '\n -+\tgit init small &&\n -+\ttest_when_finished \"rm -fr small\" &&\n ++\tgit init empty &&\n ++\ttest_when_finished \"rm -fr empty\" &&\n +\t(\n -+\t\tcd small &&\n ++\t\tcd empty &&\n +\n +\t\tgit commit --allow-empty -m \"initial commit\" &&\n +\n @@ t/t4216-log-bloom.sh: test_expect_success 'correctly report changes over limit'\n +\t\t\tgit commit-graph write --reachable --changed-paths &&\n +\t\ttest_filter_computed 1 trace.event &&\n +\t\ttest_filter_not_computed 0 trace.event &&\n -+\t\ttest_filter_trunc_small 1 trace.event &&\n ++\t\ttest_filter_trunc_empty 1 trace.event &&\n +\t\ttest_filter_trunc_large 0 trace.event\n +\t)\n +'\n2: 11db600d51 = 2: 1b4c861e68 commit-graph: rename 'split_commit_graph_opts'\n3: cf49598137 ! 3: d6c1bd395e builtin/commit-graph.c: introduce '--max-new-filters='\n @@ t/t4216-log-bloom.sh: test_expect_success 'correctly report commits with no chan\n +\n +\t\ttest_filter_computed 2 trace.event &&\n +\t\ttest_filter_not_computed 3 trace.event &&\n -+\t\ttest_filter_trunc_small 0 trace.event &&\n ++\t\ttest_filter_trunc_empty 0 trace.event &&\n +\t\ttest_filter_trunc_large 0 trace.event\n +\t)\n +'\n @@ t/t4216-log-bloom.sh: test_expect_success 'correctly report commits with no chan\n +\t\t\t\t--split=replace --max-new-filters=1 &&\n +\t\ttest_filter_computed 1 trace.event &&\n +\t\ttest_filter_not_computed 4 trace.event &&\n -+\t\ttest_filter_trunc_small 0 trace.event &&\n ++\t\ttest_filter_trunc_empty 0 trace.event &&\n +\t\ttest_filter_trunc_large 0 trace.event\n +\t)\n +'\n @@ t/t4216-log-bloom.sh: test_expect_success 'correctly report commits with no chan\n +\t\t\t\t\t--changed-paths --max-new-filters=2 &&\n +\t\t\ttest_filter_computed 2 trace.event &&\n +\t\t\ttest_filter_not_computed 4 trace.event &&\n -+\t\t\ttest_filter_trunc_small 2 trace.event &&\n ++\t\t\ttest_filter_trunc_empty 2 trace.event &&\n +\t\t\ttest_filter_trunc_large 0 trace.event\n +\t\tdone &&\n +\n @@ t/t4216-log-bloom.sh: test_expect_success 'correctly report commits with no chan\n +\t\t\t\t--changed-paths --max-new-filters=2 &&\n +\t\ttest_filter_computed 0 trace.event &&\n +\t\ttest_filter_not_computed 6 trace.event &&\n -+\t\ttest_filter_trunc_small 0 trace.event &&\n ++\t\ttest_filter_trunc_empty 0 trace.event &&\n +\t\ttest_filter_trunc_large 0 trace.event\n +\t)\n +'\n4: 1bc82cd008 ! 4: b0d51fb04a commit-graph: introduce 'commitGraph.maxNewFilters'\n @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation is limited by --max-\n +\t\t\t\t--split=replace &&\n \t\ttest_filter_computed 1 trace.event &&\n \t\ttest_filter_not_computed 4 trace.event &&\n - \t\ttest_filter_trunc_small 0 trace.event &&\n + \t\ttest_filter_trunc_empty 0 trace.event &&\n @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills previously-skipped filters' '\n \t)\n '\n @@ t/t4216-log-bloom.sh: test_expect_success 'Bloom generation backfills previously\n +\t\t\t\t--max-new-filters=1 &&\n +\t\ttest_filter_computed 1 trace.event &&\n +\t\ttest_filter_not_computed 1 trace.event &&\n -+\t\ttest_filter_trunc_small 0 trace.event &&\n ++\t\ttest_filter_trunc_empty 0 trace.event &&\n +\t\ttest_filter_trunc_large 0 trace.event\n +\t)\n +'\n"},{"id":"405817","messageId":"20200918011517.GA2013074@nand.local","threadId":"54216","inReplyTo":"xmqqo8m3oqis.fsf@gitster.c.googlers.com","subject":"Re: [PATCH v2 10/13] bloom: encode out-of-bounds filters as non-empty","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T01:15:17Z","receivedAt":"2020-09-18T01:15:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 17, 2020 at 05:52:11PM -0700, Junio C Hamano wrote:\n> Taylor Blau writes:\n>\n> >> > In practice, these boundary commits likely occupy a small proportion of\n> >> > the overall number of commits, and so the size penalty is likely smaller\n> >> > than storing a bitmap for all commits.\n> >>\n> >> | Percentage of\n> >> | commits modifying\n> >> | 0 path | >= 512 paths\n> >> ---------------+------------+----------------\n> >> android-base | 13.20% | 0.13%\n> >> cmssw | 0.15% | 0.23%\n> >> cpython | 3.07% | 0.01%\n> >> elasticsearch | 0.70% | 1.00%\n> >> gcc | 0.00% | 0.08%\n> >> gecko-dev | 0.14% | 0.64%\n> >> git | 0.11% | 0.02%\n> >> glibc | 0.02% | 0.10%\n> >> go | 0.00% | 0.07%\n> >> homebrew-cask | 0.40% | 0.02%\n> >> homebrew-core | 0.01% | 0.01%\n> >> jdk | 0.26% | 5.64%\n> >> linux | 0.01% | 0.51%\n> >> llvm-project | 0.12% | 0.03%\n> >> rails | 0.10% | 0.10%\n> >> rust | 0.07% | 0.17%\n> >> tensorflow | 0.09% | 1.02%\n> >> webkit | 0.05% | 0.31%\n> >\n> > This is very useful information to have! Without the total number of\n> > commits, it's impossible to know whether or not this is a win over the\n> > BFXL chunk. But, since the number of commits is probably \"large\" versus\n> > the percentage of boundary commits which is \"small\", it's almost\n> > certainly an advantage.\n>\n> Do you want to include it in either the log message in one of the\n> commits, in code comment, or a technical doc?\n\nLet's put it in the commit message. I think that it's useful enough that\npeople interested enough to dig through the commits would want it, but\ntoo detailed to be as visible as in the technical documentation.\n\n> > So, I think that if it's truly misleading, we could revisit this after\n> > the topic is merged. But, I'm not planning on changing anything at this\n> > point.\n>\n> If you do not want to help us go the last-mile to completion, that\n> is sad, but I do not want to see a basically good patch stall like\n> that, so let's find somebody else who can do the helping ;-)\n\nI'm sorry to give off the impression that I do not want to help; that's\nnot the case. After reading Gàbor's email, I thought that the changes he\nwas suggesting were minor, and was trying to say \"let's do this on top\nand apply the important bug fixes first,\" not, \"I am never going to\ntouch this again\".\n\n> Here is what my trial rebase produced. I'll queue it to 'seen' (if\n> you prefer I can send a full v3 patch, but I expect that you know\n> how to fetch from 'seen' and review locally) after checking if the\n> result passes the tests locally; an extra set of eyeballs to verify\n> the result is pretty much appreciated.\n\nIt looks off to a great start; and thanks for taking the care to make up\nfor my laziness. Everything you did looks good to me. I touched up\n\"bloom: encode out-of-bounds filters as non-empty\" locally to include\nthe table above (and some commentary around it), so I'll send a v3 to\nthe list once I have finished running the tests.\n\n> Thanks.\n\nThank you, and sorry again.\n\nThanks,\nTaylor\n"},{"id":"405822","messageId":"cover.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1599664389.git.me@ttaylorr.com","subject":"[PATCH v3 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:58:35Z","receivedAt":"2020-09-18T02:58:50Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Hi again,\n\nHere's a few more changes to the \"Bloom filter improvements\" topic,\nsent as one brand-new re-roll in order to simplify queuing. It\nincorporates:\n\n - Junio's changes from applying to 'seen' (namely, dropping references\n to \"too-small\" commits in favor of the much more clear \"empty\"\n commits).\n\n - On top, I applied Gàbor's data gathered in [1] to 10/13 (and added a\n little more detail on the absolute and relative size differences of\n the resulting commit-graph files built before/after that patch).\n\nThanks to everyone who has helped out along the way with this series\n(Stolee, Gàbor, Junio, Jakub, and I am sure that I am forgetting some...).\n\nSorry to have given anyone the impression that I was abandoning this\ntopic; I'm definitely not ;-).\n\n[1]: https://lore.kernel.org/git/20200917221302.GC23146@szeder.dev/\n\nDerrick Stolee (1):\n bloom/diff: properly short-circuit on max_changes\n\nTaylor Blau (12):\n commit-graph: introduce 'get_bloom_filter_settings()'\n t4216: use an '&&'-chain\n commit-graph: pass a 'struct repository *' in more places\n t/helper/test-read-graph.c: prepare repo settings\n commit-graph: respect 'commitGraph.readChangedPaths'\n commit-graph.c: store maximum changed paths\n bloom: split 'get_bloom_filter()' in two\n bloom: use provided 'struct bloom_filter_settings'\n bloom: encode out-of-bounds filters as non-empty\n commit-graph: rename 'split_commit_graph_opts'\n builtin/commit-graph.c: introduce '--max-new-filters='\n commit-graph: introduce 'commitGraph.maxNewFilters'\n\n Documentation/config.txt | 2 +\n Documentation/config/commitgraph.txt | 8 +\n Documentation/git-commit-graph.txt | 6 +\n .../technical/commit-graph-format.txt | 2 +-\n blame.c | 8 +-\n bloom.c | 59 +++--\n bloom.h | 29 ++-\n builtin/commit-graph.c | 63 ++++-\n commit-graph.c | 141 +++++++---\n commit-graph.h | 17 +-\n diff.h | 2 -\n fuzz-commit-graph.c | 5 +-\n line-log.c | 2 +-\n repo-settings.c | 3 +\n repository.h | 1 +\n revision.c | 7 +-\n t/helper/test-bloom.c | 4 +-\n t/helper/test-read-graph.c | 3 +-\n t/t0095-bloom.sh | 8 +-\n t/t4216-log-bloom.sh | 242 ++++++++++++++++--\n t/t5324-split-commit-graph.sh | 13 +\n tree-diff.c | 5 +-\n 22 files changed, 507 insertions(+), 123 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\n--\n2.28.0.510.g375ecf1f36\n"},{"id":"405823","messageId":"8f8a82224364ba6ca487d99d0e2cafeabe2edc5a.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 01/13] commit-graph: introduce 'get_bloom_filter_settings()'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:58:52Z","receivedAt":"2020-09-18T02:59:01Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Many places in the code often need a pointer to the commit-graph's\n'struct bloom_filter_settings', in which case they often take the value\nfrom the top-most commit-graph.\n\nIn the non-split case, this works as expected. In the split case,\nhowever, things get a little tricky. Not all layers in a chain of\nincremental commit-graphs are required to themselves have Bloom data,\nand so whether or not some part of the code uses Bloom filters depends\nentirely on whether or not the top-most level of the commit-graph chain\nhas Bloom filters.\n\nThis has been the behavior since Bloom filters were introduced, and has\nbeen codified into the tests since a759bfa9ee (t4216: add end to end\ntests for git log with Bloom filters, 2020-04-06). In fact, t4216.130\nrequires that Bloom filters are not used in exactly the case described\nearlier.\n\nThere is no reason that this needs to be the case, since it is perfectly\nvalid for commits in an earlier layer to have Bloom filters when commits\nin a newer layer do not.\n\nSince Bloom settings are guaranteed in practice to be the same for any\nlayer in a chain that has Bloom data, it is sufficient to traverse the\n'->base_graph' pointer until either (1) a non-null 'struct\nbloom_filter_settings *' is found, or (2) until we are at the root of\nthe commit-graph chain.\n\nIntroduce a 'get_bloom_filter_settings()' function that does just this,\nand use it instead of purely dereferencing the top-most graph's\n'->bloom_filter_settings' pointer.\n\nWhile we're at it, add an additional test in t5324 to guard against code\nin the commit-graph writing machinery that doesn't correctly handle a\nNULL 'struct bloom_filter *'.\n\nCo-authored-by: Derrick Stolee \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n blame.c | 6 ++++--\n bloom.c | 6 +++---\n commit-graph.c | 11 +++++++++++\n commit-graph.h | 2 ++\n revision.c | 5 +----\n t/t4216-log-bloom.sh | 9 ++++++---\n t/t5324-split-commit-graph.sh | 13 +++++++++++++\n 7 files changed, 40 insertions(+), 12 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 1be1cd82a2..903e23af23 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -2892,16 +2892,18 @@ void setup_blame_bloom_data(struct blame_scoreboard *sb,\n \t\t\t const char *path)\n {\n \tstruct blame_bloom_data *bd;\n+\tstruct bloom_filter_settings *bs;\n \n \tif (!sb->repo->objects->commit_graph)\n \t\treturn;\n \n-\tif (!sb->repo->objects->commit_graph->bloom_filter_settings)\n+\tbs = get_bloom_filter_settings(sb->repo);\n+\tif (!bs)\n \t\treturn;\n \n \tbd = xmalloc(sizeof(struct blame_bloom_data));\n \n-\tbd->settings = sb->repo->objects->commit_graph->bloom_filter_settings;\n+\tbd->settings = bs;\n \n \tbd->alloc = 4;\n \tbd->nr = 0;\ndiff --git a/bloom.c b/bloom.c\nindex 1a573226e7..cd9380ac62 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -38,7 +38,7 @@ static int load_bloom_filter_from_graph(struct commit_graph *g,\n \twhile (graph_pos < g->num_commits_in_base)\n \t\tg = g->base_graph;\n \n-\t/* The commit graph commit 'c' lives in doesn't carry bloom filters. */\n+\t/* The commit graph commit 'c' lives in doesn't carry Bloom filters. */\n \tif (!g->chunk_bloom_indexes)\n \t\treturn 0;\n \n@@ -195,8 +195,8 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n \t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tr->objects->commit_graph->chunk_bloom_indexes)\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n+\t\t\t\treturn filter;\n \t}\n \n \tif (filter->data)\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 0ed003e218..6a36ed0b06 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -667,6 +667,17 @@ int generation_numbers_enabled(struct repository *r)\n \treturn !!first_generation;\n }\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r)\n+{\n+\tstruct commit_graph *g = r->objects->commit_graph;\n+\twhile (g) {\n+\t\tif (g->bloom_filter_settings)\n+\t\t\treturn g->bloom_filter_settings;\n+\t\tg = g->base_graph;\n+\t}\n+\treturn NULL;\n+}\n+\n static void close_commit_graph_one(struct commit_graph *g)\n {\n \tif (!g)\ndiff --git a/commit-graph.h b/commit-graph.h\nindex 09a97030dc..0677dd1031 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -87,6 +87,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n */\n int generation_numbers_enabled(struct repository *r);\n \n+struct bloom_filter_settings *get_bloom_filter_settings(struct repository *r);\n+\n enum commit_graph_write_flags {\n \tCOMMIT_GRAPH_WRITE_APPEND = (1 << 0),\n \tCOMMIT_GRAPH_WRITE_PROGRESS = (1 << 1),\ndiff --git a/revision.c b/revision.c\nindex 1239023f93..6aeb764821 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -681,10 +681,7 @@ static void prepare_to_use_bloom_filter(struct rev_info *revs)\n \n \trepo_parse_commit(revs->repo, revs->commits->item);\n \n-\tif (!revs->repo->objects->commit_graph)\n-\t\treturn;\n-\n-\trevs->bloom_filter_settings = revs->repo->objects->commit_graph->bloom_filter_settings;\n+\trevs->bloom_filter_settings = get_bloom_filter_settings(revs->repo);\n \tif (!revs->bloom_filter_settings)\n \t\treturn;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 4bb9e9dbe2..715912ad0f 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -65,7 +65,7 @@ setup () {\n \n test_bloom_filters_used () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":0,\\\"maybe\\\"\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":${2:-0},\\\"maybe\\\"\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom &&\n@@ -139,8 +139,11 @@ test_expect_success 'setup - add commit-graph to the chain without Bloom filters\n \ttest_line_count = 2 .git/objects/info/commit-graphs/commit-graph-chain\n '\n \n-test_expect_success 'Do not use Bloom filters if the latest graph does not have Bloom filters.' '\n-\ttest_bloom_filters_not_used \"-- A/B\"\n+test_expect_success 'use Bloom filters even if the latest graph does not have Bloom filters' '\n+\t# Ensure that the number of empty filters is equal to the number of\n+\t# filters in the latest graph layer to prove that they are loaded (and\n+\t# ignored).\n+\ttest_bloom_filters_used \"-- A/B\" 3\n '\n \n test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\ndiff --git a/t/t5324-split-commit-graph.sh b/t/t5324-split-commit-graph.sh\nindex 18216463c7..c334ee9155 100755\n--- a/t/t5324-split-commit-graph.sh\n+++ b/t/t5324-split-commit-graph.sh\n@@ -427,4 +427,17 @@ done <<\\EOF\n 0600 -r--------\n EOF\n \n+test_expect_success '--split=replace with partial Bloom data' '\n+\trm -rf $graphdir $infodir/commit-graph &&\n+\tgit reset --hard commits/3 &&\n+\tgit rev-list -1 HEAD~2 >a &&\n+\tgit rev-list -1 HEAD~1 >b &&\n+\tgit commit-graph write --split=no-merge --stdin-commits --changed-paths graph-files &&\n+\ttest_line_count = 1 graph-files &&\n+\tverify_chain_files_exist $graphdir\n+'\n+\n test_done\n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405824","messageId":"1644544221150c6d642b109d38c5607e600153ed.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 03/13] commit-graph: pass a 'struct repository *' in more places","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:02Z","receivedAt":"2020-09-18T02:59:11Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a future commit, some commit-graph internals will want access to\n'r->settings', but we only have the 'struct object_directory *'\ncorresponding to that repository.\n\nAdd an additional parameter to pass the repository around in more\nplaces.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n builtin/commit-graph.c | 2 +-\n commit-graph.c | 17 ++++++++++-------\n commit-graph.h | 6 ++++--\n fuzz-commit-graph.c | 5 +++--\n 4 files changed, 18 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 523501f217..ba5584463f 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -106,7 +106,7 @@ static int graph_verify(int argc, const char **argv)\n \tFREE_AND_NULL(graph_name);\n \n \tif (open_ok)\n-\t\tgraph = load_commit_graph_one_fd_st(fd, &st, odb);\n+\t\tgraph = load_commit_graph_one_fd_st(the_repository, fd, &st, odb);\n \telse\n \t\tgraph = read_commit_graph_one(the_repository, odb);\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex 6a36ed0b06..72a838bd00 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -231,7 +231,8 @@ int open_commit_graph(const char *graph_file, int *fd, struct stat *st)\n \treturn 1;\n }\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tvoid *graph_map;\n@@ -247,7 +248,7 @@ struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n \t}\n \tgraph_map = xmmap(NULL, graph_size, PROT_READ, MAP_PRIVATE, fd, 0);\n \tclose(fd);\n-\tret = parse_commit_graph(graph_map, graph_size);\n+\tret = parse_commit_graph(r, graph_map, graph_size);\n \n \tif (ret)\n \t\tret->odb = odb;\n@@ -287,7 +288,8 @@ static int verify_commit_graph_lite(struct commit_graph *g)\n \treturn 0;\n }\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size)\n {\n \tconst unsigned char *data, *chunk_lookup;\n \tuint32_t i;\n@@ -452,7 +454,8 @@ struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size)\n \treturn NULL;\n }\n \n-static struct commit_graph *load_commit_graph_one(const char *graph_file,\n+static struct commit_graph *load_commit_graph_one(struct repository *r,\n+\t\t\t\t\t\t const char *graph_file,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \n@@ -464,7 +467,7 @@ static struct commit_graph *load_commit_graph_one(const char *graph_file,\n \tif (!open_ok)\n \t\treturn NULL;\n \n-\tg = load_commit_graph_one_fd_st(fd, &st, odb);\n+\tg = load_commit_graph_one_fd_st(r, fd, &st, odb);\n \n \tif (g)\n \t\tg->filename = xstrdup(graph_file);\n@@ -476,7 +479,7 @@ static struct commit_graph *load_commit_graph_v1(struct repository *r,\n \t\t\t\t\t\t struct object_directory *odb)\n {\n \tchar *graph_name = get_commit_graph_filename(odb);\n-\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \tfree(graph_name);\n \n \treturn g;\n@@ -557,7 +560,7 @@ static struct commit_graph *load_commit_graph_chain(struct repository *r,\n \t\tvalid = 0;\n \t\tfor (odb = r->objects->odb; odb; odb = odb->next) {\n \t\t\tchar *graph_name = get_split_graph_filename(odb, line.buf);\n-\t\t\tstruct commit_graph *g = load_commit_graph_one(graph_name, odb);\n+\t\t\tstruct commit_graph *g = load_commit_graph_one(r, graph_name, odb);\n \n \t\t\tfree(graph_name);\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex 0677dd1031..d9acb22bac 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -75,11 +75,13 @@ struct commit_graph {\n \tstruct bloom_filter_settings *bloom_filter_settings;\n };\n \n-struct commit_graph *load_commit_graph_one_fd_st(int fd, struct stat *st,\n+struct commit_graph *load_commit_graph_one_fd_st(struct repository *r,\n+\t\t\t\t\t\t int fd, struct stat *st,\n \t\t\t\t\t\t struct object_directory *odb);\n struct commit_graph *read_commit_graph_one(struct repository *r,\n \t\t\t\t\t struct object_directory *odb);\n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n /*\n * Return 1 if and only if the repository has a commit-graph\ndiff --git a/fuzz-commit-graph.c b/fuzz-commit-graph.c\nindex 430817214d..e7cf6d5b0f 100644\n--- a/fuzz-commit-graph.c\n+++ b/fuzz-commit-graph.c\n@@ -1,7 +1,8 @@\n #include \"commit-graph.h\"\n #include \"repository.h\"\n \n-struct commit_graph *parse_commit_graph(void *graph_map, size_t graph_size);\n+struct commit_graph *parse_commit_graph(struct repository *r,\n+\t\t\t\t\tvoid *graph_map, size_t graph_size);\n \n int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size);\n \n@@ -10,7 +11,7 @@ int LLVMFuzzerTestOneInput(const uint8_t *data, size_t size)\n \tstruct commit_graph *g;\n \n \tinitialize_the_repository();\n-\tg = parse_commit_graph((void *)data, size);\n+\tg = parse_commit_graph(the_repository, (void *)data, size);\n \trepo_clear(the_repository);\n \tfree_commit_graph(g);\n \n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405825","messageId":"6c2a8c9c2f19c6a41aa22f25f3bb54defae4835e.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 02/13] t4216: use an '&&'-chain","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:58:57Z","receivedAt":"2020-09-18T02:59:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In a759bfa9ee (t4216: add end to end tests for git log with Bloom\nfilters, 2020-04-06), a 'rm' invocation was added without a\ncorresponding '&&' chain.\n\nWhen 'trace.perf' already exists, everything works fine. However, the\nfunction can be executed without 'trace.perf' on disk (eg., when the\nsubset of tests run is altered with '--run'), and so the bare 'rm'\ncomplains about a missing file.\n\nTo remove some noise from the test log, invoke 'rm' with '-f', at which\npoint it is sensible to place the 'rm -f' in an '&&'-chain, which is\nboth (1) our usual style, and (2) avoids a broken chain in the future if\nmore commands are added at the beginning of the function.\n\nHelped-by: Eric Sunshine \nHelped-by: Jeff King \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n t/t4216-log-bloom.sh | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 715912ad0f..cd89c75002 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -58,7 +58,7 @@ sane_unset GIT_TRACE2_PERF_BRIEF\n sane_unset GIT_TRACE2_CONFIG_PARAMS\n \n setup () {\n-\trm \"$TRASH_DIRECTORY/trace.perf\"\n+\trm -f \"$TRASH_DIRECTORY/trace.perf\" &&\n \tgit -c core.commitGraph=false log --pretty=\"format:%s\" $1 >log_wo_bloom &&\n \tGIT_TRACE2_PERF=\"$TRASH_DIRECTORY/trace.perf\" git -c core.commitGraph=true log --pretty=\"format:%s\" $1 >log_w_bloom\n }\n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405826","messageId":"2e79a470319612b293748cdc0c1fe601cf2e20f1.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 05/13] commit-graph: respect 'commitGraph.readChangedPaths'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:11Z","receivedAt":"2020-09-18T02:59:18Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Git uses the 'core.commitGraph' configuration value to control whether\nor not the commit graph is used when parsing commits or performing a\ntraversal.\n\nNow that commit-graphs can also contain a section for changed-path Bloom\nfilters, administrators that already have commit-graphs may find it\nconvenient to use those graphs without relying on their changed-path\nBloom filters. This can happen, for example, during a staged roll-out,\nor in the event of an incident.\n\nIntroduce 'commitGraph.readChangedPaths' to control whether or not Bloom\nfilters are read. Note that this configuration is independent from both:\n\n - 'core.commitGraph', to allow flexibility in using all parts of a\n commit-graph _except_ for its Bloom filters.\n\n - The '--changed-paths' option for 'git commit-graph write', to allow\n reading and writing Bloom filters to be controlled independently.\n\nWhen the variable is set, pretend as if no Bloom data was specified at\nall. This avoids adding additional special-casing outside of the\ncommit-graph internals.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n Documentation/config.txt | 2 ++\n Documentation/config/commitgraph.txt | 4 ++++\n commit-graph.c | 6 ++++--\n repo-settings.c | 3 +++\n repository.h | 1 +\n t/t4216-log-bloom.sh | 4 +++-\n 6 files changed, 17 insertions(+), 3 deletions(-)\n create mode 100644 Documentation/config/commitgraph.txt\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 3042d80978..770ae79b82 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -340,6 +340,8 @@ include::config/column.txt[]\n \n include::config/commit.txt[]\n \n+include::config/commitgraph.txt[]\n+\n include::config/credential.txt[]\n \n include::config/completion.txt[]\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nnew file mode 100644\nindex 0000000000..cff0797b54\n--- /dev/null\n+++ b/Documentation/config/commitgraph.txt\n@@ -0,0 +1,4 @@\n+commitGraph.readChangedPaths::\n+\tIf true, then git will use the changed-path Bloom filters in the\n+\tcommit-graph file (if it exists, and they are present). Defaults to\n+\ttrue. See linkgit:git-commit-graph[1] for more information.\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 72a838bd00..ea54d108b9 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -327,6 +327,8 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\treturn NULL;\n \t}\n \n+\tprepare_repo_settings(r);\n+\n \tgraph = alloc_commit_graph();\n \n \tgraph->hash_len = the_hash_algo->rawsz;\n@@ -403,14 +405,14 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\tcase GRAPH_CHUNKID_BLOOMINDEXES:\n \t\t\tif (graph->chunk_bloom_indexes)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths)\n \t\t\t\tgraph->chunk_bloom_indexes = data + chunk_offset;\n \t\t\tbreak;\n \n \t\tcase GRAPH_CHUNKID_BLOOMDATA:\n \t\t\tif (graph->chunk_bloom_data)\n \t\t\t\tchunk_repeated = 1;\n-\t\t\telse {\n+\t\t\telse if (r->settings.commit_graph_read_changed_paths) {\n \t\t\t\tuint32_t hash_version;\n \t\t\t\tgraph->chunk_bloom_data = data + chunk_offset;\n \t\t\t\thash_version = get_be32(data + chunk_offset);\ndiff --git a/repo-settings.c b/repo-settings.c\nindex aa61a35338..88ccce2036 100644\n--- a/repo-settings.c\n+++ b/repo-settings.c\n@@ -17,9 +17,12 @@ void prepare_repo_settings(struct repository *r)\n \n \tif (!repo_config_get_bool(r, \"core.commitgraph\", &value))\n \t\tr->settings.core_commit_graph = value;\n+\tif (!repo_config_get_bool(r, \"commitgraph.readchangedpaths\", &value))\n+\t\tr->settings.commit_graph_read_changed_paths = value;\n \tif (!repo_config_get_bool(r, \"gc.writecommitgraph\", &value))\n \t\tr->settings.gc_write_commit_graph = value;\n \tUPDATE_DEFAULT_BOOL(r->settings.core_commit_graph, 1);\n+\tUPDATE_DEFAULT_BOOL(r->settings.commit_graph_read_changed_paths, 1);\n \tUPDATE_DEFAULT_BOOL(r->settings.gc_write_commit_graph, 1);\n \n \tif (!repo_config_get_int(r, \"index.version\", &value))\ndiff --git a/repository.h b/repository.h\nindex 628c834367..bacf843d46 100644\n--- a/repository.h\n+++ b/repository.h\n@@ -30,6 +30,7 @@ struct repo_settings {\n \tint initialized;\n \n \tint core_commit_graph;\n+\tint commit_graph_read_changed_paths;\n \tint gc_write_commit_graph;\n \tint fetch_write_commit_graph;\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex cd89c75002..fc7693806c 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -95,7 +95,9 @@ do\n \t\t \"--ancestry-path side..master\"\n \tdo\n \t\ttest_expect_success \"git log option: $option for path: $path\" '\n-\t\t\ttest_bloom_filters_used \"$option -- $path\"\n+\t\t\ttest_bloom_filters_used \"$option -- $path\" &&\n+\t\t\ttest_config commitgraph.readChangedPaths false &&\n+\t\t\ttest_bloom_filters_not_used \"$option -- $path\"\n \t\t'\n \tdone\n done\n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405827","messageId":"e0a6484b693e46f848b2d86c4c3c9d6368d56e93.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 04/13] t/helper/test-read-graph.c: prepare repo settings","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:07Z","receivedAt":"2020-09-18T02:59:24Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"The read-graph test-tool is used by a number of the commit-graph test to\nassert various properties about a commit-graph. Previously, this program\nnever ran 'prepare_repo_settings()'. There was no need to do so, since\nnone of the commit-graph machinery is affected by the repo settings.\n\nIn the next patch, the commit-graph machinery's behavior will become\ndependent on the repo settings, and so loading them before running the\nrest of the test tool is critical.\n\nAs such, teach the test tool to call 'prepare_repo_settings()'.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n t/helper/test-read-graph.c | 3 ++-\n 1 file changed, 2 insertions(+), 1 deletion(-)\n\ndiff --git a/t/helper/test-read-graph.c b/t/helper/test-read-graph.c\nindex 6d0c962438..5f585a1725 100644\n--- a/t/helper/test-read-graph.c\n+++ b/t/helper/test-read-graph.c\n@@ -12,11 +12,12 @@ int cmd__read_graph(int argc, const char **argv)\n \tsetup_git_directory();\n \todb = the_repository->objects->odb;\n \n+\tprepare_repo_settings(the_repository);\n+\n \tgraph = read_commit_graph_one(the_repository, odb);\n \tif (!graph)\n \t\treturn 1;\n \n-\n \tprintf(\"header: %08x %d %d %d %d\\n\",\n \t\tntohl(*(uint32_t*)graph->data),\n \t\t*(unsigned char*)(graph->data + 4),\n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405828","messageId":"3745baf8ef8810fe8e6031d06d2f6b8d967ef13c.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 08/13] bloom: use provided 'struct bloom_filter_settings'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:27Z","receivedAt":"2020-09-18T02:59:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When 'get_or_compute_bloom_filter()' needs to compute a Bloom filter\nfrom scratch, it looks to the default 'struct bloom_filter_settings' in\norder to determine the maximum number of changed paths, number of bits\nper entry, and so on.\n\nAll of these values have so far been constant, and so there was no need\nto pass in a pointer from the caller (eg., the one that is stored in the\n'struct write_commit_graph_context').\n\nStart passing in a 'struct bloom_filter_settings *' instead of using the\ndefault values to respect graph-specific settings (eg., in the case of\nsetting 'GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS').\n\nIn order to have an initialized value for these settings, move its\ninitialization to earlier in the commit-graph write.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n bloom.c | 13 ++++++-------\n bloom.h | 3 ++-\n commit-graph.c | 21 ++++++++++-----------\n t/helper/test-bloom.c | 1 +\n 4 files changed, 19 insertions(+), 19 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 393c61b4bc..2d6aef9098 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -180,13 +180,12 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t enum bloom_filter_computed *computed)\n {\n \tstruct bloom_filter *filter;\n-\tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \tint i;\n \tstruct diff_options diffopt;\n-\tint max_changes = 512;\n \n \tif (computed)\n \t\t*computed = BLOOM_NOT_COMPUTED;\n@@ -211,7 +210,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \trepo_diff_setup(r, &diffopt);\n \tdiffopt.flags.recursive = 1;\n \tdiffopt.detect_rename = 0;\n-\tdiffopt.max_changes = max_changes;\n+\tdiffopt.max_changes = settings->max_changed_paths;\n \tdiff_setup_done(&diffopt);\n \n \t/* ensure commit is parsed so we have parent information */\n@@ -223,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= max_changes) {\n+\tif (diffopt.num_changes <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -260,13 +259,13 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n-\t\tfilter->len = (hashmap_get_size(&pathmap) * settings.bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n \t\t\tstruct bloom_key key;\n-\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, &settings);\n-\t\t\tadd_key_to_filter(&key, filter, &settings);\n+\t\t\tfill_bloom_key(e->path, strlen(e->path), &key, settings);\n+\t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\ndiff --git a/bloom.h b/bloom.h\nindex e2e035ad14..c6d77e8393 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -98,10 +98,11 @@ enum bloom_filter_computed {\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t const struct bloom_filter_settings *settings,\n \t\t\t\t\t\t enum bloom_filter_computed *computed);\n \n #define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n-\t(r), (c), 0, NULL)\n+\t(r), (c), 0, NULL, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ecdab89e93..50519eb968 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1435,6 +1435,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\tctx->r,\n \t\t\tc,\n \t\t\t1,\n+\t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n \t\t\tctx->count_bloom_filter_computed++;\n@@ -1692,17 +1693,6 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \tint num_chunks = 3;\n \tuint64_t chunk_offset;\n \tstruct object_id file_hash;\n-\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n-\n-\tif (!ctx->bloom_settings) {\n-\t\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n-\t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n-\t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n-\t\t\t\t\t\t\t bloom_settings.num_hashes);\n-\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n-\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n-\t\tctx->bloom_settings = &bloom_settings;\n-\t}\n \n \tif (ctx->split) {\n \t\tstruct strbuf tmp_file = STRBUF_INIT;\n@@ -2148,6 +2138,7 @@ int write_commit_graph(struct object_directory *odb,\n \tuint32_t i, count_distinct = 0;\n \tint res = 0;\n \tint replace = 0;\n+\tstruct bloom_filter_settings bloom_settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn 0;\n@@ -2161,6 +2152,14 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->split_opts = split_opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n+\tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n+\t\t\t\t\t\t bloom_settings.bits_per_entry);\n+\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n+\t\t\t\t\t\t bloom_settings.num_hashes);\n+\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n+\tctx->bloom_settings = &bloom_settings;\n+\n \tif (flags & COMMIT_GRAPH_WRITE_BLOOM_FILTERS)\n \t\tctx->changed_paths = 1;\n \tif (!(flags & COMMIT_GRAPH_NO_WRITE_BLOOM_FILTERS)) {\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 9f7bb729fc..46e97b04eb 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -40,6 +40,7 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n \tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t &settings,\n \t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405829","messageId":"407731b9425ef942c678adb10235224698d19ee4.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 07/13] bloom: split 'get_bloom_filter()' in two","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:22Z","receivedAt":"2020-09-18T02:59:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"'get_bloom_filter' takes a flag to control whether it will compute a\nBloom filter if the requested one is missing. In the next patch, we'll\nadd yet another parameter to this method, which would force all but one\ncaller to specify an extra 'NULL' parameter at the end.\n\nInstead of doing this, split 'get_bloom_filter' into two functions:\n'get_bloom_filter' and 'get_or_compute_bloom_filter'. The former only\nlooks up a Bloom filter (and does not compute one if it's missing,\nthus dropping the 'compute_if_not_present' flag). The latter does\ncompute missing Bloom filters, with an additional parameter to store\nwhether or not it needed to do so.\n\nThis simplifies many call-sites, since the majority of existing callers\nto 'get_bloom_filter' do not want missing Bloom filters to be computed\n(so they can drop the parameter entirely and use the simpler version of\nthe function).\n\nWhile we're at it, instrument the new 'get_or_compute_bloom_filter()'\nwith counters in the 'write_commit_graph_context' struct which store\nthe number of filters that we did and didn't compute, as well as filters\nthat were truncated.\n\nIt would be nice to drop the 'compute_if_not_present' flag entirely,\nsince all remaining callers of 'get_or_compute_bloom_filter' pass it as\n'1', but this will change in a future patch and hence cannot be removed.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n blame.c | 2 +-\n bloom.c | 16 +++++++++++++---\n bloom.h | 16 +++++++++++++---\n commit-graph.c | 34 +++++++++++++++++++++++++++++++---\n line-log.c | 2 +-\n revision.c | 2 +-\n t/helper/test-bloom.c | 3 ++-\n 7 files changed, 62 insertions(+), 13 deletions(-)\n\ndiff --git a/blame.c b/blame.c\nindex 903e23af23..e5ba35dbd1 100644\n--- a/blame.c\n+++ b/blame.c\n@@ -1276,7 +1276,7 @@ static int maybe_changed_path(struct repository *r,\n \tif (commit_graph_generation(origin->commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn 1;\n \n-\tfilter = get_bloom_filter(r, origin->commit, 0);\n+\tfilter = get_bloom_filter(r, origin->commit);\n \n \tif (!filter)\n \t\treturn 1;\ndiff --git a/bloom.c b/bloom.c\nindex cd9380ac62..393c61b4bc 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,9 +177,10 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present)\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t enum bloom_filter_computed *computed)\n {\n \tstruct bloom_filter *filter;\n \tstruct bloom_filter_settings settings = DEFAULT_BLOOM_FILTER_SETTINGS;\n@@ -187,6 +188,9 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \tstruct diff_options diffopt;\n \tint max_changes = 512;\n \n+\tif (computed)\n+\t\t*computed = BLOOM_NOT_COMPUTED;\n+\n \tif (!bloom_filters.slab_size)\n \t\treturn NULL;\n \n@@ -271,8 +275,14 @@ struct bloom_filter *get_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\tfilter->data = NULL;\n \t\tfilter->len = 0;\n+\n+\t\tif (computed)\n+\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n \t}\n \n+\tif (computed)\n+\t\t*computed |= BLOOM_COMPUTED;\n+\n \tfree(diff_queued_diff.queue);\n \tDIFF_QUEUE_CLEAR(&diff_queued_diff);\n \ndiff --git a/bloom.h b/bloom.h\nindex 0b9b59a6fe..e2e035ad14 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -89,9 +89,19 @@ void add_key_to_filter(const struct bloom_key *key,\n \n void init_bloom_filters(void);\n \n-struct bloom_filter *get_bloom_filter(struct repository *r,\n-\t\t\t\t struct commit *c,\n-\t\t\t\t int compute_if_not_present);\n+enum bloom_filter_computed {\n+\tBLOOM_NOT_COMPUTED = (1 << 0),\n+\tBLOOM_COMPUTED = (1 << 1),\n+\tBLOOM_TRUNC_LARGE = (1 << 2),\n+};\n+\n+struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n+\t\t\t\t\t\t struct commit *c,\n+\t\t\t\t\t\t int compute_if_not_present,\n+\t\t\t\t\t\t enum bloom_filter_computed *computed);\n+\n+#define get_bloom_filter(r, c) get_or_compute_bloom_filter( \\\n+\t(r), (c), 0, NULL)\n \n int bloom_filter_contains(const struct bloom_filter *filter,\n \t\t\t const struct bloom_key *key,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex ba6d4a4c6c..ecdab89e93 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -972,6 +972,10 @@ struct write_commit_graph_context {\n \tconst struct split_commit_graph_opts *split_opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n+\n+\tint count_bloom_filter_computed;\n+\tint count_bloom_filter_not_computed;\n+\tint count_bloom_filter_trunc_large;\n };\n \n static int write_graph_chunk_fanout(struct hashfile *f,\n@@ -1183,7 +1187,7 @@ static int write_graph_chunk_bloom_indexes(struct hashfile *f,\n \tuint32_t cur_pos = 0;\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \t\tcur_pos += len;\n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1223,7 +1227,7 @@ static int write_graph_chunk_bloom_data(struct hashfile *f,\n \thashwrite_be32(f, ctx->bloom_settings->bits_per_entry);\n \n \twhile (list < last) {\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list, 0);\n+\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, *list);\n \t\tsize_t len = filter ? filter->len : 0;\n \n \t\tdisplay_progress(ctx->progress, ++ctx->progress_cnt);\n@@ -1393,6 +1397,16 @@ static void compute_generation_numbers(struct write_commit_graph_context *ctx)\n \tstop_progress(&ctx->progress);\n }\n \n+static void trace2_bloom_filter_write_statistics(struct write_commit_graph_context *ctx)\n+{\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-computed\",\n+\t\t\t ctx->count_bloom_filter_computed);\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-not-computed\",\n+\t\t\t ctx->count_bloom_filter_not_computed);\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-large\",\n+\t\t\t ctx->count_bloom_filter_trunc_large);\n+}\n+\n static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n {\n \tint i;\n@@ -1415,12 +1429,26 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n \n \tfor (i = 0; i < ctx->commits.nr; i++) {\n+\t\tenum bloom_filter_computed computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n-\t\tstruct bloom_filter *filter = get_bloom_filter(ctx->r, c, 1);\n+\t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n+\t\t\tctx->r,\n+\t\t\tc,\n+\t\t\t1,\n+\t\t\t&computed);\n+\t\tif (computed & BLOOM_COMPUTED) {\n+\t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (computed & BLOOM_TRUNC_LARGE)\n+\t\t\t\tctx->count_bloom_filter_trunc_large++;\n+\t\t} else if (computed & BLOOM_NOT_COMPUTED)\n+\t\t\tctx->count_bloom_filter_not_computed++;\n \t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \n+\tif (trace2_is_enabled())\n+\t\ttrace2_bloom_filter_write_statistics(ctx);\n+\n \tfree(sorted_commits);\n \tstop_progress(&progress);\n }\ndiff --git a/line-log.c b/line-log.c\nindex bf73ea95ac..68eeb425f8 100644\n--- a/line-log.c\n+++ b/line-log.c\n@@ -1159,7 +1159,7 @@ static int bloom_filter_check(struct rev_info *rev,\n \t\treturn 1;\n \n \tif (!rev->bloom_filter_settings ||\n-\t !(filter = get_bloom_filter(rev->repo, commit, 0)))\n+\t !(filter = get_bloom_filter(rev->repo, commit)))\n \t\treturn 1;\n \n \tif (!range)\ndiff --git a/revision.c b/revision.c\nindex 6aeb764821..c53ab99e59 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -752,7 +752,7 @@ static int check_maybe_different_in_bloom_filter(struct rev_info *revs,\n \tif (commit_graph_generation(commit) == GENERATION_NUMBER_INFINITY)\n \t\treturn -1;\n \n-\tfilter = get_bloom_filter(revs->repo, commit, 0);\n+\tfilter = get_bloom_filter(revs->repo, commit);\n \n \tif (!filter) {\n \t\tcount_bloom_filter_not_present++;\ndiff --git a/t/helper/test-bloom.c b/t/helper/test-bloom.c\nindex 5e77d56f59..9f7bb729fc 100644\n--- a/t/helper/test-bloom.c\n+++ b/t/helper/test-bloom.c\n@@ -39,7 +39,8 @@ static void get_bloom_filter_for_commit(const struct object_id *commit_oid)\n \tstruct bloom_filter *filter;\n \tsetup_git_directory();\n \tc = lookup_commit(the_repository, commit_oid);\n-\tfilter = get_bloom_filter(the_repository, c, 1);\n+\tfilter = get_or_compute_bloom_filter(the_repository, c, 1,\n+\t\t\t\t\t NULL);\n \tprint_bloom_filter(filter);\n }\n \n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405830","messageId":"d4e9714c167117882488bf179343ec1652e1cd27.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 06/13] commit-graph.c: store maximum changed paths","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:16Z","receivedAt":"2020-09-18T02:59:51Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"For now, we assume that there is a fixed constant describing the\nmaximum number of changed paths we are willing to store in a Bloom\nfilter.\n\nPrepare for that to (at least partially) not be the case by making it a\nmember of the 'struct bloom_filter_settings'. This will be helpful in\nthe subsequent patches by reducing the size of test cases that exercise\nstoring too many changed paths, as well as preparing for an eventual\nfuture in which this value might change.\n\nThis patch alone does not cause newly generated Bloom filters to use\na custom upper-bound on the maximum number of changed paths a single\nBloom filter can hold, that will occur in a later patch.\n\nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n bloom.h | 11 ++++++++++-\n commit-graph.c | 4 ++++\n t/t4216-log-bloom.sh | 4 ++--\n 3 files changed, 16 insertions(+), 3 deletions(-)\n\ndiff --git a/bloom.h b/bloom.h\nindex d8fbb0fbf1..0b9b59a6fe 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -28,9 +28,18 @@ struct bloom_filter_settings {\n \t * that contain n*b bits.\n \t */\n \tuint32_t bits_per_entry;\n+\n+\t/*\n+\t * The maximum number of changed paths per commit\n+\t * before declaring a Bloom filter to be too-large.\n+\t *\n+\t * Not written to the commit-graph file.\n+\t */\n+\tuint32_t max_changed_paths;\n };\n \n-#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10 }\n+#define DEFAULT_BLOOM_MAX_CHANGES 512\n+#define DEFAULT_BLOOM_FILTER_SETTINGS { 1, 7, 10, DEFAULT_BLOOM_MAX_CHANGES }\n #define BITS_PER_WORD 8\n #define BLOOMDATA_CHUNK_HEADER_SIZE 3 * sizeof(uint32_t)\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex ea54d108b9..ba6d4a4c6c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -424,6 +424,7 @@ struct commit_graph *parse_commit_graph(struct repository *r,\n \t\t\t\tgraph->bloom_filter_settings->hash_version = hash_version;\n \t\t\t\tgraph->bloom_filter_settings->num_hashes = get_be32(data + chunk_offset + 4);\n \t\t\t\tgraph->bloom_filter_settings->bits_per_entry = get_be32(data + chunk_offset + 8);\n+\t\t\t\tgraph->bloom_filter_settings->max_changed_paths = DEFAULT_BLOOM_MAX_CHANGES;\n \t\t\t}\n \t\t\tbreak;\n \t\t}\n@@ -1201,6 +1202,7 @@ static void trace2_bloom_filter_settings(struct write_commit_graph_context *ctx)\n \tjw_object_intmax(&jw, \"hash_version\", ctx->bloom_settings->hash_version);\n \tjw_object_intmax(&jw, \"num_hashes\", ctx->bloom_settings->num_hashes);\n \tjw_object_intmax(&jw, \"bits_per_entry\", ctx->bloom_settings->bits_per_entry);\n+\tjw_object_intmax(&jw, \"max_changed_paths\", ctx->bloom_settings->max_changed_paths);\n \tjw_end(&jw);\n \n \ttrace2_data_json(\"bloom\", ctx->r, \"settings\", &jw);\n@@ -1669,6 +1671,8 @@ static int write_commit_graph_file(struct write_commit_graph_context *ctx)\n \t\t\t\t\t\t\t bloom_settings.bits_per_entry);\n \t\tbloom_settings.num_hashes = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_NUM_HASHES\",\n \t\t\t\t\t\t\t bloom_settings.num_hashes);\n+\t\tbloom_settings.max_changed_paths = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS\",\n+\t\t\t\t\t\t\t bloom_settings.max_changed_paths);\n \t\tctx->bloom_settings = &bloom_settings;\n \t}\n \ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex fc7693806c..593571358d 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -174,11 +174,11 @@ test_expect_success 'persist filter settings' '\n \t\tGIT_TEST_BLOOM_SETTINGS_NUM_HASHES=9 \\\n \t\tGIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY=15 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2.txt &&\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2.txt &&\n \tGIT_TRACE2_EVENT=\"$(pwd)/trace2-auto.txt\" \\\n \t\tGIT_TRACE2_EVENT_NESTING=5 \\\n \t\tgit commit-graph write --reachable --changed-paths &&\n-\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15}\" trace2-auto.txt\n+\tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2-auto.txt\n '\n \n test_expect_success 'correctly report changes over limit' '\n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405831","messageId":"b1da2564fac38259d2c8150732ea3bb5232c7aff.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 10/13] bloom: encode out-of-bounds filters as non-empty","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:44Z","receivedAt":"2020-09-18T03:00:21Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"When a changed-path Bloom filter has either zero, or more than a\ncertain number (commonly 512) of entries, the commit-graph machinery\nencodes it as \"missing\". More specifically, it sets the indices adjacent\nin the BIDX chunk as equal to each other to indicate a \"length 0\"\nfilter; that is, that the filter occupies zero bytes on disk.\n\nThis has heretofore been fine, since the commit-graph machinery has no\nneed to care about these filters with too few or too many changed paths.\nBoth cases act like no filter has been generated at all, and so there is\nno need to store them.\n\nIn a subsequent commit, however, the commit-graph machinery will learn\nto only compute Bloom filters for some commits in the current\ncommit-graph layer. This is a change from the current implementation\nwhich computes Bloom filters for all commits that are in the layer being\nwritten. Critically for this patch, only computing some of the Bloom\nfilters means adding a third state for length 0 Bloom filters: zero\nentries, too many entries, or \"hasn't been computed\".\n\nIt will be important for that future patch to distinguish between \"not\nrepresentable\" (i.e., zero or too-many changed paths), and \"hasn't been\ncomputed\". In particular, we don't want to waste time recomputing\nfilters that have already been computed.\n\nTo that end, change how we store Bloom filters in the \"computed but not\nrepresentable\" category:\n\n - Bloom filters with no entries are stored as a single byte with all\n bits low (i.e., all queries to that Bloom filter will return\n \"definitely not\")\n\n - Bloom filters with too many entries are stored as a single byte with\n all bits set high (i.e., all queries to that Bloom filter will\n return \"maybe\").\n\nThese rules are sufficient to not incur a behavior change by changing\nthe on-disk representation of these two classes. Likewise, no\nspecification changes are necessary for the commit-graph format, either:\n\n - Filters that were previously empty will be recomputed and stored\n according to the new rules, and\n\n - old clients reading filters generated by new clients will interpret\n the filters correctly and be none the wiser to how they were\n generated.\n\nClients will invoke the Bloom machinery in more cases than before, but\nthis can be addressed by returning a NULL filter when all bits are set\nhigh. This can be addressed in a future patch.\n\nNote that this does increase the size of on-disk commit-graphs, but far\nless than other proposals. In particular, this is generally more\nefficient than storing a bitmap for which commits haven't computed their\nBloom filters. Storing a bitmap incurs a penalty of one bit per commit,\nwhereas storing explicit filters as above incurs a penalty of one byte\nper too-large or empty commit.\n\nIn practice, these boundary commits likely occupy a small proportion of\nthe overall number of commits, and so the size penalty is likely smaller\nthan storing a bitmap for all commits.\n\nSee, for example, these relative proportions of such boundary commits\n(collected by SZEDER Gábor):\n\n | Percentage of | commit-graph | |\n | commits modifying | file size | |\n ├────────┬──────────────┼───────────────────┤ pct. |\n | 0 path | >= 512 paths | before | after | change |\n ┌────────────────┼────────┼──────────────┼─────────┼─────────┼───────────┤\n | android-base | 13.20% | 0.13% | 37.468M | 37.534M | +0.1741 % |\n | cmssw | 0.15% | 0.23% | 17.118M | 17.119M | +0.0091 % |\n | cpython | 3.07% | 0.01% | 7.967M | 7.971M | +0.0423 % |\n | elasticsearch | 0.70% | 1.00% | 8.833M | 8.835M | +0.0128 % |\n | gcc | 0.00% | 0.08% | 16.073M | 16.074M | +0.0030 % |\n | gecko-dev | 0.14% | 0.64% | 59.868M | 59.874M | +0.0105 % |\n | git | 0.11% | 0.02% | 3.895M | 3.895M | +0.0020 % |\n | glibc | 0.02% | 0.10% | 3.555M | 3.555M | +0.0021 % |\n | go | 0.00% | 0.07% | 3.186M | 3.186M | +0.0018 % |\n | homebrew-cask | 0.40% | 0.02% | 7.035M | 7.035M | +0.0065 % |\n | homebrew-core | 0.01% | 0.01% | 11.611M | 11.611M | +0.0002 % |\n | jdk | 0.26% | 5.64% | 5.537M | 5.540M | +0.0590 % |\n | linux | 0.01% | 0.51% | 63.735M | 63.740M | +0.0073 % |\n | llvm-project | 0.12% | 0.03% | 25.515M | 25.516M | +0.0050 % |\n | rails | 0.10% | 0.10% | 6.252M | 6.252M | +0.0027 % |\n | rust | 0.07% | 0.17% | 9.364M | 9.364M | +0.0033 % |\n | tensorflow | 0.09% | 1.02% | 7.009M | 7.010M | +0.0158 % |\n | webkit | 0.05% | 0.31% | 17.405M | 17.406M | +0.0047 % |\n\n(where the above increase is determined by computing a non-split\ncommit-graph before and after this patch).\n\nGiven that these projects are all \"large\" by commit count, the storage\ncost by writing these filters explicitly is negligible. In the most\nextreme example, android-base (which has 494,848 commits at the time of\nwriting) would have its commit-graph increase by a modest 68.4 KB.\n\nFinally, a test to exercise filters which contain too many changed path\nentries will be introduced in a subsequent patch.\n\nSuggested-by: SZEDER Gábor \nSuggested-by: Jakub Narębski \nHelped-by: Derrick Stolee \nHelped-by: SZEDER Gábor \nHelped-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n .../technical/commit-graph-format.txt | 2 +-\n bloom.c | 16 ++++++++--\n bloom.h | 1 +\n commit-graph.c | 5 +++\n t/t0095-bloom.sh | 8 ++---\n t/t4216-log-bloom.sh | 31 +++++++++++++++++--\n 6 files changed, 54 insertions(+), 9 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 6ddbceba15..6585f1948a 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -125,7 +125,7 @@ CHUNK DATA:\n * The rest of the chunk is the concatenation of all the computed Bloom\n filters for the commits in lexicographic order.\n * Note: Commits with no changes or more than 512 changes have Bloom filters\n- of length zero.\n+ of length one, with either all bits set to zero or one respectively.\n * The BDAT chunk is present if and only if BIDX is present.\n \n Base Graphs List (ID: {'B', 'A', 'S', 'E'}) [Optional]\ndiff --git a/bloom.c b/bloom.c\nindex db9fb82437..d234551ce0 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -177,6 +177,13 @@ static int pathmap_cmp(const void *hashmap_cmp_fn_data,\n \treturn strcmp(e1->path, e2->path);\n }\n \n+static void init_truncated_large_filter(struct bloom_filter *filter)\n+{\n+\tfilter->data = xmalloc(1);\n+\tfilter->data[0] = 0xFF;\n+\tfilter->len = 1;\n+}\n+\n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\t\t\t\t struct commit *c,\n \t\t\t\t\t\t int compute_if_not_present,\n@@ -260,12 +267,18 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t}\n \n \t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths) {\n+\t\t\tinit_truncated_large_filter(filter);\n \t\t\tif (computed)\n \t\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n \t\t\tgoto cleanup;\n \t\t}\n \n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n+\t\tif (!filter->len) {\n+\t\t\tif (computed)\n+\t\t\t\t*computed |= BLOOM_TRUNC_EMPTY;\n+\t\t\tfilter->len = 1;\n+\t\t}\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n \t\thashmap_for_each_entry(&pathmap, &iter, e, entry) {\n@@ -279,8 +292,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n-\t\tfilter->data = NULL;\n-\t\tfilter->len = 0;\n+\t\tinit_truncated_large_filter(filter);\n \n \t\tif (computed)\n \t\t\t*computed |= BLOOM_TRUNC_LARGE;\ndiff --git a/bloom.h b/bloom.h\nindex c6d77e8393..adde6dfe21 100644\n--- a/bloom.h\n+++ b/bloom.h\n@@ -93,6 +93,7 @@ enum bloom_filter_computed {\n \tBLOOM_NOT_COMPUTED = (1 << 0),\n \tBLOOM_COMPUTED = (1 << 1),\n \tBLOOM_TRUNC_LARGE = (1 << 2),\n+\tBLOOM_TRUNC_EMPTY = (1 << 3),\n };\n \n struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 50519eb968..a71663c28c 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -975,6 +975,7 @@ struct write_commit_graph_context {\n \n \tint count_bloom_filter_computed;\n \tint count_bloom_filter_not_computed;\n+\tint count_bloom_filter_trunc_empty;\n \tint count_bloom_filter_trunc_large;\n };\n \n@@ -1403,6 +1404,8 @@ static void trace2_bloom_filter_write_statistics(struct write_commit_graph_conte\n \t\t\t ctx->count_bloom_filter_computed);\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-not-computed\",\n \t\t\t ctx->count_bloom_filter_not_computed);\n+\ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-empty\",\n+\t\t\t ctx->count_bloom_filter_trunc_empty);\n \ttrace2_data_intmax(\"commit-graph\", ctx->r, \"filter-trunc-large\",\n \t\t\t ctx->count_bloom_filter_trunc_large);\n }\n@@ -1439,6 +1442,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n \t\t\tctx->count_bloom_filter_computed++;\n+\t\t\tif (computed & BLOOM_TRUNC_EMPTY)\n+\t\t\t\tctx->count_bloom_filter_trunc_empty++;\n \t\t\tif (computed & BLOOM_TRUNC_LARGE)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\ndiff --git a/t/t0095-bloom.sh b/t/t0095-bloom.sh\nindex 232ba2c485..7e4ab1795f 100755\n--- a/t/t0095-bloom.sh\n+++ b/t/t0095-bloom.sh\n@@ -71,8 +71,8 @@ test_expect_success 'get bloom filters for commit with no changes' '\n \tgit init &&\n \tgit commit --allow-empty -m \"c0\" &&\n \tcat >expect <<-\\EOF &&\n-\tFilter_Length:0\n-\tFilter_Data:\n+\tFilter_Length:1\n+\tFilter_Data:00|\n \tEOF\n \ttest-tool bloom get_filter_for_commit \"$(git rev-parse HEAD)\" >actual &&\n \ttest_cmp expect actual\n@@ -107,8 +107,8 @@ test_expect_success EXPENSIVE 'get bloom filter for commit with 513 changes' '\n \tgit add bigDir &&\n \tgit commit -m \"commit with 513 changes\" &&\n \tcat >expect <<-\\EOF &&\n-\tFilter_Length:0\n-\tFilter_Data:\n+\tFilter_Length:1\n+\tFilter_Data:ff|\n \tEOF\n \ttest-tool bloom get_filter_for_commit \"$(git rev-parse HEAD)\" >actual &&\n \ttest_cmp expect actual\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 123b46a0db..15bbb626a1 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -30,6 +30,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \trm file_to_be_deleted &&\n \tgit add . &&\n \tgit commit -m \"file removed\" &&\n+\tgit commit --allow-empty -m \"empty\" &&\n \tgit commit-graph write --reachable --changed-paths &&\n \n \ttest_oid_cache <<-EOF\n@@ -37,6 +38,7 @@ test_expect_success 'setup test - repo, commits, commit graph, log outputs' '\n \toid_version sha256:2\n \tEOF\n '\n+\n graph_read_expect () {\n \tNUM_CHUNKS=5\n \tcat >expect <<- EOF\n@@ -49,7 +51,7 @@ graph_read_expect () {\n }\n \n test_expect_success 'commit-graph write wrote out the bloom chunks' '\n-\tgraph_read_expect 15\n+\tgraph_read_expect 16\n '\n \n # Turn off any inherited trace2 settings for this test.\n@@ -156,7 +158,7 @@ test_expect_success 'setup - add commit-graph to the chain with Bloom filters' '\n \n test_bloom_filters_used_when_some_filters_are_missing () {\n \tlog_args=$1\n-\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":8\"\n+\tbloom_trace_prefix=\"statistics:{\\\"filter_not_present\\\":3,\\\"maybe\\\":6,\\\"definitely_not\\\":9\"\n \tsetup \"$log_args\" &&\n \tgrep -q \"$bloom_trace_prefix\" \"$TRASH_DIRECTORY/trace.perf\" &&\n \ttest_cmp log_wo_bloom log_w_bloom\n@@ -185,10 +187,18 @@ test_max_changed_paths () {\n \tgrep \"\\\"max_changed_paths\\\":$1\" $2\n }\n \n+test_filter_not_computed () {\n+\tgrep \"\\\"key\\\":\\\"filter-not-computed\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n test_filter_computed () {\n \tgrep \"\\\"key\\\":\\\"filter-computed\\\",\\\"value\\\":\\\"$1\\\"\" $2\n }\n \n+test_filter_trunc_empty () {\n+\tgrep \"\\\"key\\\":\\\"filter-trunc-empty\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n test_filter_trunc_large () {\n \tgrep \"\\\"key\\\":\\\"filter-trunc-large\\\",\\\"value\\\":\\\"$1\\\"\" $2\n }\n@@ -283,4 +293,21 @@ test_expect_success 'correctly report changes over limit' '\n \t)\n '\n \n+test_expect_success 'correctly report commits with no changed paths' '\n+\tgit init empty &&\n+\ttest_when_finished \"rm -fr empty\" &&\n+\t(\n+\t\tcd empty &&\n+\n+\t\tgit commit --allow-empty -m \"initial commit\" &&\n+\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\ttest_filter_computed 1 trace.event &&\n+\t\ttest_filter_not_computed 0 trace.event &&\n+\t\ttest_filter_trunc_empty 1 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n test_done\n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405832","messageId":"2a125953d0be5da3ebf722219f0ebdbfbb8921df.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 11/13] commit-graph: rename 'split_commit_graph_opts'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:49Z","receivedAt":"2020-09-18T03:00:22Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"In the subsequent commit, additional options will be added to the\ncommit-graph API which have nothing to do with splitting.\n\nRename the 'split_commit_graph_opts' structure to the more-generic\n'commit_graph_opts' to encompass both. Likewise, rename the 'flags'\nmember to instead be 'split_flags' to clarify that it only has to do\nwith the behavior implied by '--split'.\n\nSuggested-by: Derrick Stolee \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n builtin/commit-graph.c | 20 ++++++++++----------\n commit-graph.c | 40 ++++++++++++++++++++--------------------\n commit-graph.h | 8 ++++----\n 3 files changed, 34 insertions(+), 34 deletions(-)\n\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex ba5584463f..f3243bd982 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -119,7 +119,7 @@ static int graph_verify(int argc, const char **argv)\n }\n \n extern int read_replace_refs;\n-static struct split_commit_graph_opts split_opts;\n+static struct commit_graph_opts write_opts;\n \n static int write_option_parse_split(const struct option *opt, const char *arg,\n \t\t\t\t int unset)\n@@ -187,24 +187,24 @@ static int graph_write(int argc, const char **argv)\n \t\tOPT_BOOL(0, \"changed-paths\", &opts.enable_changed_paths,\n \t\t\tN_(\"enable computation for changed paths\")),\n \t\tOPT_BOOL(0, \"progress\", &opts.progress, N_(\"force progress reporting\")),\n-\t\tOPT_CALLBACK_F(0, \"split\", &split_opts.flags, NULL,\n+\t\tOPT_CALLBACK_F(0, \"split\", &write_opts.split_flags, NULL,\n \t\t\tN_(\"allow writing an incremental commit-graph file\"),\n \t\t\tPARSE_OPT_OPTARG | PARSE_OPT_NONEG,\n \t\t\twrite_option_parse_split),\n-\t\tOPT_INTEGER(0, \"max-commits\", &split_opts.max_commits,\n+\t\tOPT_INTEGER(0, \"max-commits\", &write_opts.max_commits,\n \t\t\tN_(\"maximum number of commits in a non-base split commit-graph\")),\n-\t\tOPT_INTEGER(0, \"size-multiple\", &split_opts.size_multiple,\n+\t\tOPT_INTEGER(0, \"size-multiple\", &write_opts.size_multiple,\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n-\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &split_opts.expire_time,\n+\t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n \t\tOPT_END(),\n \t};\n \n \topts.progress = isatty(2);\n \topts.enable_changed_paths = -1;\n-\tsplit_opts.size_multiple = 2;\n-\tsplit_opts.max_commits = 0;\n-\tsplit_opts.expire_time = 0;\n+\twrite_opts.size_multiple = 2;\n+\twrite_opts.max_commits = 0;\n+\twrite_opts.expire_time = 0;\n \n \ttrace2_cmd_mode(\"write\");\n \n@@ -232,7 +232,7 @@ static int graph_write(int argc, const char **argv)\n \todb = find_odb(the_repository, opts.obj_dir);\n \n \tif (opts.reachable) {\n-\t\tif (write_commit_graph_reachable(odb, flags, &split_opts))\n+\t\tif (write_commit_graph_reachable(odb, flags, &write_opts))\n \t\t\treturn 1;\n \t\treturn 0;\n \t}\n@@ -261,7 +261,7 @@ static int graph_write(int argc, const char **argv)\n \t\t\t opts.stdin_packs ? &pack_indexes : NULL,\n \t\t\t opts.stdin_commits ? &commits : NULL,\n \t\t\t flags,\n-\t\t\t &split_opts))\n+\t\t\t &write_opts))\n \t\tresult = 1;\n \n cleanup:\ndiff --git a/commit-graph.c b/commit-graph.c\nindex a71663c28c..d2a2f78776 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -969,7 +969,7 @@ struct write_commit_graph_context {\n \t\t changed_paths:1,\n \t\t order_by_pack:1;\n \n-\tconst struct split_commit_graph_opts *split_opts;\n+\tconst struct commit_graph_opts *opts;\n \tsize_t total_bloom_filter_data_size;\n \tconst struct bloom_filter_settings *bloom_settings;\n \n@@ -1293,8 +1293,8 @@ static void close_reachable(struct write_commit_graph_context *ctx)\n {\n \tint i;\n \tstruct commit *commit;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tif (ctx->report_progress)\n \t\tctx->progress = start_delayed_progress(\n@@ -1483,7 +1483,7 @@ static int add_ref_to_set(const char *refname,\n \n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t\t\t const struct commit_graph_opts *opts)\n {\n \tstruct oidset commits = OIDSET_INIT;\n \tstruct refs_cb_data data;\n@@ -1500,7 +1500,7 @@ int write_commit_graph_reachable(struct object_directory *odb,\n \tstop_progress(&data.progress);\n \n \tresult = write_commit_graph(odb, NULL, &commits,\n-\t\t\t\t flags, split_opts);\n+\t\t\t\t flags, opts);\n \n \toidset_clear(&commits);\n \treturn result;\n@@ -1615,8 +1615,8 @@ static uint32_t count_distinct_commits(struct write_commit_graph_context *ctx)\n static void copy_oids_to_commits(struct write_commit_graph_context *ctx)\n {\n \tuint32_t i;\n-\tenum commit_graph_split_flags flags = ctx->split_opts ?\n-\t\tctx->split_opts->flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n+\tenum commit_graph_split_flags flags = ctx->opts ?\n+\t\tctx->opts->split_flags : COMMIT_GRAPH_SPLIT_UNSPECIFIED;\n \n \tctx->num_extra_edges = 0;\n \tif (ctx->report_progress)\n@@ -1901,13 +1901,13 @@ static void split_graph_merge_strategy(struct write_commit_graph_context *ctx)\n \tint max_commits = 0;\n \tint size_mult = 2;\n \n-\tif (ctx->split_opts) {\n-\t\tmax_commits = ctx->split_opts->max_commits;\n+\tif (ctx->opts) {\n+\t\tmax_commits = ctx->opts->max_commits;\n \n-\t\tif (ctx->split_opts->size_multiple)\n-\t\t\tsize_mult = ctx->split_opts->size_multiple;\n+\t\tif (ctx->opts->size_multiple)\n+\t\t\tsize_mult = ctx->opts->size_multiple;\n \n-\t\tflags = ctx->split_opts->flags;\n+\t\tflags = ctx->opts->split_flags;\n \t}\n \n \tg = ctx->r->objects->commit_graph;\n@@ -2085,8 +2085,8 @@ static void expire_commit_graphs(struct write_commit_graph_context *ctx)\n \tsize_t dirnamelen;\n \ttimestamp_t expire_time = time(NULL);\n \n-\tif (ctx->split_opts && ctx->split_opts->expire_time)\n-\t\texpire_time = ctx->split_opts->expire_time;\n+\tif (ctx->opts && ctx->opts->expire_time)\n+\t\texpire_time = ctx->opts->expire_time;\n \tif (!ctx->split) {\n \t\tchar *chain_file_name = get_chain_filename(ctx->odb);\n \t\tunlink(chain_file_name);\n@@ -2137,7 +2137,7 @@ int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts)\n+\t\t const struct commit_graph_opts *opts)\n {\n \tstruct write_commit_graph_context *ctx;\n \tuint32_t i, count_distinct = 0;\n@@ -2154,7 +2154,7 @@ int write_commit_graph(struct object_directory *odb,\n \tctx->append = flags & COMMIT_GRAPH_WRITE_APPEND ? 1 : 0;\n \tctx->report_progress = flags & COMMIT_GRAPH_WRITE_PROGRESS ? 1 : 0;\n \tctx->split = flags & COMMIT_GRAPH_WRITE_SPLIT ? 1 : 0;\n-\tctx->split_opts = split_opts;\n+\tctx->opts = opts;\n \tctx->total_bloom_filter_data_size = 0;\n \n \tbloom_settings.bits_per_entry = git_env_ulong(\"GIT_TEST_BLOOM_SETTINGS_BITS_PER_ENTRY\",\n@@ -2202,15 +2202,15 @@ int write_commit_graph(struct object_directory *odb,\n \t\t\t}\n \t\t}\n \n-\t\tif (ctx->split_opts)\n-\t\t\treplace = ctx->split_opts->flags & COMMIT_GRAPH_SPLIT_REPLACE;\n+\t\tif (ctx->opts)\n+\t\t\treplace = ctx->opts->split_flags & COMMIT_GRAPH_SPLIT_REPLACE;\n \t}\n \n \tctx->approx_nr_objects = approximate_object_count();\n \tctx->oids.alloc = ctx->approx_nr_objects / 32;\n \n-\tif (ctx->split && split_opts && ctx->oids.alloc > split_opts->max_commits)\n-\t\tctx->oids.alloc = split_opts->max_commits;\n+\tif (ctx->split && opts && ctx->oids.alloc > opts->max_commits)\n+\t\tctx->oids.alloc = opts->max_commits;\n \n \tif (ctx->append) {\n \t\tprepare_commit_graph_one(ctx->r, ctx->odb);\ndiff --git a/commit-graph.h b/commit-graph.h\nindex d9acb22bac..b7914b0a7a 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -105,11 +105,11 @@ enum commit_graph_split_flags {\n \tCOMMIT_GRAPH_SPLIT_REPLACE = 2\n };\n \n-struct split_commit_graph_opts {\n+struct commit_graph_opts {\n \tint size_multiple;\n \tint max_commits;\n \ttimestamp_t expire_time;\n-\tenum commit_graph_split_flags flags;\n+\tenum commit_graph_split_flags split_flags;\n };\n \n /*\n@@ -120,12 +120,12 @@ struct split_commit_graph_opts {\n */\n int write_commit_graph_reachable(struct object_directory *odb,\n \t\t\t\t enum commit_graph_write_flags flags,\n-\t\t\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t\t\t const struct commit_graph_opts *opts);\n int write_commit_graph(struct object_directory *odb,\n \t\t struct string_list *pack_indexes,\n \t\t struct oidset *commits,\n \t\t enum commit_graph_write_flags flags,\n-\t\t const struct split_commit_graph_opts *split_opts);\n+\t\t const struct commit_graph_opts *opts);\n \n #define COMMIT_GRAPH_VERIFY_SHALLOW\t(1 << 0)\n \n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405833","messageId":"375ecf1f3651b250982216d40167e4c23b8e15ff.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 13/13] commit-graph: introduce 'commitGraph.maxNewFilters'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:57Z","receivedAt":"2020-09-18T03:00:23Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a configuration variable to specify a default value for the\nrecently-introduce '--max-new-filters' option of 'git commit-graph\nwrite'.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n Documentation/config/commitgraph.txt | 4 ++++\n Documentation/git-commit-graph.txt | 3 ++-\n builtin/commit-graph.c | 14 ++++++++++++++\n t/t4216-log-bloom.sh | 24 +++++++++++++++++++++++-\n 4 files changed, 43 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex cff0797b54..4582c39fc4 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitGraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 8357846d30..60df4e4bfa 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -70,7 +70,8 @@ data.\n With the `--max-new-filters=` option, generate at most `n` new Bloom\n filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n enforced. Commits whose filters are not calculated are stored as a\n-length zero Bloom filter.\n+length zero Bloom filter. Overrides the `commitGraph.maxNewFilters`\n+configuration.\n +\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 5df9b2ef80..988445abdf 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -181,6 +181,18 @@ static int write_option_max_new_filters(const struct option *opt,\n \treturn 0;\n }\n \n+static int git_commit_graph_write_config(const char *var, const char *value,\n+\t\t\t\t\t void *cb)\n+{\n+\tif (!strcmp(var, \"commitgraph.maxnewfilters\"))\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t/*\n+\t * No need to fall-back to 'git_default_config', since this was already\n+\t * called in 'cmd_commit_graph()'.\n+\t */\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -231,6 +243,8 @@ static int graph_write(int argc, const char **argv)\n \n \ttrace2_cmd_mode(\"write\");\n \n+\tgit_config(git_commit_graph_write_config, &opts);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex af0975c841..d11040ce41 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -330,13 +330,15 @@ test_expect_success 'Bloom generation is limited by --max-new-filters' '\n '\n \n test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t# Check specifying commitGraph.maxNewFilters over \"git config\" works.\n+\ttest_config -C limits commitGraph.maxNewFilters 1 &&\n \t(\n \t\tcd limits &&\n \n \t\trm -f trace.event &&\n \t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n \t\t\tgit commit-graph write --reachable --changed-paths \\\n-\t\t\t\t--split=replace --max-new-filters=1 &&\n+\t\t\t\t--split=replace &&\n \t\ttest_filter_computed 1 trace.event &&\n \t\ttest_filter_not_computed 4 trace.event &&\n \t\ttest_filter_trunc_empty 0 trace.event &&\n@@ -344,6 +346,26 @@ test_expect_success 'Bloom generation backfills previously-skipped filters' '\n \t)\n '\n \n+test_expect_success '--max-new-filters overrides configuration' '\n+\tgit init override &&\n+\ttest_when_finished \"rm -fr override\" &&\n+\ttest_config -C override commitGraph.maxNewFilters 2 &&\n+\t(\n+\t\tcd override &&\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t\t--max-new-filters=1 &&\n+\t\ttest_filter_computed 1 trace.event &&\n+\t\ttest_filter_not_computed 1 trace.event &&\n+\t\ttest_filter_trunc_empty 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n test_expect_success 'Bloom generation backfills empty commits' '\n \tgit init empty &&\n \ttest_when_finished \"rm -fr empty\" &&\n-- \n2.28.0.510.g375ecf1f36\n"},{"id":"405834","messageId":"4549f0f747017a96d612d0f6ddd117684cf90b8f.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 12/13] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:53Z","receivedAt":"2020-09-18T03:00:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Introduce a command-line flag to specify the maximum number of new Bloom\nfilters that a 'git commit-graph write' is willing to compute from\nscratch.\n\nPrior to this patch, a commit-graph write with '--changed-paths' would\ncompute Bloom filters for all selected commits which haven't already\nbeen computed (i.e., by a previous commit-graph write with '--split'\nsuch that a roll-up or replacement is performed).\n\nThis behavior can cause prohibitively-long commit-graph writes for a\nvariety of reasons:\n\n * There may be lots of filters whose diffs take a long time to\n generate (for example, they have close to the maximum number of\n changes, diffing itself takes a long time, etc).\n\n * Old-style commit-graphs (which encode filters with too many entries\n as not having been computed at all) cause us to waste time\n recomputing filters that appear to have not been computed only to\n discover that they are too-large.\n\nThis can make the upper-bound of the time it takes for 'git commit-graph\nwrite --changed-paths' to be rather unpredictable.\n\nTo make this command behave more predictably, introduce\n'--max-new-filters=' to allow computing at most '' Bloom filters\nfrom scratch. This lets \"computing\" already-known filters proceed\nquickly, while bounding the number of slow tasks that Git is willing to\ndo.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-commit-graph.txt | 5 +++\n bloom.c | 7 ++-\n builtin/commit-graph.c | 27 +++++++++++-\n commit-graph.c | 9 +++-\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 70 ++++++++++++++++++++++++++++++\n 6 files changed, 111 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..8357846d30 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,11 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Commits whose filters are not calculated are stored as a\n+length zero Bloom filter.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex d234551ce0..68c73200a5 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -204,12 +204,11 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n-\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n+\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n \n-\tif (filter->data)\n+\tif (filter->data && filter->len)\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f3243bd982..5df9b2ef80 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n \n@@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n \n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n \n@@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n \n \ttrace2_cmd_mode(\"write\");\n \ndiff --git a/commit-graph.c b/commit-graph.c\nindex d2a2f78776..d10e730f29 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1415,6 +1415,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tstruct commit **sorted_commits;\n+\tint max_new_filters;\n \n \tinit_bloom_filters();\n \n@@ -1431,13 +1432,16 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \telse\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n \n+\tmax_new_filters = ctx->opts && ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tenum bloom_filter_computed computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\tctx->r,\n \t\t\tc,\n-\t\t\t1,\n+\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n \t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n@@ -1448,7 +1452,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n \t\t\tctx->count_bloom_filter_not_computed++;\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\tctx->total_bloom_filter_data_size += filter\n+\t\t\t? sizeof(unsigned char) * filter->len : 0;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n \ndiff --git a/commit-graph.h b/commit-graph.h\nindex b7914b0a7a..a22bd86701 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -110,6 +110,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags split_flags;\n+\tint max_new_filters;\n };\n \n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 15bbb626a1..af0975c841 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -310,4 +310,74 @@ test_expect_success 'correctly report commits with no changed paths' '\n \t)\n '\n \n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --split=replace \\\n+\t\t\t\t--changed-paths --max-new-filters=2 &&\n+\n+\t\ttest_filter_computed 2 trace.event &&\n+\t\ttest_filter_not_computed 3 trace.event &&\n+\t\ttest_filter_trunc_empty 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t(\n+\t\tcd limits &&\n+\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t\t--split=replace --max-new-filters=1 &&\n+\t\ttest_filter_computed 1 trace.event &&\n+\t\ttest_filter_not_computed 4 trace.event &&\n+\t\ttest_filter_trunc_empty 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills empty commits' '\n+\tgit init empty &&\n+\ttest_when_finished \"rm -fr empty\" &&\n+\t(\n+\t\tcd empty &&\n+\t\tfor i in $(test_seq 1 6)\n+\t\tdo\n+\t\t\tgit commit --allow-empty -m \"$i\"\n+\t\tdone &&\n+\n+\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n+\t\tfor i in $(test_seq 1 3)\n+\t\tdo\n+\t\t\trm -f trace.event &&\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\t\tgit commit-graph write --reachable \\\n+\t\t\t\t\t--changed-paths --max-new-filters=2 &&\n+\t\t\ttest_filter_computed 2 trace.event &&\n+\t\t\ttest_filter_not_computed 4 trace.event &&\n+\t\t\ttest_filter_trunc_empty 2 trace.event &&\n+\t\t\ttest_filter_trunc_large 0 trace.event\n+\t\tdone &&\n+\n+\t\t# Finally, make sure that once all commits have filters, that\n+\t\t# none are subsequently recomputed.\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable \\\n+\t\t\t\t--changed-paths --max-new-filters=2 &&\n+\t\ttest_filter_computed 0 trace.event &&\n+\t\ttest_filter_not_computed 6 trace.event &&\n+\t\ttest_filter_trunc_empty 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n test_done\n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405835","messageId":"2e4944996ec486b5828f6bc2e62014424c8d6fef.1600397826.git.me@ttaylorr.com","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"[PATCH v3 09/13] bloom/diff: properly short-circuit on max_changes","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T02:59:40Z","receivedAt":"2020-09-18T03:00:25Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"From: Derrick Stolee \n\nCommit e3696980 (diff: halt tree-diff early after max_changes,\n2020-03-30) intended to create a mechanism to short-circuit a diff\ncalculation after a certain number of paths were modified. By\nincrementing a \"num_changes\" counter throughout the recursive\nll_diff_tree_paths(), this was supposed to match the number of changes\nthat would be written into the changed-path Bloom filters.\nUnfortunately, this was not implemented correctly and instead misses\nsimple cases like file modifications. This then does not stop very\nlarge changed-path filters from being written (unless they add or remove\nmany files).\n\nTo start, change the implementation in ll_diff_tree_paths() to instead\nuse the global diff_queue_diff struct's 'nr' member as the count. This\nis a way to simplify the logic instead of making more mistakes in the\ncomplicated diff code.\n\nThis has a drawback: the diff_queue_diff struct only lists the paths\ncorresponding to blob changes, not their leading directories. Thus,\nget_or_compute_bloom_filter() needs an additional check to see if the\nhashmap with the leading directories becomes too large.\n\nOne reason why this was not caught by test cases was that the test in\nt4216-log-bloom.sh that was supposed to check this \"too many changes\"\ncondition only checked this on the initial commit of a repository. The\nold logic counted these values correctly. Update this test in a few\nways:\n\n1. Use GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS to reduce the limit,\n allowing smaller commits to engage with this logic.\n\n2. Create several interesting cases of edits, adds, removes, and mode\n changes (in the second commit). By testing both sides of the\n inequality with the *_MAX_CHANGED_PATHS variable, we can see that\n the count is exactly correct, so none of these changes are missed\n or over-counted.\n\n3. Use the trace2 data value filter_found_large to verify that these\n commits are on the correct side of the limit.\n\nAnother way to verify the behavior is correct is through performance\ntests. By testing on my local copies of the Git repository and the Linux\nkernel repository, I could measure the effect of these short-circuits\nwhen computing a fresh commit-graph file with changed-path Bloom filters\nusing the command\n\n GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=N time \\\n git commit-graph write --reachable --changed-paths\n\nand reporting the wall time and resulting commit-graph size.\n\nFor Git, the results are\n\n| | N=1 | N=10 | N=512 |\n|--------|----------------|----------------|----------------|\n| HEAD~1 | 10.90s 9.18MB | 11.11s 9.34MB | 11.31s 9.35MB |\n| HEAD | 9.21s 8.62MB | 11.11s 9.29MB | 11.29s 9.34MB |\n\nFor Linux, the results are\n\n| | N=1 | N=20 | N=512 |\n|--------|----------------|---------------|---------------|\n| HEAD~1 | 61.28s 64.3MB | 76.9s 72.6MB | 77.6s 72.6MB |\n| HEAD | 49.44s 56.3MB | 68.7s 65.9MB | 69.2s 65.9MB |\n\nNaturally, the improvement becomes much less as the limit grows, as\nfewer commits satisfy the short-circuit.\n\nReported-by: SZEDER Gábor \nSigned-off-by: Derrick Stolee \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n bloom.c | 9 +++-\n diff.h | 2 -\n t/t4216-log-bloom.sh | 100 +++++++++++++++++++++++++++++++++++++++----\n tree-diff.c | 5 +--\n 4 files changed, 100 insertions(+), 16 deletions(-)\n\ndiff --git a/bloom.c b/bloom.c\nindex 2d6aef9098..db9fb82437 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -222,7 +222,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\tdiff_tree_oid(NULL, &c->object.oid, \"\", &diffopt);\n \tdiffcore_std(&diffopt);\n \n-\tif (diffopt.num_changes <= settings->max_changed_paths) {\n+\tif (diff_queued_diff.nr <= settings->max_changed_paths) {\n \t\tstruct hashmap pathmap;\n \t\tstruct pathmap_hash_entry *e;\n \t\tstruct hashmap_iter iter;\n@@ -259,6 +259,12 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tdiff_free_filepair(diff_queued_diff.queue[i]);\n \t\t}\n \n+\t\tif (hashmap_get_size(&pathmap) > settings->max_changed_paths) {\n+\t\t\tif (computed)\n+\t\t\t\t*computed |= BLOOM_TRUNC_LARGE;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\n \t\tfilter->len = (hashmap_get_size(&pathmap) * settings->bits_per_entry + BITS_PER_WORD - 1) / BITS_PER_WORD;\n \t\tfilter->data = xcalloc(filter->len, sizeof(unsigned char));\n \n@@ -268,6 +274,7 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n \t\t\tadd_key_to_filter(&key, filter, settings);\n \t\t}\n \n+\tcleanup:\n \t\thashmap_free_entries(&pathmap, struct pathmap_hash_entry, entry);\n \t} else {\n \t\tfor (i = 0; i < diff_queued_diff.nr; i++)\ndiff --git a/diff.h b/diff.h\nindex e0c0af6286..1d32b71885 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -287,8 +287,6 @@ struct diff_options {\n \n \t/* If non-zero, then stop computing after this many changes. */\n \tint max_changes;\n-\t/* For internal use only. */\n-\tint num_changes;\n \n \tint ita_invisible_in_index;\n /* white-space error highlighting */\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 593571358d..123b46a0db 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -181,21 +181,103 @@ test_expect_success 'persist filter settings' '\n \tgrep \"{\\\"hash_version\\\":1,\\\"num_hashes\\\":9,\\\"bits_per_entry\\\":15,\\\"max_changed_paths\\\":512\" trace2-auto.txt\n '\n \n+test_max_changed_paths () {\n+\tgrep \"\\\"max_changed_paths\\\":$1\" $2\n+}\n+\n+test_filter_computed () {\n+\tgrep \"\\\"key\\\":\\\"filter-computed\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n+test_filter_trunc_large () {\n+\tgrep \"\\\"key\\\":\\\"filter-trunc-large\\\",\\\"value\\\":\\\"$1\\\"\" $2\n+}\n+\n test_expect_success 'correctly report changes over limit' '\n-\tgit init 513changes &&\n+\tgit init limits &&\n \t(\n-\t\tcd 513changes &&\n-\t\tfor i in $(test_seq 1 513)\n+\t\tcd limits &&\n+\t\tmkdir d &&\n+\t\tmkdir d/e &&\n+\n+\t\tfor i in $(test_seq 1 2)\n \t\tdo\n-\t\t\techo $i >file$i.txt || return 1\n+\t\t\tprintf $i >d/file$i.txt &&\n+\t\t\tprintf $i >d/e/file$i.txt || return 1\n \t\tdone &&\n-\t\tgit add . &&\n+\n+\t\tmkdir mode &&\n+\t\tprintf bash >mode/script.sh &&\n+\n+\t\tmkdir foo &&\n+\t\ttouch foo/bar &&\n+\t\ttouch foo.txt &&\n+\n+\t\tgit add d foo foo.txt mode &&\n \t\tgit commit -m \"files\" &&\n-\t\tgit commit-graph write --reachable --changed-paths &&\n-\t\tfor i in $(test_seq 1 513)\n+\n+\t\t# Commit has 7 file and 4 directory adds\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\ttest_max_changed_paths 10 trace &&\n+\t\ttest_filter_computed 1 trace &&\n+\t\ttest_filter_trunc_large 1 trace &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n \t\tdo\n-\t\t\tgit -c core.commitGraph=false log -- file$i.txt >expect &&\n-\t\t\tgit log -- file$i.txt >actual &&\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# Make a variety of path changes\n+\t\tprintf new1 >d/e/file1.txt &&\n+\t\tprintf new2 >d/file2.txt &&\n+\t\trm d/e/file2.txt &&\n+\t\trm -r foo &&\n+\t\tprintf text >foo &&\n+\t\tmkdir f &&\n+\t\tprintf new1 >f/file1.txt &&\n+\n+\t\t# including a mode-only change (counts as modified)\n+\t\tgit update-index --chmod=+x mode/script.sh &&\n+\n+\t\tgit add foo d f &&\n+\t\tgit commit -m \"complicated\" &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=10 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-edit\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\ttest_max_changed_paths 10 trace-edit &&\n+\t\ttest_filter_computed 2 trace-edit &&\n+\t\ttest_filter_trunc_large 2 trace-edit &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n+\t\t\ttest_cmp expect actual || return 1\n+\t\tdone &&\n+\n+\t\t# start from scratch and rebuild\n+\t\trm -f .git/objects/info/commit-graph &&\n+\t\tGIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS=11 \\\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace-update\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths &&\n+\t\ttest_max_changed_paths 11 trace-update &&\n+\t\ttest_filter_computed 2 trace-update &&\n+\t\ttest_filter_trunc_large 0 trace-update &&\n+\n+\t\tfor path in $(git ls-tree -r --name-only HEAD)\n+\t\tdo\n+\t\t\tgit -c commitGraph.readChangedPaths=false log \\\n+\t\t\t\t-- $path >expect &&\n+\t\t\tgit log -- $path >actual &&\n \t\t\ttest_cmp expect actual || return 1\n \t\tdone\n \t)\ndiff --git a/tree-diff.c b/tree-diff.c\nindex 6ebad1a46f..7cebbb327e 100644\n--- a/tree-diff.c\n+++ b/tree-diff.c\n@@ -434,7 +434,7 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tif (diff_can_quit_early(opt))\n \t\t\tbreak;\n \n-\t\tif (opt->max_changes && opt->num_changes > opt->max_changes)\n+\t\tif (opt->max_changes && diff_queued_diff.nr > opt->max_changes)\n \t\t\tbreak;\n \n \t\tif (opt->pathspec.nr) {\n@@ -521,7 +521,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \n \t\t\t/* t↓ */\n \t\t\tupdate_tree_entry(&t);\n-\t\t\topt->num_changes++;\n \t\t}\n \n \t\t/* t > p[imin] */\n@@ -539,7 +538,6 @@ static struct combine_diff_path *ll_diff_tree_paths(\n \t\tskip_emit_tp:\n \t\t\t/* ∀ pi=p[imin] pi↓ */\n \t\t\tupdate_tp_entries(tp, nparent);\n-\t\t\topt->num_changes++;\n \t\t}\n \t}\n \n@@ -557,7 +555,6 @@ struct combine_diff_path *diff_tree_paths(\n \tconst struct object_id **parents_oid, int nparent,\n \tstruct strbuf *base, struct diff_options *opt)\n {\n-\topt->num_changes = 0;\n \tp = ll_diff_tree_paths(p, oid, parents_oid, nparent, base, opt);\n \n \t/*\n-- \n2.28.0.510.g375ecf1f36\n\n"},{"id":"405843","messageId":"20200918092036.GA24813@szeder.dev","threadId":"54216","inReplyTo":"ccb6482feb8d8606d82b5ab97e33184f26d6c5b6.1600279373.git.me@ttaylorr.com","subject":"Re: [PATCH v2 12/13] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-09-18T09:23:24Z","receivedAt":"2020-09-18T09:23:36Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Wed, Sep 16, 2020 at 02:08:10PM -0400, Taylor Blau wrote:\n> Introduce a command-line flag to specify the maximum number of new Bloom\n> filters that a 'git commit-graph write' is willing to compute from\n> scratch.\n> \n> Prior to this patch, a commit-graph write with '--changed-paths' would\n> compute Bloom filters for all selected commits which haven't already\n> been computed (i.e., by a previous commit-graph write with '--split'\n> such that a roll-up or replacement is performed).\n> \n> This behavior can cause prohibitively-long commit-graph writes for a\n> variety of reasons:\n> \n> * There may be lots of filters whose diffs take a long time to\n> generate (for example, they have close to the maximum number of\n> changes, diffing itself takes a long time, etc).\n> \n> * Old-style commit-graphs (which encode filters with too many entries\n> as not having been computed at all) cause us to waste time\n> recomputing filters that appear to have not been computed only to\n> discover that they are too-large.\n> \n> This can make the upper-bound of the time it takes for 'git commit-graph\n> write --changed-paths' to be rather unpredictable.\n> \n> To make this command behave more predictably, introduce\n> '--max-new-filters=' to allow computing at most '' Bloom filters\n> from scratch. This lets \"computing\" already-known filters proceed\n> quickly, while bounding the number of slow tasks that Git is willing to\n> do.\n> \n> Signed-off-by: Taylor Blau \n> ---\n> Documentation/git-commit-graph.txt | 5 +++\n> bloom.c | 7 ++-\n> builtin/commit-graph.c | 27 +++++++++++-\n> commit-graph.c | 9 +++-\n> commit-graph.h | 1 +\n> t/t4216-log-bloom.sh | 70 ++++++++++++++++++++++++++++++\n> 6 files changed, 111 insertions(+), 8 deletions(-)\n> \n> diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n> index 17405c73a9..8357846d30 100644\n> --- a/Documentation/git-commit-graph.txt\n> +++ b/Documentation/git-commit-graph.txt\n> @@ -67,6 +67,11 @@ this option is given, future commit-graph writes will automatically assume\n> that this option was intended. Use `--no-changed-paths` to stop storing this\n> data.\n> +\n> +With the `--max-new-filters=` option, generate at most `n` new Bloom\n> +filters (if `--changed-paths` is specified).\n\nIf '--max-new-filters=42' is specified without '--changed-paths', then\nit is silently ignored instead of erroring out. Is that intentional?\n\n> If `n` is `-1`, no limit is\n> +enforced. Commits whose filters are not calculated are stored as a\n> +length zero Bloom filter.\n\nThis last sentence goes into implementation details of the file\nformat, so I don't think it should be included here.\n\nWhat should be documented here instead is what happens later with\nthose commits for which no modified path Bloom filters were computed.\nWill they ever be computed? If yes, which commands will compute them\n(even implicitly), and thus will run slower? What command(s) should\nthe users run if they want to compute the missing filters?\n\nLet's suppose that running\n\n git commit-graph write --reachable --changed-paths --max-new-filters= --split\n\nwrites a commit-graph layer containing N1 commits, and N1 > L, so N1-L\ncommits won't have an associated modified path Bloom filter. After\ncreating and fetching new commits, this command is executed again to\nadd information about N2 commits to the commit-graph, and N2 < L.\n\n - If this second invocation writes a new commit-graph layer, then it\n will only compute Bloom filters for commits in this new layer,\n even though it still has some \"Bloom-computing-capacity\" left.\n (This will happen even if this second write were invoked without\n '--max-new-filters'.)\n\n - If the merge strategy decides that, instead of writing a new\n layer, the new N2 commits should be merged with the layer\n containing the previous N1 commits, then it will compute Bloom\n filters for some or all of those N1 commits without filters,\n depending on whether N2-L is smaller or larger than N1-L.\n\nIs this how it is supposed to work? We can't tell, because neither\nthe commit messages nor the documentation updates talk about this.\n\n> ++\n> With the `--split[=]` option, write the commit-graph as a\n> chain of multiple commit-graph files stored in\n> `/info/commit-graphs`. Commit-graph layers are merged based on the\n"},{"id":"405857","messageId":"20200918132727.GB1600256@nand.local","threadId":"54216","inReplyTo":"20200918092036.GA24813@szeder.dev","subject":"Re: [PATCH v2 12/13] builtin/commit-graph.c: introduce '--max-new-filters='","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T13:27:27Z","receivedAt":"2020-09-18T13:27:35Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 18, 2020 at 11:23:24AM +0200, SZEDER Gábor wrote:\n> On Wed, Sep 16, 2020 at 02:08:10PM -0400, Taylor Blau wrote:\n> > +With the `--max-new-filters=` option, generate at most `n` new Bloom\n> > +filters (if `--changed-paths` is specified).\n>\n> If '--max-new-filters=42' is specified without '--changed-paths', then\n> it is silently ignored instead of erroring out. Is that intentional?\n\nYeah, that is intentional. The rationale is that if you're specifying \"I\nwon't compute any more than N filters\" but don't say \"compute and write\nchanged-path Bloom filters\", then you'll compute 0 < N filters, which is\nfine.\n\nI considered making this a warning, but ultimately decided against it.\nIn particular, I had users who might set 'commitGraph.maxNewFilters' to\nsome value, but only sometimes write a commit-graph with\n'--changed-paths'.\n\n> > If `n` is `-1`, no limit is\n> > +enforced. Commits whose filters are not calculated are stored as a\n> > +length zero Bloom filter.\n>\n> This last sentence goes into implementation details of the file\n> format, so I don't think it should be included here.\n\nGood idea, I'll drop that.\n\n> What should be documented here instead is what happens later with\n> those commits for which no modified path Bloom filters were computed.\n> Will they ever be computed? If yes, which commands will compute them\n> (even implicitly), and thus will run slower? What command(s) should\n> the users run if they want to compute the missing filters?\n>\n> Let's suppose that running\n>\n> git commit-graph write --reachable --changed-paths --max-new-filters= --split\n>\n> writes a commit-graph layer containing N1 commits, and N1 > L, so N1-L\n> commits won't have an associated modified path Bloom filter. After\n> creating and fetching new commits, this command is executed again to\n> add information about N2 commits to the commit-graph, and N2 < L.\n>\n> - If this second invocation writes a new commit-graph layer, then it\n> will only compute Bloom filters for commits in this new layer,\n> even though it still has some \"Bloom-computing-capacity\" left.\n> (This will happen even if this second write were invoked without\n> '--max-new-filters'.)\n>\n> - If the merge strategy decides that, instead of writing a new\n> layer, the new N2 commits should be merged with the layer\n> containing the previous N1 commits, then it will compute Bloom\n> filters for some or all of those N1 commits without filters,\n> depending on whether N2-L is smaller or larger than N1-L.\n>\n> Is this how it is supposed to work? We can't tell, because neither\n> the commit messages nor the documentation updates talk about this.\n\nThat's a wonderful question, and yes: that's how it's supposed to work.\nImplicit in all of this is that these commit-graph options are only\nacting over the new layers that we are going to write. So, if we're\ngoing to roll-up older layers, we'll start with their uncomputed\nfilters before computing new filters in later layers.\n\nThis is the same thing as running 'git commit-graph write --reachable\n--changed-paths --split' (omitting '--max-new-filters' is not a typo\nthere). Because the commit-graph command only acts over new layers, old\nlayers do not suddenly get Bloom filters computed and those filters\nrewritten. If a user wants that to happen, they can pass\n'--split=replace', which will force all existing graphs to reside in a\nnew, single layer chain.\n\nLet's use the following version of the patch which I think will clarify\nyour questions:\n\n--- >8 ---\n\nSubject: [PATCH] builtin/commit-graph.c: introduce '--max-new-filters='\n\nIntroduce a command-line flag to specify the maximum number of new Bloom\nfilters that a 'git commit-graph write' is willing to compute from\nscratch.\n\nPrior to this patch, a commit-graph write with '--changed-paths' would\ncompute Bloom filters for all selected commits which haven't already\nbeen computed (i.e., by a previous commit-graph write with '--split'\nsuch that a roll-up or replacement is performed).\n\nThis behavior can cause prohibitively-long commit-graph writes for a\nvariety of reasons:\n\n * There may be lots of filters whose diffs take a long time to\n generate (for example, they have close to the maximum number of\n changes, diffing itself takes a long time, etc).\n\n * Old-style commit-graphs (which encode filters with too many entries\n as not having been computed at all) cause us to waste time\n recomputing filters that appear to have not been computed only to\n discover that they are too-large.\n\nThis can make the upper-bound of the time it takes for 'git commit-graph\nwrite --changed-paths' to be rather unpredictable.\n\nTo make this command behave more predictably, introduce\n'--max-new-filters=' to allow computing at most '' Bloom filters\nfrom scratch. This lets \"computing\" already-known filters proceed\nquickly, while bounding the number of slow tasks that Git is willing to\ndo.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n Documentation/git-commit-graph.txt | 6 +++\n bloom.c | 7 ++-\n builtin/commit-graph.c | 27 +++++++++++-\n commit-graph.c | 9 +++-\n commit-graph.h | 1 +\n t/t4216-log-bloom.sh | 70 ++++++++++++++++++++++++++++++\n 6 files changed, 112 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 17405c73a9..8c75855782 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -67,6 +67,12 @@ this option is given, future commit-graph writes will automatically assume\n that this option was intended. Use `--no-changed-paths` to stop storing this\n data.\n +\n+With the `--max-new-filters=` option, generate at most `n` new Bloom\n+filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n+enforced. Only commits present in the new layer count against this\n+limit. To retroactively compute Bloom filters over earlier layers, it is\n+advised to use `--split=replace`.\n++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n `/info/commit-graphs`. Commit-graph layers are merged based on the\ndiff --git a/bloom.c b/bloom.c\nindex d234551ce0..68c73200a5 100644\n--- a/bloom.c\n+++ b/bloom.c\n@@ -204,12 +204,11 @@ struct bloom_filter *get_or_compute_bloom_filter(struct repository *r,\n\n \tif (!filter->data) {\n \t\tload_commit_graph_info(r, c);\n-\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH &&\n-\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c))\n-\t\t\t\treturn filter;\n+\t\tif (commit_graph_position(c) != COMMIT_NOT_FROM_GRAPH)\n+\t\t\tload_bloom_filter_from_graph(r->objects->commit_graph, filter, c);\n \t}\n\n-\tif (filter->data)\n+\tif (filter->data && filter->len)\n \t\treturn filter;\n \tif (!compute_if_not_present)\n \t\treturn NULL;\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex f3243bd982..5df9b2ef80 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -13,7 +13,8 @@ static char const * const builtin_commit_graph_usage[] = {\n \tN_(\"git commit-graph verify [--object-dir ] [--shallow] [--[no-]progress]\"),\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n\n@@ -25,7 +26,8 @@ static const char * const builtin_commit_graph_verify_usage[] = {\n static const char * const builtin_commit_graph_write_usage[] = {\n \tN_(\"git commit-graph write [--object-dir ] [--append] \"\n \t \"[--split[=]] [--reachable|--stdin-packs|--stdin-commits] \"\n-\t \"[--changed-paths] [--[no-]progress] \"),\n+\t \"[--changed-paths] [--[no-]max-new-filters ] [--[no-]progress] \"\n+\t \"\"),\n \tNULL\n };\n\n@@ -162,6 +164,23 @@ static int read_one_commit(struct oidset *commits, struct progress *progress,\n \treturn 0;\n }\n\n+static int write_option_max_new_filters(const struct option *opt,\n+\t\t\t\t\tconst char *arg,\n+\t\t\t\t\tint unset)\n+{\n+\tint *to = opt->value;\n+\tif (unset)\n+\t\t*to = -1;\n+\telse {\n+\t\tconst char *s;\n+\t\t*to = strtol(arg, (char **)&s, 10);\n+\t\tif (*s)\n+\t\t\treturn error(_(\"%s expects a numerical value\"),\n+\t\t\t\t optname(opt, opt->flags));\n+\t}\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -197,6 +216,9 @@ static int graph_write(int argc, const char **argv)\n \t\t\tN_(\"maximum ratio between two levels of a split commit-graph\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire-time\", &write_opts.expire_time,\n \t\t\tN_(\"only expire files older than a given date-time\")),\n+\t\tOPT_CALLBACK_F(0, \"max-new-filters\", &write_opts.max_new_filters,\n+\t\t\tNULL, N_(\"maximum number of changed-path Bloom filters to compute\"),\n+\t\t\t0, write_option_max_new_filters),\n \t\tOPT_END(),\n \t};\n\n@@ -205,6 +227,7 @@ static int graph_write(int argc, const char **argv)\n \twrite_opts.size_multiple = 2;\n \twrite_opts.max_commits = 0;\n \twrite_opts.expire_time = 0;\n+\twrite_opts.max_new_filters = -1;\n\n \ttrace2_cmd_mode(\"write\");\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex d2a2f78776..d10e730f29 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -1415,6 +1415,7 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \tint i;\n \tstruct progress *progress = NULL;\n \tstruct commit **sorted_commits;\n+\tint max_new_filters;\n\n \tinit_bloom_filters();\n\n@@ -1431,13 +1432,16 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \telse\n \t\tQSORT(sorted_commits, ctx->commits.nr, commit_gen_cmp);\n\n+\tmax_new_filters = ctx->opts && ctx->opts->max_new_filters >= 0 ?\n+\t\tctx->opts->max_new_filters : ctx->commits.nr;\n+\n \tfor (i = 0; i < ctx->commits.nr; i++) {\n \t\tenum bloom_filter_computed computed = 0;\n \t\tstruct commit *c = sorted_commits[i];\n \t\tstruct bloom_filter *filter = get_or_compute_bloom_filter(\n \t\t\tctx->r,\n \t\t\tc,\n-\t\t\t1,\n+\t\t\tctx->count_bloom_filter_computed < max_new_filters,\n \t\t\tctx->bloom_settings,\n \t\t\t&computed);\n \t\tif (computed & BLOOM_COMPUTED) {\n@@ -1448,7 +1452,8 @@ static void compute_bloom_filters(struct write_commit_graph_context *ctx)\n \t\t\t\tctx->count_bloom_filter_trunc_large++;\n \t\t} else if (computed & BLOOM_NOT_COMPUTED)\n \t\t\tctx->count_bloom_filter_not_computed++;\n-\t\tctx->total_bloom_filter_data_size += sizeof(unsigned char) * filter->len;\n+\t\tctx->total_bloom_filter_data_size += filter\n+\t\t\t? sizeof(unsigned char) * filter->len : 0;\n \t\tdisplay_progress(progress, i + 1);\n \t}\n\ndiff --git a/commit-graph.h b/commit-graph.h\nindex b7914b0a7a..a22bd86701 100644\n--- a/commit-graph.h\n+++ b/commit-graph.h\n@@ -110,6 +110,7 @@ struct commit_graph_opts {\n \tint max_commits;\n \ttimestamp_t expire_time;\n \tenum commit_graph_split_flags split_flags;\n+\tint max_new_filters;\n };\n\n /*\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex 15bbb626a1..af0975c841 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -310,4 +310,74 @@ test_expect_success 'correctly report commits with no changed paths' '\n \t)\n '\n\n+test_expect_success 'Bloom generation is limited by --max-new-filters' '\n+\t(\n+\t\tcd limits &&\n+\t\ttest_commit c2 filter &&\n+\t\ttest_commit c3 filter &&\n+\t\ttest_commit c4 no-filter &&\n+\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --split=replace \\\n+\t\t\t\t--changed-paths --max-new-filters=2 &&\n+\n+\t\ttest_filter_computed 2 trace.event &&\n+\t\ttest_filter_not_computed 3 trace.event &&\n+\t\ttest_filter_trunc_empty 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t(\n+\t\tcd limits &&\n+\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t\t--split=replace --max-new-filters=1 &&\n+\t\ttest_filter_computed 1 trace.event &&\n+\t\ttest_filter_not_computed 4 trace.event &&\n+\t\ttest_filter_trunc_empty 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n+test_expect_success 'Bloom generation backfills empty commits' '\n+\tgit init empty &&\n+\ttest_when_finished \"rm -fr empty\" &&\n+\t(\n+\t\tcd empty &&\n+\t\tfor i in $(test_seq 1 6)\n+\t\tdo\n+\t\t\tgit commit --allow-empty -m \"$i\"\n+\t\tdone &&\n+\n+\t\t# Generate Bloom filters for empty commits 1-6, two at a time.\n+\t\tfor i in $(test_seq 1 3)\n+\t\tdo\n+\t\t\trm -f trace.event &&\n+\t\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\t\tgit commit-graph write --reachable \\\n+\t\t\t\t\t--changed-paths --max-new-filters=2 &&\n+\t\t\ttest_filter_computed 2 trace.event &&\n+\t\t\ttest_filter_not_computed 4 trace.event &&\n+\t\t\ttest_filter_trunc_empty 2 trace.event &&\n+\t\t\ttest_filter_trunc_large 0 trace.event\n+\t\tdone &&\n+\n+\t\t# Finally, make sure that once all commits have filters, that\n+\t\t# none are subsequently recomputed.\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable \\\n+\t\t\t\t--changed-paths --max-new-filters=2 &&\n+\t\ttest_filter_computed 0 trace.event &&\n+\t\ttest_filter_not_computed 6 trace.event &&\n+\t\ttest_filter_trunc_empty 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n test_done\n--\n2.28.0.510.g375ecf1f36\n\n"},{"id":"405858","messageId":"20200918132937.GA1601745@nand.local","threadId":"54216","inReplyTo":"375ecf1f3651b250982216d40167e4c23b8e15ff.1600397826.git.me@ttaylorr.com","subject":"Re: [PATCH v3 13/13] commit-graph: introduce 'commitGraph.maxNewFilters'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T13:29:37Z","receivedAt":"2020-09-18T13:29:43Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Thu, Sep 17, 2020 at 10:59:57PM -0400, Taylor Blau wrote:\n> Introduce a configuration variable to specify a default value for the\n> recently-introduce '--max-new-filters' option of 'git commit-graph\n> write'.\n\nThe replacement in [1] causes a conflict when applying this patch after\nit, so here is a suggested resolution:\n\n[1]: https://lore.kernel.org/git/cover.1600397826.git.me@ttaylorr.com/\n\n--- >8 ---\n\nSubject: [PATCH] commit-graph: introduce 'commitGraph.maxNewFilters'\n\nIntroduce a configuration variable to specify a default value for the\nrecently-introduce '--max-new-filters' option of 'git commit-graph\nwrite'.\n\nHelped-by: Junio C Hamano \nSigned-off-by: Taylor Blau \nSigned-off-by: Junio C Hamano \n---\n Documentation/config/commitgraph.txt | 4 ++++\n Documentation/git-commit-graph.txt | 3 ++-\n builtin/commit-graph.c | 14 ++++++++++++++\n t/t4216-log-bloom.sh | 24 +++++++++++++++++++++++-\n 4 files changed, 43 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/config/commitgraph.txt b/Documentation/config/commitgraph.txt\nindex cff0797b54..4582c39fc4 100644\n--- a/Documentation/config/commitgraph.txt\n+++ b/Documentation/config/commitgraph.txt\n@@ -1,3 +1,7 @@\n+commitGraph.maxNewFilters::\n+\tSpecifies the default value for the `--max-new-filters` option of `git\n+\tcommit-graph write` (c.f., linkgit:git-commit-graph[1]).\n+\n commitGraph.readChangedPaths::\n \tIf true, then git will use the changed-path Bloom filters in the\n \tcommit-graph file (if it exists, and they are present). Defaults to\ndiff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\nindex 8c75855782..b6ddbb4700 100644\n--- a/Documentation/git-commit-graph.txt\n+++ b/Documentation/git-commit-graph.txt\n@@ -71,7 +71,8 @@ With the `--max-new-filters=` option, generate at most `n` new Bloom\n filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n enforced. Only commits present in the new layer count against this\n limit. To retroactively compute Bloom filters over earlier layers, it is\n-advised to use `--split=replace`.\n+advised to use `--split=replace`. Overrides the\n+`commitGraph.maxNewFilters` configuration.\n +\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\ndiff --git a/builtin/commit-graph.c b/builtin/commit-graph.c\nindex 5df9b2ef80..988445abdf 100644\n--- a/builtin/commit-graph.c\n+++ b/builtin/commit-graph.c\n@@ -181,6 +181,18 @@ static int write_option_max_new_filters(const struct option *opt,\n \treturn 0;\n }\n\n+static int git_commit_graph_write_config(const char *var, const char *value,\n+\t\t\t\t\t void *cb)\n+{\n+\tif (!strcmp(var, \"commitgraph.maxnewfilters\"))\n+\t\twrite_opts.max_new_filters = git_config_int(var, value);\n+\t/*\n+\t * No need to fall-back to 'git_default_config', since this was already\n+\t * called in 'cmd_commit_graph()'.\n+\t */\n+\treturn 0;\n+}\n+\n static int graph_write(int argc, const char **argv)\n {\n \tstruct string_list pack_indexes = STRING_LIST_INIT_NODUP;\n@@ -231,6 +243,8 @@ static int graph_write(int argc, const char **argv)\n\n \ttrace2_cmd_mode(\"write\");\n\n+\tgit_config(git_commit_graph_write_config, &opts);\n+\n \targc = parse_options(argc, argv, NULL,\n \t\t\t builtin_commit_graph_write_options,\n \t\t\t builtin_commit_graph_write_usage, 0);\ndiff --git a/t/t4216-log-bloom.sh b/t/t4216-log-bloom.sh\nindex af0975c841..d11040ce41 100755\n--- a/t/t4216-log-bloom.sh\n+++ b/t/t4216-log-bloom.sh\n@@ -330,13 +330,15 @@ test_expect_success 'Bloom generation is limited by --max-new-filters' '\n '\n\n test_expect_success 'Bloom generation backfills previously-skipped filters' '\n+\t# Check specifying commitGraph.maxNewFilters over \"git config\" works.\n+\ttest_config -C limits commitGraph.maxNewFilters 1 &&\n \t(\n \t\tcd limits &&\n\n \t\trm -f trace.event &&\n \t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n \t\t\tgit commit-graph write --reachable --changed-paths \\\n-\t\t\t\t--split=replace --max-new-filters=1 &&\n+\t\t\t\t--split=replace &&\n \t\ttest_filter_computed 1 trace.event &&\n \t\ttest_filter_not_computed 4 trace.event &&\n \t\ttest_filter_trunc_empty 0 trace.event &&\n@@ -344,6 +346,26 @@ test_expect_success 'Bloom generation backfills previously-skipped filters' '\n \t)\n '\n\n+test_expect_success '--max-new-filters overrides configuration' '\n+\tgit init override &&\n+\ttest_when_finished \"rm -fr override\" &&\n+\ttest_config -C override commitGraph.maxNewFilters 2 &&\n+\t(\n+\t\tcd override &&\n+\t\ttest_commit one &&\n+\t\ttest_commit two &&\n+\n+\t\trm -f trace.event &&\n+\t\tGIT_TRACE2_EVENT=\"$(pwd)/trace.event\" \\\n+\t\t\tgit commit-graph write --reachable --changed-paths \\\n+\t\t\t\t--max-new-filters=1 &&\n+\t\ttest_filter_computed 1 trace.event &&\n+\t\ttest_filter_not_computed 1 trace.event &&\n+\t\ttest_filter_trunc_empty 0 trace.event &&\n+\t\ttest_filter_trunc_large 0 trace.event\n+\t)\n+'\n+\n test_expect_success 'Bloom generation backfills empty commits' '\n \tgit init empty &&\n \ttest_when_finished \"rm -fr empty\" &&\n--\n2.28.0.510.g375ecf1f36\n\n"},{"id":"405859","messageId":"20200918133140.GB1601745@nand.local","threadId":"54216","inReplyTo":"cover.1600397826.git.me@ttaylorr.com","subject":"Re: [PATCH v3 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T13:31:40Z","receivedAt":"2020-09-18T13:31:46Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"Junio,\n\nTwo replacements for this version, if it ends up being the one you\nqueue. Gàbor suggested some helpful changes on 12/13, which in turn\ncause a conflict when applying 13/13.\n\nWhen queueing, please take:\n\n - The patch in [1] instead of v3's original 12/13, and\n - The patch in [2] as a suggested resolution when applying v3's\n original 13/13 on top.\n\n[1]: https://lore.kernel.org/git/20200918132727.GB1600256@nand.local/\n[2]: https://lore.kernel.org/git/20200918132937.GA1601745@nand.local/\n\nThanks.\n"},{"id":"405860","messageId":"20200918133413.GA1602165@nand.local","threadId":"54216","inReplyTo":"20200918133140.GB1601745@nand.local","subject":"Re: [PATCH v3 00/13] more miscellaneous Bloom filter improvements, redux","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T13:34:13Z","receivedAt":"2020-09-18T13:34:19Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 18, 2020 at 09:31:40AM -0400, Taylor Blau wrote:\n> Junio,\n>\n> Two replacements for this version, if it ends up being the one you\n> queue. Gàbor suggested some helpful changes on 12/13, which in turn\n> cause a conflict when applying 13/13.\n\nI should mention, the changes are purely an alteration to the new\ndocumentation introduced by this series. Here's a range-diff:\n\n12: 4549f0f747 ! 12: 1c3f6b5c96 builtin/commit-graph.c: introduce '--max-new-filters='\n @@ Documentation/git-commit-graph.txt: this option is given, future commit-graph wr\n +\n +With the `--max-new-filters=` option, generate at most `n` new Bloom\n +filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n -+enforced. Commits whose filters are not calculated are stored as a\n -+length zero Bloom filter.\n ++enforced. Only commits present in the new layer count against this\n ++limit. To retroactively compute Bloom filters over earlier layers, it is\n ++advised to use `--split=replace`.\n ++\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n13: 375ecf1f36 ! 13: a7330ee850 commit-graph: introduce 'commitGraph.maxNewFilters'\n @@ Documentation/config/commitgraph.txt\n \tcommit-graph file (if it exists, and they are present). Defaults to\n\n ## Documentation/git-commit-graph.txt ##\n -@@ Documentation/git-commit-graph.txt: data.\n - With the `--max-new-filters=` option, generate at most `n` new Bloom\n +@@ Documentation/git-commit-graph.txt: With the `--max-new-filters=` option, generate at most `n` new Bloom\n filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n - enforced. Commits whose filters are not calculated are stored as a\n --length zero Bloom filter.\n -+length zero Bloom filter. Overrides the `commitGraph.maxNewFilters`\n -+configuration.\n + enforced. Only commits present in the new layer count against this\n + limit. To retroactively compute Bloom filters over earlier layers, it is\n +-advised to use `--split=replace`.\n ++advised to use `--split=replace`. Overrides the\n ++`commitGraph.maxNewFilters` configuration.\n +\n With the `--split[=]` option, write the commit-graph as a\n chain of multiple commit-graph files stored in\n\nThanks,\nTaylor\n"},{"id":"405877","messageId":"20200918162740.GA25148@szeder.dev","threadId":"54216","inReplyTo":"3745baf8ef8810fe8e6031d06d2f6b8d967ef13c.1600397826.git.me@ttaylorr.com","subject":"Re: [PATCH v3 08/13] bloom: use provided 'struct bloom_filter_settings'","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2020-09-18T16:27:40Z","receivedAt":"2020-09-18T16:27:49Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"On Thu, Sep 17, 2020 at 10:59:27PM -0400, Taylor Blau wrote:\n> When 'get_or_compute_bloom_filter()' needs to compute a Bloom filter\n> from scratch, it looks to the default 'struct bloom_filter_settings' in\n> order to determine the maximum number of changed paths, number of bits\n> per entry, and so on.\n> \n> All of these values have so far been constant, and so there was no need\n> to pass in a pointer from the caller (eg., the one that is stored in the\n> 'struct write_commit_graph_context').\n> \n> Start passing in a 'struct bloom_filter_settings *' instead of using the\n> default values to respect graph-specific settings (eg., in the case of\n> setting 'GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS').\n\nI think this description misses the most important aspect of this\npatch: it fixes, even if only partially, the half-broken fix in\n0087a87ba8 (commit-graph: persist existence of changed-paths,\n2020-07-01).\n\nThat commit, among other things, tried to make sure that we use the\nsame Bloom filter settings to compute new Bloom filters that have been\nused in already existing filters. However, it only read those\nsettings from the header of the existing BDAT chunk and wrote them to\nthe header of the new BDAT chunk (and printed them to trace2 output).\nUnfortunately, it didn't actually use those settings to compute Bloom\nfilters, because it left get_bloom_filter() unchanged to use the\nhardcoded default Bloom filter settings. This can result in bogus\ncommits-graphs and, in turn, pathspec-limited revision walks omitting\ncommits that do modify the specified path.\n\n"},{"id":"405878","messageId":"20200918163242.GA138001@nand.local","threadId":"54216","inReplyTo":"20200918162740.GA25148@szeder.dev","subject":"Re: [PATCH v3 08/13] bloom: use provided 'struct bloom_filter_settings'","fromName":"Taylor Blau","fromEmail":"me@ttaylorr.com","sentAt":"2020-09-18T16:32:42Z","receivedAt":"2020-09-18T16:32:48Z","isPatch":true,"sender":{"key":"me@ttaylorr.com","avatar":"https://avatars.githubusercontent.com/u/301000140?v=4"},"body":"On Fri, Sep 18, 2020 at 06:27:40PM +0200, SZEDER Gábor wrote:\n> On Thu, Sep 17, 2020 at 10:59:27PM -0400, Taylor Blau wrote:\n> > When 'get_or_compute_bloom_filter()' needs to compute a Bloom filter\n> > from scratch, it looks to the default 'struct bloom_filter_settings' in\n> > order to determine the maximum number of changed paths, number of bits\n> > per entry, and so on.\n> >\n> > All of these values have so far been constant, and so there was no need\n> > to pass in a pointer from the caller (eg., the one that is stored in the\n> > 'struct write_commit_graph_context').\n> >\n> > Start passing in a 'struct bloom_filter_settings *' instead of using the\n> > default values to respect graph-specific settings (eg., in the case of\n> > setting 'GIT_TEST_BLOOM_SETTINGS_MAX_CHANGED_PATHS').\n>\n> I think this description misses the most important aspect of this\n> patch: it fixes, even if only partially, the half-broken fix in\n> 0087a87ba8 (commit-graph: persist existence of changed-paths,\n> 2020-07-01).\n\nCould you suggest an amendment? I understand what you are saying below,\nbut I'm not sure how or if you want it incorporated into this patch.\n\nIf you feel this is critical to change (I have no opinion either way),\nthen I'm happy to send a re-roll once review has stabilized. When do you\nplan on finishing your read-through of this v3?\n\nThanks,\nTaylor\n"},{"id":"405893","messageId":"xmqqft7fm14b.fsf@gitster.c.googlers.com","threadId":"54216","inReplyTo":"20200918132937.GA1601745@nand.local","subject":"Re: [PATCH v3 13/13] commit-graph: introduce 'commitGraph.maxNewFilters'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-09-18T17:43:48Z","receivedAt":"2020-09-18T17:43:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Taylor Blau writes:\n\n> On Thu, Sep 17, 2020 at 10:59:57PM -0400, Taylor Blau wrote:\n>> Introduce a configuration variable to specify a default value for the\n>> recently-introduce '--max-new-filters' option of 'git commit-graph\n>> write'.\n>\n> The replacement in [1] causes a conflict when applying this patch after\n> it, so here is a suggested resolution:\n>\n> [1]: https://lore.kernel.org/git/cover.1600397826.git.me@ttaylorr.com/\n\nI think you meant <20200918132727.GB1600256@nand.local> not the\ncover letter. Here is what I ended up with:\n\n $ git checkout tb/bloom-improvements\n $ git diff @{yesterday}\n diff --git a/Documentation/git-commit-graph.txt b/Documentation/git-commit-graph.txt\n index 60df4e4bfa..de6b6de230 100644\n --- a/Documentation/git-commit-graph.txt\n +++ b/Documentation/git-commit-graph.txt\n @@ -69,8 +69,9 @@ data.\n +\n With the `--max-new-filters=` option, generate at most `n` new Bloom\n filters (if `--changed-paths` is specified). If `n` is `-1`, no limit is\n -enforced. Commits whose filters are not calculated are stored as a\n -length zero Bloom filter. Overrides the `commitGraph.maxNewFilters`\n +enforced. Only commits present in the new layer count against this\n +limit. To retroactively compute Bloom filters over earlier layers, it is\n +advised to use `--split=replace`. Overrides the `commitGraph.maxNewFilters`\n configuration.\n +\n With the `--split[=]` option, write the commit-graph as a\n\nThanks.\n"}]}