From: Mark C. Chu-Carroll Date: Mon, 26 Oct 2026 17:43:53 GMT Subject: [PATCH v2 0/3] repo: add degree-of-concern reporting to structure Message-ID: <20261026-b4-repo-structure-concern-v2-0-e0df34228dc8@fastmail.com> In-Reply-To: <20261021-b4-repo-structure-concern-v1-0-e3010552292c@fastmail.com> "git repo structure" computes a set of statistics that are useful for diagnosing performance issues in a git repository. But for a typical user who hasn't encountered any issues yet, it can be difficult to interpret the data - when should a value be considered concerning? This patchset introduces a feature to "git repo structure" that reports on how concerning certain measures of storage structure are for a repository. In table mode, concern is displayed as a column containing a number of stars indicating the degree of concern; in line or nul mode, concerns are displayed as additionaL lines with a numeric value. To select defaults for the thresholds where a value becomes concern, a (hopefully) representative selection of repositories were analyzed. Based on empirical exploration of the data, the measurement that seemed closest to accurately assessing when a repository's structure becomes concerning was based on the base 2 logarithm of the value. Across all of the sample repositories, for all of the measurements, the point at which things become problematic was when a base 2 logarithm of a value was more than 1 standard deviation from the mean of the log_2 values in the sample set. Based on this, the concern threshold was set for each of the measures. While this seems like a reasonable approach for setting default thresholds, it's obviously still very ad-hoc. It's likely that over time, the default values will evolve; and in the short term, projects may want to set their own thresholds. So this patchset includes support for configuring the thresholds on a per-reposiry basis. Overview of the change: * The first commit adds the command-line option "--concern", and creates the basic implementation of concern reporting. * The next commit adds configurability for the concern thresholds. * The final commit update the documentation for the command and configuration options. Signed-off-by: Mark C. Chu-Carroll --- Changes in v2: - Fix memory leak in concern string generation. - Fix CI/CD test issue involving zlib versions. - Updating the naming of the config value prefix. - Link to v1: https://patch.msgid.link/20261021-b4-repo-structure-concern-v1-0-e3010552292c@fastmail.com --- Mark C. Chu-Carroll (3): repo: add level of concern reporting repo: make concern thresholds configurable repo: add documentation for level-of-concern. Documentation/config/repo.adoc | 59 +++++++ Documentation/git-repo.adoc | 27 +++- builtin/repo.c | 343 ++++++++++++++++++++++++++++++++++++++--- t/t1901-repo-structure.sh | 137 ++++++++++++++++ 4 files changed, 543 insertions(+), 23 deletions(-) Range-diff versus v1: 1: c2b2a79967 = 1: e8971a4050 repo: add level of concern reporting 2: 67d77cfcd9 ! 2: 7a78eee517 repo: make concern thresholds configurable @@ builtin/repo.c #include "environment.h" #include "hash.h" #include "hex.h" +@@ + #define REPO_STRUCTURE_USAGE \ + "git repo structure [--format=(table|lines|nul) | -z] [--[no-]concern]" + ++#define REPO_STRUCTURE_CONFIG_PREFIX "repo.concern." ++ + static const char *const repo_usage[] = { + REPO_INFO_USAGE, + REPO_STRUCTURE_USAGE, @@ builtin/repo.c: enum concern_stat { CONCERN_STAT_TREE_INFLATED_SIZE, CONCERN_STAT_BLOB_INFLATED_SIZE, @@ builtin/repo.c: enum concern_stat { - { .name = "tag_inflated_size", .threshold = 25} +struct concern_stat_threshold concern_stat_thresholds[] = { + { .name = "commit-count", .threshold = 19 }, -+ {.name = "tree-count", .threshold = 21 }, ++ { .name = "tree-count", .threshold = 21 }, + { .name = "blob-count", .threshold = 22 }, + { .name = "tag-count", .threshold = 15 }, + { .name = "commit-size", .threshold = 28 }, @@ builtin/repo.c: enum concern_stat { }; /* + * Concern thresholds aren't universal. We try to set reasonable default + * values, but projects may want to set these thresholds differently. To + * support that, they can set the thresholds for their project in the +- * project's git config, in the section "repo.structure.concern". ++ * project's git config, in the section "repo.concern". + */ + static int config_concern_thresholds(const char *prefixed_var, + const char *str_val, + const struct config_context *ctx, + void *cb UNUSED) { +- const char *config_prefix = "repo.structure.concern."; +- if (!strncmp(config_prefix, prefixed_var, strlen(config_prefix))) { +- const char *concern_name = &prefixed_var[strlen(config_prefix)]; ++ if (!strncmp(REPO_STRUCTURE_CONFIG_PREFIX, prefixed_var, ++ strlen(REPO_STRUCTURE_CONFIG_PREFIX))) { ++ const char *concern_name = ++ &prefixed_var[strlen(REPO_STRUCTURE_CONFIG_PREFIX)]; + for (int i = 0; concern_stat_thresholds[i].name != 0; i++) { + if (!strcmp(concern_stat_thresholds[i].name, concern_name)) { + int value = git_config_int(prefixed_var, str_val, ctx->kvi); +@@ builtin/repo.c: static int config_concern_thresholds(const char *prefixed_var, + return 0; + } + +-static size_t approx_log2(long value) { ++static size_t approx_log2(size_t value) { + int position = 0; + if (value <= 0) { + return 0; @@ builtin/repo.c: static size_t approx_log2(long value) { static size_t repo_structure_concern_level(enum concern_stat stat_id, size_t value) { @@ builtin/repo.c: static size_t approx_log2(long value) { BUG("unexpected repository stat type"); log_value = approx_log2(value); if (log_value > concern_stat_thresholds[stat_id].threshold) +@@ builtin/repo.c: static void stats_table_print_structure(const struct stats_table *table, + strbuf_utf8_align(&buf, ALIGN_LEFT, concern_col_width, concern_str.buf); + strbuf_addstr(&buf, " |"); + } ++ strbuf_release(&concern_str); + printf("%s\n", buf.buf); + } + @@ builtin/repo.c: static void structure_keyvalue_print(struct repo_structure *stats, print_keyvalue("objects.blobs.disk_size", key_delim, stats->objects.disk_sizes.blobs, value_delim); @@ builtin/repo.c: static void structure_keyvalue_print(struct repo_structure *stat repo_structure_concern_level(CONCERN_STAT_BLOB_SIZE, stats->objects.disk_sizes.blobs), value_delim); +@@ builtin/repo.c: static void structure_keyvalue_print(struct repo_structure *stats, + stats->objects.largest.commit_size.value), + value_delim); + print_object_data("objects.trees.max_size", key_delim, +- &stats->objects.largest.tree_size, value_delim); ++ &stats->objects.largest.tree_size, value_delim); + if (show_concern) + print_keyvalue("objects.trees.max_size.concern_level", key_delim, + repo_structure_concern_level(CONCERN_STAT_TREE_LARGEST, +@@ builtin/repo.c: static void structure_keyvalue_print(struct repo_structure *stats, + stats->objects.largest.blob_size.value), + value_delim); + print_object_data("objects.tags.max_size", key_delim, +- &stats->objects.largest.tag_size, value_delim); ++ &stats->objects.largest.tag_size, value_delim); + if (show_concern) + print_keyvalue("objects.tags.max_size.concern_level", key_delim, + repo_structure_concern_level(CONCERN_STAT_TAG_LARGEST, @@ builtin/repo.c: static int cmd_repo_structure(int argc, const char **argv, const char *prefix, argc = parse_options(argc, argv, prefix, options, repo_structure_usage, 0); if (argc) @@ t/t1901-repo-structure.sh: test_expect_success 'empty repository' ' + git init repo && + ( + cd repo && ++ # differences between zlib and zlib-ng can cause this ++ # test to fail; so we just turn it off. ++ git config core.compression 0 && + # configure thresholds to get concern indications -+ git config repo.structure.concern.commit-count 7 && -+ git config repo.structure.concern.commit-size 14 && -+ git config repo.structure.concern.tree-size 10 && -+ git config repo.structure.concern.blob-size 8 && -+ git config repo.structure.concern.blob-inflated-size 10 && -+ git config repo.structure.concern.tag-size 2 && -+ git config repo.structure.concern.commit-largest 4 && -+ git config repo.structure.concern.blob-largest 2 && -+ touch configured && ++ git config repo.concern.commit-count 7 && ++ git config repo.concern.commit-size 14 && ++ git config repo.concern.tree-size 10 && ++ git config repo.concern.blob-size 8 && ++ git config repo.concern.blob-inflated-size 10 && ++ git config repo.concern.tag-size 2 && ++ git config repo.concern.commit-largest 4 && ++ git config repo.concern.blob-largest 2 && + test_commit_bulk 1005 && + git tag -a foo -m bar && + oid="$(git rev-parse HEAD)" && @@ t/t1901-repo-structure.sh: test_expect_success 'empty repository' ' + | * Commits | $(object_type_disk_usage commit true) | *** | + | * Trees | $(object_type_disk_usage tree true) | ******** | + | * Blobs | $(object_type_disk_usage blob true) | ****** | -+ | * Tags | $(object_type_disk_usage tag) B | **** | ++ | * Tags | $(object_type_disk_usage tag) B | ***** | + | | | | + | * Largest objects | | | + | * Commits | | | @@ t/t1901-repo-structure.sh: test_expect_success 'empty repository' ' test_expect_success SHA1 'lines and nul format' ' test_when_finished "rm -rf repo" && git init repo && +@@ t/t1901-repo-structure.sh: test_expect_success SHA1 'lines format with concerns' ' + git init repo && + ( + cd repo && ++ # differences between zlib and zlib-ng can cause this ++ # test to fail; so we just turn it off. ++ git config core.compression 0 && + # configure thresholds to get concern indications +- git config repo.structure.concern.commit-count 7 && +- git config repo.structure.concern.commit-size 14 && +- git config repo.structure.concern.tree-size 10 && +- git config repo.structure.concern.blob-size 8 && +- git config repo.structure.concern.blob-inflated-size 10 && +- git config repo.structure.concern.tag-size 2 && +- git config repo.structure.concern.commit-largest 4 && +- git config repo.structure.concern.blob-largest 2 && +- touch configured && ++ git config repo.concern.commit-count 7 && ++ git config repo.concern.commit-size 14 && ++ git config repo.concern.tree-size 10 && ++ git config repo.concern.blob-size 8 && ++ git config repo.concern.blob-inflated-size 10 && ++ git config repo.concern.tag-size 2 && ++ git config repo.concern.commit-largest 4 && ++ git config repo.concern.blob-largest 2 && + test_commit_bulk 1005 && + git tag -a foo -m bar && + oid="$(git rev-parse HEAD)" && +@@ t/t1901-repo-structure.sh: test_expect_success SHA1 'lines format with concerns' ' + # Also creates a commit, tree, and blob. + git notes add -m foo && + +- + # The tags disk size is handled specially due to the + # git-rev-list(1) --disk-usage=human option printing the full + # "byte/bytes" unit string instead of just "B". @@ t/t1901-repo-structure.sh: test_expect_success SHA1 'lines format with concerns' ' objects.tags.inflated_size.concern_level=0 objects.commits.disk_size.concern_level=3 objects.trees.disk_size.concern_level=8 - objects.blobs.blob_size.concern_level=6 +- objects.tags.disk_size.concern_level=4 + objects.blobs.disk_size.concern_level=6 - objects.tags.disk_size.concern_level=4 ++ objects.tags.disk_size.concern_level=5 objects.commits.max_size.concern_level=3 objects.trees.max_size.concern_level=0 + objects.blobs.max_size.concern_level=1 3: ac0adb7c9f ! 3: cc78eaeca0 repo: add documentation for level-of-concern. @@ Commit message ## Documentation/config/repo.adoc (new) ## @@ -+`repo.structure.concern.commit-count`:: ++`repo.concern.commit-count`:: + Sets the value of the concern threshold for commit count. + -+`repo.structure.concern.tree-count`:: ++`repo.concern.tree-count`:: + Sets the value of the concern threshold for tree count. + -+`repo.structure.concern.blob-count`:: ++`repo.concern.blob-count`:: + Sets the value of the concern threshold for blob count. + -+`repo.structure.concern.tag-count`:: ++`repo.concern.tag-count`:: + Sets the value of the concern threshold for tag count. + -+`repo.structure.concern.commit-size`:: ++`repo.concern.commit-size`:: + Sets the value of the concern threshold for total commit size. + -+`repo.structure.concern.tree-size`:: ++`repo.concern.tree-size`:: + Sets the value of the concern threshold for total tree size. + -+`repo.structure.concern.blob-size`:: ++`repo.concern.blob-size`:: + Sets the value of the concern threshold for total blob size. + -+`repo.structure.concern.tag-size`:: ++`repo.concern.tag-size`:: + Sets the value of the concern threshold for total tag size. + -+`repo.structure.concern.commit-inflated-size`:: ++`repo.concern.commit-inflated-size`:: + Sets the value of the concern threshold for total size + of all inflated commits. + -+`repo.structure.concern.tree-inflated-size`:: ++`repo.concern.tree-inflated-size`:: + Sets the value of the concern threshold for total size of + all inflated trees. + -+`repo.structure.concern.blob-inflated-size`:: ++`repo.concern.blob-inflated-size`:: + Sets the value of the concern threshold for total size of + all inflated blobs. + -+`repo.structure.concern.tag-inflated-size`:: ++`repo.concern.tag-inflated-size`:: + Sets the value of the concern threshold for the total size + of all inflated tags. + -+`repo.structure.concern.commit-largest`:: ++`repo.concern.commit-largest`:: + Sets the value of the concern threshold the size of the largest + commit. + -+`repo.structure.concern.tree-largest`:: ++`repo.concern.tree-largest`:: + Sets the value of the concern threshold for the size of the + largest tree. + -+`repo.structure.concern.blob-largest`:: ++`repo.concern.blob-largest`:: + Sets the value of the concern threshold for the size of the + largest blob. + -+`repo.structure.concern.tag-largest`:: ++`repo.concern.tag-largest`:: + Sets the value of the concern threshold for the size of the + largest tag. -++ -+All of the concern thresholds are integer values for the ++ ++All of the concern thresholds are integer values for the +floor of the base-2 log of the threshold value. The default values +of each threshold are set to heuristically determined values. -+ ## Documentation/git-repo.adoc ## @@ Documentation/git-repo.adoc: SYNOPSIS @@ Documentation/git-repo.adoc: supported: +users should be concerned about, adding '--concern' will provide a +level of concern to be added to the output. In 'table' mode, this will +be a table column indicating the level of concern as a number of stars; -+in 'lines` or `nul' mode, a `concern_level` line will be added for each ++in `lines` or `nul` mode, a `concern_level` line will be added for each +statistic with a numeric value. ++ -+The level of corcern for each statistic is computed using the base_2 log of the ++The level of concern for each statistic is computed using the log_2 of the +statistic value. The value is considered concerning if it exceeds a heuristically +chosen threshold value; the degree of concern is the difference between the -+base_2 log of the value, and the base_2 log of the corresponding threshold. ++log_2 of the value, and the log_2 of the corresponding threshold. ++ +The threshold values are configurable; see the configuration section below. --- base-commit: 3bc0341126508f78f5869cbfc0005e987efdf0c7 change-id: 20261019-b4-repo-structure-concern-f1d6f6326095