[PATCH v3 0/2] repo: add degree-of-concern reporting to structure

"git repo structure" computes a set of statistics that are useful for diagnosing performance issues in a git repository. But for a typical user who hasn't encountered any issues yet, it can be difficult to interpret the data - when should a value be considered concerning?

This patchset introduces a feature to "git repo structure" that reports on how concerning certain measures of storage structure are for a repository. In table mode, concern is displayed as a column containing a number of stars indicating the degree of concern; in line or nul mode, concerns are displayed as additionaL lines with a numeric value.

To select defaults for the thresholds where a value becomes concern, a (hopefully) representative selection of repositories were analyzed. Based on empirical exploration of the data, the measurement that seemed closest to accurately assessing when a repository's structure becomes concerning was based on the base 2 logarithm of the value. Across all of the sample repositories, for all of the measurements, the point at which things become problematic was when a base 2 logarithm of a value was more than 1 standard deviation from the mean of the log_2 values in the sample set. Based on this, the concern threshold was set for each of the measures.

While this seems like a reasonable approach for setting default thresholds, it's obviously still very ad-hoc. It's likely that over time, the default values will evolve; and in the short term, projects may want to set their own thresholds. So this patchset includes support for configuring the thresholds on a per-reposiry basis.

Overview of the change:

* The first commit adds the command-line option "--concern", and creates the basic implementation of concern reporting.
* The next commit adds configurability for the concern thresholds.

Signed-off-by: Mark C. Chu-Carroll <markchucarroll@fastmail.com>
---
Changes in v3:
- Merge the documentation commit backwards into the other two commits.
- Link to v2: https://patch.msgid.link/20261026-b4-repo-structure-concern-v2-0-e0df34228dc8@fastmail.com

Changes in v2:
- Fix memory leak in concern string generation.
- Fix CI/CD test issue involving zlib versions.
- Updating the naming of the config value prefix.
- Link to v1: https://patch.msgid.link/20261021-b4-repo-structure-concern-v1-0-e3010552292c@fastmail.com

---
Mark C. Chu-Carroll (2):
      repo: add level of concern reporting
      repo: make concern thresholds configurable

 Documentation/config/repo.adoc |  59 +++++++
 Documentation/git-repo.adoc    |  27 +++-
 builtin/repo.c                 | 343 ++++++++++++++++++++++++++++++++++++++---
 t/t1901-repo-structure.sh      | 137 ++++++++++++++++
 4 files changed, 543 insertions(+), 23 deletions(-)

Range-diff versus v2:

1:  b1ef56dd47 ! 1:  11c60f6d6d repo: add level of concern reporting
    @@ Commit message
         when a stat values has reached a point where it might be
         concerning, and indicate how severe the concern should be.
     
    + ## Documentation/git-repo.adoc ##
    +@@ Documentation/git-repo.adoc: SYNOPSIS
    + [synopsis]
    + git repo info [--format=(lines|nul) | -z] [--all | <key>...]
    + git repo info --keys [--format=(lines|nul) | -z]
    +-git repo structure [--format=(table|lines|nul) | -z]
    ++git repo structure [--format=(table|lines|nul) | -z] [--[no-]concern]
    + 
    + DESCRIPTION
    + -----------
    +@@ Documentation/git-repo.adoc: supported:
    + `nul`:::
    + 	Similar to `lines`, but using a _NUL_ character after each value.
    + 
    +-`structure [--format=(table|lines|nul) | -z]`::
    ++`structure [--format=(table|lines|nul) | -z] [--[no-]concern]`::
    + 	Retrieve statistics about the current repository structure. The
    + 	following kinds of information are reported:
    + +
    +@@ Documentation/git-repo.adoc: supported:
    + 	quoted.
    + +
    + `-z` is an alias for `--format=nul`.
    +++
    ++To make it easier to know when one of the values is something that
    ++users should be concerned about, adding '--concern' will provide a
    ++level of concern to be added to the output. In 'table' mode, this will
    ++be a table column indicating the level of concern as a number of stars;
    ++in `lines` or `nul` mode, a `concern_level` line will be added for each
    ++statistic with a numeric value.
    +++
    ++The level of concern for each statistic is computed using the log_2 of the
    ++statistic value. The value is considered concerning if it exceeds a heuristically
    ++chosen threshold value; the degree of concern is the difference between the
    ++log_2 of the value, and the log_2 of the corresponding threshold.
    + 
    + INFO KEYS
    + ---------
    +
      ## builtin/repo.c ##
     @@
      	"git repo info --keys [--format=(lines|nul) | -z]"
2:  446391b362 ! 2:  4eba5072b0 repo: make concern thresholds configurable
    @@ Commit message
         configure per-repo concern levels for "git repo structure"
         concern reporting.
     
    + ## Documentation/config/repo.adoc (new) ##
    +@@
    ++`repo.concern.commit-count`::
    ++	Sets the value of the concern threshold for commit count.
    ++
    ++`repo.concern.tree-count`::
    ++	Sets the value of the concern threshold for tree count.
    ++
    ++`repo.concern.blob-count`::
    ++	Sets the value of the concern threshold for blob count.
    ++
    ++`repo.concern.tag-count`::
    ++	Sets the value of the concern threshold for tag count.
    ++
    ++`repo.concern.commit-size`::
    ++	Sets the value of the concern threshold for total commit size.
    ++
    ++`repo.concern.tree-size`::
    ++	Sets the value of the concern threshold for total tree size.
    ++
    ++`repo.concern.blob-size`::
    ++	Sets the value of the concern threshold for total blob size.
    ++
    ++`repo.concern.tag-size`::
    ++	Sets the value of the concern threshold for total tag size.
    ++
    ++`repo.concern.commit-inflated-size`::
    ++	Sets the value of the concern threshold for total size
    ++	of all inflated commits.
    ++
    ++`repo.concern.tree-inflated-size`::
    ++	Sets the value of the concern threshold for total size of
    ++	all inflated trees.
    ++
    ++`repo.concern.blob-inflated-size`::
    ++	Sets the value of the concern threshold for total size of
    ++	all inflated blobs.
    ++
    ++`repo.concern.tag-inflated-size`::
    ++	Sets the value of the concern threshold for the total size
    ++	of all inflated tags.
    ++
    ++`repo.concern.commit-largest`::
    ++	Sets the value of the concern threshold the size of the largest
    ++	commit.
    ++
    ++`repo.concern.tree-largest`::
    ++	Sets the value of the concern threshold for the size of the
    ++	largest tree.
    ++
    ++`repo.concern.blob-largest`::
    ++	Sets the value of the concern threshold for the size of the
    ++	largest blob.
    ++
    ++`repo.concern.tag-largest`::
    ++	Sets the value of the concern threshold for the size of the
    ++	largest tag.
    ++
    ++All of the concern thresholds are integer values for the
    ++floor of the base-2 log of the threshold value. The default values
    ++of each threshold are set to heuristically determined values.
    +
    + ## Documentation/git-repo.adoc ##
    +@@ Documentation/git-repo.adoc: statistic value. The value is considered concerning if it exceeds a heuristicall
    + chosen threshold value; the degree of concern is the difference between the
    + log_2 of the value, and the log_2 of the corresponding threshold.
    + 
    ++The threshold values are configurable; see the configuration section below.
    ++
    + INFO KEYS
    + ---------
    + In order to obtain a set of values from `git repo info`, you should provide
    +@@ Documentation/git-repo.adoc: using the `nul` format:
    + git repo info --format=nul layout.bare layout.shallow
    + ------------
    + 
    ++CONFIGURATION
    ++-------------
    ++
    ++include::includes/cmd-config-section-all.adoc[]
    ++
    ++:git-repo: 1
    ++include::config/repo.adoc[]
    ++
    ++
    + SEE ALSO
    + --------
    + linkgit:git-rev-parse[1]
    +
      ## builtin/repo.c ##
     @@
      
3:  92f1fafbcd < -:  ---------- repo: add documentation for level-of-concern.

--- base-commit: 3bc0341126508f78f5869cbfc0005e987efdf0c7 change-id: 20261019-b4-repo-structure-concern-f1d6f6326095