[RFC PATCH v7 0/10] diff: add provider interface and initial providers

Every in-process diff in Git reduces, at one point, to a single question: given two blobs and the settings the diff runs under, which line ranges changed? The answer is the diff's hunks: for each change, the position and length of the range on the old side and on the new. Each consumer asks in its own shape:

- blame diffs each suspect's blob against its parent's, taking only the
  coordinates through xdiff's hunk callback;
- the stat formats keep only the added and deleted counts;
- patch output emits from the hunks, with xdiff interleaving context and
  content around them;
- log -L maps the tracked range across each commit from the coordinates.

In every case the answer is computed the same way: load both blobs and run xdiff. That is the only source, so nothing that already holds the answer, or that would answer differently on purpose, can supply it instead. Sometimes that is what we want, which is why patch-id and format-patch stay on the builtin computation throughout: patch-id needs identical hashes on every machine, and a format-patch must apply for recipients who share none of the sender's configuration. Other times another source would be useful.

This RFC sketches a direction. The unified series shows one interface carrying two example providers and their interaction; it is not shaped to merge as one topic. If the direction holds, the work returns as separate reviewable series (see Roadmap). The two examples are demonstrations, each an RFC on its own: diff.<driver>.process, the RFC cooking as mm/diff-process-hunks, lets a configured external process answer with its own notion of which lines changed, and the diff-hunks store, new in this thread, remembers what xdiff computed and serves it back. One is authoritative and external, one a cache and in-process.

Three pieces:

- A hunk provider interface (diff-provider.h) is the point of the
  series.  A provider is an alternate source for the answer: asked with
  the pair's object ids and the diff settings, before any blob is
  loaded, it may supply the hunks in place of the builtin computation.
  A miss falls through to that computation, and every answer passes one
  shared validity check first.  The providers form a chain the
  repository owns, built on first consultation and released in
  repo_clear(), so provider state such as a running process never
  outlives its repository.  Chain order is the authority, and the
  terminal provider is the builtin computation itself, so the interface
  never exists without an implementor: patch 02 ships it answering every
  request the way the consumers did before.  A consumer states its
  request in one struct and reads one set of outcomes (answered,
  unanswered, or failed); it never names a provider, and a provider
  added later maps onto those outcomes inside the interface, so consumer
  code is written once.  Because every diff now walks the chain even
  with no store or process configured, the default path was measured
  against the pre-series base and runs within noise (a 5000-commit
  log --stat and a long-history blame, ratio 1.00 either way).

- The diff-hunks store shows the non-authoritative side: an in-process
  cache at $GIT_DIR/objects/info/diff-hunks that may only reproduce the
  builtin diff, so serving from it never changes a command's output.  It
  is read by default and written only when a repository owner opts in,
  warming it as a side effect of diff work the command already does:

      GIT_DIFF_HUNKS_WRITE=1 git log --all --stat >/dev/null

  A warmed store then serves the stat formats and blame from stored
  coordinates instead of a fresh diff: on git.git a 5000-commit log
  --stat runs about 1.9x faster, and blame reads the same entries
  opportunistically (full numbers in [1]).  Its format and keying, what
  it may not serve, and how it handles corruption and staleness are in
  git-diff-hunks(1), gitformat-diff-hunks(5), and [2].  The interface
  point is small: a cache drops in as the provider that stands aside
  wherever an authoritative one answers.

- diff.<driver>.process shows the authoritative side: an external
  process, configured per driver, whose answers may deliberately differ
  from the builtin diff and outrank the store.  Git asks it for a pair by
  object names alone, so it answers before any blob is read, which suits
  a cache or a process that fetches the blobs itself.  Consulting is
  opt-in per command, following the allow_textconv precedent, and a pair
  the process cannot answer falls back to the builtin diff.  The
  protocol, the per-command gate, how failures are handled, and the
  versioning that lets it grow are in gitattributes(5) and footnotes [3]
  and [4].  The interface point, again, is small: an external,
  authoritative provider joins the same chain ahead of the cache, and
  neither consumer learns it is there.  A later content-carrying request
  would extend it to the pairs and consumers this identity-only form
  leaves on the builtin diff.

The series stops at the coordinates. A consumer that needs the changed text, such as patch output, would have only its hunk selection replaced, with xdiff still emitting content from the blobs; that machinery is the content enrichment sketched in the Roadmap. Establishing the framework on coordinates first keeps this series one design: the question, the interface, and two providers answering by identity.

Shape of the series:

  01     documentation: how external diff drivers relate to the
         features layered on the diff
  02     the provider interface: the request and outcome types, the
         emit entry point, the shared validity check, and the
         repository-owned chain with its terminal builtin provider
  03     the store: on-disk format, library, and the diff-hunks command
  04     recording: the stat walk computes, sums, and records
         trim-stable pairs (writes gated off by default)
  05     reading: the consult entry point and the store's registration
         as a provider; the request gains the object ids and diff
         options
  06     blame reading through the interface's emit path
  07-09  process preparation: sub-process lifecycle split, a gentle
         status read for an optional process, and the
         diff.<driver>.process config
  10     the process provider, oid-only, at the head of the chain, with
         the per-command gate; the request gains the path

Roadmap:

This RFC asks whether the direction is right, not for these ten patches to merge as one topic. If it holds, the work returns in reviewable pieces:

- the interface and the store (patches 01 through 06): a cache with
  measured numbers and no external-process machinery

- the process provider (patches 07 through 10) on the same interface

- the content enrichment (the content-carrying request, patch output and
  log -L consulting, and the xdiff machinery that feeds a provider's
  hunks into emission) once the identity-keyed framework settles.

Several design questions are left for those series.

mm/diff-process-hunks in seen would be dropped in favor of this thread and its split.

The series applies on the line-log topic (mm/line-log-limited-ops) rebased onto current master. The topic rewrites the same builtin_diffstat() region this series touches, and current master includes 061a68e443 (sub-process: use gentle handshake to avoid die() on startup failure), which this topic leans on: a process that dies during the handshake degrades to the builtin diff like every other failure. A trial merge against seen shows no interaction with other topics beyond the mm/diff-process-hunks replacement above.

The base (line-log topic on current master) and the full series are available at:

  git fetch https://github.com/mmontalbo/git mm/line-log-stat-formats-followup
  git fetch https://github.com/mmontalbo/git mm/hunk-providers-oid-first

Changes since v6:

This is a restructuring, not an incremental reroll, so a range-diff against v6 is unreadable; the map of what changed:

- The series now leads with the hunk provider interface and brings the
  diff-hunks store in as its in-process implementation (patches 02
  through 06, new to this thread).  It keeps only the identity-keyed
  half of the external diff process protocol from mm/diff-process-hunks.
- v6's gitattributes documentation, sub-process split, and userdiff
  config return close to their v6 form as patches 01, 07, and 09.  Patch
  08 is new: a gentle status read so a protocol error in an optional
  process degrades to the builtin diff instead of dying.
- v6's protocol patch returns as patch 10, reduced to the oid-only
  request, consulting through the interface, and carrying a per-command
  gate (v6's bypass patch folds into it).
- v6's blame and stat consults return as identity-keyed consults
  (patches 05, 06, and 10); their content legs, along with v6's xdiff
  external-hunks machinery, content-carrying request, and line-log
  consult, are withheld for the content enrichment.

Footnotes:

[1] Store numbers, measured with hyperfine against the same build with
    core.diffHunks=false.  The warm is a full cold build of the store;
    the blame speedup is file-dependent (see the coverage limitation):

    git.git (82,912 commits, --all)
      warm log --all --stat      20.9 s        store 28 MB, verify 39 ms
      log --stat -5000           1.91x    (1.38 s -> 0.72 s)
      blame diff.c               1.26x     (509 ms -> 403 ms)
      blame hit rate             54% (896 of 1653 pairs)

    linux (1,445,548 commits, --all)
      warm log --all --stat      714 s       store 298 MB, verify 415 ms
      log --stat -5000           1.43x    (2.29 s -> 1.60 s)
      blame kernel/sched/core.c  1.43x     (1.59 s -> 1.11 s)
      blame hit rate             74% (2414 of 3263 pairs)

[2] The store is its own file because nothing existing is addressed by a
    blob pair: notes attach to single objects, commit-graph chunks to
    commits.  One entry per pair, keyed by (old blob, new blob,
    xdl_opts) and recorded only when the pair's trimmed and untrimmed
    diffs agree, serves blame at zero context and the stat formats at
    any -U (divergent pairs are 0.4-0.5% of a warm and always compute).
    The writer fsyncs and commits atomically, and a reader bounds-checks
    every record and treats an unparsable file as absent; the trailing
    checksum is checked by git diff-hunks verify, not on every read, the
    same read-time trust the commit-graph and multi-pack-index take.
    There is deliberately no fsck integration, expiry, or background
    maintenance: the store is derivable at any time, so the recovery
    path is git diff-hunks clear and a re-warm.  New commits make it
    incomplete, not wrong; a later warm seeds from the file and pays
    only for what is new.

[3] Consulting the process is allowed per command, like textconv: git
    diff, git log and git show, and git blame consult it; the plumbing
    diff commands do not unless --ext-diff or --diff-process is given,
    and the interactive-patch machinery, format-patch, and range-diff
    stay builtin.  Options the process is never told about select no
    process, and an object id is sent only when it names the exact bytes
    diffed (a pair under an active object replacement is not sent).  The
    command comes from local configuration, as with filter.<name>
    .process: attributes select only a driver name, so cloning cannot
    cause a process to run.  gitattributes(5) has the full gate.

[4] The protocol is versioned and capability-negotiated, and extends
    without breaking deployed processes: a process ignores request keys
    it does not know, Git ignores trailing tokens on a hunk line so
    fields can be appended, and new request forms arrive as capabilities
    a process may decline.  Announcing a capability Git did not request
    aborts the command, the filter protocol's handshake rule.  The
    content-carrying request is the natural first extension; markers for
    formatting-only changes and function or token boundaries are
    candidates beyond it.

Michael Montalbo (10):
  gitattributes: document how external diff drivers relate to diff
    features
  diff: introduce a hunk provider interface
  diff-hunks: add the store format, library, and command
  diff: record precomputed hunks during stat output
  diff: read precomputed hunks for stat output
  blame: read precomputed hunks
  sub-process: separate process lifecycle from hashmap management
  sub-process: add a gentle status read
  userdiff: add diff.<driver>.process config
  diff: consult oid-only hunk providers via diff.<driver>.process

 .gitignore                               |    1 +
 Documentation/Makefile                   |    1 +
 Documentation/config.adoc                |    2 +
 Documentation/config/core.adoc           |   10 +-
 Documentation/config/diff-hunks.adoc     |    8 +
 Documentation/config/diff.adoc           |    6 +
 Documentation/diff-algorithm-option.adoc |    3 +
 Documentation/diff-options.adoc          |   15 +-
 Documentation/git-diff-hunks.adoc        |  146 +++
 Documentation/gitattributes.adoc         |  171 ++++
 Documentation/gitformat-diff-hunks.adoc  |  129 +++
 Documentation/meson.build                |    2 +
 Makefile                                 |    5 +
 blame.c                                  |   81 +-
 builtin.h                                |    1 +
 builtin/blame.c                          |    9 +-
 builtin/diff-hunks.c                     |   53 ++
 builtin/diff-tree.c                      |    3 +
 builtin/diff.c                           |   11 +
 builtin/log.c                            |   19 +
 chunk-format.c                           |   62 +-
 chunk-format.h                           |   14 +
 command-list.txt                         |    2 +
 diff-hunks.c                             | 1034 ++++++++++++++++++++++
 diff-hunks.h                             |  141 +++
 diff-process.c                           |  669 ++++++++++++++
 diff-provider-internal.h                 |  130 +++
 diff-provider.c                          |  190 ++++
 diff-provider.h                          |  159 ++++
 diff.c                                   |  294 +++++-
 diff.h                                   |   47 +
 environment.c                            |    1 +
 git.c                                    |    1 +
 meson.build                              |    4 +
 odb.c                                    |    2 +
 odb.h                                    |    4 +
 range-diff.c                             |    6 +
 repo-settings.c                          |    1 +
 repo-settings.h                          |    1 +
 repository.c                             |    3 +
 repository.h                             |    8 +
 sub-process.c                            |   52 +-
 sub-process.h                            |   19 +-
 t/helper/meson.build                     |    1 +
 t/helper/test-diff-process-backend.c     |  349 ++++++++
 t/helper/test-tool.c                     |    1 +
 t/helper/test-tool.h                     |    1 +
 t/meson.build                            |    3 +
 t/perf/p4218-diff-hunks.sh               |   48 +
 t/t4080-diff-process.sh                  |  593 +++++++++++++
 t/t4220-diff-hunks.sh                    |  819 +++++++++++++++++
 t/t4220/README                           |   55 ++
 t/t4220/trim-divergent-new               |  319 +++++++
 t/t4220/trim-divergent-old               |  316 +++++++
 userdiff.c                               |    7 +
 userdiff.h                               |    2 +
 write-or-die.h                           |    7 +-
 xdiff-interface.h                        |   12 +
 58 files changed, 5989 insertions(+), 64 deletions(-)
 create mode 100644 Documentation/config/diff-hunks.adoc
 create mode 100644 Documentation/git-diff-hunks.adoc
 create mode 100644 Documentation/gitformat-diff-hunks.adoc
 create mode 100644 builtin/diff-hunks.c
 create mode 100644 diff-hunks.c
 create mode 100644 diff-hunks.h
 create mode 100644 diff-process.c
 create mode 100644 diff-provider-internal.h
 create mode 100644 diff-provider.c
 create mode 100644 diff-provider.h
 create mode 100644 t/helper/test-diff-process-backend.c
 create mode 100755 t/perf/p4218-diff-hunks.sh
 create mode 100755 t/t4080-diff-process.sh
 create mode 100755 t/t4220-diff-hunks.sh
 create mode 100644 t/t4220/README
 create mode 100644 t/t4220/trim-divergent-new
 create mode 100644 t/t4220/trim-divergent-old

base-commit: 9a0c4701dcd5725c4184599322b52933ff5005ca prerequisite-patch-id: 6270dea79c9f06530737cefa3e1a0a39a1be7877 prerequisite-patch-id: 46fcc16a7a2ed760a1134d2a92c87699f3ec7bdb prerequisite-patch-id: c1e3da243003d060e429bc2196ae02b3453f01f9 prerequisite-patch-id: 4ad4e273494d4e8503706c21bfdc90a5d7ce116a prerequisite-patch-id: f7fa1367756daafa83f4f030a5c7b6dc3dbb70d7 prerequisite-patch-id: 829e76c9fec655a07f9383086a35bff3290b1c74 prerequisite-patch-id: 5c5a0d61ae9b6d628d05f1eb5df046758f3111a8