[RFC PATCH 0/4] sign a SHA-256 digest of the tree in commits and tags

I'm concerned about the ecosystem impact of moving the `git init` default hashing function to SHA-256 in 3.0. I have suggested that it may be more feasible with similar benefits to add the ability to inject an independently calculated and verifiable tree content sha into signed objects instead.

This RFC series is meant to demonstrate how this might work.

It adds the ability to directly rehash the full tree contents when signing a commit or tag with SHA-256 without the repository needing to be in the sha256 object format.

In this series "git tag -s --hash=sha256" and "git commit -S --hash=sha256" compute a SHA-256 digest over every file in the tree (and submodules) and put that additional hash in a header before signing:

  object 78bd45828aa36fbde3161f49da15272dff3d06f5
  type commit
  tag v1.0
  tagger A U Thor <author@example.com> 1790749714 +0200
  tree-sha256 775aff90d07c9a73f19ef83ab89bd8e95d1d835325d53cc57a2232b015783d89

  Release 1.0
  -----BEGIN SSH SIGNATURE-----

For a commit it goes after "committer", before "gpgsig". Setting gpg.treeHash=sha256 makes it the default for everything you sign.

The digest is SHA-256 over one record per file, sorted by path:

  <hex sha256 of content> SP <path> NUL

The file mode isn't included. Submodules are followed into their own repositories and contribute "<hex digest of their tree> SP <path>/ NUL", so the signature covers their contents too; if a submodule isn't available, we fail rather than sign something we can't vouch for.

Old versions of Git are fine with the new header: fsck ignores extra headers after "tagger" by default (and always for commits), and "git tag -v" and "git verify-commit" check the signature as before.

  - Patch 1 adds the digest, with a test-tool helper so it can be
    tested on its own.
  - Patches 2 and 3 add --hash to "git tag" and "git commit".
  - Patch 4 adds gpg.treeHash.

From a speed perspective, it's not fast but it's not slow. The default build on my M5 is 245ms for a git.git signed tag call, ~5s for the Linux tree. An accelerated OpenSSL build is 135ms for git.git, 1.8s for Linux.

However, this is single threaded. We could easily do parallel hashing which should make it many times faster - my previous tests in Rust on 18 threads on my M5 did git.git in 36ms and Linux tree in 0.5s (verified the same hash).

Not in this series, and what I'd like opinions on:

  - Any interest? Would the list find this approach a viable alternative
    to not switching the default hash function to sha-256 in 3.0? Not that
    it wouldn't be an available object format, but that it wouldn't need to
    be the default one.

  - Verification. "git tag -v" and "git verify-commit" don't recompute
    the digest yet. I'd like to agree on the format before adding that.

  - Excluding submodules. Large projects can have submodules that most
    people never check out, and they can't sign with --hash today. One
    option is an "excluded:<commit>" header that still covers the
    pinned commit but not its contents, with a header listing the
    excluded paths so that verification can report them.

  - Naming. The header is "tree-sha256", the option "--hash", and the
    config "gpg.treeHash". I'm not attached to any of them.

Scott Chacon (4):
  tree-sha256: hash the contents of a tree with SHA-256
  tag: add --hash=sha256 to sign a tree-sha256 header
  commit: add --hash=sha256 to sign a tree-sha256 header
  gpg: add gpg.treeHash to sign a tree-sha256 header by default

 Documentation/config/gpg.adoc |   6 +
 Documentation/git-commit.adoc |  11 +-
 Documentation/git-tag.adoc    |  11 +-
 Makefile                      |   2 +
 builtin/commit.c              |  46 ++++++-
 builtin/tag.c                 |  41 +++++-
 meson.build                   |   1 +
 t/helper/meson.build          |   1 +
 t/helper/test-tool.c          |   1 +
 t/helper/test-tool.h          |   1 +
 t/helper/test-tree-sha256.c   |  31 +++++
 t/meson.build                 |   2 +
 t/t1018-tree-sha256.sh        | 123 +++++++++++++++++
 t/t7032-tree-sha256-signed.sh | 169 +++++++++++++++++++++++
 tree-sha256.c                 | 247 ++++++++++++++++++++++++++++++++++
 tree-sha256.h                 |  36 +++++
 16 files changed, 720 insertions(+), 9 deletions(-)
 create mode 100644 t/helper/test-tree-sha256.c
 create mode 100755 t/t1018-tree-sha256.sh
 create mode 100755 t/t7032-tree-sha256-signed.sh
 create mode 100644 tree-sha256.c
 create mode 100644 tree-sha256.h

base-commit: a018953688f1b10bddf91bff8747068f5f4746a4