[PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses

This series adds a build-time knob to allow selecting an alternative SHA-1 implementation for non-cryptographic hashing within Git, starting with the `hashwrite()` family of functions.

This series is the result of starting to roll out verbatim multi-pack reuse within GitHub's infrastructure. I noticed that on larger repositories, it is harder thus far to measure a CPU speed-up on clones where multi-pack reuse is enabled.

After some profiling, I noticed that we spend a significant amount of time in hashwrite(), which is not all that surprising. But much of that time is wasted in GitHub's infrastructure, since we are using the same collision-detecting SHA-1 implementation to produce a trailing checksum for the pack which does not need to be cryptographically secure.

This series teaches a new set of build-time knobs: OPENSSL_SHA1_FAST, BLK_SHA1_FAST, and APPLE_COMMON_CRYPTO_SHA1_FAST, which can be used to select an alterantive SHA-1 implementation for non-cryptographic uses within Git.

The series is laid out as follows:

  - The first two patches are preparatory, allowing us to include
    multiple SHA-1 wrapper headers and adding scaffolding functions for
    the _fast() variants, respectively.

  - The third patch introduces the build-time knobs for selecting which
    SHA-1 implementation is used for non-cryptographic purposes.

  - The fourth and final patch updates the hashwrite() implementation to
    use the _fast() variants.

From the final commit, a clone of the kernel which took 19.219 seconds on my machine with no build-time modifications now only takes 11.597 seconds when compiling with OPENSSL_SHA1_FAST=1, for a ~39.7% speed-up. On a freshly-repacked copy of git.git, the time goes from 758ms to 333ms for a ~56% speed-up.

Thanks in advance for your review!

Taylor Blau (4):
  sha1: do not redefine `platform_SHA_CTX` and friends
  hash.h: scaffolding for _fast hashing variants
  Makefile: allow specifying a SHA-1 for non-cryptographic uses
  csum-file.c: use fast SHA-1 implementation when available

 Makefile          | 25 ++++++++++++++++++
 block-sha1/sha1.h |  2 ++
 csum-file.c       | 18 ++++++-------
 hash.h            | 67 +++++++++++++++++++++++++++++++++++++++++++++++
 object-file.c     | 42 +++++++++++++++++++++++++++++
 sha1/openssl.h    |  2 ++
 sha1dc_git.h      |  3 +++
 7 files changed, 150 insertions(+), 9 deletions(-)

base-commit: 159f2d50e75c17382c9f4eb7cbda671a6fa612d1