From: Taylor Blau Date: Sun, 01 Sep 2024 16:03:15 GMT Subject: [PATCH 0/4] hash.h: support choosing a separate SHA-1 for non-cryptographic uses Message-ID: This series adds a build-time knob to allow selecting an alternative SHA-1 implementation for non-cryptographic hashing within Git, starting with the `hashwrite()` family of functions. This series is the result of starting to roll out verbatim multi-pack reuse within GitHub's infrastructure. I noticed that on larger repositories, it is harder thus far to measure a CPU speed-up on clones where multi-pack reuse is enabled. After some profiling, I noticed that we spend a significant amount of time in hashwrite(), which is not all that surprising. But much of that time is wasted in GitHub's infrastructure, since we are using the same collision-detecting SHA-1 implementation to produce a trailing checksum for the pack which does not need to be cryptographically secure. This series teaches a new set of build-time knobs: OPENSSL_SHA1_FAST, BLK_SHA1_FAST, and APPLE_COMMON_CRYPTO_SHA1_FAST, which can be used to select an alterantive SHA-1 implementation for non-cryptographic uses within Git. The series is laid out as follows: - The first two patches are preparatory, allowing us to include multiple SHA-1 wrapper headers and adding scaffolding functions for the _fast() variants, respectively. - The third patch introduces the build-time knobs for selecting which SHA-1 implementation is used for non-cryptographic purposes. - The fourth and final patch updates the hashwrite() implementation to use the _fast() variants. From the final commit, a clone of the kernel which took 19.219 seconds on my machine with no build-time modifications now only takes 11.597 seconds when compiling with OPENSSL_SHA1_FAST=1, for a ~39.7% speed-up. On a freshly-repacked copy of git.git, the time goes from 758ms to 333ms for a ~56% speed-up. Thanks in advance for your review! Taylor Blau (4): sha1: do not redefine `platform_SHA_CTX` and friends hash.h: scaffolding for _fast hashing variants Makefile: allow specifying a SHA-1 for non-cryptographic uses csum-file.c: use fast SHA-1 implementation when available Makefile | 25 ++++++++++++++++++ block-sha1/sha1.h | 2 ++ csum-file.c | 18 ++++++------- hash.h | 67 +++++++++++++++++++++++++++++++++++++++++++++++ object-file.c | 42 +++++++++++++++++++++++++++++ sha1/openssl.h | 2 ++ sha1dc_git.h | 3 +++ 7 files changed, 150 insertions(+), 9 deletions(-) base-commit: 159f2d50e75c17382c9f4eb7cbda671a6fa612d1 -- 2.46.0.425.ge8f5cbd280c