Re: SHA-256 transition

From: Jeff King <peff@peff.net>
Date: Jun 24, 2022, 10:52 UTC
Message-ID: <YrWXdNGZGN7gXL40@coredump.intra.peff.net>
In-Reply-To: <YrI9dvfoc5NYgVDq@tapette.crustytoothpaste.net>

On Wed, Jun 22, 2022 at 12:29:59AM +0000, brian m. carlson wrote:

Show 10 quoted lines

> > We've since migrated our default hash function from SHA-1 to SHA-1DC
> > (except on vanilla OSX, see [2]). It's a variant SHA-1 that detects the
> > SHAttered attack implemented by the same researchers. I'm not aware of a
> > current viable SHA-1 collision against the variant of SHA-1 that we
> > actually use these days.
> 
> That's true, but that still doesn't let you store the data.  There is
> some data that you can't store in a SHA-1 repository, and SHA-1DC is
> extremely slow.  Using SHA-256 can make things like indexing packs
> substantially faster.

I'm curious if you have numbers on this. I naively converted linux.git to sha256 by doing "fast-export | fast-import" (the latter in a sha256 repo, of course, and then both repacked with "-f --window=250" to get reasonable apples-to-apples packs).

Running "index-pack --verify" on the result takes about the same time (this is on an 8-core system, hence the real/user differences):

  [sha1dc]
  real	2m43.754s
  user	10m52.452s
  sys	0m36.745s

  [sha256]
  real	2m41.884s
  user	12m23.344s
  sys	0m35.222s

The sha256 repo actually has about 10% fewer objects (I didn't investigate, but this is perhaps due to cutting out tags and a few other things to convince fast-export to finish running). I'm not sure about the extra user time (multicore timings here are funny because of frequency scaling, so I think the "real" line is more interesting). So sha256 actually comes out a bit worse here. On the other hand, this is just using our blk_SHA256 implementation. There may be faster alternatives (including ones with hardware support).

I wouldn't be at all surprised if the difference isn't substantial in the long run, though. The repo is on the order of 100GB of object data. That's a lot to hash, but it's also just a lot to deal with at all (zlib inflating, applying deltas, etc).

Anyway, this is a pretty rough cut at an experiment. I was mostly curious if you had done something more advanced, and/or gotten different results.

-Peff