Re: [PATCH] Put sha1dc on a diet

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mar 1, 2017, 23:38 UTC
Message-ID: <CA+55aFz4ixVKVURki8FeXjL5H51A_cQXsZpzKJ-N9n574Yy1rg@mail.gmail.com>
In-Reply-To: <20170301203427.e5xa5ej3czli7c3o@sigill.intra.peff.net>

On Wed, Mar 1, 2017 at 12:34 PM, Jeff King <peff@peff.net> wrote:

>
> I don't think that helps. The sha1 over the pack-file takes about 1.3s
> with openssl, and 5s with sha1dc. So we already know the increase there
> is only a few seconds, not a few minutes.

Yeah, I did a few statistics by adding just logging of "SHA1_Init()" calls. For that network clone situation, the call distribution is

      1        SHA1: Init at builtin/index-pack.c:326
 841228        SHA1: Init at builtin/index-pack.c:450
      2        SHA1: Init at csum-file.c:152
4415756        SHA1: Init at sha1_file.c:3218

(the line numbers are a bit off from 'pu', because I obviously have the logging code).

The big number (one for every object) is from write_sha1_file_prepare(), which we'd want to be the strong collision checking version because those are things we're about to create git objects out of. It's called from

 - hash_sha1_file() - doesn't actually write the object, but is used
to calculate the sha for incoming data after applying the delta, for
example.

 - write_sha1_file() - many uses, actually writes the object

 - hash_sha1_file_literally() - git hash-object

and that index-pack.c:450 is from unpack_entry_data() for the base non-delta objects (which should also be the strong kind).

So all of them should check against collision attacks, so none of them seem to be things you'd want to optimize away..

So I was wrong in thinking that there were a lot of unnecessary SHA1 calculations in that load. They all look like they should be done with the slower checking code.

Oh well.

                      Linus