Re: [PATCH] Put sha1dc on a diet

From: Linus Torvalds <torvalds@linux-foundation.org>
Date: Mar 2, 2017, 16:35 UTC
Message-ID: <CA+55aFzscLaviJac-SB65WFYViY=wyAF3EWOnhHSuzSuFLdPTA@mail.gmail.com>
In-Reply-To: <alpine.DEB.2.20.1703021539330.3767@virtualbox>

On Thu, Mar 2, 2017 at 6:45 AM, Johannes Schindelin <Johannes.Schindelin@gmx.de> wrote:

Show 5 quoted lines

>
> It would probably make sense to switch the index integrity check away from
> SHA-1 because we really only care about detecting bit flips there, and we
> have no need for the computational overhead of using a full-blown
> cryptographic hash for that purpose.

Which index do you actually see as being a problem, btw? The main file index (.git/index) or the pack-file indexes?

We definitely don't need the checking version of sha1 for either of those, but as Jeff already did the math, at least the pack-file index is almost negligible, because the pack-file operations that update it end up doing SHA1 over the objects - and the object SHA1 calculations are much bigger.

And I don't think we even check the pack-file index hashes except on fsck.

Now, if your _file_ index is 300-400MB (and I do think we check the SHA fingerprint on that even on just reading it - verify_hdr() in do_read_index()), then that's going to be a somewhat noticeable hit on every normal "git diff" etc.

But I'd have expected the stat() calls of all the files listed by that index to be the _much_ bigger problem in that case. Or do you just turn those off with assume-unchanged?

Yeah, those stat calls are threaded when preloading, but even so..

Anyway, the file index SHA1 checking could probably just be disabled entirely (with a config flag). It's a corruption check that simply isn't that important. So if that's your main SHA1 issue, that would be easy to fix.

Everything else - like pack-file generation etc for a big clone() may end up using a ton of SHA1 too, but the SHA1 costs all scale with the other costs that drown them out (ie zlib, network, etc).

I'd love to see a profile if you have one.

                      Linus