# SHA-accelerated Git 4 messages from 2019-06-02 to 2019-06-03. Participants: Jeffrey Walton, Michal Suchánek, Johannes Schindelin. Thread: https://gitlist.dev/t/51222 ## Jeffrey Walton, 2019-06-02 10:43 Subject: SHA-accelerated Git Message-ID: URL: https://gitlist.dev/e/CAH8yC8nMcVuLeLWrVumLaz2Pbi3GugzgY-Bw3SMipjJkaph85Q%40mail.gmail.com ``` Hi Everyone, I have a lot of experience with cutting in SHA acceleration. I have no experience with Git. If someone would setup a fork and provide a call like: // https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/cpufeature.h // Or whatever the project prefers has_x86_sha = cpu_feature_enabled(X86_FEATURE_SHA_NI); if (has_x86_sha) { sha1_hash_block(state, ptr, size); } And provide a stub: sha1_hash_block(uint32_t* state, uint8_t* ptr, size_t size) { } Then I would be happy to fill in the pieces. I can also help with ARMv8. In fact, if you want to use Andy Polyakov's asm from Cryptogams (https://www.openssl.org/~appro/cryptogams/), then I can help with that, too. Andy's skills are renowned and his code is used in the kernel. Jeff ``` ## Michal Suchánek, 2019-06-02 11:21 Subject: Re: SHA-accelerated Git Message-ID: <20190602132150.4c1b320f@naga.suse.cz> URL: https://gitlist.dev/e/20190602132150.4c1b320f%40naga.suse.cz In-Reply-To: ``` Hello, On Sun, 2 Jun 2019 06:43:07 -0400 Jeffrey Walton wrote: > Hi Everyone, > > I have a lot of experience with cutting in SHA acceleration. I have no > experience with Git. sha1 is dead. Git is planning to move away from it. It will not happen overnight but it probably does not make sense to accelerate it at this point. https://github.com/git/git/blob/next/Documentation/technical/hash-function-transition.txt > > If someone would setup a fork and provide a call like: > > // https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/cpufeature.h > // Or whatever the project prefers > has_x86_sha = cpu_feature_enabled(X86_FEATURE_SHA_NI); > > if (has_x86_sha) > { > sha1_hash_block(state, ptr, size); > } > > And provide a stub: > > sha1_hash_block(uint32_t* state, uint8_t* ptr, size_t size) > { > } > > Then I would be happy to fill in the pieces. > > I can also help with ARMv8. > > In fact, if you want to use Andy Polyakov's asm from Cryptogams > (https://www.openssl.org/~appro/cryptogams/), then I can help with > that, too. Andy's skills are renowned and his code is used in the > kernel. And buggy. I am not referring to the sha1 implementation in particular but to the hand-crafted assembly in general. Do you have some performance data that shows significant improvement of common tasks that makes maintaining this hackery worthwhile? Thanks Michal ``` ## Jeffrey Walton, 2019-06-02 11:34 Subject: Re: SHA-accelerated Git Message-ID: URL: https://gitlist.dev/e/CAH8yC8n8XN1wKiF4FgTm6xfnRHUDi%2B1%3DiuWXE8F91P%3D7mLNsLg%40mail.gmail.com In-Reply-To: <20190602132150.4c1b320f@naga.suse.cz> ``` On Sun, Jun 2, 2019 at 7:21 AM Michal Suchánek wrote: > > Hello, > > On Sun, 2 Jun 2019 06:43:07 -0400 > Jeffrey Walton wrote: > > > Hi Everyone, > > > > I have a lot of experience with cutting in SHA acceleration. I have no > > experience with Git. > > sha1 is dead. Git is planning to move away from it. It will not happen > overnight but it probably does not make sense to accelerate it at this > point. > > https://github.com/git/git/blob/next/Documentation/technical/hash-function-transition.txt SHA-1 has been dead for years. That has not stopped Git yet :) My guess is, Git will have to carry around a SHA-1 implementation for a decade or so for interop purposes. > > If someone would setup a fork and provide a call like: > > > > // https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/cpufeature.h > > // Or whatever the project prefers > > has_x86_sha = cpu_feature_enabled(X86_FEATURE_SHA_NI); > > > > if (has_x86_sha) > > { > > sha1_hash_block(state, ptr, size); > > } > > > > And provide a stub: > > > > sha1_hash_block(uint32_t* state, uint8_t* ptr, size_t size) > > { > > } > > > > Then I would be happy to fill in the pieces. > > > > I can also help with ARMv8. > > > > In fact, if you want to use Andy Polyakov's asm from Cryptogams > > (https://www.openssl.org/~appro/cryptogams/), then I can help with > > that, too. Andy's skills are renowned and his code is used in the > > kernel. > > And buggy. I am not referring to the sha1 implementation in particular > but to the hand-crafted assembly in general. > > Do you have some performance data that shows significant improvement of > common tasks that makes maintaining this hackery worthwhile? Well, considering Git carries along an SHA-1 implementation with undefined behavior, anything would be an improvement since it can't pass acceptance testing out of the box. And Git does that for a 0.01 to 0.03 cpb increase in spperformance. For SHA-1 intrinsics on x86, performance increases to about 1.7 cpb. For SHA-256 performance increases to 3.8 cpb. On ARMv8, performance will increase to about 2.0 cpb on A-53 cpu's, and about 1.5 to 1.0 cpb on A-57 cpus. Comparing against a standard C/C++ implementation, Git will see a 3x to 6x speedup depending on the platform and algorithm. Jeff ``` ## Johannes Schindelin, 2019-06-03 10:14 Subject: Re: SHA-accelerated Git Message-ID: URL: https://gitlist.dev/e/nycvar.QRO.7.76.6.1906031211470.48%40tvgsbejvaqbjf.bet In-Reply-To: ``` Hi Jeffrey, On Sun, 2 Jun 2019, Jeffrey Walton wrote: > On Sun, Jun 2, 2019 at 7:21 AM Michal Suchánek wrote: > > > > On Sun, 2 Jun 2019 06:43:07 -0400 > > Jeffrey Walton wrote: > > > > > I have a lot of experience with cutting in SHA acceleration. I have > > > no experience with Git. > > > > sha1 is dead. Git is planning to move away from it. It will not happen > > overnight but it probably does not make sense to accelerate it at this > > point. > > > > https://github.com/git/git/blob/next/Documentation/technical/hash-function-transition.txt > > SHA-1 has been dead for years. That has not stopped Git yet :) Even worse. Git has changed from an OpenSSL accelerated SHA-1 to a collision-detecting SHA-1 (sha1dc). As long as your accelerated SHA-1 cannot guarantee the same level of safeguards as sha1dc, there is a negligible likelihood that your patches will be accepted. > Comparing against a standard C/C++ implementation, Git will see a 3x to > 6x speedup depending on the platform and algorithm. Indeed, this reflects my findings. It's just that we struck the balance on the other side of performance, in favor of preventing possible attack vectors as best as we can. Ciao, Johannes ```