{"thread":{"id":"51222","subject":"SHA-accelerated Git","startedAt":"2019-06-02T10:43:34Z","lastAt":"2019-06-03T10:14:17Z","messageCount":4,"participants":["Jeffrey Walton","Michal Suchánek","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"376542","messageId":"CAH8yC8nMcVuLeLWrVumLaz2Pbi3GugzgY-Bw3SMipjJkaph85Q@mail.gmail.com","threadId":"51222","inReplyTo":null,"subject":"SHA-accelerated Git","fromName":"Jeffrey Walton","fromEmail":"noloader@gmail.com","sentAt":"2019-06-02T10:43:07Z","receivedAt":"2019-06-02T10:43:34Z","isPatch":false,"sender":{"key":"noloader@gmail.com","avatar":null},"body":"Hi Everyone,\n\nI have a lot of experience with cutting in SHA acceleration. I have no\nexperience with Git.\n\nIf someone would setup a fork and provide a call like:\n\n // https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/cpufeature.h\n // Or whatever the project prefers\n has_x86_sha = cpu_feature_enabled(X86_FEATURE_SHA_NI);\n\n if (has_x86_sha)\n {\n sha1_hash_block(state, ptr, size);\n }\n\nAnd provide a stub:\n\n sha1_hash_block(uint32_t* state, uint8_t* ptr, size_t size)\n {\n }\n\nThen I would be happy to fill in the pieces.\n\nI can also help with ARMv8.\n\nIn fact, if you want to use Andy Polyakov's asm from Cryptogams\n(https://www.openssl.org/~appro/cryptogams/), then I can help with\nthat, too. Andy's skills are renowned and his code is used in the\nkernel.\n\nJeff\n"},{"id":"376545","messageId":"20190602132150.4c1b320f@naga.suse.cz","threadId":"51222","inReplyTo":"CAH8yC8nMcVuLeLWrVumLaz2Pbi3GugzgY-Bw3SMipjJkaph85Q@mail.gmail.com","subject":"Re: SHA-accelerated Git","fromName":"Michal Suchánek","fromEmail":"msuchanek@suse.de","sentAt":"2019-06-02T11:21:50Z","receivedAt":"2019-06-02T11:21:55Z","isPatch":false,"sender":{"key":"msuchanek@suse.de","avatar":"https://avatars.githubusercontent.com/u/787652?v=4"},"body":"Hello,\n\nOn Sun, 2 Jun 2019 06:43:07 -0400\nJeffrey Walton wrote:\n\n> Hi Everyone,\n> \n> I have a lot of experience with cutting in SHA acceleration. I have no\n> experience with Git.\n\nsha1 is dead. Git is planning to move away from it. It will not happen\novernight but it probably does not make sense to accelerate it at this\npoint.\n\nhttps://github.com/git/git/blob/next/Documentation/technical/hash-function-transition.txt\n\n\n> \n> If someone would setup a fork and provide a call like:\n> \n> // https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/cpufeature.h\n> // Or whatever the project prefers\n> has_x86_sha = cpu_feature_enabled(X86_FEATURE_SHA_NI);\n> \n> if (has_x86_sha)\n> {\n> sha1_hash_block(state, ptr, size);\n> }\n> \n> And provide a stub:\n> \n> sha1_hash_block(uint32_t* state, uint8_t* ptr, size_t size)\n> {\n> }\n> \n> Then I would be happy to fill in the pieces.\n> \n> I can also help with ARMv8.\n> \n> In fact, if you want to use Andy Polyakov's asm from Cryptogams\n> (https://www.openssl.org/~appro/cryptogams/), then I can help with\n> that, too. Andy's skills are renowned and his code is used in the\n> kernel.\n\nAnd buggy. I am not referring to the sha1 implementation in particular\nbut to the hand-crafted assembly in general.\n\nDo you have some performance data that shows significant improvement of\ncommon tasks that makes maintaining this hackery worthwhile?\n\nThanks\n\nMichal\n"},{"id":"376547","messageId":"CAH8yC8n8XN1wKiF4FgTm6xfnRHUDi+1=iuWXE8F91P=7mLNsLg@mail.gmail.com","threadId":"51222","inReplyTo":"20190602132150.4c1b320f@naga.suse.cz","subject":"Re: SHA-accelerated Git","fromName":"Jeffrey Walton","fromEmail":"noloader@gmail.com","sentAt":"2019-06-02T11:34:12Z","receivedAt":"2019-06-02T11:34:40Z","isPatch":false,"sender":{"key":"noloader@gmail.com","avatar":null},"body":"On Sun, Jun 2, 2019 at 7:21 AM Michal Suchánek wrote:\n>\n> Hello,\n>\n> On Sun, 2 Jun 2019 06:43:07 -0400\n> Jeffrey Walton wrote:\n>\n> > Hi Everyone,\n> >\n> > I have a lot of experience with cutting in SHA acceleration. I have no\n> > experience with Git.\n>\n> sha1 is dead. Git is planning to move away from it. It will not happen\n> overnight but it probably does not make sense to accelerate it at this\n> point.\n>\n> https://github.com/git/git/blob/next/Documentation/technical/hash-function-transition.txt\n\nSHA-1 has been dead for years. That has not stopped Git yet :)\n\nMy guess is, Git will have to carry around a SHA-1 implementation for\na decade or so for interop purposes.\n\n> > If someone would setup a fork and provide a call like:\n> >\n> > // https://github.com/torvalds/linux/blob/master/arch/x86/include/asm/cpufeature.h\n> > // Or whatever the project prefers\n> > has_x86_sha = cpu_feature_enabled(X86_FEATURE_SHA_NI);\n> >\n> > if (has_x86_sha)\n> > {\n> > sha1_hash_block(state, ptr, size);\n> > }\n> >\n> > And provide a stub:\n> >\n> > sha1_hash_block(uint32_t* state, uint8_t* ptr, size_t size)\n> > {\n> > }\n> >\n> > Then I would be happy to fill in the pieces.\n> >\n> > I can also help with ARMv8.\n> >\n> > In fact, if you want to use Andy Polyakov's asm from Cryptogams\n> > (https://www.openssl.org/~appro/cryptogams/), then I can help with\n> > that, too. Andy's skills are renowned and his code is used in the\n> > kernel.\n>\n> And buggy. I am not referring to the sha1 implementation in particular\n> but to the hand-crafted assembly in general.\n>\n> Do you have some performance data that shows significant improvement of\n> common tasks that makes maintaining this hackery worthwhile?\n\nWell, considering Git carries along an SHA-1 implementation with\nundefined behavior, anything would be an improvement since it can't\npass acceptance testing out of the box. And Git does that for a 0.01\nto 0.03 cpb increase in spperformance.\n\nFor SHA-1 intrinsics on x86, performance increases to about 1.7 cpb.\nFor SHA-256 performance increases to 3.8 cpb. On ARMv8, performance\nwill increase to about 2.0 cpb on A-53 cpu's, and about 1.5 to 1.0 cpb\non A-57 cpus.\n\nComparing against a standard C/C++ implementation, Git will see a 3x\nto 6x speedup depending on the platform and algorithm.\n\nJeff\n"},{"id":"376561","messageId":"nycvar.QRO.7.76.6.1906031211470.48@tvgsbejvaqbjf.bet","threadId":"51222","inReplyTo":"CAH8yC8n8XN1wKiF4FgTm6xfnRHUDi+1=iuWXE8F91P=7mLNsLg@mail.gmail.com","subject":"Re: SHA-accelerated Git","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2019-06-03T10:14:18Z","receivedAt":"2019-06-03T10:14:17Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jeffrey,\n\nOn Sun, 2 Jun 2019, Jeffrey Walton wrote:\n\n> On Sun, Jun 2, 2019 at 7:21 AM Michal Suchánek wrote:\n> >\n> > On Sun, 2 Jun 2019 06:43:07 -0400\n> > Jeffrey Walton wrote:\n> >\n> > > I have a lot of experience with cutting in SHA acceleration. I have\n> > > no experience with Git.\n> >\n> > sha1 is dead. Git is planning to move away from it. It will not happen\n> > overnight but it probably does not make sense to accelerate it at this\n> > point.\n> >\n> > https://github.com/git/git/blob/next/Documentation/technical/hash-function-transition.txt\n>\n> SHA-1 has been dead for years. That has not stopped Git yet :)\n\nEven worse. Git has changed from an OpenSSL accelerated SHA-1 to a\ncollision-detecting SHA-1 (sha1dc).\n\nAs long as your accelerated SHA-1 cannot guarantee the same level of\nsafeguards as sha1dc, there is a negligible likelihood that your patches\nwill be accepted.\n\n> Comparing against a standard C/C++ implementation, Git will see a 3x to\n> 6x speedup depending on the platform and algorithm.\n\nIndeed, this reflects my findings.\n\nIt's just that we struck the balance on the other side of performance, in\nfavor of preventing possible attack vectors as best as we can.\n\nCiao,\nJohannes\n"}]}