{"thread":{"id":"60356","subject":"[PATCH] Add x64 SHA-NI implementation to sha1 and sha256 in git ?","startedAt":"2023-10-12T20:05:06Z","lastAt":"2023-10-15T17:30:47Z","messageCount":5,"participants":["Feiyang Xue","brian m. carlson","Junio C Hamano","Beat Bolli","rsbecker@nexbridge.com"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"483169","messageId":"20231012200447.3553757-1-hi@fxue.dev","threadId":"60356","inReplyTo":null,"subject":"[PATCH] Add x64 SHA-NI implementation to sha1 and sha256 in git ?","fromName":"Feiyang Xue","fromEmail":"hi@fxue.dev","sentAt":"2023-10-12T20:04:47Z","receivedAt":"2023-10-12T20:05:06Z","isPatch":true,"sender":{"key":"hi@fxue.dev","avatar":null},"body":"Looking for comments on if this is a good idea to add SHA-NI option to git, \nAnd if so, what'd be a good implementation.\n\nThe attached patch is more of a proof of concept,  using a sha-ni example \nfound on internet and have it tapped into git's \"hash-ll.h\" when it sees make \nflags \"SHA1_SHANI=1\" and/or \"SHA256_SHANI=1\" for sha1/sha256 respectively.\n\n\"git hash-object\" is about 3-ish times faster for me\n\nThere are few topics that i'm uncertain about. \n\n1. Is it good idea to have machine-specific asm codes in git. I read there was \ncommit fd1ec82547 which removed assembly implementation for PPC_SHA1. Does that \nimply maintainers doesn't want machine-specific assembly in source?\n\n2. If it is cool to have machine-specific codes in git, what's a good logic for\ntheir on/off? Build time flag? Run time flag? Run time automatic detection to \ncorresponding instructions? \n\n3. If i can include machine-specific codes, what'd be a good way/idea to cleanup\nthe asm? I suppose git maintainers do not want someone else's license header?\n\nRegards,\nFeiyang\n\n\n---\n Makefile                                      |  28 ++\n hash-ll.h                                     |   8 +-\n .../intel_sha_extensions_sha1_assembly.asm    | 308 +++++++++++++++\n sha1-x64shani/sha1_x64.c                      |  71 ++++\n sha1-x64shani/sha1_x64.h                      |  21 +\n .../intel_sha_extensions_sha256_assembly.asm  | 371 ++++++++++++++++++\n sha256-x64shani/sha256_x64.c                  |  82 ++++\n sha256-x64shani/sha256_x64.h                  |  23 ++\n 8 files changed, 910 insertions(+), 2 deletions(-)\n create mode 100644 sha1-x64shani/intel_sha_extensions_sha1_assembly.asm\n create mode 100644 sha1-x64shani/sha1_x64.c\n create mode 100644 sha1-x64shani/sha1_x64.h\n create mode 100644 sha256-x64shani/intel_sha_extensions_sha256_assembly.asm\n create mode 100644 sha256-x64shani/sha256_x64.c\n create mode 100644 sha256-x64shani/sha256_x64.h\n\ndiff --git a/Makefile b/Makefile\nindex e440728c24..6892af2ee5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -1917,6 +1917,11 @@ ifdef PPC_SHA1\n $(error the PPC_SHA1 flag has been removed along with the PowerPC-specific SHA-1 implementation.)\n endif\n \n+ifdef SHA1_SHANI\n+\tLIB_OBJS += sha1-x64shani/sha1_x64.o\n+\tLIB_OBJS += sha1-x64shani/intel_sha_extensions_sha1_assembly.o\n+\tBASIC_CFLAGS += -DSHA1_SHANI\n+else\n ifdef OPENSSL_SHA1\n \tEXTLIBS += $(LIB_4_CRYPTO)\n \tBASIC_CFLAGS += -DSHA1_OPENSSL\n@@ -1957,7 +1962,13 @@ endif\n endif\n endif\n endif\n+endif\n \n+ifdef SHA256_SHANI\n+\tLIB_OBJS += sha256-x64shani/sha256_x64.o\n+\tLIB_OBJS += sha256-x64shani/intel_sha_extensions_sha256_assembly.o\n+\tBASIC_CFLAGS += -DSHA256_SHANI\n+else\n ifdef OPENSSL_SHA256\n \tEXTLIBS += $(LIB_4_CRYPTO)\n \tBASIC_CFLAGS += -DSHA256_OPENSSL\n@@ -1975,6 +1986,9 @@ else\n endif\n endif\n endif\n+endif\n+\n+$(info $$LIB_OBJS is [${LIB_OBJS}])\n \n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\n@@ -2711,6 +2725,20 @@ missing_compdb_dir =\n compdb_args =\n endif\n \n+$(info $$OBJECTS is [${OBJECTS}])\n+\n+### for SHA-NI \n+YASM_SRC := $(wildcard $(OBJECTS:o=asm))\n+YASM_OBJ := $(YASM_SRC:asm=o)\n+OBJECTS := $(filter-out $(YASM_OBJ),$(OBJECTS))\n+\n+$(info $$YASM_SRC is [${YASM_SRC}])\n+$(info $$YASM_OBJ is [${YASM_OBJ}])\n+$(info $$OBJECTS is [${OBJECTS}])\n+\n+$(YASM_OBJ): %.o: %.asm $(missing_dep_dirs)\n+\t$(QUIET_CC)yasm $< -f elf64 -X gnu -g dwarf2 -o $@\n+\n $(OBJECTS): %.o: %.c GIT-CFLAGS $(missing_dep_dirs) $(missing_compdb_dir)\n \t$(QUIET_CC)$(CC) -o $*.o -c $(dep_args) $(compdb_args) $(ALL_CFLAGS) $(EXTRA_CPPFLAGS) $<\n \ndiff --git a/hash-ll.h b/hash-ll.h\nindex 8050925137..31d9e91f26 100644\n--- a/hash-ll.h\n+++ b/hash-ll.h\n@@ -1,7 +1,9 @@\n #ifndef HASH_LL_H\n #define HASH_LL_H\n \n-#if defined(SHA1_APPLE)\n+#if defined(SHA1_SHANI)\n+#include \"sha1-x64shani/sha1_x64.h\"\n+#elif defined(SHA1_APPLE)\n #include <CommonCrypto/CommonDigest.h>\n #elif defined(SHA1_OPENSSL)\n #include <openssl/sha.h>\n@@ -11,7 +13,9 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n-#if defined(SHA256_NETTLE)\n+#if defined(SHA256_SHANI)\n+#include \"sha256-x64shani/sha256_x64.h\"\n+#elif defined(SHA256_NETTLE)\n #include \"sha256/nettle.h\"\n #elif defined(SHA256_GCRYPT)\n #define SHA256_NEEDS_CLONE_HELPER\ndiff --git a/sha1-x64shani/intel_sha_extensions_sha1_assembly.asm b/sha1-x64shani/intel_sha_extensions_sha1_assembly.asm\nnew file mode 100644\nindex 0000000000..ba59825966\n--- /dev/null\n+++ b/sha1-x64shani/intel_sha_extensions_sha1_assembly.asm\n@@ -0,0 +1,308 @@\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+; Copyright (c) 2013, Intel Corporation \n+; \n+; All rights reserved. \n+; \n+; Redistribution and use in source and binary forms, with or without\n+; modification, are permitted provided that the following conditions are\n+; met: \n+; \n+; * Redistributions of source code must retain the above copyright\n+;   notice, this list of conditions and the following disclaimer.  \n+; \n+; * Redistributions in binary form must reproduce the above copyright\n+;   notice, this list of conditions and the following disclaimer in the\n+;   documentation and/or other materials provided with the\n+;   distribution. \n+; \n+; * Neither the name of the Intel Corporation nor the names of its\n+;   contributors may be used to endorse or promote products derived from\n+;   this software without specific prior written permission. \n+; \n+; \n+; THIS SOFTWARE IS PROVIDED BY INTEL CORPORATION \"\"AS IS\"\" AND ANY\n+; EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n+; IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n+; PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL INTEL CORPORATION OR\n+; CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,\n+; EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\n+; PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR\n+; PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF\n+; LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING\n+; NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\n+; SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+;\n+; Intel SHA Extensions optimized implementation of a SHA-1 update function \n+;\n+; The function takes a pointer to the current hash values, a pointer to the \n+; input data, and a number of 64 byte blocks to process.  Once all blocks have \n+; been processed, the digest pointer is  updated with the resulting hash value.\n+; The function only processes complete blocks, there is no functionality to \n+; store partial blocks.  All message padding and hash value initialization must\n+; be done outside the update function.  \n+;\n+; The indented lines in the loop are instructions related to rounds processing.\n+; The non-indented lines are instructions related to the message schedule.\n+;\n+; Author: Sean Gulley <sean.m.gulley@intel.com>\n+; Date:   July 2013\n+;\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+;\n+; Example YASM command lines:\n+; Windows:  \n+;  yasm -Xvc -f x64 -rnasm -pnasm -DWINABI -o intel_sha_extensions_sha1_assembly.obj -g cv8 intel_sha_extensions_sha1_assembly.asm\n+; Linux:    \n+;  yasm -f elf64 -X gnu -g dwarf2 -o intel_sha_extensions_sha1_assembly.o intel_sha_extensions_sha1_assembly.asm\n+;\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+\n+%ifdef WINABI\n+%define DIGEST_PTR\trcx \t; 1st arg\n+%define DATA_PTR\trdx \t; 2nd arg\n+%define NUM_BLKS \tr8\t; 3rd arg\n+%else\n+%define DIGEST_PTR\trdi\t; 1st arg\n+%define DATA_PTR\trsi\t; 2nd arg\n+%define NUM_BLKS\trdx\t; 3rd arg\n+%endif\n+\n+%define RSPSAVE\t\trax\n+\n+struc frame\n+%ifdef WINABI\n+.xmm_save:\tresdq\t2\n+%endif\n+.hash_save:\tresdq\t2\n+endstruc\n+\n+%define ABCD\t\txmm0\n+%define E0\t\txmm1\t; Need two E's b/c they ping pong\n+%define E1\t\txmm2\n+%define MSG0\t\txmm3\n+%define MSG1\t\txmm4\n+%define MSG2\t\txmm5\n+%define MSG3\t\txmm6\n+%define SHUF_MASK\txmm7\n+\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+;; void sha1_update(uint32_t *digest, const void *data, uint32_t numBlocks);\n+;; arg 1 : pointer to digest\n+;; arg 2 : pointer to input data\n+;; arg 3 : Number of blocks to process\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+section .text\n+global sha1_update:function\n+align 32\n+sha1_update:\n+\tmov\t\tRSPSAVE, rsp\n+\tsub\t\trsp, frame_size\n+\tand\t\trsp, ~0xF\n+\n+%ifdef WINABI\n+\tmovdqa\t\t[rsp + frame.xmm_save + 0*16], xmm6\n+\tmovdqa\t\t[rsp + frame.xmm_save + 1*16], xmm7\n+%endif\n+\n+\tshl\t\tNUM_BLKS, 6\t\t; convert to bytes\n+\tjz\t\t.done_hash\n+\tadd\t\tNUM_BLKS, DATA_PTR\t; pointer to end of data\n+\n+\t;; load initial hash values\n+\tpinsrd\t\tE0,   [DIGEST_PTR + 1*16], 3\n+\tmovdqu\t\tABCD, [DIGEST_PTR + 0*16]\n+\tpand\t\tE0,   [UPPER_WORD_MASK wrt rip]\n+\tpshufd\t\tABCD, ABCD, 0x1B\n+\n+\tmovdqa\t\tSHUF_MASK, [PSHUFFLE_BYTE_FLIP_MASK wrt rip]\n+\n+.loop0:\n+\t;; Save hash values for addition after rounds\n+\tmovdqa\t\t[rsp + frame.hash_save + 0*16], E0\n+\tmovdqa\t\t[rsp + frame.hash_save + 1*16], ABCD\n+\n+\t;; Rounds 0-3\n+\tmovdqu\t\tMSG0, [DATA_PTR + 0*16]\n+\tpshufb\t\tMSG0, SHUF_MASK\n+\t\tpaddd\t\tE0, MSG0\n+\t\tmovdqa\t\tE1, ABCD\n+\t\tsha1rnds4\tABCD, E0, 0\n+\n+\t;; Rounds 4-7\n+\tmovdqu\t\tMSG1, [DATA_PTR + 1*16]\n+\tpshufb\t\tMSG1, SHUF_MASK\n+\t\tsha1nexte\tE1, MSG1\n+\t\tmovdqa\t\tE0, ABCD\n+\t\tsha1rnds4\tABCD, E1, 0\n+\tsha1msg1\tMSG0, MSG1\n+\n+\t;; Rounds 8-11\n+\tmovdqu\t\tMSG2, [DATA_PTR + 2*16]\n+\tpshufb\t\tMSG2, SHUF_MASK\n+\t\tsha1nexte\tE0, MSG2\n+\t\tmovdqa\t\tE1, ABCD\n+\t\tsha1rnds4\tABCD, E0, 0\n+\tsha1msg1\tMSG1, MSG2\n+\tpxor\t\tMSG0, MSG2\n+\n+\t;; Rounds 12-15\n+\tmovdqu\t\tMSG3, [DATA_PTR + 3*16]\n+\tpshufb\t\tMSG3, SHUF_MASK\n+\t\tsha1nexte\tE1, MSG3\n+\t\tmovdqa\t\tE0, ABCD\n+\tsha1msg2\tMSG0, MSG3\n+\t\tsha1rnds4\tABCD, E1, 0\n+\tsha1msg1\tMSG2, MSG3\n+\tpxor\t\tMSG1, MSG3\n+\n+\t;; Rounds 16-19\n+\t\tsha1nexte\tE0, MSG0\n+\t\tmovdqa\t\tE1, ABCD\n+\tsha1msg2\tMSG1, MSG0\n+\t\tsha1rnds4\tABCD, E0, 0\n+\tsha1msg1\tMSG3, MSG0\n+\tpxor\t\tMSG2, MSG0\n+\t\n+\t;; Rounds 20-23\n+\t\tsha1nexte\tE1, MSG1\n+\t\tmovdqa\t\tE0, ABCD\n+\tsha1msg2\tMSG2, MSG1\n+\t\tsha1rnds4\tABCD, E1, 1\n+\tsha1msg1\tMSG0, MSG1\n+\tpxor\t\tMSG3, MSG1\n+\t\n+\t;; Rounds 24-27\n+\t\tsha1nexte\tE0, MSG2\n+\t\tmovdqa\t\tE1, ABCD\n+\tsha1msg2\tMSG3, MSG2\n+\t\tsha1rnds4\tABCD, E0, 1\n+\tsha1msg1\tMSG1, MSG2\n+\tpxor\t\tMSG0, MSG2\n+\t\n+\t;; Rounds 28-31\n+\t\tsha1nexte\tE1, MSG3\n+\t\tmovdqa\t\tE0, ABCD\n+\tsha1msg2\tMSG0, MSG3\n+\t\tsha1rnds4\tABCD, E1, 1\n+\tsha1msg1\tMSG2, MSG3\n+\tpxor\t\tMSG1, MSG3\n+\n+\t;; Rounds 32-35\n+\t\tsha1nexte\tE0, MSG0\n+\t\tmovdqa\t\tE1, ABCD\n+\tsha1msg2\tMSG1, MSG0\n+\t\tsha1rnds4\tABCD, E0, 1\n+\tsha1msg1\tMSG3, MSG0\n+\tpxor\t\tMSG2, MSG0\n+\t\n+\t;; Rounds 36-39\n+\t\tsha1nexte\tE1, MSG1\n+\t\tmovdqa\t\tE0, ABCD\n+\tsha1msg2\tMSG2, MSG1\n+\t\tsha1rnds4\tABCD, E1, 1\n+\tsha1msg1\tMSG0, MSG1\n+\tpxor\t\tMSG3, MSG1\n+\t\n+\t;; Rounds 40-43\n+\t\tsha1nexte\tE0, MSG2\n+\t\tmovdqa\t\tE1, ABCD\n+\tsha1msg2\tMSG3, MSG2\n+\t\tsha1rnds4\tABCD, E0, 2\n+\tsha1msg1\tMSG1, MSG2\n+\tpxor\t\tMSG0, MSG2\n+\t\n+\t;; Rounds 44-47\n+\t\tsha1nexte\tE1, MSG3\n+\t\tmovdqa\t\tE0, ABCD\n+\tsha1msg2\tMSG0, MSG3\n+\t\tsha1rnds4\tABCD, E1, 2\n+\tsha1msg1\tMSG2, MSG3\n+\tpxor\t\tMSG1, MSG3\n+\n+\t;; Rounds 48-51\n+\t\tsha1nexte\tE0, MSG0\n+\t\tmovdqa\t\tE1, ABCD\n+\tsha1msg2\tMSG1, MSG0\n+\t\tsha1rnds4\tABCD, E0, 2\n+\tsha1msg1\tMSG3, MSG0\n+\tpxor\t\tMSG2, MSG0\n+\t\n+\t;; Rounds 52-55\n+\t\tsha1nexte\tE1, MSG1\n+\t\tmovdqa\t\tE0, ABCD\n+\tsha1msg2\tMSG2, MSG1\n+\t\tsha1rnds4\tABCD, E1, 2\n+\tsha1msg1\tMSG0, MSG1\n+\tpxor\t\tMSG3, MSG1\n+\t\n+\t;; Rounds 56-59\n+\t\tsha1nexte\tE0, MSG2\n+\t\tmovdqa\t\tE1, ABCD\n+\tsha1msg2\tMSG3, MSG2\n+\t\tsha1rnds4\tABCD, E0, 2\n+\tsha1msg1\tMSG1, MSG2\n+\tpxor\t\tMSG0, MSG2\n+\t\n+\t;; Rounds 60-63\n+\t\tsha1nexte\tE1, MSG3\n+\t\tmovdqa\t\tE0, ABCD\n+\tsha1msg2\tMSG0, MSG3\n+\t\tsha1rnds4\tABCD, E1, 3\n+\tsha1msg1\tMSG2, MSG3\n+\tpxor\t\tMSG1, MSG3\n+\n+\t;; Rounds 64-67\n+\t\tsha1nexte\tE0, MSG0\n+\t\tmovdqa\t\tE1, ABCD\n+\tsha1msg2\tMSG1, MSG0\n+\t\tsha1rnds4\tABCD, E0, 3\n+\tsha1msg1\tMSG3, MSG0\n+\tpxor\t\tMSG2, MSG0\n+\t\n+\t;; Rounds 68-71\n+\t\tsha1nexte\tE1, MSG1\n+\t\tmovdqa\t\tE0, ABCD\n+\tsha1msg2\tMSG2, MSG1\n+\t\tsha1rnds4\tABCD, E1, 3\n+\tpxor\t\tMSG3, MSG1\n+\t\n+\t;; Rounds 72-75\n+\t\tsha1nexte\tE0, MSG2\n+\t\tmovdqa\t\tE1, ABCD\n+\tsha1msg2\tMSG3, MSG2\n+\t\tsha1rnds4\tABCD, E0, 3\n+\t\n+\t;; Rounds 76-79\n+\t\tsha1nexte\tE1, MSG3\n+\t\tmovdqa\t\tE0, ABCD\n+\t\tsha1rnds4\tABCD, E1, 3\n+\n+\t;; Add current hash values with previously saved\n+\tsha1nexte\tE0,   [rsp + frame.hash_save + 0*16]\n+\tpaddd\t\tABCD, [rsp + frame.hash_save + 1*16]\n+\n+\t;; Increment data pointer and loop if more to process\n+\tadd\t\tDATA_PTR, 64\n+\tcmp\t\tDATA_PTR, NUM_BLKS\n+\tjne\t\t.loop0\n+\n+\t;; Write hash values back in the correct order\n+\tpshufd\t\tABCD, ABCD, 0x1B\n+\tmovdqu\t\t[DIGEST_PTR + 0*16], ABCD\n+\tpextrd\t\t[DIGEST_PTR + 1*16], E0, 3\n+\n+.done_hash:\n+%ifdef WINABI\n+\tmovdqa\t\txmm6, [rsp + frame.xmm_save + 0*16]\n+\tmovdqa\t\txmm7, [rsp + frame.xmm_save + 1*16]\n+%endif\n+\tmov\t\trsp, RSPSAVE\n+\n+\tret\t\n+\n+section .data\n+align 64\n+PSHUFFLE_BYTE_FLIP_MASK: ddq 0x000102030405060708090a0b0c0d0e0f\n+UPPER_WORD_MASK:         ddq 0xFFFFFFFF000000000000000000000000\n+\ndiff --git a/sha1-x64shani/sha1_x64.c b/sha1-x64shani/sha1_x64.c\nnew file mode 100644\nindex 0000000000..0ed0d49ed6\n--- /dev/null\n+++ b/sha1-x64shani/sha1_x64.c\n@@ -0,0 +1,71 @@\n+#include \"../git-compat-util.h\"\n+#include \"sha1_x64.h\"\n+#include <byteswap.h>\n+\n+// static inline void shani_SHA1_Init(shani_SHA_CTX *c);\n+\n+void shani_SHA1_Init(shani_SHA_CTX *c)\n+{\n+    static const uint32_t sha1InitialDigest[5] = {\n+           0x67452301, 0xefcdab89, 0x98badcfe, 0x10325476, 0xc3d2e1f0\n+    };\n+    memcpy(c->h, sha1InitialDigest, 20);\n+    c->len = 0;\n+    c->total_len = 0;\n+}\n+\n+void sha1_update(void *c, const void *d, int l);\n+void shani_SHA1_Update(shani_SHA_CTX *c, const void *data, unsigned long len)\n+{\n+    while (len > 0)\n+    {\n+        if (c->len == 0 && len >= 64)\n+        {\n+            sha1_update(c->h, data, len/64);\n+            unsigned long bytes_hashed = (len/64)*64;\n+            data += bytes_hashed;\n+            len -= bytes_hashed;\n+            c->total_len += bytes_hashed;\n+            continue;\n+        }\n+#define MIN(x, y) (((x) < (y)) ? (x) : (y))\n+        size_t cpy = MIN(64 - c->len, len);\n+#undef MIN\n+        memcpy(c->msgbuf + c->len, data, cpy);\n+        data += cpy;\n+        len -= cpy;\n+        c->len += cpy;\n+        c->total_len += cpy;\n+\n+        if (c->len == 64)\n+        {\n+            sha1_update(c->h, c->msgbuf, 1);\n+            c->len = 0;\n+        }\n+    }\n+}\n+\n+void shani_SHA1_Final(unsigned char *md, shani_SHA_CTX *c)\n+{\n+    c->msgbuf[c->len] = 0x80;\n+    ++c->len;\n+    memset(c->msgbuf + c->len, 0, 64 - c->len);\n+    if (c->len > (64-8))\n+    {\n+        sha1_update(c->h, c->msgbuf, 1);\n+        memset(c->msgbuf, 0, 64);\n+        c->len = 0;\n+    }\n+\n+    for (int ibyte = 0; ibyte < 8; ++ibyte)\n+       c->msgbuf[63-ibyte] = (uint8_t)((c->total_len * 8ULL) >> (ibyte*8));\n+\n+    sha1_update(c->h, c->msgbuf, 1);\n+    c->len = 0;    \n+\n+    // flip endianness\n+    int i;\n+    for (i = 0; i < 20; i += 4) {\n+        *(uint32_t *)(md + i) = __bswap_32(c->h[i / 4]);\n+    }\n+}\ndiff --git a/sha1-x64shani/sha1_x64.h b/sha1-x64shani/sha1_x64.h\nnew file mode 100644\nindex 0000000000..ca1ec116cf\n--- /dev/null\n+++ b/sha1-x64shani/sha1_x64.h\n@@ -0,0 +1,21 @@\n+/*\n+ * SHA1 routine using x64 SHA-NI instructions\n+ */\n+\n+typedef struct shani_SHA_CTX\n+{\n+    uint32_t h[5];\n+    int len;\n+    uint64_t total_len;\n+    char msgbuf[64];\n+} shani_SHA_CTX;\n+\n+\n+void shani_SHA1_Init(shani_SHA_CTX *ctx);\n+void shani_SHA1_Update(shani_SHA_CTX *ctx, const void *dataIn, size_t len);\n+void shani_SHA1_Final(unsigned char hashout[20], shani_SHA_CTX *ctx);\n+\n+#define platform_SHA_CTX\tshani_SHA_CTX\n+#define platform_SHA1_Init\tshani_SHA1_Init\n+#define platform_SHA1_Update\tshani_SHA1_Update\n+#define platform_SHA1_Final\tshani_SHA1_Final\ndiff --git a/sha256-x64shani/intel_sha_extensions_sha256_assembly.asm b/sha256-x64shani/intel_sha_extensions_sha256_assembly.asm\nnew file mode 100644\nindex 0000000000..b587e82154\n--- /dev/null\n+++ b/sha256-x64shani/intel_sha_extensions_sha256_assembly.asm\n@@ -0,0 +1,371 @@\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+; Copyright (c) 2013, Intel Corporation \n+; \n+; All rights reserved. \n+; \n+; Redistribution and use in source and binary forms, with or without\n+; modification, are permitted provided that the following conditions are\n+; met: \n+; \n+; * Redistributions of source code must retain the above copyright\n+;   notice, this list of conditions and the following disclaimer.  \n+; \n+; * Redistributions in binary form must reproduce the above copyright\n+;   notice, this list of conditions and the following disclaimer in the\n+;   documentation and/or other materials provided with the\n+;   distribution. \n+; \n+; * Neither the name of the Intel Corporation nor the names of its\n+;   contributors may be used to endorse or promote products derived from\n+;   this software without specific prior written permission. \n+; \n+; \n+; THIS SOFTWARE IS PROVIDED BY INTEL CORPORATION \"\"AS IS\"\" AND ANY\n+; EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE\n+; IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR\n+; PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL INTEL CORPORATION OR\n+; CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,\n+; EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,\n+; PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR\n+; PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF\n+; LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING\n+; NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS\n+; SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+;\n+; Intel SHA Extensions optimized implementation of a SHA-256 update function \n+;\n+; The function takes a pointer to the current hash values, a pointer to the \n+; input data, and a number of 64 byte blocks to process.  Once all blocks have \n+; been processed, the digest pointer is  updated with the resulting hash value.\n+; The function only processes complete blocks, there is no functionality to \n+; store partial blocks.  All message padding and hash value initialization must\n+; be done outside the update function.  \n+;\n+; The indented lines in the loop are instructions related to rounds processing.\n+; The non-indented lines are instructions related to the message schedule.\n+;\n+; Author: Sean Gulley <sean.m.gulley@intel.com>\n+; Date:   July 2013\n+;\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+;\n+; Example YASM command lines:\n+; Windows:  yasm -Xvc -f x64 -rnasm -pnasm -DWINABI -o intel_sha_extensions_sha256_assembly.obj -g cv8 intel_sha_extensions_sha256_assembly.asm\n+; Linux:    yasm -f elf64 -X gnu -g dwarf2 -o intel_sha_extensions_sha256_assembly.o intel_sha_extensions_sha256_assembly.asm\n+;\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+\n+%ifdef WINABI\n+%define DIGEST_PTR\trcx \t; 1st arg\n+%define DATA_PTR\trdx \t; 2nd arg\n+%define NUM_BLKS \tr8\t; 3rd arg\n+%else\n+%define DIGEST_PTR\trdi\t; 1st arg\n+%define DATA_PTR\trsi\t; 2nd arg\n+%define NUM_BLKS\trdx\t; 3rd arg\n+%endif\n+\n+%define SHA256CONSTANTS\trax\n+\n+%ifdef WINABI\n+%define RSPSAVE\t\tr9\n+\n+struc frame\n+.xmm_save:\tresdq\t5\n+endstruc\n+%endif\n+\n+%define MSG\t\txmm0\n+%define STATE0\t\txmm1\n+%define STATE1\t\txmm2\n+%define MSGTMP0\t\txmm3\n+%define MSGTMP1\t\txmm4\n+%define MSGTMP2\t\txmm5\n+%define MSGTMP3\t\txmm6\n+%define MSGTMP4\t\txmm7\n+\n+%define SHUF_MASK\txmm8\n+\n+%define ABEF_SAVE\txmm9\n+%define CDGH_SAVE\txmm10\n+\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+;; void sha256_update(uint32_t *digest, const void *data, uint32_t numBlocks);\n+;; arg 1 : pointer to digest\n+;; arg 2 : pointer to input data\n+;; arg 3 : Number of blocks to process\n+;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;\n+section .text\n+global sha256_update:function\n+align 32\n+sha256_update:\n+%ifdef WINABI\n+\tmov\t\tRSPSAVE, rsp\n+\tsub\t\trsp, frame_size\n+\tand\t\trsp, ~0xF\n+\n+\tmovdqa\t\t[rsp + frame.xmm_save + 0*16], xmm6\n+\tmovdqa\t\t[rsp + frame.xmm_save + 1*16], xmm7\n+\tmovdqa\t\t[rsp + frame.xmm_save + 2*16], xmm8\n+\tmovdqa\t\t[rsp + frame.xmm_save + 3*16], xmm9\n+\tmovdqa\t\t[rsp + frame.xmm_save + 4*16], xmm10\n+%endif\n+\n+\tshl\t\tNUM_BLKS, 6\t\t; convert to bytes\n+\tjz\t\t.done_hash\n+\tadd\t\tNUM_BLKS, DATA_PTR\t; pointer to end of data\n+\n+\t;; load initial hash values\n+\t;; Need to reorder these appropriately\n+\t;; DCBA, HGFE -> ABEF, CDGH\n+\tmovdqu\t\tSTATE0, [DIGEST_PTR + 0*16]\n+\tmovdqu\t\tSTATE1, [DIGEST_PTR + 1*16]\n+\n+\tpshufd\t\tSTATE0,  STATE0,  0xB1\t; CDAB\n+\tpshufd\t\tSTATE1,  STATE1,  0x1B\t; EFGH\n+\tmovdqa\t\tMSGTMP4, STATE0\n+\tpalignr\t\tSTATE0,  STATE1,  8\t; ABEF\n+\tpblendw\t\tSTATE1,  MSGTMP4, 0xF0\t; CDGH\n+\n+\tmovdqa\t\tSHUF_MASK, [PSHUFFLE_BYTE_FLIP_MASK wrt rip]\n+\tlea\t\tSHA256CONSTANTS,[K256 wrt rip]\n+\n+.loop0:\n+\t;; Save hash values for addition after rounds\n+\tmovdqa\t\tABEF_SAVE, STATE0\n+\tmovdqa\t\tCDGH_SAVE, STATE1\n+\n+\t;; Rounds 0-3\n+\tmovdqu\t\tMSG, [DATA_PTR + 0*16]\n+\tpshufb\t\tMSG, SHUF_MASK\n+\tmovdqa\t\tMSGTMP0, MSG\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 0*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\n+\t;; Rounds 4-7\n+\tmovdqu\t\tMSG, [DATA_PTR + 1*16]\n+\tpshufb\t\tMSG, SHUF_MASK\n+\tmovdqa\t\tMSGTMP1, MSG\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 1*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP0, MSGTMP1\n+\n+\t;; Rounds 8-11\n+\tmovdqu\t\tMSG, [DATA_PTR + 2*16]\n+\tpshufb\t\tMSG, SHUF_MASK\n+\tmovdqa\t\tMSGTMP2, MSG\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 2*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP1, MSGTMP2\n+\n+\t;; Rounds 12-15\n+\tmovdqu\t\tMSG, [DATA_PTR + 3*16]\n+\tpshufb\t\tMSG, SHUF_MASK\n+\tmovdqa\t\tMSGTMP3, MSG\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 3*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP3\n+\tpalignr\t\tMSGTMP4, MSGTMP2, 4\n+\tpaddd\t\tMSGTMP0, MSGTMP4\n+\tsha256msg2\tMSGTMP0, MSGTMP3\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP2, MSGTMP3\n+\n+\t;; Rounds 16-19\n+\tmovdqa\t\tMSG, MSGTMP0\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 4*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP0\n+\tpalignr\t\tMSGTMP4, MSGTMP3, 4\n+\tpaddd\t\tMSGTMP1, MSGTMP4\n+\tsha256msg2\tMSGTMP1, MSGTMP0\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP3, MSGTMP0\n+\n+\t;; Rounds 20-23\n+\tmovdqa\t\tMSG, MSGTMP1\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 5*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP1\n+\tpalignr\t\tMSGTMP4, MSGTMP0, 4\n+\tpaddd\t\tMSGTMP2, MSGTMP4\n+\tsha256msg2\tMSGTMP2, MSGTMP1\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP0, MSGTMP1\n+\n+\t;; Rounds 24-27\n+\tmovdqa\t\tMSG, MSGTMP2\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 6*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP2\n+\tpalignr\t\tMSGTMP4, MSGTMP1, 4\n+\tpaddd\t\tMSGTMP3, MSGTMP4\n+\tsha256msg2\tMSGTMP3, MSGTMP2\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP1, MSGTMP2\n+\n+\t;; Rounds 28-31\n+\tmovdqa\t\tMSG, MSGTMP3\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 7*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP3\n+\tpalignr\t\tMSGTMP4, MSGTMP2, 4\n+\tpaddd\t\tMSGTMP0, MSGTMP4\n+\tsha256msg2\tMSGTMP0, MSGTMP3\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP2, MSGTMP3\n+\n+\t;; Rounds 32-35\n+\tmovdqa\t\tMSG, MSGTMP0\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 8*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP0\n+\tpalignr\t\tMSGTMP4, MSGTMP3, 4\n+\tpaddd\t\tMSGTMP1, MSGTMP4\n+\tsha256msg2\tMSGTMP1, MSGTMP0\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP3, MSGTMP0\n+\n+\t;; Rounds 36-39\n+\tmovdqa\t\tMSG, MSGTMP1\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 9*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP1\n+\tpalignr\t\tMSGTMP4, MSGTMP0, 4\n+\tpaddd\t\tMSGTMP2, MSGTMP4\n+\tsha256msg2\tMSGTMP2, MSGTMP1\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP0, MSGTMP1\n+\n+\t;; Rounds 40-43\n+\tmovdqa\t\tMSG, MSGTMP2\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 10*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP2\n+\tpalignr\t\tMSGTMP4, MSGTMP1, 4\n+\tpaddd\t\tMSGTMP3, MSGTMP4\n+\tsha256msg2\tMSGTMP3, MSGTMP2\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP1, MSGTMP2\n+\n+\t;; Rounds 44-47\n+\tmovdqa\t\tMSG, MSGTMP3\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 11*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP3\n+\tpalignr\t\tMSGTMP4, MSGTMP2, 4\n+\tpaddd\t\tMSGTMP0, MSGTMP4\n+\tsha256msg2\tMSGTMP0, MSGTMP3\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP2, MSGTMP3\n+\n+\t;; Rounds 48-51\n+\tmovdqa\t\tMSG, MSGTMP0\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 12*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP0\n+\tpalignr\t\tMSGTMP4, MSGTMP3, 4\n+\tpaddd\t\tMSGTMP1, MSGTMP4\n+\tsha256msg2\tMSGTMP1, MSGTMP0\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\tsha256msg1\tMSGTMP3, MSGTMP0\n+\n+\t;; Rounds 52-55\n+\tmovdqa\t\tMSG, MSGTMP1\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 13*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP1\n+\tpalignr\t\tMSGTMP4, MSGTMP0, 4\n+\tpaddd\t\tMSGTMP2, MSGTMP4\n+\tsha256msg2\tMSGTMP2, MSGTMP1\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\n+\t;; Rounds 56-59\n+\tmovdqa\t\tMSG, MSGTMP2\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 14*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\tmovdqa\t\tMSGTMP4, MSGTMP2\n+\tpalignr\t\tMSGTMP4, MSGTMP1, 4\n+\tpaddd\t\tMSGTMP3, MSGTMP4\n+\tsha256msg2\tMSGTMP3, MSGTMP2\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\n+\t;; Rounds 60-63\n+\tmovdqa\t\tMSG, MSGTMP3\n+\t\tpaddd\t\tMSG, [SHA256CONSTANTS + 15*16]\n+\t\tsha256rnds2\tSTATE1, STATE0\n+\t\tpshufd \t\tMSG, MSG, 0x0E\n+\t\tsha256rnds2\tSTATE0, STATE1\n+\n+\t;; Add current hash values with previously saved\n+\tpaddd\t\tSTATE0, ABEF_SAVE\n+\tpaddd\t\tSTATE1, CDGH_SAVE\n+\n+\t;; Increment data pointer and loop if more to process\n+\tadd\t\tDATA_PTR, 64\n+\tcmp\t\tDATA_PTR, NUM_BLKS\n+\tjne\t\t.loop0\n+\n+\t;; Write hash values back in the correct order\n+\tpshufd\t\tSTATE0,  STATE0,  0x1B\t; FEBA\n+\tpshufd\t\tSTATE1,  STATE1,  0xB1\t; DCHG\n+\tmovdqa\t\tMSGTMP4, STATE0\n+\tpblendw\t\tSTATE0,  STATE1,  0xF0\t; DCBA\n+\tpalignr\t\tSTATE1,  MSGTMP4, 8\t; HGFE\n+\n+\tmovdqu\t\t[DIGEST_PTR + 0*16], STATE0\n+\tmovdqu\t\t[DIGEST_PTR + 1*16], STATE1\n+\n+.done_hash:\n+%ifdef WINABI\n+\tmovdqa\t\txmm6,  [rsp + frame.xmm_save + 0*16]\n+\tmovdqa\t\txmm7,  [rsp + frame.xmm_save + 1*16]\n+\tmovdqa\t\txmm8,  [rsp + frame.xmm_save + 2*16]\n+\tmovdqa\t\txmm9,  [rsp + frame.xmm_save + 3*16]\n+\tmovdqa\t\txmm10, [rsp + frame.xmm_save + 4*16]\n+\tmov\t\trsp, RSPSAVE\n+%endif\n+\n+\tret\t\n+\t\n+section .data\n+align 64\n+K256:\n+\tdd\t0x428a2f98,0x71374491,0xb5c0fbcf,0xe9b5dba5\n+\tdd\t0x3956c25b,0x59f111f1,0x923f82a4,0xab1c5ed5\n+\tdd\t0xd807aa98,0x12835b01,0x243185be,0x550c7dc3\n+\tdd\t0x72be5d74,0x80deb1fe,0x9bdc06a7,0xc19bf174\n+\tdd\t0xe49b69c1,0xefbe4786,0x0fc19dc6,0x240ca1cc\n+\tdd\t0x2de92c6f,0x4a7484aa,0x5cb0a9dc,0x76f988da\n+\tdd\t0x983e5152,0xa831c66d,0xb00327c8,0xbf597fc7\n+\tdd\t0xc6e00bf3,0xd5a79147,0x06ca6351,0x14292967\n+\tdd\t0x27b70a85,0x2e1b2138,0x4d2c6dfc,0x53380d13\n+\tdd\t0x650a7354,0x766a0abb,0x81c2c92e,0x92722c85\n+\tdd\t0xa2bfe8a1,0xa81a664b,0xc24b8b70,0xc76c51a3\n+\tdd\t0xd192e819,0xd6990624,0xf40e3585,0x106aa070\n+\tdd\t0x19a4c116,0x1e376c08,0x2748774c,0x34b0bcb5\n+\tdd\t0x391c0cb3,0x4ed8aa4a,0x5b9cca4f,0x682e6ff3\n+\tdd\t0x748f82ee,0x78a5636f,0x84c87814,0x8cc70208\n+\tdd\t0x90befffa,0xa4506ceb,0xbef9a3f7,0xc67178f2\n+\n+PSHUFFLE_BYTE_FLIP_MASK: ddq 0x0c0d0e0f08090a0b0405060700010203\n+\ndiff --git a/sha256-x64shani/sha256_x64.c b/sha256-x64shani/sha256_x64.c\nnew file mode 100644\nindex 0000000000..01f3417719\n--- /dev/null\n+++ b/sha256-x64shani/sha256_x64.c\n@@ -0,0 +1,82 @@\n+#include \"../git-compat-util.h\"\n+#include \"sha256_x64.h\"\n+#include <byteswap.h>\n+\n+static int shani = -1;\n+\n+void shani_SHA256_Init(shani_SHA256_CTX *ctx)\n+{\n+\tctx->total_len = 0;\n+\tctx->len = 0;\n+\tctx->h[0] = 0x6a09e667ul;\n+\tctx->h[1] = 0xbb67ae85ul;\n+\tctx->h[2] = 0x3c6ef372ul;\n+\tctx->h[3] = 0xa54ff53aul;\n+\tctx->h[4] = 0x510e527ful;\n+\tctx->h[5] = 0x9b05688cul;\n+\tctx->h[6] = 0x1f83d9abul;\n+\tctx->h[7] = 0x5be0cd19ul;\n+\n+    if (shani != 1) {\n+        shani = 1;\n+        printf(\"set shani to 1\\n\");\n+    } else {\n+        printf(\"shani is 1 already\\n\");\n+    }\n+}\n+\n+void sha256_update(uint32_t *digest, const void *data, uint32_t numBlocks);\n+void shani_SHA256_Update(shani_SHA256_CTX *c, const void *data, size_t len)\n+{\n+    while (len > 0)\n+    {\n+        if (c->len == 0 && len >= 64)\n+        {\n+            sha256_update(c->h, data, len/64);\n+            unsigned long bytes_hashed = (len/64)*64;\n+            data += bytes_hashed;\n+            len -= bytes_hashed;\n+            c->total_len += bytes_hashed;\n+            continue;\n+        }\n+#define MIN(x, y) (((x) < (y)) ? (x) : (y))\n+        size_t cpy = MIN(64 - c->len, len);\n+#undef MIN\n+        memcpy(c->msgbuf + c->len, data, cpy);\n+        data += cpy;\n+        len -= cpy;\n+        c->len += cpy;\n+        c->total_len += cpy;\n+\n+        if (c->len == 64)\n+        {\n+            sha256_update(c->h, c->msgbuf, 1);\n+            c->len = 0;\n+        }\n+    }\n+}\n+\n+void shani_SHA256_Final(unsigned char *digest, shani_SHA256_CTX *c)\n+{\n+    c->msgbuf[c->len] = 0x80;\n+    ++c->len;\n+    memset(c->msgbuf + c->len, 0, 64 - c->len);\n+    if (c->len > (64-8))\n+    {\n+        sha256_update(c->h, c->msgbuf, 1);\n+        memset(c->msgbuf, 0, 64);\n+        c->len = 0;\n+    }\n+\n+    for (int ibyte = 0; ibyte < 8; ++ibyte)\n+       c->msgbuf[63-ibyte] = (uint8_t)((c->total_len * 8ULL) >> (ibyte*8));\n+\n+    sha256_update(c->h, c->msgbuf, 1);\n+    c->len = 0;\n+    \n+    // flip endianness\n+    int i;\n+    for (i = 0; i < 32; i += 4) {\n+        *(uint32_t *)(digest + i) = __bswap_32(c->h[i / 4]);\n+    }\n+}\n\\ No newline at end of file\ndiff --git a/sha256-x64shani/sha256_x64.h b/sha256-x64shani/sha256_x64.h\nnew file mode 100644\nindex 0000000000..25f1ff67ba\n--- /dev/null\n+++ b/sha256-x64shani/sha256_x64.h\n@@ -0,0 +1,23 @@\n+/*\n+ * SHA256 routine using x64 SHA-NI instructions\n+ */\n+\n+#define SHANI_SHA256_BLKSIZE 64\n+\n+struct shani_SHA256_CTX {\n+    uint32_t h[8];\n+    int len;\n+    uint64_t total_len;\n+    char msgbuf[SHANI_SHA256_BLKSIZE];\n+};\n+\n+typedef struct shani_SHA256_CTX shani_SHA256_CTX;\n+\n+void shani_SHA256_Init(shani_SHA256_CTX *ctx);\n+void shani_SHA256_Update(shani_SHA256_CTX *c, const void *data, size_t len);\n+void shani_SHA256_Final(unsigned char *digest, shani_SHA256_CTX *c);\n+\n+#define platform_SHA256_CTX\t    shani_SHA256_CTX\n+#define platform_SHA256_Init\tshani_SHA256_Init\n+#define platform_SHA256_Update\tshani_SHA256_Update\n+#define platform_SHA256_Final\tshani_SHA256_Final\n\\ No newline at end of file\n-- \n2.41.0\n\n"},{"id":"483236","messageId":"ZSmxFopwJyBGmZY-@tapette.crustytoothpaste.net","threadId":"60356","inReplyTo":"20231012200447.3553757-1-hi@fxue.dev","subject":"Re: [PATCH] Add x64 SHA-NI implementation to sha1 and sha256 in git ?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-10-13T21:05:26Z","receivedAt":"2023-10-13T21:05:30Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-10-12 at 20:04:47, Feiyang Xue wrote:\n> Looking for comments on if this is a good idea to add SHA-NI option to git,\n> And if so, what'd be a good implementation.\n> \n> The attached patch is more of a proof of concept,  using a sha-ni example\n> found on internet and have it tapped into git's \"hash-ll.h\" when it sees make\n> flags \"SHA1_SHANI=1\" and/or \"SHA256_SHANI=1\" for sha1/sha256 respectively.\n> \n> \"git hash-object\" is about 3-ish times faster for me\n\nYou almost certainly don't want to use SHA-1 using native instructions\nbecause it doesn't detect collisions, unlike the default implementation\n(SHA-1-DC).  Since SHA-1 collisions are relatively cheap (less than USD\n45,000) to make, not detecting collisions would be a security issue\nworthy of a CVE.\n\nIt's fine for SHA-256, but see below.\n\n> There are few topics that i'm uncertain about.\n> \n> 1. Is it good idea to have machine-specific asm codes in git. I read there was\n> commit fd1ec82547 which removed assembly implementation for PPC_SHA1. Does that\n> imply maintainers doesn't want machine-specific assembly in source?\n\nI'd prefer we didn't.\n\nNettle has SHA-NI extensions on systems that support it, and so does\nOpenSSL.  OpenSSL can't be used by distros for licensing reasons, but\nNettle can, and both of those libraries automatically detect the best\ncode to use based on the chip.  One of those libraries is almost always\ngoing to be linked into Git at some point because of libcurl anyway.\n\nIf we ship the code, then someone has to maintain it, and I don't know\nif we have any assembly experts.  We might also have a bug that produced\nincorrect results, which would be pretty disastrous for the affected\nusers.  It's much better for maintainability if we push that off onto\nthe cryptographic library maintainers who are much more competent in\nthat regard than I am (I will not speak for others) and let distro\nmaintainers simply link the appropriate library, which, as I said above,\nthey're already effectively doing.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"483237","messageId":"xmqqil7axczo.fsf@gitster.g","threadId":"60356","inReplyTo":"ZSmxFopwJyBGmZY-@tapette.crustytoothpaste.net","subject":"Re: [PATCH] Add x64 SHA-NI implementation to sha1 and sha256 in git ?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-10-14T00:29:47Z","receivedAt":"2023-10-14T00:29:54Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n\n> If we ship the code, then someone has to maintain it, and I don't know\n> if we have any assembly experts.  We might also have a bug that produced\n> incorrect results, which would be pretty disastrous for the affected\n> users.  It's much better for maintainability if we push that off onto\n> the cryptographic library maintainers who are much more competent in\n> that regard than I am (I will not speak for others) and let distro\n> maintainers simply link the appropriate library, which, as I said above,\n> they're already effectively doing.\n\nThanks for a well reasoned response.  Very much appreciated.\n"},{"id":"483287","messageId":"fd59abd6-49a5-4f6a-8d6c-c608cd3350f7@drbeat.li","threadId":"60356","inReplyTo":"ZSmxFopwJyBGmZY-@tapette.crustytoothpaste.net","subject":"Re: [PATCH] Add x64 SHA-NI implementation to sha1 and sha256 in git ?","fromName":"Beat Bolli","fromEmail":"dev+git@drbeat.li","sentAt":"2023-10-15T16:37:11Z","receivedAt":"2023-10-15T16:42:49Z","isPatch":true,"sender":{"key":"dev+git@drbeat.li","avatar":"https://avatars.githubusercontent.com/u/21444?v=4"},"body":"Hi\n\nOn 13.10.23 23:05, brian m. carlson wrote:\n> On 2023-10-12 at 20:04:47, Feiyang Xue wrote:\n>> Looking for comments on if this is a good idea to add SHA-NI option to git,\n>> And if so, what'd be a good implementation.\n>>\n>> The attached patch is more of a proof of concept,  using a sha-ni example\n>> found on internet and have it tapped into git's \"hash-ll.h\" when it sees make\n>> flags \"SHA1_SHANI=1\" and/or \"SHA256_SHANI=1\" for sha1/sha256 respectively.\n>>\n>> \"git hash-object\" is about 3-ish times faster for me\n> \n> You almost certainly don't want to use SHA-1 using native instructions\n> because it doesn't detect collisions, unlike the default implementation\n> (SHA-1-DC).  Since SHA-1 collisions are relatively cheap (less than USD\n> 45,000) to make, not detecting collisions would be a security issue\n> worthy of a CVE.\n> \n> It's fine for SHA-256, but see below.\n> \n>> There are few topics that i'm uncertain about.\n>>\n>> 1. Is it good idea to have machine-specific asm codes in git. I read there was\n>> commit fd1ec82547 which removed assembly implementation for PPC_SHA1. Does that\n>> imply maintainers doesn't want machine-specific assembly in source?\n> \n> I'd prefer we didn't.\n> \n> Nettle has SHA-NI extensions on systems that support it, and so does\n> OpenSSL.  OpenSSL can't be used by distros for licensing reasons, but\n> Nettle can, and both of those libraries automatically detect the best\n> code to use based on the chip.  One of those libraries is almost always\n> going to be linked into Git at some point because of libcurl anyway.\n> \n> If we ship the code, then someone has to maintain it, and I don't know\n> if we have any assembly experts.  We might also have a bug that produced\n> incorrect results, which would be pretty disastrous for the affected\n> users.  It's much better for maintainability if we push that off onto\n> the cryptographic library maintainers who are much more competent in\n> that regard than I am (I will not speak for others) and let distro\n> maintainers simply link the appropriate library, which, as I said above,\n> they're already effectively doing.\n\nIn addition to all the valid points brian makes, the patch uses a \nnon-standard assembler (yasm, see http://yasm.tortall.net/) that's not \npart of the default GNU toolchain.\n\nCheers, Beat\n"},{"id":"483290","messageId":"022101d9ff8d$4dc63fd0$e952bf70$@nexbridge.com","threadId":"60356","inReplyTo":"fd59abd6-49a5-4f6a-8d6c-c608cd3350f7@drbeat.li","subject":"RE: [PATCH] Add x64 SHA-NI implementation to sha1 and sha256 in git ?","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2023-10-15T17:30:28Z","receivedAt":"2023-10-15T17:30:47Z","isPatch":true,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On Sunday, October 15, 2023 12:37 PM, Beat Bolli wrote:\n>On 13.10.23 23:05, brian m. carlson wrote:\n>> On 2023-10-12 at 20:04:47, Feiyang Xue wrote:\n>>> Looking for comments on if this is a good idea to add SHA-NI option\n>>> to git, And if so, what'd be a good implementation.\n>>>\n>>> The attached patch is more of a proof of concept,  using a sha-ni\n>>> example found on internet and have it tapped into git's \"hash-ll.h\"\n>>> when it sees make flags \"SHA1_SHANI=1\" and/or \"SHA256_SHANI=1\" for\n>sha1/sha256 respectively.\n>>>\n>>> \"git hash-object\" is about 3-ish times faster for me\n>>\n>> You almost certainly don't want to use SHA-1 using native instructions\n>> because it doesn't detect collisions, unlike the default\n>> implementation (SHA-1-DC).  Since SHA-1 collisions are relatively\n>> cheap (less than USD\n>> 45,000) to make, not detecting collisions would be a security issue\n>> worthy of a CVE.\n>>\n>> It's fine for SHA-256, but see below.\n>>\n>>> There are few topics that i'm uncertain about.\n>>>\n>>> 1. Is it good idea to have machine-specific asm codes in git. I read\n>>> there was commit fd1ec82547 which removed assembly implementation for\n>>> PPC_SHA1. Does that imply maintainers doesn't want machine-specific assembly\n>in source?\n>>\n>> I'd prefer we didn't.\n>>\n>> Nettle has SHA-NI extensions on systems that support it, and so does\n>> OpenSSL.  OpenSSL can't be used by distros for licensing reasons, but\n>> Nettle can, and both of those libraries automatically detect the best\n>> code to use based on the chip.  One of those libraries is almost\n>> always going to be linked into Git at some point because of libcurl anyway.\n>>\n>> If we ship the code, then someone has to maintain it, and I don't know\n>> if we have any assembly experts.  We might also have a bug that\n>> produced incorrect results, which would be pretty disastrous for the\n>> affected users.  It's much better for maintainability if we push that\n>> off onto the cryptographic library maintainers who are much more\n>> competent in that regard than I am (I will not speak for others) and\n>> let distro maintainers simply link the appropriate library, which, as\n>> I said above, they're already effectively doing.\n>\n>In addition to all the valid points brian makes, the patch uses a non-standard\n>assembler (yasm, see http://yasm.tortall.net/) that's not part of the default GNU\n>toolchain.\n\nAside from that, not everyone uses the GNU toolchain. I would suggest that if someone needs ASM support for SHA-1, they could reimplement SHA-1-DC using rotate ASM primitives instead of directly using SHA1 assembly, but I question whether that is significantly faster than having the compiler optimizer generate the equivalent rotates. Still, maintaining any ASM code requires out of box skillsets that are unlikely sustainable.\n--Randall\n\n"}]}