{"thread":{"id":"49668","subject":"[PATCH v4 00/12] Base SHA-256 implementation","startedAt":"2018-10-25T02:40:17Z","lastAt":"2018-11-14T04:10:32Z","messageCount":58,"participants":["brian m. carlson","Carlo Arenas","Christian Couder","Junio C Hamano","Ævar Arnfjörð Bjarmason","Derrick Stolee","Duy Nguyen","Ramsay Jones","Jeff King"],"isPatch":true,"patchVersion":4,"patchTotal":12},"messages":[{"id":"361448","messageId":"20181025024005.154208-1-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":null,"subject":"[PATCH v4 00/12] Base SHA-256 implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:39:53Z","receivedAt":"2018-10-25T02:40:17Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This series provides a functional SHA-256 implementation and wires it\nup, along with some housekeeping patches to make it suitable for\ntesting.\n\nWhile I was fixing the macros, I wondered if I could make the code a bit\ncleaner by using inline functions. I tried it and found that not only\ndid it make the code cleaner, it made the code significantly faster\nacross all sizes of output, with larger gains on larger chunks (e.g.,\n214 MiB/s on 16 KiB chunks vs 151 MiB/s). I'm unsure why this effect\noccurs, but I figured nobody would complain about improved performance.\n\nChanges from v3:\n* Switch to using inline functions instead of macros in many cases.\n* Undefine remaining macros at the top.\n\nChanges from v2:\n* Improve commit messages to include timing and performance information.\n* Improve commit messages to be less ambiguous and more friendly to a\n wider variety of English speakers.\n* Prefer functions taking struct git_hash_algo in hex.c.\n* Port pieces of the block-sha1 implementation over to the block-sha256\n implementation for better compatibility.\n* Drop patch 13 in favor of further discussion about the best way\n forward for versioning commit graph.\n* Rename the test so as to have a different number from other tests.\n* Rebase on master.\n\nChanges from v1:\n* Add a hash_to_hex function mirroring sha1_to_hex, but for\n the_hash_algo.\n* Strip commit message explanation about why we chose SHA-256.\n* Rebase on master\n* Strip leading whitespace from commit message.\n* Improve commit-graph patch to cover new code added since v1.\n* Be more honest about the scope of work involved in porting the SHA-256\n implementation out of libtomcrypt.\n* Revert change to limit hashcmp to 20 bytes.\n\nbrian m. carlson (12):\n sha1-file: rename algorithm to \"sha1\"\n sha1-file: provide functions to look up hash algorithms\n hex: introduce functions to print arbitrary hashes\n cache: make hashcmp and hasheq work with larger hashes\n t: add basic tests for our SHA-1 implementation\n t: make the sha1 test-tool helper generic\n sha1-file: add a constant for hash block size\n t/helper: add a test helper to compute hash speed\n commit-graph: convert to using the_hash_algo\n Add a base implementation of SHA-256 support\n sha256: add an SHA-256 implementation using libgcrypt\n hash: add an SHA-256 implementation using OpenSSL\n\n Makefile | 22 +++\n cache.h | 51 ++++---\n commit-graph.c | 33 +++--\n hash.h | 41 +++++-\n hex.c | 32 +++-\n sha1-file.c | 70 ++++++++-\n sha256/block/sha256.c | 201 ++++++++++++++++++++++++++\n sha256/block/sha256.h | 26 ++++\n sha256/gcrypt.h | 30 ++++\n t/helper/test-hash-speed.c | 61 ++++++++\n t/helper/{test-sha1.c => test-hash.c} | 19 +--\n t/helper/test-sha1.c | 52 +------\n t/helper/test-sha256.c | 7 +\n t/helper/test-tool.c | 2 +\n t/helper/test-tool.h | 4 +\n t/t0015-hash.sh | 54 +++++++\n 16 files changed, 601 insertions(+), 104 deletions(-)\n create mode 100644 sha256/block/sha256.c\n create mode 100644 sha256/block/sha256.h\n create mode 100644 sha256/gcrypt.h\n create mode 100644 t/helper/test-hash-speed.c\n copy t/helper/{test-sha1.c => test-hash.c} (65%)\n create mode 100644 t/helper/test-sha256.c\n create mode 100755 t/t0015-hash.sh\n\nRange-diff against v3:\n 1: a004a4c982 < -: ---------- :hash-impl\n 2: cf9f7f5620 = 1: cf9f7f5620 sha1-file: rename algorithm to \"sha1\"\n 3: 0144deaebe = 2: 0144deaebe sha1-file: provide functions to look up hash algorithms\n 4: b74858fb03 = 3: b74858fb03 hex: introduce functions to print arbitrary hashes\n 5: e9703017a4 = 4: e9703017a4 cache: make hashcmp and hasheq work with larger hashes\n 6: ab85a834fd = 5: ab85a834fd t: add basic tests for our SHA-1 implementation\n 7: 962f6d8903 = 6: 962f6d8903 t: make the sha1 test-tool helper generic\n 8: 53addf4d58 = 7: 53addf4d58 sha1-file: add a constant for hash block size\n 9: 9ace10faa2 = 8: 9ace10faa2 t/helper: add a test helper to compute hash speed\n10: 9adc56d01e = 9: 9adc56d01e commit-graph: convert to using the_hash_algo\n11: 8e82cb0dfb ! 10: f48cb1ad27 Add a base implementation of SHA-256 support\n @@ -207,6 +207,9 @@\n +#include \"git-compat-util.h\"\n +#include \"./sha256.h\"\n +\n ++#undef RND\n ++#undef BLKSIZE\n ++\n +#define BLKSIZE blk_SHA256_BLKSIZE\n +\n +void blk_SHA256_Init(blk_SHA256_CTX *ctx)\n @@ -228,14 +231,35 @@\n +\treturn (x >> n) | (x << (32 - n));\n +}\n +\n -+#define Ch(x,y,z) (z ^ (x & (y ^ z)))\n -+#define Maj(x,y,z) (((x | y) & z) | (x & y))\n -+#define S(x, n) ror((x),(n))\n -+#define R(x, n) ((x)>>(n))\n -+#define Sigma0(x) (S(x, 2) ^ S(x, 13) ^ S(x, 22))\n -+#define Sigma1(x) (S(x, 6) ^ S(x, 11) ^ S(x, 25))\n -+#define Gamma0(x) (S(x, 7) ^ S(x, 18) ^ R(x, 3))\n -+#define Gamma1(x) (S(x, 17) ^ S(x, 19) ^ R(x, 10))\n ++static inline uint32_t ch(uint32_t x, uint32_t y, uint32_t z)\n ++{\n ++\treturn (z ^ (x & (y ^ z)));\n ++}\n ++\n ++static inline uint32_t maj(uint32_t x, uint32_t y, uint32_t z)\n ++{\n ++\treturn (((x | y) & z) | (x & y));\n ++}\n ++\n ++static inline uint32_t sigma0(uint32_t x)\n ++{\n ++\treturn ror(x, 2) ^ ror(x, 13) ^ ror(x, 22);\n ++}\n ++\n ++static inline uint32_t sigma1(uint32_t x)\n ++{\n ++\treturn ror(x, 6) ^ ror(x, 11) ^ ror(x, 25);\n ++}\n ++\n ++static inline uint32_t gamma0(uint32_t x)\n ++{\n ++\treturn ror(x, 7) ^ ror(x, 18) ^ (x >> 3);\n ++}\n ++\n ++static inline uint32_t gamma1(uint32_t x)\n ++{\n ++\treturn ror(x, 17) ^ ror(x, 19) ^ (x >> 10);\n ++}\n +\n +static void blk_SHA256_Transform(blk_SHA256_CTX *ctx, const unsigned char *buf)\n +{\n @@ -255,12 +279,12 @@\n +\n +\t/* fill W[16..63] */\n +\tfor (i = 16; i < 64; i++) {\n -+\t\tW[i] = Gamma1(W[i - 2]) + W[i - 7] + Gamma0(W[i - 15]) + W[i - 16];\n ++\t\tW[i] = gamma1(W[i - 2]) + W[i - 7] + gamma0(W[i - 15]) + W[i - 16];\n +\t}\n +\n +#define RND(a,b,c,d,e,f,g,h,i,ki) \\\n -+\tt0 = h + Sigma1(e) + Ch(e, f, g) + ki + W[i]; \\\n -+\tt1 = Sigma0(a) + Maj(a, b, c); \\\n ++\tt0 = h + sigma1(e) + ch(e, f, g) + ki + W[i]; \\\n ++\tt1 = sigma0(a) + maj(a, b, c); \\\n +\td += t0; \\\n +\th = t0 + t1;\n +\n @@ -329,15 +353,6 @@\n +\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],62,0xbef9a3f7);\n +\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],63,0xc67178f2);\n +\n -+#undef RND\n -+#undef Ch\n -+#undef Maj\n -+#undef S\n -+#undef R\n -+#undef Sigma0\n -+#undef Sigma1\n -+#undef Gamma0\n -+#undef Gamma1\n +\n +\tfor (i = 0; i < 8; i++) {\n +\t\tctx->state[i] = ctx->state[i] + S[i];\n12: 9e0061bd74 = 11: fe8f2ba01c sha256: add an SHA-256 implementation using libgcrypt\n13: 128d6b8150 = 12: 38142d8fc6 hash: add an SHA-256 implementation using OpenSSL\n"},{"id":"361449","messageId":"20181025024005.154208-2-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 01/12] sha1-file: rename algorithm to \"sha1\"","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:39:54Z","receivedAt":"2018-10-25T02:40:17Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"The transition plan anticipates us using a syntax such as \"^{sha1}\" for\ndisambiguation. Since this is a syntax some people will be typing a\nlot, it makes sense to provide a short, easy-to-type syntax. Omitting\nthe dash doesn't create any ambiguity; however, it does make the syntax\nshorter and easier to type, especially for touch typists. In addition,\nthe transition plan already uses \"sha1\" in this context.\n\nRename the name of SHA-1 implementation to \"sha1\".\n\nNote that this change creates no backwards compatibility concerns, since\nwe haven't yet used this field in any configuration settings.\n\nSigned-off-by: brian m. carlson \n---\n sha1-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sha1-file.c b/sha1-file.c\nindex dd0b6aa873..91311ebb3d 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -97,7 +97,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\tNULL,\n \t},\n \t{\n-\t\t\"sha-1\",\n+\t\t\"sha1\",\n \t\t/* \"sha1\", big-endian */\n \t\t0x73686131,\n \t\tGIT_SHA1_RAWSZ,\n"},{"id":"361450","messageId":"20181025024005.154208-3-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:39:55Z","receivedAt":"2018-10-25T02:40:19Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"There are several ways we might refer to a hash algorithm: by name, such\nas in the config file; by format ID, such as in a pack; or internally,\nby a pointer to the hash_algos array. Provide functions to look up hash\nalgorithms based on these various forms and return the internal constant\nused for them. If conversion to another form is necessary, this\ninternal constant can be used to look up the proper data in the\nhash_algos array.\n\nSigned-off-by: brian m. carlson \n---\n hash.h | 13 +++++++++++++\n sha1-file.c | 21 +++++++++++++++++++++\n 2 files changed, 34 insertions(+)\n\ndiff --git a/hash.h b/hash.h\nindex 7c8238bc2e..80881eea47 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -98,4 +98,17 @@ struct git_hash_algo {\n };\n extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n \n+/*\n+ * Return a GIT_HASH_* constant based on the name. Returns GIT_HASH_UNKNOWN if\n+ * the name doesn't match a known algorithm.\n+ */\n+int hash_algo_by_name(const char *name);\n+/* Identical, except based on the format ID. */\n+int hash_algo_by_id(uint32_t format_id);\n+/* Identical, except for a pointer to struct git_hash_algo. */\n+static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n+{\n+\treturn p - hash_algos;\n+}\n+\n #endif\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 91311ebb3d..7e9dedc744 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -122,6 +122,27 @@ const char *empty_blob_oid_hex(void)\n \treturn oid_to_hex_r(buf, the_hash_algo->empty_blob);\n }\n \n+int hash_algo_by_name(const char *name)\n+{\n+\tint i;\n+\tif (!name)\n+\t\treturn GIT_HASH_UNKNOWN;\n+\tfor (i = 1; i < GIT_HASH_NALGOS; i++)\n+\t\tif (!strcmp(name, hash_algos[i].name))\n+\t\t\treturn i;\n+\treturn GIT_HASH_UNKNOWN;\n+}\n+\n+int hash_algo_by_id(uint32_t format_id)\n+{\n+\tint i;\n+\tfor (i = 1; i < GIT_HASH_NALGOS; i++)\n+\t\tif (format_id == hash_algos[i].format_id)\n+\t\t\treturn i;\n+\treturn GIT_HASH_UNKNOWN;\n+}\n+\n+\n /*\n * This is meant to hold a *small* number of objects that you would\n * want read_sha1_file() to be able to return, but yet you do not want\n"},{"id":"361451","messageId":"20181025024005.154208-4-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 03/12] hex: introduce functions to print arbitrary hashes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:39:56Z","receivedAt":"2018-10-25T02:40:22Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Currently, we have functions that turn an arbitrary SHA-1 value or an\nobject ID into hex format, either using a static buffer or with a\nuser-provided buffer. Add variants of these functions that can handle\nan arbitrary hash algorithm, specified by constant. Update the\ndocumentation as well.\n\nWhile we're at it, remove the \"extern\" declaration from this family of\nfunctions, since it's not needed and our style now recommends against\nit.\n\nWe use the variant taking the algorithm structure pointer as the\ninternal variant, since taking an algorithm pointer is the easiest way\nto handle all of the variants in use.\n\nNote that we maintain these functions because there are hashes which\nmust change based on the hash algorithm in use but are not object IDs\n(such as pack checksums).\n\nSigned-off-by: brian m. carlson \n---\n cache.h | 15 +++++++++------\n hex.c | 32 ++++++++++++++++++++++++--------\n 2 files changed, 33 insertions(+), 14 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 59c8a93046..51580c4b77 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1364,9 +1364,9 @@ extern int get_oid_hex(const char *hex, struct object_id *sha1);\n extern int hex_to_bytes(unsigned char *binary, const char *hex, size_t len);\n \n /*\n- * Convert a binary sha1 to its hex equivalent. The `_r` variant is reentrant,\n+ * Convert a binary hash to its hex equivalent. The `_r` variant is reentrant,\n * and writes the NUL-terminated output to the buffer `out`, which must be at\n- * least `GIT_SHA1_HEXSZ + 1` bytes, and returns a pointer to out for\n+ * least `GIT_MAX_HEXSZ + 1` bytes, and returns a pointer to out for\n * convenience.\n *\n * The non-`_r` variant returns a static buffer, but uses a ring of 4\n@@ -1374,10 +1374,13 @@ extern int hex_to_bytes(unsigned char *binary, const char *hex, size_t len);\n *\n * printf(\"%s -> %s\", sha1_to_hex(one), sha1_to_hex(two));\n */\n-extern char *sha1_to_hex_r(char *out, const unsigned char *sha1);\n-extern char *oid_to_hex_r(char *out, const struct object_id *oid);\n-extern char *sha1_to_hex(const unsigned char *sha1);\t/* static buffer result! */\n-extern char *oid_to_hex(const struct object_id *oid);\t/* same static buffer as sha1_to_hex */\n+char *hash_to_hex_algop_r(char *buffer, const unsigned char *hash, const struct git_hash_algo *);\n+char *sha1_to_hex_r(char *out, const unsigned char *sha1);\n+char *oid_to_hex_r(char *out, const struct object_id *oid);\n+char *hash_to_hex_algop(const unsigned char *hash, const struct git_hash_algo *);\t/* static buffer result! */\n+char *sha1_to_hex(const unsigned char *sha1);\t\t\t\t\t\t/* same static buffer */\n+char *hash_to_hex(const unsigned char *hash);\t\t\t\t\t\t/* same static buffer */\n+char *oid_to_hex(const struct object_id *oid);\t\t\t\t\t\t/* same static buffer */\n \n /*\n * Parse a 40-character hexadecimal object ID starting from hex, updating the\ndiff --git a/hex.c b/hex.c\nindex 10af1a29e8..d2e8bb9540 100644\n--- a/hex.c\n+++ b/hex.c\n@@ -73,14 +73,15 @@ int parse_oid_hex(const char *hex, struct object_id *oid, const char **end)\n \treturn ret;\n }\n \n-char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n+inline char *hash_to_hex_algop_r(char *buffer, const unsigned char *hash,\n+\t\t\t\t\tconst struct git_hash_algo *algop)\n {\n \tstatic const char hex[] = \"0123456789abcdef\";\n \tchar *buf = buffer;\n \tint i;\n \n-\tfor (i = 0; i < the_hash_algo->rawsz; i++) {\n-\t\tunsigned int val = *sha1++;\n+\tfor (i = 0; i < algop->rawsz; i++) {\n+\t\tunsigned int val = *hash++;\n \t\t*buf++ = hex[val >> 4];\n \t\t*buf++ = hex[val & 0xf];\n \t}\n@@ -89,20 +90,35 @@ char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n \treturn buffer;\n }\n \n-char *oid_to_hex_r(char *buffer, const struct object_id *oid)\n+char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n {\n-\treturn sha1_to_hex_r(buffer, oid->hash);\n+\treturn hash_to_hex_algop_r(buffer, sha1, &hash_algos[GIT_HASH_SHA1]);\n }\n \n-char *sha1_to_hex(const unsigned char *sha1)\n+char *oid_to_hex_r(char *buffer, const struct object_id *oid)\n+{\n+\treturn hash_to_hex_algop_r(buffer, oid->hash, the_hash_algo);\n+}\n+\n+char *hash_to_hex_algop(const unsigned char *hash, const struct git_hash_algo *algop)\n {\n \tstatic int bufno;\n \tstatic char hexbuffer[4][GIT_MAX_HEXSZ + 1];\n \tbufno = (bufno + 1) % ARRAY_SIZE(hexbuffer);\n-\treturn sha1_to_hex_r(hexbuffer[bufno], sha1);\n+\treturn hash_to_hex_algop_r(hexbuffer[bufno], hash, algop);\n+}\n+\n+char *sha1_to_hex(const unsigned char *sha1)\n+{\n+\treturn hash_to_hex_algop(sha1, &hash_algos[GIT_HASH_SHA1]);\n+}\n+\n+char *hash_to_hex(const unsigned char *hash)\n+{\n+\treturn hash_to_hex_algop(hash, the_hash_algo);\n }\n \n char *oid_to_hex(const struct object_id *oid)\n {\n-\treturn sha1_to_hex(oid->hash);\n+\treturn hash_to_hex_algop(oid->hash, the_hash_algo);\n }\n"},{"id":"361452","messageId":"20181025024005.154208-5-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 04/12] cache: make hashcmp and hasheq work with larger hashes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:39:57Z","receivedAt":"2018-10-25T02:40:22Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In 183a638b7d (\"hashcmp: assert constant hash size\", 2018-08-23), we\nmodified hashcmp to assert that the hash size was always 20 to help it\noptimize and inline calls to memcmp. In a future series, we replaced\nmany calls to hashcmp and oidcmp with calls to hasheq and oideq to\nimprove inlining further.\n\nHowever, we want to support hash algorithms other than SHA-1, namely\nSHA-256. When doing so, we must handle the case where these values are\n32 bytes long as well as 20. Adjust hashcmp to handle two cases:\n20-byte matches, and maximum-size matches. Therefore, when we include\nSHA-256, we'll automatically handle it properly, while at the same time\nteaching the compiler that there are only two possible options to\nconsider. This will allow the compiler to write the most efficient\npossible code.\n\nCopy similar code into hasheq and perform an identical transformation.\nAt least with GCC 8.2.0, making hasheq defer to hashcmp when there are\ntwo branches prevents the compiler from inlining the comparison, while\nthe code in this patch is inlined properly. Add a comment to avoid an\naccidental performance regression from well-intentioned refactoring.\n\nSigned-off-by: brian m. carlson \n---\n cache.h | 22 ++++++++++++----------\n 1 file changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 51580c4b77..bab8e8964f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1027,16 +1027,12 @@ extern const struct object_id null_oid;\n static inline int hashcmp(const unsigned char *sha1, const unsigned char *sha2)\n {\n \t/*\n-\t * This is a temporary optimization hack. By asserting the size here,\n-\t * we let the compiler know that it's always going to be 20, which lets\n-\t * it turn this fixed-size memcmp into a few inline instructions.\n-\t *\n-\t * This will need to be extended or ripped out when we learn about\n-\t * hashes of different sizes.\n+\t * Teach the compiler that there are only two possibilities of hash size\n+\t * here, so that it can optimize for this case as much as possible.\n \t */\n-\tif (the_hash_algo->rawsz != 20)\n-\t\tBUG(\"hash size not yet supported by hashcmp\");\n-\treturn memcmp(sha1, sha2, the_hash_algo->rawsz);\n+\tif (the_hash_algo->rawsz == GIT_MAX_RAWSZ)\n+\t\treturn memcmp(sha1, sha2, GIT_MAX_RAWSZ);\n+\treturn memcmp(sha1, sha2, GIT_SHA1_RAWSZ);\n }\n \n static inline int oidcmp(const struct object_id *oid1, const struct object_id *oid2)\n@@ -1046,7 +1042,13 @@ static inline int oidcmp(const struct object_id *oid1, const struct object_id *o\n \n static inline int hasheq(const unsigned char *sha1, const unsigned char *sha2)\n {\n-\treturn !hashcmp(sha1, sha2);\n+\t/*\n+\t * We write this here instead of deferring to hashcmp so that the\n+\t * compiler can properly inline it and avoid calling memcmp.\n+\t */\n+\tif (the_hash_algo->rawsz == GIT_MAX_RAWSZ)\n+\t\treturn !memcmp(sha1, sha2, GIT_MAX_RAWSZ);\n+\treturn !memcmp(sha1, sha2, GIT_SHA1_RAWSZ);\n }\n \n static inline int oideq(const struct object_id *oid1, const struct object_id *oid2)\n"},{"id":"361453","messageId":"20181025024005.154208-6-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 05/12] t: add basic tests for our SHA-1 implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:39:58Z","receivedAt":"2018-10-25T02:40:22Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We have in the past had some unfortunate endianness issues with some\nSHA-1 implementations we ship, especially on big-endian machines. Add\nan explicit test using the test helper to catch these issues and point\nthem out prominently. This test can also be used as a staging ground\nfor people testing additional algorithms to verify that their\nimplementations are working as expected.\n\nSigned-off-by: brian m. carlson \n---\n t/t0015-hash.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n create mode 100755 t/t0015-hash.sh\n\ndiff --git a/t/t0015-hash.sh b/t/t0015-hash.sh\nnew file mode 100755\nindex 0000000000..8e763c2c3d\n--- /dev/null\n+++ b/t/t0015-hash.sh\n@@ -0,0 +1,29 @@\n+#!/bin/sh\n+\n+test_description='test basic hash implementation'\n+. ./test-lib.sh\n+\n+\n+test_expect_success 'test basic SHA-1 hash values' '\n+\ttest-tool sha1 actual &&\n+\tgrep da39a3ee5e6b4b0d3255bfef95601890afd80709 actual &&\n+\tprintf \"a\" | test-tool sha1 >actual &&\n+\tgrep 86f7e437faa5a7fce15d1ddcb9eaeaea377667b8 actual &&\n+\tprintf \"abc\" | test-tool sha1 >actual &&\n+\tgrep a9993e364706816aba3e25717850c26c9cd0d89d actual &&\n+\tprintf \"message digest\" | test-tool sha1 >actual &&\n+\tgrep c12252ceda8be8994d5fa0290a47231c1d16aae3 actual &&\n+\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha1 >actual &&\n+\tgrep 32d10c7b8cf96570ca04ce37f2a19d84240d3a89 actual &&\n+\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n+\t\ttest-tool sha1 >actual &&\n+\tgrep 34aa973cd4c4daa4f61eeb2bdbad27316534016f actual &&\n+\tprintf \"blob 0\\0\" | test-tool sha1 >actual &&\n+\tgrep e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 actual &&\n+\tprintf \"blob 3\\0abc\" | test-tool sha1 >actual &&\n+\tgrep f2ba8f84ab5c1bce84a7b441cb1959cfc7093b7f actual &&\n+\tprintf \"tree 0\\0\" | test-tool sha1 >actual &&\n+\tgrep 4b825dc642cb6eb9a060e54bf8d69288fbee4904 actual\n+'\n+\n+test_done\n"},{"id":"361454","messageId":"20181025024005.154208-7-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 06/12] t: make the sha1 test-tool helper generic","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:39:59Z","receivedAt":"2018-10-25T02:40:23Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Since we're going to have multiple hash algorithms to test, it makes\nsense to share as much of the test code as possible. Convert the sha1\nhelper for the test-tool to be generic and move it out into its own\nmodule. This will allow us to share most of this code with our NewHash\nimplementation.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 1 +\n t/helper/{test-sha1.c => test-hash.c} | 19 +++++-----\n t/helper/test-sha1.c | 52 +--------------------------\n t/helper/test-tool.h | 2 ++\n 4 files changed, 14 insertions(+), 60 deletions(-)\n copy t/helper/{test-sha1.c => test-hash.c} (65%)\n\ndiff --git a/Makefile b/Makefile\nindex d18ab0fe78..81dc9ac819 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -714,6 +714,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n+TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-hash.c\nsimilarity index 65%\ncopy from t/helper/test-sha1.c\ncopy to t/helper/test-hash.c\nindex 1ba0675c75..0a31de66f3 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-hash.c\n@@ -1,13 +1,14 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n \n-int cmd__sha1(int ac, const char **av)\n+int cmd_hash_impl(int ac, const char **av, int algo)\n {\n-\tgit_SHA_CTX ctx;\n-\tunsigned char sha1[20];\n+\tgit_hash_ctx ctx;\n+\tunsigned char hash[GIT_MAX_HEXSZ];\n \tunsigned bufsz = 8192;\n \tint binary = 0;\n \tchar *buffer;\n+\tconst struct git_hash_algo *algop = &hash_algos[algo];\n \n \tif (ac == 2) {\n \t\tif (!strcmp(av[1], \"-b\"))\n@@ -26,7 +27,7 @@ int cmd__sha1(int ac, const char **av)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\tgit_SHA1_Init(&ctx);\n+\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -38,20 +39,20 @@ int cmd__sha1(int ac, const char **av)\n \t\t\tif (sz == 0)\n \t\t\t\tbreak;\n \t\t\tif (sz < 0)\n-\t\t\t\tdie_errno(\"test-sha1\");\n+\t\t\t\tdie_errno(\"test-hash\");\n \t\t\tthis_sz += sz;\n \t\t\tcp += sz;\n \t\t\troom -= sz;\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\tgit_SHA1_Update(&ctx, buffer, this_sz);\n+\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\tgit_SHA1_Final(sha1, &ctx);\n+\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n-\t\tfwrite(sha1, 1, 20, stdout);\n+\t\tfwrite(hash, 1, algop->rawsz, stdout);\n \telse\n-\t\tputs(sha1_to_hex(sha1));\n+\t\tputs(hash_to_hex_algop(hash, algop));\n \texit(0);\n }\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-sha1.c\nindex 1ba0675c75..d860c387c3 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-sha1.c\n@@ -3,55 +3,5 @@\n \n int cmd__sha1(int ac, const char **av)\n {\n-\tgit_SHA_CTX ctx;\n-\tunsigned char sha1[20];\n-\tunsigned bufsz = 8192;\n-\tint binary = 0;\n-\tchar *buffer;\n-\n-\tif (ac == 2) {\n-\t\tif (!strcmp(av[1], \"-b\"))\n-\t\t\tbinary = 1;\n-\t\telse\n-\t\t\tbufsz = strtoul(av[1], NULL, 10) * 1024 * 1024;\n-\t}\n-\n-\tif (!bufsz)\n-\t\tbufsz = 8192;\n-\n-\twhile ((buffer = malloc(bufsz)) == NULL) {\n-\t\tfprintf(stderr, \"bufsz %u is too big, halving...\\n\", bufsz);\n-\t\tbufsz /= 2;\n-\t\tif (bufsz < 1024)\n-\t\t\tdie(\"OOPS\");\n-\t}\n-\n-\tgit_SHA1_Init(&ctx);\n-\n-\twhile (1) {\n-\t\tssize_t sz, this_sz;\n-\t\tchar *cp = buffer;\n-\t\tunsigned room = bufsz;\n-\t\tthis_sz = 0;\n-\t\twhile (room) {\n-\t\t\tsz = xread(0, cp, room);\n-\t\t\tif (sz == 0)\n-\t\t\t\tbreak;\n-\t\t\tif (sz < 0)\n-\t\t\t\tdie_errno(\"test-sha1\");\n-\t\t\tthis_sz += sz;\n-\t\t\tcp += sz;\n-\t\t\troom -= sz;\n-\t\t}\n-\t\tif (this_sz == 0)\n-\t\t\tbreak;\n-\t\tgit_SHA1_Update(&ctx, buffer, this_sz);\n-\t}\n-\tgit_SHA1_Final(sha1, &ctx);\n-\n-\tif (binary)\n-\t\tfwrite(sha1, 1, 20, stdout);\n-\telse\n-\t\tputs(sha1_to_hex(sha1));\n-\texit(0);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1);\n }\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e4890566da..29ac7b0b0d 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -50,4 +50,6 @@ int cmd__windows_named_pipe(int argc, const char **argv);\n #endif\n int cmd__write_cache(int argc, const char **argv);\n \n+int cmd_hash_impl(int ac, const char **av, int algo);\n+\n #endif\n"},{"id":"361455","messageId":"20181025024005.154208-8-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 07/12] sha1-file: add a constant for hash block size","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:40:00Z","receivedAt":"2018-10-25T02:40:25Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"There is one place we need the hash algorithm block size: the HMAC code\nfor push certs. Expose this constant in struct git_hash_algo and expose\nvalues for SHA-1 and for the largest value of any hash.\n\nSigned-off-by: brian m. carlson \n---\n cache.h | 4 ++++\n hash.h | 3 +++\n sha1-file.c | 2 ++\n 3 files changed, 9 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex bab8e8964f..9e5d1dd85a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -45,10 +45,14 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n+/* The block size of SHA-1. */\n+#define GIT_SHA1_BLKSZ 64\n \n /* The length in byte and in hex digits of the largest possible hash value. */\n #define GIT_MAX_RAWSZ GIT_SHA1_RAWSZ\n #define GIT_MAX_HEXSZ GIT_SHA1_HEXSZ\n+/* The largest possible block size for any supported hash. */\n+#define GIT_MAX_BLKSZ GIT_SHA1_BLKSZ\n \n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/hash.h b/hash.h\nindex 80881eea47..1bcf7ab6fd 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -81,6 +81,9 @@ struct git_hash_algo {\n \t/* The length of the hash in hex characters. */\n \tsize_t hexsz;\n \n+\t/* The block size of the hash. */\n+\tsize_t blksz;\n+\n \t/* The hash initialization function. */\n \tgit_hash_init_fn init_fn;\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex 7e9dedc744..9bdd04ea45 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -90,6 +90,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t0x00000000,\n \t\t0,\n \t\t0,\n+\t\t0,\n \t\tgit_hash_unknown_init,\n \t\tgit_hash_unknown_update,\n \t\tgit_hash_unknown_final,\n@@ -102,6 +103,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t0x73686131,\n \t\tGIT_SHA1_RAWSZ,\n \t\tGIT_SHA1_HEXSZ,\n+\t\tGIT_SHA1_BLKSZ,\n \t\tgit_hash_sha1_init,\n \t\tgit_hash_sha1_update,\n \t\tgit_hash_sha1_final,\n"},{"id":"361456","messageId":"20181025024005.154208-9-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 08/12] t/helper: add a test helper to compute hash speed","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:40:01Z","receivedAt":"2018-10-25T02:40:26Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a utility (which is less for the testsuite and more for developers)\nthat can compute hash speeds for whatever hash algorithms are\nimplemented. This allows developers to test their personal systems to\ndetermine the performance characteristics of various algorithms.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 1 +\n t/helper/test-hash-speed.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 64 insertions(+)\n create mode 100644 t/helper/test-hash-speed.c\n\ndiff --git a/Makefile b/Makefile\nindex 81dc9ac819..68169a7abb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -716,6 +716,7 @@ TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n+TEST_BUILTINS_OBJS += test-hash-speed.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\n TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\ndiff --git a/t/helper/test-hash-speed.c b/t/helper/test-hash-speed.c\nnew file mode 100644\nindex 0000000000..432233c7f0\n--- /dev/null\n+++ b/t/helper/test-hash-speed.c\n@@ -0,0 +1,61 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+#define NUM_SECONDS 3\n+\n+static inline void compute_hash(const struct git_hash_algo *algo, git_hash_ctx *ctx, uint8_t *final, const void *p, size_t len)\n+{\n+\talgo->init_fn(ctx);\n+\talgo->update_fn(ctx, p, len);\n+\talgo->final_fn(final, ctx);\n+}\n+\n+int cmd__hash_speed(int ac, const char **av)\n+{\n+\tgit_hash_ctx ctx;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n+\tclock_t initial, start, end;\n+\tunsigned bufsizes[] = { 64, 256, 1024, 8192, 16384 };\n+\tint i;\n+\tvoid *p;\n+\tconst struct git_hash_algo *algo = NULL;\n+\n+\tif (ac == 2) {\n+\t\tfor (i = 1; i < GIT_HASH_NALGOS; i++) {\n+\t\t\tif (!strcmp(av[1], hash_algos[i].name)) {\n+\t\t\t\talgo = &hash_algos[i];\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (!algo)\n+\t\tdie(\"usage: test-tool hash-speed algo_name\");\n+\n+\t/* Use this as an offset to make overflow less likely. */\n+\tinitial = clock();\n+\n+\tprintf(\"algo: %s\\n\", algo->name);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(bufsizes); i++) {\n+\t\tunsigned long j, kb;\n+\t\tdouble kb_per_sec;\n+\t\tp = xcalloc(1, bufsizes[i]);\n+\t\tstart = end = clock() - initial;\n+\t\tfor (j = 0; ((end - start) / CLOCKS_PER_SEC) < NUM_SECONDS; j++) {\n+\t\t\tcompute_hash(algo, &ctx, hash, p, bufsizes[i]);\n+\n+\t\t\t/*\n+\t\t\t * Only check elapsed time every 128 iterations to avoid\n+\t\t\t * dominating the runtime with system calls.\n+\t\t\t */\n+\t\t\tif (!(j & 127))\n+\t\t\t\tend = clock() - initial;\n+\t\t}\n+\t\tkb = j * bufsizes[i];\n+\t\tkb_per_sec = kb / (1024 * ((double)end - start) / CLOCKS_PER_SEC);\n+\t\tprintf(\"size %u: %lu iters; %lu KiB; %0.2f KiB/s\\n\", bufsizes[i], j, kb, kb_per_sec);\n+\t\tfree(p);\n+\t}\n+\n+\texit(0);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 6b5836dc1b..e009c8186d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -20,6 +20,7 @@ static struct test_cmd cmds[] = {\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"hashmap\", cmd__hashmap },\n+\t{ \"hash-speed\", cmd__hash_speed },\n \t{ \"index-version\", cmd__index_version },\n \t{ \"json-writer\", cmd__json_writer },\n \t{ \"lazy-init-name-hash\", cmd__lazy_init_name_hash },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 29ac7b0b0d..19a7e8332a 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -16,6 +16,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n+int cmd__hash_speed(int argc, const char **argv);\n int cmd__index_version(int argc, const char **argv);\n int cmd__json_writer(int argc, const char **argv);\n int cmd__lazy_init_name_hash(int argc, const char **argv);\n"},{"id":"361457","messageId":"20181025024005.154208-10-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 09/12] commit-graph: convert to using the_hash_algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:40:02Z","receivedAt":"2018-10-25T02:40:27Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Instead of using hard-coded constants for object sizes, use\nthe_hash_algo to look them up. In addition, use a function call to look\nup the object ID version and produce the correct value. For now, we use\nversion 1, which means to use the default algorithm used in the rest of\nthe repository.\n\nSigned-off-by: brian m. carlson \n---\n commit-graph.c | 33 +++++++++++++++++----------------\n 1 file changed, 17 insertions(+), 16 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..6763d19288 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -23,16 +23,11 @@\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n #define GRAPH_CHUNKID_LARGEEDGES 0x45444745 /* \"EDGE\" */\n \n-#define GRAPH_DATA_WIDTH 36\n+#define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n #define GRAPH_VERSION_1 0x1\n #define GRAPH_VERSION GRAPH_VERSION_1\n \n-#define GRAPH_OID_VERSION_SHA1 1\n-#define GRAPH_OID_LEN_SHA1 GIT_SHA1_RAWSZ\n-#define GRAPH_OID_VERSION GRAPH_OID_VERSION_SHA1\n-#define GRAPH_OID_LEN GRAPH_OID_LEN_SHA1\n-\n #define GRAPH_OCTOPUS_EDGES_NEEDED 0x80000000\n #define GRAPH_PARENT_MISSING 0x7fffffff\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n@@ -44,13 +39,18 @@\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n #define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n-\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n+\t\t\t+ GRAPH_FANOUT_SIZE + the_hash_algo->rawsz)\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static uint8_t oid_version(void)\n+{\n+\treturn 1;\n+}\n+\n static struct commit_graph *alloc_commit_graph(void)\n {\n \tstruct commit_graph *g = xcalloc(1, sizeof(*g));\n@@ -125,15 +125,15 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \t}\n \n \thash_version = *(unsigned char*)(data + 5);\n-\tif (hash_version != GRAPH_OID_VERSION) {\n+\tif (hash_version != oid_version()) {\n \t\terror(_(\"hash version %X does not match version %X\"),\n-\t\t hash_version, GRAPH_OID_VERSION);\n+\t\t hash_version, oid_version());\n \t\tgoto cleanup_fail;\n \t}\n \n \tgraph = alloc_commit_graph();\n \n-\tgraph->hash_len = GRAPH_OID_LEN;\n+\tgraph->hash_len = the_hash_algo->rawsz;\n \tgraph->num_chunks = *(unsigned char*)(data + 6);\n \tgraph->graph_fd = fd;\n \tgraph->data = graph_map;\n@@ -149,7 +149,7 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \n \t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n \n-\t\tif (chunk_offset > graph_size - GIT_MAX_RAWSZ) {\n+\t\tif (chunk_offset > graph_size - the_hash_algo->rawsz) {\n \t\t\terror(_(\"improper chunk offset %08x%08x\"), (uint32_t)(chunk_offset >> 32),\n \t\t\t (uint32_t)chunk_offset);\n \t\t\tgoto cleanup_fail;\n@@ -764,6 +764,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n+\tconst unsigned hashsz = the_hash_algo->rawsz;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -909,7 +910,7 @@ void write_commit_graph(const char *obj_dir,\n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, GRAPH_OID_VERSION);\n+\thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, num_chunks);\n \thashwrite_u8(f, 0); /* unused padding byte */\n \n@@ -924,8 +925,8 @@ void write_commit_graph(const char *obj_dir,\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\n \tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n@@ -938,8 +939,8 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\twrite_graph_chunk_oids(f, hashsz, commits.list, commits.nr);\n+\twrite_graph_chunk_data(f, hashsz, commits.list, commits.nr);\n \twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n"},{"id":"361458","messageId":"20181025024005.154208-11-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 10/12] Add a base implementation of SHA-256 support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:40:03Z","receivedAt":"2018-10-25T02:40:29Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"SHA-1 is weak and we need to transition to a new hash function. For\nsome time, we have referred to this new function as NewHash. Recently,\nwe decided to pick SHA-256 as NewHash.\n\nAdd a basic implementation of SHA-256 based off libtomcrypt, which is in\nthe public domain. Optimize it and restructure it to meet our coding\nstandards. Pull in the update and final functions from the SHA-1 block\nimplementation, as we know these function correctly with all compilers.\nThis implementation is slower than SHA-1, but more performant\nimplementations will be introduced in future commits.\n\nWire up SHA-256 in the list of hash algorithms, and add a test that the\nalgorithm works correctly.\n\nNote that with this patch, it is still not possible to switch to using\nSHA-256 in Git. Additional patches are needed to prepare the code to\nhandle a larger hash algorithm and further test fixes are needed.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 4 +\n cache.h | 12 ++-\n hash.h | 19 +++-\n sha1-file.c | 45 +++++++++\n sha256/block/sha256.c | 201 +++++++++++++++++++++++++++++++++++++++++\n sha256/block/sha256.h | 26 ++++++\n t/helper/test-sha256.c | 7 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0015-hash.sh | 25 +++++\n 10 files changed, 337 insertions(+), 4 deletions(-)\n create mode 100644 sha256/block/sha256.c\n create mode 100644 sha256/block/sha256.h\n create mode 100644 t/helper/test-sha256.c\n\ndiff --git a/Makefile b/Makefile\nindex 68169a7abb..e99b7712f6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -739,6 +739,7 @@ TEST_BUILTINS_OBJS += test-run-command.o\n TEST_BUILTINS_OBJS += test-scrap-cache-tree.o\n TEST_BUILTINS_OBJS += test-sha1.o\n TEST_BUILTINS_OBJS += test-sha1-array.o\n+TEST_BUILTINS_OBJS += test-sha256.o\n TEST_BUILTINS_OBJS += test-sigchain.o\n TEST_BUILTINS_OBJS += test-strcmp-offset.o\n TEST_BUILTINS_OBJS += test-string-list.o\n@@ -1633,6 +1634,9 @@ endif\n endif\n endif\n \n+LIB_OBJS += sha256/block/sha256.o\n+BASIC_CFLAGS += -DSHA256_BLK\n+\n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\n \tBASIC_CFLAGS += -DSHA1_MAX_BLOCK_SIZE=\"$(SHA1_MAX_BLOCK_SIZE)\"\ndiff --git a/cache.h b/cache.h\nindex 9e5d1dd85a..48ce1565e6 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -48,11 +48,17 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n /* The block size of SHA-1. */\n #define GIT_SHA1_BLKSZ 64\n \n+/* The length in bytes and in hex digits of an object name (SHA-256 value). */\n+#define GIT_SHA256_RAWSZ 32\n+#define GIT_SHA256_HEXSZ (2 * GIT_SHA256_RAWSZ)\n+/* The block size of SHA-256. */\n+#define GIT_SHA256_BLKSZ 64\n+\n /* The length in byte and in hex digits of the largest possible hash value. */\n-#define GIT_MAX_RAWSZ GIT_SHA1_RAWSZ\n-#define GIT_MAX_HEXSZ GIT_SHA1_HEXSZ\n+#define GIT_MAX_RAWSZ GIT_SHA256_RAWSZ\n+#define GIT_MAX_HEXSZ GIT_SHA256_HEXSZ\n /* The largest possible block size for any supported hash. */\n-#define GIT_MAX_BLKSZ GIT_SHA1_BLKSZ\n+#define GIT_MAX_BLKSZ GIT_SHA256_BLKSZ\n \n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/hash.h b/hash.h\nindex 1bcf7ab6fd..a9bc624020 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -15,6 +15,8 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n+#include \"sha256/block/sha256.h\"\n+\n #ifndef platform_SHA_CTX\n /*\n * platform's underlying implementation of SHA-1; could be OpenSSL,\n@@ -34,6 +36,18 @@\n #define git_SHA1_Update\t\tplatform_SHA1_Update\n #define git_SHA1_Final\t\tplatform_SHA1_Final\n \n+#ifndef platform_SHA256_CTX\n+#define platform_SHA256_CTX\tSHA256_CTX\n+#define platform_SHA256_Init\tSHA256_Init\n+#define platform_SHA256_Update\tSHA256_Update\n+#define platform_SHA256_Final\tSHA256_Final\n+#endif\n+\n+#define git_SHA256_CTX\t\tplatform_SHA256_CTX\n+#define git_SHA256_Init\t\tplatform_SHA256_Init\n+#define git_SHA256_Update\tplatform_SHA256_Update\n+#define git_SHA256_Final\tplatform_SHA256_Final\n+\n #ifdef SHA1_MAX_BLOCK_SIZE\n #include \"compat/sha1-chunked.h\"\n #undef git_SHA1_Update\n@@ -52,12 +66,15 @@\n #define GIT_HASH_UNKNOWN 0\n /* SHA-1 */\n #define GIT_HASH_SHA1 1\n+/* SHA-256 */\n+#define GIT_HASH_SHA256 2\n /* Number of algorithms supported (including unknown). */\n-#define GIT_HASH_NALGOS (GIT_HASH_SHA1 + 1)\n+#define GIT_HASH_NALGOS (GIT_HASH_SHA256 + 1)\n \n /* A suitably aligned type for stack allocations of hash contexts. */\n union git_hash_ctx {\n \tgit_SHA_CTX sha1;\n+\tgit_SHA256_CTX sha256;\n };\n typedef union git_hash_ctx git_hash_ctx;\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex 9bdd04ea45..c97d93a14a 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -40,10 +40,20 @@\n #define EMPTY_TREE_SHA1_BIN_LITERAL \\\n \t \"\\x4b\\x82\\x5d\\xc6\\x42\\xcb\\x6e\\xb9\\xa0\\x60\" \\\n \t \"\\xe5\\x4b\\xf8\\xd6\\x92\\x88\\xfb\\xee\\x49\\x04\"\n+#define EMPTY_TREE_SHA256_BIN_LITERAL \\\n+\t\"\\x6e\\xf1\\x9b\\x41\\x22\\x5c\\x53\\x69\\xf1\\xc1\" \\\n+\t\"\\x04\\xd4\\x5d\\x8d\\x85\\xef\\xa9\\xb0\\x57\\xb5\" \\\n+\t\"\\x3b\\x14\\xb4\\xb9\\xb9\\x39\\xdd\\x74\\xde\\xcc\" \\\n+\t\"\\x53\\x21\"\n \n #define EMPTY_BLOB_SHA1_BIN_LITERAL \\\n \t\"\\xe6\\x9d\\xe2\\x9b\\xb2\\xd1\\xd6\\x43\\x4b\\x8b\" \\\n \t\"\\x29\\xae\\x77\\x5a\\xd8\\xc2\\xe4\\x8c\\x53\\x91\"\n+#define EMPTY_BLOB_SHA256_BIN_LITERAL \\\n+\t\"\\x47\\x3a\\x0f\\x4c\\x3b\\xe8\\xa9\\x36\\x81\\xa2\" \\\n+\t\"\\x67\\xe3\\xb1\\xe9\\xa7\\xdc\\xda\\x11\\x85\\x43\" \\\n+\t\"\\x6f\\xe1\\x41\\xf7\\x74\\x91\\x20\\xa3\\x03\\x72\" \\\n+\t\"\\x18\\x13\"\n \n const unsigned char null_sha1[GIT_MAX_RAWSZ];\n const struct object_id null_oid;\n@@ -53,6 +63,12 @@ static const struct object_id empty_tree_oid = {\n static const struct object_id empty_blob_oid = {\n \tEMPTY_BLOB_SHA1_BIN_LITERAL\n };\n+static const struct object_id empty_tree_oid_sha256 = {\n+\tEMPTY_TREE_SHA256_BIN_LITERAL\n+};\n+static const struct object_id empty_blob_oid_sha256 = {\n+\tEMPTY_BLOB_SHA256_BIN_LITERAL\n+};\n \n static void git_hash_sha1_init(git_hash_ctx *ctx)\n {\n@@ -69,6 +85,22 @@ static void git_hash_sha1_final(unsigned char *hash, git_hash_ctx *ctx)\n \tgit_SHA1_Final(hash, &ctx->sha1);\n }\n \n+\n+static void git_hash_sha256_init(git_hash_ctx *ctx)\n+{\n+\tgit_SHA256_Init(&ctx->sha256);\n+}\n+\n+static void git_hash_sha256_update(git_hash_ctx *ctx, const void *data, size_t len)\n+{\n+\tgit_SHA256_Update(&ctx->sha256, data, len);\n+}\n+\n+static void git_hash_sha256_final(unsigned char *hash, git_hash_ctx *ctx)\n+{\n+\tgit_SHA256_Final(hash, &ctx->sha256);\n+}\n+\n static void git_hash_unknown_init(git_hash_ctx *ctx)\n {\n \tBUG(\"trying to init unknown hash\");\n@@ -110,6 +142,19 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t&empty_tree_oid,\n \t\t&empty_blob_oid,\n \t},\n+\t{\n+\t\t\"sha256\",\n+\t\t/* \"s256\", big-endian */\n+\t\t0x73323536,\n+\t\tGIT_SHA256_RAWSZ,\n+\t\tGIT_SHA256_HEXSZ,\n+\t\tGIT_SHA256_BLKSZ,\n+\t\tgit_hash_sha256_init,\n+\t\tgit_hash_sha256_update,\n+\t\tgit_hash_sha256_final,\n+\t\t&empty_tree_oid_sha256,\n+\t\t&empty_blob_oid_sha256,\n+\t}\n };\n \n const char *empty_tree_oid_hex(void)\ndiff --git a/sha256/block/sha256.c b/sha256/block/sha256.c\nnew file mode 100644\nindex 0000000000..fa9f80b350\n--- /dev/null\n+++ b/sha256/block/sha256.c\n@@ -0,0 +1,201 @@\n+#include \"git-compat-util.h\"\n+#include \"./sha256.h\"\n+\n+#undef RND\n+#undef BLKSIZE\n+\n+#define BLKSIZE blk_SHA256_BLKSIZE\n+\n+void blk_SHA256_Init(blk_SHA256_CTX *ctx)\n+{\n+\tctx->offset = 0;\n+\tctx->size = 0;\n+\tctx->state[0] = 0x6A09E667UL;\n+\tctx->state[1] = 0xBB67AE85UL;\n+\tctx->state[2] = 0x3C6EF372UL;\n+\tctx->state[3] = 0xA54FF53AUL;\n+\tctx->state[4] = 0x510E527FUL;\n+\tctx->state[5] = 0x9B05688CUL;\n+\tctx->state[6] = 0x1F83D9ABUL;\n+\tctx->state[7] = 0x5BE0CD19UL;\n+}\n+\n+static inline uint32_t ror(uint32_t x, unsigned n)\n+{\n+\treturn (x >> n) | (x << (32 - n));\n+}\n+\n+static inline uint32_t ch(uint32_t x, uint32_t y, uint32_t z)\n+{\n+\treturn (z ^ (x & (y ^ z)));\n+}\n+\n+static inline uint32_t maj(uint32_t x, uint32_t y, uint32_t z)\n+{\n+\treturn (((x | y) & z) | (x & y));\n+}\n+\n+static inline uint32_t sigma0(uint32_t x)\n+{\n+\treturn ror(x, 2) ^ ror(x, 13) ^ ror(x, 22);\n+}\n+\n+static inline uint32_t sigma1(uint32_t x)\n+{\n+\treturn ror(x, 6) ^ ror(x, 11) ^ ror(x, 25);\n+}\n+\n+static inline uint32_t gamma0(uint32_t x)\n+{\n+\treturn ror(x, 7) ^ ror(x, 18) ^ (x >> 3);\n+}\n+\n+static inline uint32_t gamma1(uint32_t x)\n+{\n+\treturn ror(x, 17) ^ ror(x, 19) ^ (x >> 10);\n+}\n+\n+static void blk_SHA256_Transform(blk_SHA256_CTX *ctx, const unsigned char *buf)\n+{\n+\n+\tuint32_t S[8], W[64], t0, t1;\n+\tint i;\n+\n+\t/* copy state into S */\n+\tfor (i = 0; i < 8; i++) {\n+\t\tS[i] = ctx->state[i];\n+\t}\n+\n+\t/* copy the state into 512-bits into W[0..15] */\n+\tfor (i = 0; i < 16; i++, buf += sizeof(uint32_t)) {\n+\t\tW[i] = get_be32(buf);\n+\t}\n+\n+\t/* fill W[16..63] */\n+\tfor (i = 16; i < 64; i++) {\n+\t\tW[i] = gamma1(W[i - 2]) + W[i - 7] + gamma0(W[i - 15]) + W[i - 16];\n+\t}\n+\n+#define RND(a,b,c,d,e,f,g,h,i,ki) \\\n+\tt0 = h + sigma1(e) + ch(e, f, g) + ki + W[i]; \\\n+\tt1 = sigma0(a) + maj(a, b, c); \\\n+\td += t0; \\\n+\th = t0 + t1;\n+\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x71374491);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcf);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba5);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25b);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],5,0x59f111f1);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],6,0x923f82a4);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],7,0xab1c5ed5);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],8,0xd807aa98);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],9,0x12835b01);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],10,0x243185be);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],11,0x550c7dc3);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],12,0x72be5d74);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],13,0x80deb1fe);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],14,0x9bdc06a7);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],15,0xc19bf174);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],16,0xe49b69c1);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],17,0xefbe4786);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],18,0x0fc19dc6);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],19,0x240ca1cc);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],20,0x2de92c6f);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],21,0x4a7484aa);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],22,0x5cb0a9dc);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],23,0x76f988da);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],24,0x983e5152);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],25,0xa831c66d);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],26,0xb00327c8);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],27,0xbf597fc7);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],28,0xc6e00bf3);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],29,0xd5a79147);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],30,0x06ca6351);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],31,0x14292967);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],32,0x27b70a85);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],33,0x2e1b2138);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],34,0x4d2c6dfc);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],35,0x53380d13);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],36,0x650a7354);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],37,0x766a0abb);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],38,0x81c2c92e);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],39,0x92722c85);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],40,0xa2bfe8a1);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],41,0xa81a664b);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],42,0xc24b8b70);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],43,0xc76c51a3);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],44,0xd192e819);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],45,0xd6990624);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],46,0xf40e3585);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],47,0x106aa070);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],48,0x19a4c116);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],49,0x1e376c08);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],50,0x2748774c);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],51,0x34b0bcb5);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],52,0x391c0cb3);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],53,0x4ed8aa4a);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],54,0x5b9cca4f);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],55,0x682e6ff3);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],56,0x748f82ee);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],57,0x78a5636f);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],58,0x84c87814);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],59,0x8cc70208);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],60,0x90befffa);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],61,0xa4506ceb);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],62,0xbef9a3f7);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],63,0xc67178f2);\n+\n+\n+\tfor (i = 0; i < 8; i++) {\n+\t\tctx->state[i] = ctx->state[i] + S[i];\n+\t}\n+}\n+\n+void blk_SHA256_Update(blk_SHA256_CTX *ctx, const void *data, size_t len)\n+{\n+\tunsigned int len_buf = ctx->size & 63;\n+\n+\tctx->size += len;\n+\n+\t/* Read the data into buf and process blocks as they get full */\n+\tif (len_buf) {\n+\t\tunsigned int left = 64 - len_buf;\n+\t\tif (len < left)\n+\t\t\tleft = len;\n+\t\tmemcpy(len_buf + ctx->buf, data, left);\n+\t\tlen_buf = (len_buf + left) & 63;\n+\t\tlen -= left;\n+\t\tdata = ((const char *)data + left);\n+\t\tif (len_buf)\n+\t\t\treturn;\n+\t\tblk_SHA256_Transform(ctx, ctx->buf);\n+\t}\n+\twhile (len >= 64) {\n+\t\tblk_SHA256_Transform(ctx, data);\n+\t\tdata = ((const char *)data + 64);\n+\t\tlen -= 64;\n+\t}\n+\tif (len)\n+\t\tmemcpy(ctx->buf, data, len);\n+}\n+\n+void blk_SHA256_Final(unsigned char *digest, blk_SHA256_CTX *ctx)\n+{\n+\tstatic const unsigned char pad[64] = { 0x80 };\n+\tunsigned int padlen[2];\n+\tint i;\n+\n+\t/* Pad with a binary 1 (ie 0x80), then zeroes, then length */\n+\tpadlen[0] = htonl((uint32_t)(ctx->size >> 29));\n+\tpadlen[1] = htonl((uint32_t)(ctx->size << 3));\n+\n+\ti = ctx->size & 63;\n+\tblk_SHA256_Update(ctx, pad, 1 + (63 & (55 - i)));\n+\tblk_SHA256_Update(ctx, padlen, 8);\n+\n+\t/* copy output */\n+\tfor (i = 0; i < 8; i++, digest += sizeof(uint32_t))\n+\t\tput_be32(digest, ctx->state[i]);\n+}\ndiff --git a/sha256/block/sha256.h b/sha256/block/sha256.h\nnew file mode 100644\nindex 0000000000..38f02f7e6c\n--- /dev/null\n+++ b/sha256/block/sha256.h\n@@ -0,0 +1,26 @@\n+#ifndef SHA256_BLOCK_SHA256_H\n+#define SHA256_BLOCK_SHA256_H\n+\n+#include \"git-compat-util.h\"\n+\n+#define blk_SHA256_BLKSIZE 64\n+\n+struct blk_SHA256_CTX {\n+\tuint32_t state[8];\n+\tuint64_t size;\n+\tuint32_t offset;\n+\tuint8_t buf[blk_SHA256_BLKSIZE];\n+};\n+\n+typedef struct blk_SHA256_CTX blk_SHA256_CTX;\n+\n+void blk_SHA256_Init(blk_SHA256_CTX *ctx);\n+void blk_SHA256_Update(blk_SHA256_CTX *ctx, const void *data, size_t len);\n+void blk_SHA256_Final(unsigned char *digest, blk_SHA256_CTX *ctx);\n+\n+#define platform_SHA256_CTX blk_SHA256_CTX\n+#define platform_SHA256_Init blk_SHA256_Init\n+#define platform_SHA256_Update blk_SHA256_Update\n+#define platform_SHA256_Final blk_SHA256_Final\n+\n+#endif\ndiff --git a/t/helper/test-sha256.c b/t/helper/test-sha256.c\nnew file mode 100644\nindex 0000000000..0ac6a99d5f\n--- /dev/null\n+++ b/t/helper/test-sha256.c\n@@ -0,0 +1,7 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+int cmd__sha256(int ac, const char **av)\n+{\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex e009c8186d..2a65193514 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -43,6 +43,7 @@ static struct test_cmd cmds[] = {\n \t{ \"scrap-cache-tree\", cmd__scrap_cache_tree },\n \t{ \"sha1\", cmd__sha1 },\n \t{ \"sha1-array\", cmd__sha1_array },\n+\t{ \"sha256\", cmd__sha256 },\n \t{ \"sigchain\", cmd__sigchain },\n \t{ \"strcmp-offset\", cmd__strcmp_offset },\n \t{ \"string-list\", cmd__string_list },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 19a7e8332a..2e66a8e47b 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -39,6 +39,7 @@ int cmd__run_command(int argc, const char **argv);\n int cmd__scrap_cache_tree(int argc, const char **argv);\n int cmd__sha1(int argc, const char **argv);\n int cmd__sha1_array(int argc, const char **argv);\n+int cmd__sha256(int argc, const char **argv);\n int cmd__sigchain(int argc, const char **argv);\n int cmd__strcmp_offset(int argc, const char **argv);\n int cmd__string_list(int argc, const char **argv);\ndiff --git a/t/t0015-hash.sh b/t/t0015-hash.sh\nindex 8e763c2c3d..f8e639743f 100755\n--- a/t/t0015-hash.sh\n+++ b/t/t0015-hash.sh\n@@ -26,4 +26,29 @@ test_expect_success 'test basic SHA-1 hash values' '\n \tgrep 4b825dc642cb6eb9a060e54bf8d69288fbee4904 actual\n '\n \n+test_expect_success 'test basic SHA-256 hash values' '\n+\ttest-tool sha256 actual &&\n+\tgrep e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 actual &&\n+\tprintf \"a\" | test-tool sha256 >actual &&\n+\tgrep ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb actual &&\n+\tprintf \"abc\" | test-tool sha256 >actual &&\n+\tgrep ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad actual &&\n+\tprintf \"message digest\" | test-tool sha256 >actual &&\n+\tgrep f7846f55cf23e14eebeab5b4e1550cad5b509e3348fbc4efa3a1413d393cb650 actual &&\n+\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha256 >actual &&\n+\tgrep 71c480df93d6ae2f1efad1447c66c9525e316218cf51fc8d9ed832f2daf18b73 actual &&\n+\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n+\t\ttest-tool sha256 >actual &&\n+\tgrep cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0 actual &&\n+\tperl -E \"for (1..100000) { print q{abcdefghijklmnopqrstuvwxyz}; }\" | \\\n+\t\ttest-tool sha256 >actual &&\n+\tgrep e406ba321ca712ad35a698bf0af8d61fc4dc40eca6bdcea4697962724ccbde35 actual &&\n+\tprintf \"blob 0\\0\" | test-tool sha256 >actual &&\n+\tgrep 473a0f4c3be8a93681a267e3b1e9a7dcda1185436fe141f7749120a303721813 actual &&\n+\tprintf \"blob 3\\0abc\" | test-tool sha256 >actual &&\n+\tgrep c1cf6e465077930e88dc5136641d402f72a229ddd996f627d60e9639eaba35a6 actual &&\n+\tprintf \"tree 0\\0\" | test-tool sha256 >actual &&\n+\tgrep 6ef19b41225c5369f1c104d45d8d85efa9b057b53b14b4b9b939dd74decc5321 actual\n+'\n+\n test_done\n"},{"id":"361459","messageId":"20181025024005.154208-12-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 11/12] sha256: add an SHA-256 implementation using libgcrypt","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:40:04Z","receivedAt":"2018-10-25T02:40:31Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Generally, one gets better performance out of cryptographic routines\nwritten in assembly than C, and this is also true for SHA-256. In\naddition, most Linux distributions cannot distribute Git linked against\nOpenSSL for licensing reasons.\n\nMost systems with GnuPG will also have libgcrypt, since it is a\ndependency of GnuPG. libgcrypt is also faster than the SHA1DC\nimplementation for messages of a few KiB and larger.\n\nFor comparison, on a Core i7-6600U, this implementation processes 16 KiB\nchunks at 355 MiB/s while SHA1DC processes equivalent chunks at 337\nMiB/s.\n\nIn addition, libgcrypt is licensed under the LGPL 2.1, which is\ncompatible with the GPL. Add an implementation of SHA-256 that uses\nlibgcrypt.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 13 +++++++++++--\n hash.h | 4 ++++\n sha256/gcrypt.h | 30 ++++++++++++++++++++++++++++++\n 3 files changed, 45 insertions(+), 2 deletions(-)\n create mode 100644 sha256/gcrypt.h\n\ndiff --git a/Makefile b/Makefile\nindex e99b7712f6..5a07e03100 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -179,6 +179,10 @@ all::\n # in one call to the platform's SHA1_Update(). e.g. APPLE_COMMON_CRYPTO\n # wants 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined.\n #\n+# Define BLK_SHA256 to use the built-in SHA-256 routines.\n+#\n+# Define GCRYPT_SHA256 to use the SHA-256 routines in libgcrypt.\n+#\n # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).\n #\n # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).\n@@ -1634,8 +1638,13 @@ endif\n endif\n endif\n \n-LIB_OBJS += sha256/block/sha256.o\n-BASIC_CFLAGS += -DSHA256_BLK\n+ifdef GCRYPT_SHA256\n+\tBASIC_CFLAGS += -DSHA256_GCRYPT\n+\tEXTLIBS += -lgcrypt\n+else\n+\tLIB_OBJS += sha256/block/sha256.o\n+\tBASIC_CFLAGS += -DSHA256_BLK\n+endif\n \n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\ndiff --git a/hash.h b/hash.h\nindex a9bc624020..2ef098052d 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -15,7 +15,11 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n+#if defined(SHA256_GCRYPT)\n+#include \"sha256/gcrypt.h\"\n+#else\n #include \"sha256/block/sha256.h\"\n+#endif\n \n #ifndef platform_SHA_CTX\n /*\ndiff --git a/sha256/gcrypt.h b/sha256/gcrypt.h\nnew file mode 100644\nindex 0000000000..09bd8bb200\n--- /dev/null\n+++ b/sha256/gcrypt.h\n@@ -0,0 +1,30 @@\n+#ifndef SHA256_GCRYPT_H\n+#define SHA256_GCRYPT_H\n+\n+#include \n+\n+#define SHA256_DIGEST_SIZE 32\n+\n+typedef gcry_md_hd_t gcrypt_SHA256_CTX;\n+\n+inline void gcrypt_SHA256_Init(gcrypt_SHA256_CTX *ctx)\n+{\n+\tgcry_md_open(ctx, GCRY_MD_SHA256, 0);\n+}\n+\n+inline void gcrypt_SHA256_Update(gcrypt_SHA256_CTX *ctx, const void *data, size_t len)\n+{\n+\tgcry_md_write(*ctx, data, len);\n+}\n+\n+inline void gcrypt_SHA256_Final(unsigned char *digest, gcrypt_SHA256_CTX *ctx)\n+{\n+\tmemcpy(digest, gcry_md_read(*ctx, GCRY_MD_SHA256), SHA256_DIGEST_SIZE);\n+}\n+\n+#define platform_SHA256_CTX gcrypt_SHA256_CTX\n+#define platform_SHA256_Init gcrypt_SHA256_Init\n+#define platform_SHA256_Update gcrypt_SHA256_Update\n+#define platform_SHA256_Final gcrypt_SHA256_Final\n+\n+#endif\n"},{"id":"361460","messageId":"20181025024005.154208-13-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v4 12/12] hash: add an SHA-256 implementation using OpenSSL","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-25T02:40:05Z","receivedAt":"2018-10-25T02:40:31Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We already have OpenSSL routines available for SHA-1, so add routines\nfor SHA-256 as well.\n\nOn a Core i7-6600U, this SHA-256 implementation compares favorably to\nthe SHA1DC SHA-1 implementation:\n\nSHA-1: 157 MiB/s (64 byte chunks); 337 MiB/s (16 KiB chunks)\nSHA-256: 165 MiB/s (64 byte chunks); 408 MiB/s (16 KiB chunks)\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 7 +++++++\n hash.h | 2 ++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 5a07e03100..36fd3a149b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -183,6 +183,8 @@ all::\n #\n # Define GCRYPT_SHA256 to use the SHA-256 routines in libgcrypt.\n #\n+# Define OPENSSL_SHA256 to use the SHA-256 routines in OpenSSL.\n+#\n # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).\n #\n # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).\n@@ -1638,6 +1640,10 @@ endif\n endif\n endif\n \n+ifdef OPENSSL_SHA256\n+\tEXTLIBS += $(LIB_4_CRYPTO)\n+\tBASIC_CFLAGS += -DSHA256_OPENSSL\n+else\n ifdef GCRYPT_SHA256\n \tBASIC_CFLAGS += -DSHA256_GCRYPT\n \tEXTLIBS += -lgcrypt\n@@ -1645,6 +1651,7 @@ else\n \tLIB_OBJS += sha256/block/sha256.o\n \tBASIC_CFLAGS += -DSHA256_BLK\n endif\n+endif\n \n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\ndiff --git a/hash.h b/hash.h\nindex 2ef098052d..adde708cf2 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -17,6 +17,8 @@\n \n #if defined(SHA256_GCRYPT)\n #include \"sha256/gcrypt.h\"\n+#elif defined(SHA256_OPENSSL)\n+#include \n #else\n #include \"sha256/block/sha256.h\"\n #endif\n"},{"id":"361461","messageId":"CAPUEspjCjFiwCO8TCM23f2jqFSpy2z05+Ea_zGATy0jh1L83iQ@mail.gmail.com","threadId":"49668","inReplyTo":"20181025024005.154208-11-sandals@crustytoothpaste.net","subject":"Re: [PATCH v4 10/12] Add a base implementation of SHA-256 support","fromName":"Carlo Arenas","fromEmail":"carenas@gmail.com","sentAt":"2018-10-25T03:02:55Z","receivedAt":"2018-10-25T03:03:08Z","isPatch":true,"sender":{"key":"carenas@gmail.com","avatar":"https://avatars.githubusercontent.com/u/76036?v=4"},"body":"On Wed, Oct 24, 2018 at 7:41 PM brian m. carlson\n wrote:\n> diff --git a/sha256/block/sha256.h b/sha256/block/sha256.h\n> new file mode 100644\n> index 0000000000..38f02f7e6c\n> --- /dev/null\n> +++ b/sha256/block/sha256.h\n> @@ -0,0 +1,26 @@\n> +#ifndef SHA256_BLOCK_SHA256_H\n> +#define SHA256_BLOCK_SHA256_H\n> +\n> +#include \"git-compat-util.h\"\n\nthis shouldn't be needed and might be discouraged as per the\ninstructions in Documentation/CodingGuidelines\n\nCarlo\n"},{"id":"361779","messageId":"CAP8UFD3oYY=brGz1HZ67vxaRrxdwCdwNaJ9n78yJy9bUncW9pQ@mail.gmail.com","threadId":"49668","inReplyTo":"20181025024005.154208-11-sandals@crustytoothpaste.net","subject":"Re: [PATCH v4 10/12] Add a base implementation of SHA-256 support","fromName":"Christian Couder","fromEmail":"christian.couder@gmail.com","sentAt":"2018-10-27T09:03:40Z","receivedAt":"2018-10-27T09:03:56Z","isPatch":true,"sender":{"key":"christian.couder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/208954?v=4"},"body":"On Thu, Oct 25, 2018 at 4:42 AM brian m. carlson\n wrote:\n\n> +static void blk_SHA256_Transform(blk_SHA256_CTX *ctx, const unsigned char *buf)\n> +{\n> +\n> + uint32_t S[8], W[64], t0, t1;\n> + int i;\n> +\n> + /* copy state into S */\n> + for (i = 0; i < 8; i++) {\n> + S[i] = ctx->state[i];\n> + }\n\nMaybe remove unnecessary brackets above and below.\n\n> + /* copy the state into 512-bits into W[0..15] */\n> + for (i = 0; i < 16; i++, buf += sizeof(uint32_t)) {\n> + W[i] = get_be32(buf);\n> + }\n> +\n> + /* fill W[16..63] */\n> + for (i = 16; i < 64; i++) {\n> + W[i] = gamma1(W[i - 2]) + W[i - 7] + gamma0(W[i - 15]) + W[i - 16];\n> + }\n\n[...]\n\n> + RND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],62,0xbef9a3f7);\n> + RND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],63,0xc67178f2);\n> +\n> +\n\nSpurious new line.\n\n> + for (i = 0; i < 8; i++) {\n> + ctx->state[i] = ctx->state[i] + S[i];\n> + }\n\nMaybe remove unnecessary brackets and use \"+=\", like:\n\n for (i = 0; i < 8; i++)\n ctx->state[i] += S[i];\n"},{"id":"361843","messageId":"20181028155206.GI6119@genre.crustytoothpaste.net","threadId":"49668","inReplyTo":"CAPUEspjCjFiwCO8TCM23f2jqFSpy2z05+Ea_zGATy0jh1L83iQ@mail.gmail.com","subject":"Re: [PATCH v4 10/12] Add a base implementation of SHA-256 support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-28T15:52:06Z","receivedAt":"2018-10-28T15:52:17Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Wed, Oct 24, 2018 at 08:02:55PM -0700, Carlo Arenas wrote:\n> On Wed, Oct 24, 2018 at 7:41 PM brian m. carlson\n> wrote:\n> > diff --git a/sha256/block/sha256.h b/sha256/block/sha256.h\n> > new file mode 100644\n> > index 0000000000..38f02f7e6c\n> > --- /dev/null\n> > +++ b/sha256/block/sha256.h\n> > @@ -0,0 +1,26 @@\n> > +#ifndef SHA256_BLOCK_SHA256_H\n> > +#define SHA256_BLOCK_SHA256_H\n> > +\n> > +#include \"git-compat-util.h\"\n> \n> this shouldn't be needed and might be discouraged as per the\n> instructions in Documentation/CodingGuidelines\n\nThis may not strictly be needed, but removing it makes the header no\nlonger self-contained, which blows up my (and others') in-editor\nlinting. I think it's okay to add this extra header here to keep it\nself-contained, even if we know that it's not going to be absolutely\nrequired.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"361854","messageId":"xmqqftwpa8my.fsf@gitster-ct.c.googlers.com","threadId":"49668","inReplyTo":"20181028155206.GI6119@genre.crustytoothpaste.net","subject":"Re: [PATCH v4 10/12] Add a base implementation of SHA-256 support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-10-29T00:39:33Z","receivedAt":"2018-10-29T00:42:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n>> > +\n>> > +#include \"git-compat-util.h\"\n>> \n>> this shouldn't be needed and might be discouraged as per the\n>> instructions in Documentation/CodingGuidelines\n>\n> This may not strictly be needed, but removing it makes the header no\n> longer self-contained, which blows up my (and others') in-editor\n> linting.\n\nThat sounds like bending backwards to please tools, though. Can't\nthese in-editor linting learn the local rules of the codebase they\nare asked to operate on?\n"},{"id":"362134","messageId":"20181031225521.GA573515@genre.crustytoothpaste.net","threadId":"49668","inReplyTo":"xmqqftwpa8my.fsf@gitster-ct.c.googlers.com","subject":"Re: [PATCH v4 10/12] Add a base implementation of SHA-256 support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-10-31T22:55:21Z","receivedAt":"2018-10-31T22:55:30Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Oct 29, 2018 at 09:39:33AM +0900, Junio C Hamano wrote:\n> \"brian m. carlson\" writes:\n> > This may not strictly be needed, but removing it makes the header no\n> > longer self-contained, which blows up my (and others') in-editor\n> > linting.\n> \n> That sounds like bending backwards to please tools, though. Can't\n> these in-editor linting learn the local rules of the codebase they\n> are asked to operate on?\n\nDoing so involves writing (in my case) Vim code to configure settings\nfor this repo.\n\nSince language linting tools invoke compilers and other language\nruntimes, you need to specify command-line arguments to those tools, and\nin general, that's not a safe thing you can read from the repository\nconfiguration, since just viewing files should not be able to execute\narbitrary code[0]. Languages such as Perl which can execute arbitrary\ncode with compile checks have to be enabled explicitly with ALE (which\nis what I'm using).\n\nI need to do a reroll to address some other issues people brought up, so\nI can remove this line.\n\n[0] Pass \"-o .bashrc\" to the preprocessor, for example.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"362149","messageId":"xmqqlg6dtlgd.fsf@gitster-ct.c.googlers.com","threadId":"49668","inReplyTo":"20181031225521.GA573515@genre.crustytoothpaste.net","subject":"Re: [PATCH v4 10/12] Add a base implementation of SHA-256 support","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-01T05:29:06Z","receivedAt":"2018-11-01T05:29:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> I need to do a reroll to address some other issues people brought up, so\n> I can remove this line.\n\nOK. I actually do not feel too strongly about this one, by the way.\n"},{"id":"362418","messageId":"20181104234458.139223-1-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181025024005.154208-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 00/12] Base SHA-256 implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:46Z","receivedAt":"2018-11-04T23:45:12Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This series provides a functional SHA-256 implementation and wires it\nup, along with some housekeeping patches to make it suitable for\ntesting.\n\nChanges from v4:\n* Downcase hex constants for consistency.\n* Remove needless parentheses in return statement.\n* Remove braces for single statement loops.\n* Switch to +=.\n* Add references to rationale for SHA-256.\n* Remove inclusion of \"git-compat-util.h\" in header.\n\nChanges from v3:\n* Switch to using inline functions instead of macros in many cases.\n* Undefine remaining macros at the top.\n\nChanges from v2:\n* Improve commit messages to include timing and performance information.\n* Improve commit messages to be less ambiguous and more friendly to a\n wider variety of English speakers.\n* Prefer functions taking struct git_hash_algo in hex.c.\n* Port pieces of the block-sha1 implementation over to the block-sha256\n implementation for better compatibility.\n* Drop patch 13 in favor of further discussion about the best way\n forward for versioning commit graph.\n* Rename the test so as to have a different number from other tests.\n* Rebase on master.\n\nChanges from v1:\n* Add a hash_to_hex function mirroring sha1_to_hex, but for\n the_hash_algo.\n* Strip commit message explanation about why we chose SHA-256.\n* Rebase on master\n* Strip leading whitespace from commit message.\n* Improve commit-graph patch to cover new code added since v1.\n* Be more honest about the scope of work involved in porting the SHA-256\n implementation out of libtomcrypt.\n* Revert change to limit hashcmp to 20 bytes.\n\nbrian m. carlson (12):\n sha1-file: rename algorithm to \"sha1\"\n sha1-file: provide functions to look up hash algorithms\n hex: introduce functions to print arbitrary hashes\n cache: make hashcmp and hasheq work with larger hashes\n t: add basic tests for our SHA-1 implementation\n t: make the sha1 test-tool helper generic\n sha1-file: add a constant for hash block size\n t/helper: add a test helper to compute hash speed\n commit-graph: convert to using the_hash_algo\n Add a base implementation of SHA-256 support\n sha256: add an SHA-256 implementation using libgcrypt\n hash: add an SHA-256 implementation using OpenSSL\n\n Makefile | 22 +++\n cache.h | 51 ++++---\n commit-graph.c | 33 ++---\n hash.h | 41 +++++-\n hex.c | 32 +++--\n sha1-file.c | 70 ++++++++-\n sha256/block/sha256.c | 196 ++++++++++++++++++++++++++\n sha256/block/sha256.h | 26 ++++\n sha256/gcrypt.h | 30 ++++\n t/helper/test-hash-speed.c | 61 ++++++++\n t/helper/{test-sha1.c => test-hash.c} | 19 +--\n t/helper/test-sha1.c | 52 +------\n t/helper/test-sha256.c | 7 +\n t/helper/test-tool.c | 2 +\n t/helper/test-tool.h | 4 +\n t/t0015-hash.sh | 54 +++++++\n 16 files changed, 596 insertions(+), 104 deletions(-)\n create mode 100644 sha256/block/sha256.c\n create mode 100644 sha256/block/sha256.h\n create mode 100644 sha256/gcrypt.h\n create mode 100644 t/helper/test-hash-speed.c\n copy t/helper/{test-sha1.c => test-hash.c} (65%)\n create mode 100644 t/helper/test-sha256.c\n create mode 100755 t/t0015-hash.sh\n\nRange-diff against v4:\n 1: a004a4c982 < -: ---------- :hash-impl\n 2: cf9f7f5620 = 1: cf9f7f5620 sha1-file: rename algorithm to \"sha1\"\n 3: 0144deaebe = 2: 0144deaebe sha1-file: provide functions to look up hash algorithms\n 4: b74858fb03 = 3: b74858fb03 hex: introduce functions to print arbitrary hashes\n 5: e9703017a4 = 4: e9703017a4 cache: make hashcmp and hasheq work with larger hashes\n 6: ab85a834fd = 5: ab85a834fd t: add basic tests for our SHA-1 implementation\n 7: 962f6d8903 = 6: 962f6d8903 t: make the sha1 test-tool helper generic\n 8: 53addf4d58 = 7: 53addf4d58 sha1-file: add a constant for hash block size\n 9: 9ace10faa2 = 8: 9ace10faa2 t/helper: add a test helper to compute hash speed\n10: 9adc56d01e = 9: 9adc56d01e commit-graph: convert to using the_hash_algo\n11: f48cb1ad27 ! 10: 90544c504c Add a base implementation of SHA-256 support\n @@ -4,7 +4,9 @@\n \n SHA-1 is weak and we need to transition to a new hash function. For\n some time, we have referred to this new function as NewHash. Recently,\n - we decided to pick SHA-256 as NewHash.\n + we decided to pick SHA-256 as NewHash. The reasons behind the choice of\n + SHA-256 are outlined in the thread starting at [1] and in the commit\n + history for the hash function transition document.\n \n Add a basic implementation of SHA-256 based off libtomcrypt, which is in\n the public domain. Optimize it and restructure it to meet our coding\n @@ -20,6 +22,8 @@\n SHA-256 in Git. Additional patches are needed to prepare the code to\n handle a larger hash algorithm and further test fixes are needed.\n \n + [1] https://public-inbox.org/git/20180609224913.GC38834@genre.crustytoothpaste.net/\n +\n Signed-off-by: brian m. carlson \n \n diff --git a/Makefile b/Makefile\n @@ -216,14 +220,14 @@\n +{\n +\tctx->offset = 0;\n +\tctx->size = 0;\n -+\tctx->state[0] = 0x6A09E667UL;\n -+\tctx->state[1] = 0xBB67AE85UL;\n -+\tctx->state[2] = 0x3C6EF372UL;\n -+\tctx->state[3] = 0xA54FF53AUL;\n -+\tctx->state[4] = 0x510E527FUL;\n -+\tctx->state[5] = 0x9B05688CUL;\n -+\tctx->state[6] = 0x1F83D9ABUL;\n -+\tctx->state[7] = 0x5BE0CD19UL;\n ++\tctx->state[0] = 0x6a09e667ul;\n ++\tctx->state[1] = 0xbb67ae85ul;\n ++\tctx->state[2] = 0x3c6ef372ul;\n ++\tctx->state[3] = 0xa54ff53aul;\n ++\tctx->state[4] = 0x510e527ful;\n ++\tctx->state[5] = 0x9b05688cul;\n ++\tctx->state[6] = 0x1f83d9abul;\n ++\tctx->state[7] = 0x5be0cd19ul;\n +}\n +\n +static inline uint32_t ror(uint32_t x, unsigned n)\n @@ -233,12 +237,12 @@\n +\n +static inline uint32_t ch(uint32_t x, uint32_t y, uint32_t z)\n +{\n -+\treturn (z ^ (x & (y ^ z)));\n ++\treturn z ^ (x & (y ^ z));\n +}\n +\n +static inline uint32_t maj(uint32_t x, uint32_t y, uint32_t z)\n +{\n -+\treturn (((x | y) & z) | (x & y));\n ++\treturn ((x | y) & z) | (x & y);\n +}\n +\n +static inline uint32_t sigma0(uint32_t x)\n @@ -268,19 +272,16 @@\n +\tint i;\n +\n +\t/* copy state into S */\n -+\tfor (i = 0; i < 8; i++) {\n ++\tfor (i = 0; i < 8; i++)\n +\t\tS[i] = ctx->state[i];\n -+\t}\n +\n +\t/* copy the state into 512-bits into W[0..15] */\n -+\tfor (i = 0; i < 16; i++, buf += sizeof(uint32_t)) {\n ++\tfor (i = 0; i < 16; i++, buf += sizeof(uint32_t))\n +\t\tW[i] = get_be32(buf);\n -+\t}\n +\n +\t/* fill W[16..63] */\n -+\tfor (i = 16; i < 64; i++) {\n ++\tfor (i = 16; i < 64; i++)\n +\t\tW[i] = gamma1(W[i - 2]) + W[i - 7] + gamma0(W[i - 15]) + W[i - 16];\n -+\t}\n +\n +#define RND(a,b,c,d,e,f,g,h,i,ki) \\\n +\tt0 = h + sigma1(e) + ch(e, f, g) + ki + W[i]; \\\n @@ -353,10 +354,8 @@\n +\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],62,0xbef9a3f7);\n +\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],63,0xc67178f2);\n +\n -+\n -+\tfor (i = 0; i < 8; i++) {\n -+\t\tctx->state[i] = ctx->state[i] + S[i];\n -+\t}\n ++\tfor (i = 0; i < 8; i++)\n ++\t\tctx->state[i] += S[i];\n +}\n +\n +void blk_SHA256_Update(blk_SHA256_CTX *ctx, const void *data, size_t len)\n12: fe8f2ba01c = 11: 467c86e878 sha256: add an SHA-256 implementation using libgcrypt\n13: 38142d8fc6 = 12: 73e4bc17d0 hash: add an SHA-256 implementation using OpenSSL\n"},{"id":"362419","messageId":"20181104234458.139223-2-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 01/12] sha1-file: rename algorithm to \"sha1\"","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:47Z","receivedAt":"2018-11-04T23:45:12Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"The transition plan anticipates us using a syntax such as \"^{sha1}\" for\ndisambiguation. Since this is a syntax some people will be typing a\nlot, it makes sense to provide a short, easy-to-type syntax. Omitting\nthe dash doesn't create any ambiguity; however, it does make the syntax\nshorter and easier to type, especially for touch typists. In addition,\nthe transition plan already uses \"sha1\" in this context.\n\nRename the name of SHA-1 implementation to \"sha1\".\n\nNote that this change creates no backwards compatibility concerns, since\nwe haven't yet used this field in any configuration settings.\n\nSigned-off-by: brian m. carlson \n---\n sha1-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sha1-file.c b/sha1-file.c\nindex dd0b6aa873..91311ebb3d 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -97,7 +97,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\tNULL,\n \t},\n \t{\n-\t\t\"sha-1\",\n+\t\t\"sha1\",\n \t\t/* \"sha1\", big-endian */\n \t\t0x73686131,\n \t\tGIT_SHA1_RAWSZ,\n"},{"id":"362420","messageId":"20181104234458.139223-3-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:48Z","receivedAt":"2018-11-04T23:45:12Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"There are several ways we might refer to a hash algorithm: by name, such\nas in the config file; by format ID, such as in a pack; or internally,\nby a pointer to the hash_algos array. Provide functions to look up hash\nalgorithms based on these various forms and return the internal constant\nused for them. If conversion to another form is necessary, this\ninternal constant can be used to look up the proper data in the\nhash_algos array.\n\nSigned-off-by: brian m. carlson \n---\n hash.h | 13 +++++++++++++\n sha1-file.c | 21 +++++++++++++++++++++\n 2 files changed, 34 insertions(+)\n\ndiff --git a/hash.h b/hash.h\nindex 7c8238bc2e..80881eea47 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -98,4 +98,17 @@ struct git_hash_algo {\n };\n extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n \n+/*\n+ * Return a GIT_HASH_* constant based on the name. Returns GIT_HASH_UNKNOWN if\n+ * the name doesn't match a known algorithm.\n+ */\n+int hash_algo_by_name(const char *name);\n+/* Identical, except based on the format ID. */\n+int hash_algo_by_id(uint32_t format_id);\n+/* Identical, except for a pointer to struct git_hash_algo. */\n+static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n+{\n+\treturn p - hash_algos;\n+}\n+\n #endif\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 91311ebb3d..7e9dedc744 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -122,6 +122,27 @@ const char *empty_blob_oid_hex(void)\n \treturn oid_to_hex_r(buf, the_hash_algo->empty_blob);\n }\n \n+int hash_algo_by_name(const char *name)\n+{\n+\tint i;\n+\tif (!name)\n+\t\treturn GIT_HASH_UNKNOWN;\n+\tfor (i = 1; i < GIT_HASH_NALGOS; i++)\n+\t\tif (!strcmp(name, hash_algos[i].name))\n+\t\t\treturn i;\n+\treturn GIT_HASH_UNKNOWN;\n+}\n+\n+int hash_algo_by_id(uint32_t format_id)\n+{\n+\tint i;\n+\tfor (i = 1; i < GIT_HASH_NALGOS; i++)\n+\t\tif (format_id == hash_algos[i].format_id)\n+\t\t\treturn i;\n+\treturn GIT_HASH_UNKNOWN;\n+}\n+\n+\n /*\n * This is meant to hold a *small* number of objects that you would\n * want read_sha1_file() to be able to return, but yet you do not want\n"},{"id":"362421","messageId":"20181104234458.139223-4-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 03/12] hex: introduce functions to print arbitrary hashes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:49Z","receivedAt":"2018-11-04T23:45:13Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Currently, we have functions that turn an arbitrary SHA-1 value or an\nobject ID into hex format, either using a static buffer or with a\nuser-provided buffer. Add variants of these functions that can handle\nan arbitrary hash algorithm, specified by constant. Update the\ndocumentation as well.\n\nWhile we're at it, remove the \"extern\" declaration from this family of\nfunctions, since it's not needed and our style now recommends against\nit.\n\nWe use the variant taking the algorithm structure pointer as the\ninternal variant, since taking an algorithm pointer is the easiest way\nto handle all of the variants in use.\n\nNote that we maintain these functions because there are hashes which\nmust change based on the hash algorithm in use but are not object IDs\n(such as pack checksums).\n\nSigned-off-by: brian m. carlson \n---\n cache.h | 15 +++++++++------\n hex.c | 32 ++++++++++++++++++++++++--------\n 2 files changed, 33 insertions(+), 14 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 59c8a93046..51580c4b77 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1364,9 +1364,9 @@ extern int get_oid_hex(const char *hex, struct object_id *sha1);\n extern int hex_to_bytes(unsigned char *binary, const char *hex, size_t len);\n \n /*\n- * Convert a binary sha1 to its hex equivalent. The `_r` variant is reentrant,\n+ * Convert a binary hash to its hex equivalent. The `_r` variant is reentrant,\n * and writes the NUL-terminated output to the buffer `out`, which must be at\n- * least `GIT_SHA1_HEXSZ + 1` bytes, and returns a pointer to out for\n+ * least `GIT_MAX_HEXSZ + 1` bytes, and returns a pointer to out for\n * convenience.\n *\n * The non-`_r` variant returns a static buffer, but uses a ring of 4\n@@ -1374,10 +1374,13 @@ extern int hex_to_bytes(unsigned char *binary, const char *hex, size_t len);\n *\n * printf(\"%s -> %s\", sha1_to_hex(one), sha1_to_hex(two));\n */\n-extern char *sha1_to_hex_r(char *out, const unsigned char *sha1);\n-extern char *oid_to_hex_r(char *out, const struct object_id *oid);\n-extern char *sha1_to_hex(const unsigned char *sha1);\t/* static buffer result! */\n-extern char *oid_to_hex(const struct object_id *oid);\t/* same static buffer as sha1_to_hex */\n+char *hash_to_hex_algop_r(char *buffer, const unsigned char *hash, const struct git_hash_algo *);\n+char *sha1_to_hex_r(char *out, const unsigned char *sha1);\n+char *oid_to_hex_r(char *out, const struct object_id *oid);\n+char *hash_to_hex_algop(const unsigned char *hash, const struct git_hash_algo *);\t/* static buffer result! */\n+char *sha1_to_hex(const unsigned char *sha1);\t\t\t\t\t\t/* same static buffer */\n+char *hash_to_hex(const unsigned char *hash);\t\t\t\t\t\t/* same static buffer */\n+char *oid_to_hex(const struct object_id *oid);\t\t\t\t\t\t/* same static buffer */\n \n /*\n * Parse a 40-character hexadecimal object ID starting from hex, updating the\ndiff --git a/hex.c b/hex.c\nindex 10af1a29e8..d2e8bb9540 100644\n--- a/hex.c\n+++ b/hex.c\n@@ -73,14 +73,15 @@ int parse_oid_hex(const char *hex, struct object_id *oid, const char **end)\n \treturn ret;\n }\n \n-char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n+inline char *hash_to_hex_algop_r(char *buffer, const unsigned char *hash,\n+\t\t\t\t\tconst struct git_hash_algo *algop)\n {\n \tstatic const char hex[] = \"0123456789abcdef\";\n \tchar *buf = buffer;\n \tint i;\n \n-\tfor (i = 0; i < the_hash_algo->rawsz; i++) {\n-\t\tunsigned int val = *sha1++;\n+\tfor (i = 0; i < algop->rawsz; i++) {\n+\t\tunsigned int val = *hash++;\n \t\t*buf++ = hex[val >> 4];\n \t\t*buf++ = hex[val & 0xf];\n \t}\n@@ -89,20 +90,35 @@ char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n \treturn buffer;\n }\n \n-char *oid_to_hex_r(char *buffer, const struct object_id *oid)\n+char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n {\n-\treturn sha1_to_hex_r(buffer, oid->hash);\n+\treturn hash_to_hex_algop_r(buffer, sha1, &hash_algos[GIT_HASH_SHA1]);\n }\n \n-char *sha1_to_hex(const unsigned char *sha1)\n+char *oid_to_hex_r(char *buffer, const struct object_id *oid)\n+{\n+\treturn hash_to_hex_algop_r(buffer, oid->hash, the_hash_algo);\n+}\n+\n+char *hash_to_hex_algop(const unsigned char *hash, const struct git_hash_algo *algop)\n {\n \tstatic int bufno;\n \tstatic char hexbuffer[4][GIT_MAX_HEXSZ + 1];\n \tbufno = (bufno + 1) % ARRAY_SIZE(hexbuffer);\n-\treturn sha1_to_hex_r(hexbuffer[bufno], sha1);\n+\treturn hash_to_hex_algop_r(hexbuffer[bufno], hash, algop);\n+}\n+\n+char *sha1_to_hex(const unsigned char *sha1)\n+{\n+\treturn hash_to_hex_algop(sha1, &hash_algos[GIT_HASH_SHA1]);\n+}\n+\n+char *hash_to_hex(const unsigned char *hash)\n+{\n+\treturn hash_to_hex_algop(hash, the_hash_algo);\n }\n \n char *oid_to_hex(const struct object_id *oid)\n {\n-\treturn sha1_to_hex(oid->hash);\n+\treturn hash_to_hex_algop(oid->hash, the_hash_algo);\n }\n"},{"id":"362422","messageId":"20181104234458.139223-6-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 05/12] t: add basic tests for our SHA-1 implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:51Z","receivedAt":"2018-11-04T23:45:14Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We have in the past had some unfortunate endianness issues with some\nSHA-1 implementations we ship, especially on big-endian machines. Add\nan explicit test using the test helper to catch these issues and point\nthem out prominently. This test can also be used as a staging ground\nfor people testing additional algorithms to verify that their\nimplementations are working as expected.\n\nSigned-off-by: brian m. carlson \n---\n t/t0015-hash.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n create mode 100755 t/t0015-hash.sh\n\ndiff --git a/t/t0015-hash.sh b/t/t0015-hash.sh\nnew file mode 100755\nindex 0000000000..8e763c2c3d\n--- /dev/null\n+++ b/t/t0015-hash.sh\n@@ -0,0 +1,29 @@\n+#!/bin/sh\n+\n+test_description='test basic hash implementation'\n+. ./test-lib.sh\n+\n+\n+test_expect_success 'test basic SHA-1 hash values' '\n+\ttest-tool sha1 actual &&\n+\tgrep da39a3ee5e6b4b0d3255bfef95601890afd80709 actual &&\n+\tprintf \"a\" | test-tool sha1 >actual &&\n+\tgrep 86f7e437faa5a7fce15d1ddcb9eaeaea377667b8 actual &&\n+\tprintf \"abc\" | test-tool sha1 >actual &&\n+\tgrep a9993e364706816aba3e25717850c26c9cd0d89d actual &&\n+\tprintf \"message digest\" | test-tool sha1 >actual &&\n+\tgrep c12252ceda8be8994d5fa0290a47231c1d16aae3 actual &&\n+\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha1 >actual &&\n+\tgrep 32d10c7b8cf96570ca04ce37f2a19d84240d3a89 actual &&\n+\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n+\t\ttest-tool sha1 >actual &&\n+\tgrep 34aa973cd4c4daa4f61eeb2bdbad27316534016f actual &&\n+\tprintf \"blob 0\\0\" | test-tool sha1 >actual &&\n+\tgrep e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 actual &&\n+\tprintf \"blob 3\\0abc\" | test-tool sha1 >actual &&\n+\tgrep f2ba8f84ab5c1bce84a7b441cb1959cfc7093b7f actual &&\n+\tprintf \"tree 0\\0\" | test-tool sha1 >actual &&\n+\tgrep 4b825dc642cb6eb9a060e54bf8d69288fbee4904 actual\n+'\n+\n+test_done\n"},{"id":"362426","messageId":"20181104234458.139223-5-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 04/12] cache: make hashcmp and hasheq work with larger hashes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:50Z","receivedAt":"2018-11-04T23:45:14Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In 183a638b7d (\"hashcmp: assert constant hash size\", 2018-08-23), we\nmodified hashcmp to assert that the hash size was always 20 to help it\noptimize and inline calls to memcmp. In a future series, we replaced\nmany calls to hashcmp and oidcmp with calls to hasheq and oideq to\nimprove inlining further.\n\nHowever, we want to support hash algorithms other than SHA-1, namely\nSHA-256. When doing so, we must handle the case where these values are\n32 bytes long as well as 20. Adjust hashcmp to handle two cases:\n20-byte matches, and maximum-size matches. Therefore, when we include\nSHA-256, we'll automatically handle it properly, while at the same time\nteaching the compiler that there are only two possible options to\nconsider. This will allow the compiler to write the most efficient\npossible code.\n\nCopy similar code into hasheq and perform an identical transformation.\nAt least with GCC 8.2.0, making hasheq defer to hashcmp when there are\ntwo branches prevents the compiler from inlining the comparison, while\nthe code in this patch is inlined properly. Add a comment to avoid an\naccidental performance regression from well-intentioned refactoring.\n\nSigned-off-by: brian m. carlson \n---\n cache.h | 22 ++++++++++++----------\n 1 file changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 51580c4b77..bab8e8964f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1027,16 +1027,12 @@ extern const struct object_id null_oid;\n static inline int hashcmp(const unsigned char *sha1, const unsigned char *sha2)\n {\n \t/*\n-\t * This is a temporary optimization hack. By asserting the size here,\n-\t * we let the compiler know that it's always going to be 20, which lets\n-\t * it turn this fixed-size memcmp into a few inline instructions.\n-\t *\n-\t * This will need to be extended or ripped out when we learn about\n-\t * hashes of different sizes.\n+\t * Teach the compiler that there are only two possibilities of hash size\n+\t * here, so that it can optimize for this case as much as possible.\n \t */\n-\tif (the_hash_algo->rawsz != 20)\n-\t\tBUG(\"hash size not yet supported by hashcmp\");\n-\treturn memcmp(sha1, sha2, the_hash_algo->rawsz);\n+\tif (the_hash_algo->rawsz == GIT_MAX_RAWSZ)\n+\t\treturn memcmp(sha1, sha2, GIT_MAX_RAWSZ);\n+\treturn memcmp(sha1, sha2, GIT_SHA1_RAWSZ);\n }\n \n static inline int oidcmp(const struct object_id *oid1, const struct object_id *oid2)\n@@ -1046,7 +1042,13 @@ static inline int oidcmp(const struct object_id *oid1, const struct object_id *o\n \n static inline int hasheq(const unsigned char *sha1, const unsigned char *sha2)\n {\n-\treturn !hashcmp(sha1, sha2);\n+\t/*\n+\t * We write this here instead of deferring to hashcmp so that the\n+\t * compiler can properly inline it and avoid calling memcmp.\n+\t */\n+\tif (the_hash_algo->rawsz == GIT_MAX_RAWSZ)\n+\t\treturn !memcmp(sha1, sha2, GIT_MAX_RAWSZ);\n+\treturn !memcmp(sha1, sha2, GIT_SHA1_RAWSZ);\n }\n \n static inline int oideq(const struct object_id *oid1, const struct object_id *oid2)\n"},{"id":"362423","messageId":"20181104234458.139223-8-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 07/12] sha1-file: add a constant for hash block size","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:53Z","receivedAt":"2018-11-04T23:45:16Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"There is one place we need the hash algorithm block size: the HMAC code\nfor push certs. Expose this constant in struct git_hash_algo and expose\nvalues for SHA-1 and for the largest value of any hash.\n\nSigned-off-by: brian m. carlson \n---\n cache.h | 4 ++++\n hash.h | 3 +++\n sha1-file.c | 2 ++\n 3 files changed, 9 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex bab8e8964f..9e5d1dd85a 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -45,10 +45,14 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n+/* The block size of SHA-1. */\n+#define GIT_SHA1_BLKSZ 64\n \n /* The length in byte and in hex digits of the largest possible hash value. */\n #define GIT_MAX_RAWSZ GIT_SHA1_RAWSZ\n #define GIT_MAX_HEXSZ GIT_SHA1_HEXSZ\n+/* The largest possible block size for any supported hash. */\n+#define GIT_MAX_BLKSZ GIT_SHA1_BLKSZ\n \n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/hash.h b/hash.h\nindex 80881eea47..1bcf7ab6fd 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -81,6 +81,9 @@ struct git_hash_algo {\n \t/* The length of the hash in hex characters. */\n \tsize_t hexsz;\n \n+\t/* The block size of the hash. */\n+\tsize_t blksz;\n+\n \t/* The hash initialization function. */\n \tgit_hash_init_fn init_fn;\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex 7e9dedc744..9bdd04ea45 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -90,6 +90,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t0x00000000,\n \t\t0,\n \t\t0,\n+\t\t0,\n \t\tgit_hash_unknown_init,\n \t\tgit_hash_unknown_update,\n \t\tgit_hash_unknown_final,\n@@ -102,6 +103,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t0x73686131,\n \t\tGIT_SHA1_RAWSZ,\n \t\tGIT_SHA1_HEXSZ,\n+\t\tGIT_SHA1_BLKSZ,\n \t\tgit_hash_sha1_init,\n \t\tgit_hash_sha1_update,\n \t\tgit_hash_sha1_final,\n"},{"id":"362425","messageId":"20181104234458.139223-7-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 06/12] t: make the sha1 test-tool helper generic","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:52Z","receivedAt":"2018-11-04T23:45:16Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Since we're going to have multiple hash algorithms to test, it makes\nsense to share as much of the test code as possible. Convert the sha1\nhelper for the test-tool to be generic and move it out into its own\nmodule. This will allow us to share most of this code with our NewHash\nimplementation.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 1 +\n t/helper/{test-sha1.c => test-hash.c} | 19 +++++-----\n t/helper/test-sha1.c | 52 +--------------------------\n t/helper/test-tool.h | 2 ++\n 4 files changed, 14 insertions(+), 60 deletions(-)\n copy t/helper/{test-sha1.c => test-hash.c} (65%)\n\ndiff --git a/Makefile b/Makefile\nindex d18ab0fe78..81dc9ac819 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -714,6 +714,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n+TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-hash.c\nsimilarity index 65%\ncopy from t/helper/test-sha1.c\ncopy to t/helper/test-hash.c\nindex 1ba0675c75..0a31de66f3 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-hash.c\n@@ -1,13 +1,14 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n \n-int cmd__sha1(int ac, const char **av)\n+int cmd_hash_impl(int ac, const char **av, int algo)\n {\n-\tgit_SHA_CTX ctx;\n-\tunsigned char sha1[20];\n+\tgit_hash_ctx ctx;\n+\tunsigned char hash[GIT_MAX_HEXSZ];\n \tunsigned bufsz = 8192;\n \tint binary = 0;\n \tchar *buffer;\n+\tconst struct git_hash_algo *algop = &hash_algos[algo];\n \n \tif (ac == 2) {\n \t\tif (!strcmp(av[1], \"-b\"))\n@@ -26,7 +27,7 @@ int cmd__sha1(int ac, const char **av)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\tgit_SHA1_Init(&ctx);\n+\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -38,20 +39,20 @@ int cmd__sha1(int ac, const char **av)\n \t\t\tif (sz == 0)\n \t\t\t\tbreak;\n \t\t\tif (sz < 0)\n-\t\t\t\tdie_errno(\"test-sha1\");\n+\t\t\t\tdie_errno(\"test-hash\");\n \t\t\tthis_sz += sz;\n \t\t\tcp += sz;\n \t\t\troom -= sz;\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\tgit_SHA1_Update(&ctx, buffer, this_sz);\n+\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\tgit_SHA1_Final(sha1, &ctx);\n+\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n-\t\tfwrite(sha1, 1, 20, stdout);\n+\t\tfwrite(hash, 1, algop->rawsz, stdout);\n \telse\n-\t\tputs(sha1_to_hex(sha1));\n+\t\tputs(hash_to_hex_algop(hash, algop));\n \texit(0);\n }\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-sha1.c\nindex 1ba0675c75..d860c387c3 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-sha1.c\n@@ -3,55 +3,5 @@\n \n int cmd__sha1(int ac, const char **av)\n {\n-\tgit_SHA_CTX ctx;\n-\tunsigned char sha1[20];\n-\tunsigned bufsz = 8192;\n-\tint binary = 0;\n-\tchar *buffer;\n-\n-\tif (ac == 2) {\n-\t\tif (!strcmp(av[1], \"-b\"))\n-\t\t\tbinary = 1;\n-\t\telse\n-\t\t\tbufsz = strtoul(av[1], NULL, 10) * 1024 * 1024;\n-\t}\n-\n-\tif (!bufsz)\n-\t\tbufsz = 8192;\n-\n-\twhile ((buffer = malloc(bufsz)) == NULL) {\n-\t\tfprintf(stderr, \"bufsz %u is too big, halving...\\n\", bufsz);\n-\t\tbufsz /= 2;\n-\t\tif (bufsz < 1024)\n-\t\t\tdie(\"OOPS\");\n-\t}\n-\n-\tgit_SHA1_Init(&ctx);\n-\n-\twhile (1) {\n-\t\tssize_t sz, this_sz;\n-\t\tchar *cp = buffer;\n-\t\tunsigned room = bufsz;\n-\t\tthis_sz = 0;\n-\t\twhile (room) {\n-\t\t\tsz = xread(0, cp, room);\n-\t\t\tif (sz == 0)\n-\t\t\t\tbreak;\n-\t\t\tif (sz < 0)\n-\t\t\t\tdie_errno(\"test-sha1\");\n-\t\t\tthis_sz += sz;\n-\t\t\tcp += sz;\n-\t\t\troom -= sz;\n-\t\t}\n-\t\tif (this_sz == 0)\n-\t\t\tbreak;\n-\t\tgit_SHA1_Update(&ctx, buffer, this_sz);\n-\t}\n-\tgit_SHA1_Final(sha1, &ctx);\n-\n-\tif (binary)\n-\t\tfwrite(sha1, 1, 20, stdout);\n-\telse\n-\t\tputs(sha1_to_hex(sha1));\n-\texit(0);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1);\n }\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e4890566da..29ac7b0b0d 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -50,4 +50,6 @@ int cmd__windows_named_pipe(int argc, const char **argv);\n #endif\n int cmd__write_cache(int argc, const char **argv);\n \n+int cmd_hash_impl(int ac, const char **av, int algo);\n+\n #endif\n"},{"id":"362424","messageId":"20181104234458.139223-9-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 08/12] t/helper: add a test helper to compute hash speed","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:54Z","receivedAt":"2018-11-04T23:45:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a utility (which is less for the testsuite and more for developers)\nthat can compute hash speeds for whatever hash algorithms are\nimplemented. This allows developers to test their personal systems to\ndetermine the performance characteristics of various algorithms.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 1 +\n t/helper/test-hash-speed.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 64 insertions(+)\n create mode 100644 t/helper/test-hash-speed.c\n\ndiff --git a/Makefile b/Makefile\nindex 81dc9ac819..68169a7abb 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -716,6 +716,7 @@ TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n+TEST_BUILTINS_OBJS += test-hash-speed.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\n TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\ndiff --git a/t/helper/test-hash-speed.c b/t/helper/test-hash-speed.c\nnew file mode 100644\nindex 0000000000..432233c7f0\n--- /dev/null\n+++ b/t/helper/test-hash-speed.c\n@@ -0,0 +1,61 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+#define NUM_SECONDS 3\n+\n+static inline void compute_hash(const struct git_hash_algo *algo, git_hash_ctx *ctx, uint8_t *final, const void *p, size_t len)\n+{\n+\talgo->init_fn(ctx);\n+\talgo->update_fn(ctx, p, len);\n+\talgo->final_fn(final, ctx);\n+}\n+\n+int cmd__hash_speed(int ac, const char **av)\n+{\n+\tgit_hash_ctx ctx;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n+\tclock_t initial, start, end;\n+\tunsigned bufsizes[] = { 64, 256, 1024, 8192, 16384 };\n+\tint i;\n+\tvoid *p;\n+\tconst struct git_hash_algo *algo = NULL;\n+\n+\tif (ac == 2) {\n+\t\tfor (i = 1; i < GIT_HASH_NALGOS; i++) {\n+\t\t\tif (!strcmp(av[1], hash_algos[i].name)) {\n+\t\t\t\talgo = &hash_algos[i];\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (!algo)\n+\t\tdie(\"usage: test-tool hash-speed algo_name\");\n+\n+\t/* Use this as an offset to make overflow less likely. */\n+\tinitial = clock();\n+\n+\tprintf(\"algo: %s\\n\", algo->name);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(bufsizes); i++) {\n+\t\tunsigned long j, kb;\n+\t\tdouble kb_per_sec;\n+\t\tp = xcalloc(1, bufsizes[i]);\n+\t\tstart = end = clock() - initial;\n+\t\tfor (j = 0; ((end - start) / CLOCKS_PER_SEC) < NUM_SECONDS; j++) {\n+\t\t\tcompute_hash(algo, &ctx, hash, p, bufsizes[i]);\n+\n+\t\t\t/*\n+\t\t\t * Only check elapsed time every 128 iterations to avoid\n+\t\t\t * dominating the runtime with system calls.\n+\t\t\t */\n+\t\t\tif (!(j & 127))\n+\t\t\t\tend = clock() - initial;\n+\t\t}\n+\t\tkb = j * bufsizes[i];\n+\t\tkb_per_sec = kb / (1024 * ((double)end - start) / CLOCKS_PER_SEC);\n+\t\tprintf(\"size %u: %lu iters; %lu KiB; %0.2f KiB/s\\n\", bufsizes[i], j, kb, kb_per_sec);\n+\t\tfree(p);\n+\t}\n+\n+\texit(0);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 6b5836dc1b..e009c8186d 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -20,6 +20,7 @@ static struct test_cmd cmds[] = {\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"hashmap\", cmd__hashmap },\n+\t{ \"hash-speed\", cmd__hash_speed },\n \t{ \"index-version\", cmd__index_version },\n \t{ \"json-writer\", cmd__json_writer },\n \t{ \"lazy-init-name-hash\", cmd__lazy_init_name_hash },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 29ac7b0b0d..19a7e8332a 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -16,6 +16,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n+int cmd__hash_speed(int argc, const char **argv);\n int cmd__index_version(int argc, const char **argv);\n int cmd__json_writer(int argc, const char **argv);\n int cmd__lazy_init_name_hash(int argc, const char **argv);\n"},{"id":"362427","messageId":"20181104234458.139223-10-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 09/12] commit-graph: convert to using the_hash_algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:55Z","receivedAt":"2018-11-04T23:45:20Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Instead of using hard-coded constants for object sizes, use\nthe_hash_algo to look them up. In addition, use a function call to look\nup the object ID version and produce the correct value. For now, we use\nversion 1, which means to use the default algorithm used in the rest of\nthe repository.\n\nSigned-off-by: brian m. carlson \n---\n commit-graph.c | 33 +++++++++++++++++----------------\n 1 file changed, 17 insertions(+), 16 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..6763d19288 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -23,16 +23,11 @@\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n #define GRAPH_CHUNKID_LARGEEDGES 0x45444745 /* \"EDGE\" */\n \n-#define GRAPH_DATA_WIDTH 36\n+#define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n #define GRAPH_VERSION_1 0x1\n #define GRAPH_VERSION GRAPH_VERSION_1\n \n-#define GRAPH_OID_VERSION_SHA1 1\n-#define GRAPH_OID_LEN_SHA1 GIT_SHA1_RAWSZ\n-#define GRAPH_OID_VERSION GRAPH_OID_VERSION_SHA1\n-#define GRAPH_OID_LEN GRAPH_OID_LEN_SHA1\n-\n #define GRAPH_OCTOPUS_EDGES_NEEDED 0x80000000\n #define GRAPH_PARENT_MISSING 0x7fffffff\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n@@ -44,13 +39,18 @@\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n #define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n-\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n+\t\t\t+ GRAPH_FANOUT_SIZE + the_hash_algo->rawsz)\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static uint8_t oid_version(void)\n+{\n+\treturn 1;\n+}\n+\n static struct commit_graph *alloc_commit_graph(void)\n {\n \tstruct commit_graph *g = xcalloc(1, sizeof(*g));\n@@ -125,15 +125,15 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \t}\n \n \thash_version = *(unsigned char*)(data + 5);\n-\tif (hash_version != GRAPH_OID_VERSION) {\n+\tif (hash_version != oid_version()) {\n \t\terror(_(\"hash version %X does not match version %X\"),\n-\t\t hash_version, GRAPH_OID_VERSION);\n+\t\t hash_version, oid_version());\n \t\tgoto cleanup_fail;\n \t}\n \n \tgraph = alloc_commit_graph();\n \n-\tgraph->hash_len = GRAPH_OID_LEN;\n+\tgraph->hash_len = the_hash_algo->rawsz;\n \tgraph->num_chunks = *(unsigned char*)(data + 6);\n \tgraph->graph_fd = fd;\n \tgraph->data = graph_map;\n@@ -149,7 +149,7 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \n \t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n \n-\t\tif (chunk_offset > graph_size - GIT_MAX_RAWSZ) {\n+\t\tif (chunk_offset > graph_size - the_hash_algo->rawsz) {\n \t\t\terror(_(\"improper chunk offset %08x%08x\"), (uint32_t)(chunk_offset >> 32),\n \t\t\t (uint32_t)chunk_offset);\n \t\t\tgoto cleanup_fail;\n@@ -764,6 +764,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n+\tconst unsigned hashsz = the_hash_algo->rawsz;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -909,7 +910,7 @@ void write_commit_graph(const char *obj_dir,\n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, GRAPH_OID_VERSION);\n+\thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, num_chunks);\n \thashwrite_u8(f, 0); /* unused padding byte */\n \n@@ -924,8 +925,8 @@ void write_commit_graph(const char *obj_dir,\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\n \tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n@@ -938,8 +939,8 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\twrite_graph_chunk_oids(f, hashsz, commits.list, commits.nr);\n+\twrite_graph_chunk_data(f, hashsz, commits.list, commits.nr);\n \twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n"},{"id":"362428","messageId":"20181104234458.139223-11-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 10/12] Add a base implementation of SHA-256 support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:56Z","receivedAt":"2018-11-04T23:45:21Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"SHA-1 is weak and we need to transition to a new hash function. For\nsome time, we have referred to this new function as NewHash. Recently,\nwe decided to pick SHA-256 as NewHash. The reasons behind the choice of\nSHA-256 are outlined in the thread starting at [1] and in the commit\nhistory for the hash function transition document.\n\nAdd a basic implementation of SHA-256 based off libtomcrypt, which is in\nthe public domain. Optimize it and restructure it to meet our coding\nstandards. Pull in the update and final functions from the SHA-1 block\nimplementation, as we know these function correctly with all compilers.\nThis implementation is slower than SHA-1, but more performant\nimplementations will be introduced in future commits.\n\nWire up SHA-256 in the list of hash algorithms, and add a test that the\nalgorithm works correctly.\n\nNote that with this patch, it is still not possible to switch to using\nSHA-256 in Git. Additional patches are needed to prepare the code to\nhandle a larger hash algorithm and further test fixes are needed.\n\n[1] https://public-inbox.org/git/20180609224913.GC38834@genre.crustytoothpaste.net/\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 4 +\n cache.h | 12 ++-\n hash.h | 19 +++-\n sha1-file.c | 45 ++++++++++\n sha256/block/sha256.c | 196 +++++++++++++++++++++++++++++++++++++++++\n sha256/block/sha256.h | 26 ++++++\n t/helper/test-sha256.c | 7 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0015-hash.sh | 25 ++++++\n 10 files changed, 332 insertions(+), 4 deletions(-)\n create mode 100644 sha256/block/sha256.c\n create mode 100644 sha256/block/sha256.h\n create mode 100644 t/helper/test-sha256.c\n\ndiff --git a/Makefile b/Makefile\nindex 68169a7abb..e99b7712f6 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -739,6 +739,7 @@ TEST_BUILTINS_OBJS += test-run-command.o\n TEST_BUILTINS_OBJS += test-scrap-cache-tree.o\n TEST_BUILTINS_OBJS += test-sha1.o\n TEST_BUILTINS_OBJS += test-sha1-array.o\n+TEST_BUILTINS_OBJS += test-sha256.o\n TEST_BUILTINS_OBJS += test-sigchain.o\n TEST_BUILTINS_OBJS += test-strcmp-offset.o\n TEST_BUILTINS_OBJS += test-string-list.o\n@@ -1633,6 +1634,9 @@ endif\n endif\n endif\n \n+LIB_OBJS += sha256/block/sha256.o\n+BASIC_CFLAGS += -DSHA256_BLK\n+\n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\n \tBASIC_CFLAGS += -DSHA1_MAX_BLOCK_SIZE=\"$(SHA1_MAX_BLOCK_SIZE)\"\ndiff --git a/cache.h b/cache.h\nindex 9e5d1dd85a..48ce1565e6 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -48,11 +48,17 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n /* The block size of SHA-1. */\n #define GIT_SHA1_BLKSZ 64\n \n+/* The length in bytes and in hex digits of an object name (SHA-256 value). */\n+#define GIT_SHA256_RAWSZ 32\n+#define GIT_SHA256_HEXSZ (2 * GIT_SHA256_RAWSZ)\n+/* The block size of SHA-256. */\n+#define GIT_SHA256_BLKSZ 64\n+\n /* The length in byte and in hex digits of the largest possible hash value. */\n-#define GIT_MAX_RAWSZ GIT_SHA1_RAWSZ\n-#define GIT_MAX_HEXSZ GIT_SHA1_HEXSZ\n+#define GIT_MAX_RAWSZ GIT_SHA256_RAWSZ\n+#define GIT_MAX_HEXSZ GIT_SHA256_HEXSZ\n /* The largest possible block size for any supported hash. */\n-#define GIT_MAX_BLKSZ GIT_SHA1_BLKSZ\n+#define GIT_MAX_BLKSZ GIT_SHA256_BLKSZ\n \n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/hash.h b/hash.h\nindex 1bcf7ab6fd..a9bc624020 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -15,6 +15,8 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n+#include \"sha256/block/sha256.h\"\n+\n #ifndef platform_SHA_CTX\n /*\n * platform's underlying implementation of SHA-1; could be OpenSSL,\n@@ -34,6 +36,18 @@\n #define git_SHA1_Update\t\tplatform_SHA1_Update\n #define git_SHA1_Final\t\tplatform_SHA1_Final\n \n+#ifndef platform_SHA256_CTX\n+#define platform_SHA256_CTX\tSHA256_CTX\n+#define platform_SHA256_Init\tSHA256_Init\n+#define platform_SHA256_Update\tSHA256_Update\n+#define platform_SHA256_Final\tSHA256_Final\n+#endif\n+\n+#define git_SHA256_CTX\t\tplatform_SHA256_CTX\n+#define git_SHA256_Init\t\tplatform_SHA256_Init\n+#define git_SHA256_Update\tplatform_SHA256_Update\n+#define git_SHA256_Final\tplatform_SHA256_Final\n+\n #ifdef SHA1_MAX_BLOCK_SIZE\n #include \"compat/sha1-chunked.h\"\n #undef git_SHA1_Update\n@@ -52,12 +66,15 @@\n #define GIT_HASH_UNKNOWN 0\n /* SHA-1 */\n #define GIT_HASH_SHA1 1\n+/* SHA-256 */\n+#define GIT_HASH_SHA256 2\n /* Number of algorithms supported (including unknown). */\n-#define GIT_HASH_NALGOS (GIT_HASH_SHA1 + 1)\n+#define GIT_HASH_NALGOS (GIT_HASH_SHA256 + 1)\n \n /* A suitably aligned type for stack allocations of hash contexts. */\n union git_hash_ctx {\n \tgit_SHA_CTX sha1;\n+\tgit_SHA256_CTX sha256;\n };\n typedef union git_hash_ctx git_hash_ctx;\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex 9bdd04ea45..c97d93a14a 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -40,10 +40,20 @@\n #define EMPTY_TREE_SHA1_BIN_LITERAL \\\n \t \"\\x4b\\x82\\x5d\\xc6\\x42\\xcb\\x6e\\xb9\\xa0\\x60\" \\\n \t \"\\xe5\\x4b\\xf8\\xd6\\x92\\x88\\xfb\\xee\\x49\\x04\"\n+#define EMPTY_TREE_SHA256_BIN_LITERAL \\\n+\t\"\\x6e\\xf1\\x9b\\x41\\x22\\x5c\\x53\\x69\\xf1\\xc1\" \\\n+\t\"\\x04\\xd4\\x5d\\x8d\\x85\\xef\\xa9\\xb0\\x57\\xb5\" \\\n+\t\"\\x3b\\x14\\xb4\\xb9\\xb9\\x39\\xdd\\x74\\xde\\xcc\" \\\n+\t\"\\x53\\x21\"\n \n #define EMPTY_BLOB_SHA1_BIN_LITERAL \\\n \t\"\\xe6\\x9d\\xe2\\x9b\\xb2\\xd1\\xd6\\x43\\x4b\\x8b\" \\\n \t\"\\x29\\xae\\x77\\x5a\\xd8\\xc2\\xe4\\x8c\\x53\\x91\"\n+#define EMPTY_BLOB_SHA256_BIN_LITERAL \\\n+\t\"\\x47\\x3a\\x0f\\x4c\\x3b\\xe8\\xa9\\x36\\x81\\xa2\" \\\n+\t\"\\x67\\xe3\\xb1\\xe9\\xa7\\xdc\\xda\\x11\\x85\\x43\" \\\n+\t\"\\x6f\\xe1\\x41\\xf7\\x74\\x91\\x20\\xa3\\x03\\x72\" \\\n+\t\"\\x18\\x13\"\n \n const unsigned char null_sha1[GIT_MAX_RAWSZ];\n const struct object_id null_oid;\n@@ -53,6 +63,12 @@ static const struct object_id empty_tree_oid = {\n static const struct object_id empty_blob_oid = {\n \tEMPTY_BLOB_SHA1_BIN_LITERAL\n };\n+static const struct object_id empty_tree_oid_sha256 = {\n+\tEMPTY_TREE_SHA256_BIN_LITERAL\n+};\n+static const struct object_id empty_blob_oid_sha256 = {\n+\tEMPTY_BLOB_SHA256_BIN_LITERAL\n+};\n \n static void git_hash_sha1_init(git_hash_ctx *ctx)\n {\n@@ -69,6 +85,22 @@ static void git_hash_sha1_final(unsigned char *hash, git_hash_ctx *ctx)\n \tgit_SHA1_Final(hash, &ctx->sha1);\n }\n \n+\n+static void git_hash_sha256_init(git_hash_ctx *ctx)\n+{\n+\tgit_SHA256_Init(&ctx->sha256);\n+}\n+\n+static void git_hash_sha256_update(git_hash_ctx *ctx, const void *data, size_t len)\n+{\n+\tgit_SHA256_Update(&ctx->sha256, data, len);\n+}\n+\n+static void git_hash_sha256_final(unsigned char *hash, git_hash_ctx *ctx)\n+{\n+\tgit_SHA256_Final(hash, &ctx->sha256);\n+}\n+\n static void git_hash_unknown_init(git_hash_ctx *ctx)\n {\n \tBUG(\"trying to init unknown hash\");\n@@ -110,6 +142,19 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t&empty_tree_oid,\n \t\t&empty_blob_oid,\n \t},\n+\t{\n+\t\t\"sha256\",\n+\t\t/* \"s256\", big-endian */\n+\t\t0x73323536,\n+\t\tGIT_SHA256_RAWSZ,\n+\t\tGIT_SHA256_HEXSZ,\n+\t\tGIT_SHA256_BLKSZ,\n+\t\tgit_hash_sha256_init,\n+\t\tgit_hash_sha256_update,\n+\t\tgit_hash_sha256_final,\n+\t\t&empty_tree_oid_sha256,\n+\t\t&empty_blob_oid_sha256,\n+\t}\n };\n \n const char *empty_tree_oid_hex(void)\ndiff --git a/sha256/block/sha256.c b/sha256/block/sha256.c\nnew file mode 100644\nindex 0000000000..37850b4e52\n--- /dev/null\n+++ b/sha256/block/sha256.c\n@@ -0,0 +1,196 @@\n+#include \"git-compat-util.h\"\n+#include \"./sha256.h\"\n+\n+#undef RND\n+#undef BLKSIZE\n+\n+#define BLKSIZE blk_SHA256_BLKSIZE\n+\n+void blk_SHA256_Init(blk_SHA256_CTX *ctx)\n+{\n+\tctx->offset = 0;\n+\tctx->size = 0;\n+\tctx->state[0] = 0x6a09e667ul;\n+\tctx->state[1] = 0xbb67ae85ul;\n+\tctx->state[2] = 0x3c6ef372ul;\n+\tctx->state[3] = 0xa54ff53aul;\n+\tctx->state[4] = 0x510e527ful;\n+\tctx->state[5] = 0x9b05688cul;\n+\tctx->state[6] = 0x1f83d9abul;\n+\tctx->state[7] = 0x5be0cd19ul;\n+}\n+\n+static inline uint32_t ror(uint32_t x, unsigned n)\n+{\n+\treturn (x >> n) | (x << (32 - n));\n+}\n+\n+static inline uint32_t ch(uint32_t x, uint32_t y, uint32_t z)\n+{\n+\treturn z ^ (x & (y ^ z));\n+}\n+\n+static inline uint32_t maj(uint32_t x, uint32_t y, uint32_t z)\n+{\n+\treturn ((x | y) & z) | (x & y);\n+}\n+\n+static inline uint32_t sigma0(uint32_t x)\n+{\n+\treturn ror(x, 2) ^ ror(x, 13) ^ ror(x, 22);\n+}\n+\n+static inline uint32_t sigma1(uint32_t x)\n+{\n+\treturn ror(x, 6) ^ ror(x, 11) ^ ror(x, 25);\n+}\n+\n+static inline uint32_t gamma0(uint32_t x)\n+{\n+\treturn ror(x, 7) ^ ror(x, 18) ^ (x >> 3);\n+}\n+\n+static inline uint32_t gamma1(uint32_t x)\n+{\n+\treturn ror(x, 17) ^ ror(x, 19) ^ (x >> 10);\n+}\n+\n+static void blk_SHA256_Transform(blk_SHA256_CTX *ctx, const unsigned char *buf)\n+{\n+\n+\tuint32_t S[8], W[64], t0, t1;\n+\tint i;\n+\n+\t/* copy state into S */\n+\tfor (i = 0; i < 8; i++)\n+\t\tS[i] = ctx->state[i];\n+\n+\t/* copy the state into 512-bits into W[0..15] */\n+\tfor (i = 0; i < 16; i++, buf += sizeof(uint32_t))\n+\t\tW[i] = get_be32(buf);\n+\n+\t/* fill W[16..63] */\n+\tfor (i = 16; i < 64; i++)\n+\t\tW[i] = gamma1(W[i - 2]) + W[i - 7] + gamma0(W[i - 15]) + W[i - 16];\n+\n+#define RND(a,b,c,d,e,f,g,h,i,ki) \\\n+\tt0 = h + sigma1(e) + ch(e, f, g) + ki + W[i]; \\\n+\tt1 = sigma0(a) + maj(a, b, c); \\\n+\td += t0; \\\n+\th = t0 + t1;\n+\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x71374491);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcf);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba5);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25b);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],5,0x59f111f1);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],6,0x923f82a4);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],7,0xab1c5ed5);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],8,0xd807aa98);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],9,0x12835b01);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],10,0x243185be);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],11,0x550c7dc3);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],12,0x72be5d74);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],13,0x80deb1fe);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],14,0x9bdc06a7);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],15,0xc19bf174);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],16,0xe49b69c1);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],17,0xefbe4786);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],18,0x0fc19dc6);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],19,0x240ca1cc);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],20,0x2de92c6f);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],21,0x4a7484aa);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],22,0x5cb0a9dc);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],23,0x76f988da);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],24,0x983e5152);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],25,0xa831c66d);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],26,0xb00327c8);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],27,0xbf597fc7);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],28,0xc6e00bf3);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],29,0xd5a79147);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],30,0x06ca6351);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],31,0x14292967);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],32,0x27b70a85);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],33,0x2e1b2138);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],34,0x4d2c6dfc);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],35,0x53380d13);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],36,0x650a7354);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],37,0x766a0abb);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],38,0x81c2c92e);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],39,0x92722c85);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],40,0xa2bfe8a1);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],41,0xa81a664b);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],42,0xc24b8b70);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],43,0xc76c51a3);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],44,0xd192e819);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],45,0xd6990624);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],46,0xf40e3585);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],47,0x106aa070);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],48,0x19a4c116);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],49,0x1e376c08);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],50,0x2748774c);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],51,0x34b0bcb5);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],52,0x391c0cb3);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],53,0x4ed8aa4a);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],54,0x5b9cca4f);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],55,0x682e6ff3);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],56,0x748f82ee);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],57,0x78a5636f);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],58,0x84c87814);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],59,0x8cc70208);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],60,0x90befffa);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],61,0xa4506ceb);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],62,0xbef9a3f7);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],63,0xc67178f2);\n+\n+\tfor (i = 0; i < 8; i++)\n+\t\tctx->state[i] += S[i];\n+}\n+\n+void blk_SHA256_Update(blk_SHA256_CTX *ctx, const void *data, size_t len)\n+{\n+\tunsigned int len_buf = ctx->size & 63;\n+\n+\tctx->size += len;\n+\n+\t/* Read the data into buf and process blocks as they get full */\n+\tif (len_buf) {\n+\t\tunsigned int left = 64 - len_buf;\n+\t\tif (len < left)\n+\t\t\tleft = len;\n+\t\tmemcpy(len_buf + ctx->buf, data, left);\n+\t\tlen_buf = (len_buf + left) & 63;\n+\t\tlen -= left;\n+\t\tdata = ((const char *)data + left);\n+\t\tif (len_buf)\n+\t\t\treturn;\n+\t\tblk_SHA256_Transform(ctx, ctx->buf);\n+\t}\n+\twhile (len >= 64) {\n+\t\tblk_SHA256_Transform(ctx, data);\n+\t\tdata = ((const char *)data + 64);\n+\t\tlen -= 64;\n+\t}\n+\tif (len)\n+\t\tmemcpy(ctx->buf, data, len);\n+}\n+\n+void blk_SHA256_Final(unsigned char *digest, blk_SHA256_CTX *ctx)\n+{\n+\tstatic const unsigned char pad[64] = { 0x80 };\n+\tunsigned int padlen[2];\n+\tint i;\n+\n+\t/* Pad with a binary 1 (ie 0x80), then zeroes, then length */\n+\tpadlen[0] = htonl((uint32_t)(ctx->size >> 29));\n+\tpadlen[1] = htonl((uint32_t)(ctx->size << 3));\n+\n+\ti = ctx->size & 63;\n+\tblk_SHA256_Update(ctx, pad, 1 + (63 & (55 - i)));\n+\tblk_SHA256_Update(ctx, padlen, 8);\n+\n+\t/* copy output */\n+\tfor (i = 0; i < 8; i++, digest += sizeof(uint32_t))\n+\t\tput_be32(digest, ctx->state[i]);\n+}\ndiff --git a/sha256/block/sha256.h b/sha256/block/sha256.h\nnew file mode 100644\nindex 0000000000..38f02f7e6c\n--- /dev/null\n+++ b/sha256/block/sha256.h\n@@ -0,0 +1,26 @@\n+#ifndef SHA256_BLOCK_SHA256_H\n+#define SHA256_BLOCK_SHA256_H\n+\n+#include \"git-compat-util.h\"\n+\n+#define blk_SHA256_BLKSIZE 64\n+\n+struct blk_SHA256_CTX {\n+\tuint32_t state[8];\n+\tuint64_t size;\n+\tuint32_t offset;\n+\tuint8_t buf[blk_SHA256_BLKSIZE];\n+};\n+\n+typedef struct blk_SHA256_CTX blk_SHA256_CTX;\n+\n+void blk_SHA256_Init(blk_SHA256_CTX *ctx);\n+void blk_SHA256_Update(blk_SHA256_CTX *ctx, const void *data, size_t len);\n+void blk_SHA256_Final(unsigned char *digest, blk_SHA256_CTX *ctx);\n+\n+#define platform_SHA256_CTX blk_SHA256_CTX\n+#define platform_SHA256_Init blk_SHA256_Init\n+#define platform_SHA256_Update blk_SHA256_Update\n+#define platform_SHA256_Final blk_SHA256_Final\n+\n+#endif\ndiff --git a/t/helper/test-sha256.c b/t/helper/test-sha256.c\nnew file mode 100644\nindex 0000000000..0ac6a99d5f\n--- /dev/null\n+++ b/t/helper/test-sha256.c\n@@ -0,0 +1,7 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+int cmd__sha256(int ac, const char **av)\n+{\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex e009c8186d..2a65193514 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -43,6 +43,7 @@ static struct test_cmd cmds[] = {\n \t{ \"scrap-cache-tree\", cmd__scrap_cache_tree },\n \t{ \"sha1\", cmd__sha1 },\n \t{ \"sha1-array\", cmd__sha1_array },\n+\t{ \"sha256\", cmd__sha256 },\n \t{ \"sigchain\", cmd__sigchain },\n \t{ \"strcmp-offset\", cmd__strcmp_offset },\n \t{ \"string-list\", cmd__string_list },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 19a7e8332a..2e66a8e47b 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -39,6 +39,7 @@ int cmd__run_command(int argc, const char **argv);\n int cmd__scrap_cache_tree(int argc, const char **argv);\n int cmd__sha1(int argc, const char **argv);\n int cmd__sha1_array(int argc, const char **argv);\n+int cmd__sha256(int argc, const char **argv);\n int cmd__sigchain(int argc, const char **argv);\n int cmd__strcmp_offset(int argc, const char **argv);\n int cmd__string_list(int argc, const char **argv);\ndiff --git a/t/t0015-hash.sh b/t/t0015-hash.sh\nindex 8e763c2c3d..f8e639743f 100755\n--- a/t/t0015-hash.sh\n+++ b/t/t0015-hash.sh\n@@ -26,4 +26,29 @@ test_expect_success 'test basic SHA-1 hash values' '\n \tgrep 4b825dc642cb6eb9a060e54bf8d69288fbee4904 actual\n '\n \n+test_expect_success 'test basic SHA-256 hash values' '\n+\ttest-tool sha256 actual &&\n+\tgrep e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 actual &&\n+\tprintf \"a\" | test-tool sha256 >actual &&\n+\tgrep ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb actual &&\n+\tprintf \"abc\" | test-tool sha256 >actual &&\n+\tgrep ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad actual &&\n+\tprintf \"message digest\" | test-tool sha256 >actual &&\n+\tgrep f7846f55cf23e14eebeab5b4e1550cad5b509e3348fbc4efa3a1413d393cb650 actual &&\n+\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha256 >actual &&\n+\tgrep 71c480df93d6ae2f1efad1447c66c9525e316218cf51fc8d9ed832f2daf18b73 actual &&\n+\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n+\t\ttest-tool sha256 >actual &&\n+\tgrep cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0 actual &&\n+\tperl -E \"for (1..100000) { print q{abcdefghijklmnopqrstuvwxyz}; }\" | \\\n+\t\ttest-tool sha256 >actual &&\n+\tgrep e406ba321ca712ad35a698bf0af8d61fc4dc40eca6bdcea4697962724ccbde35 actual &&\n+\tprintf \"blob 0\\0\" | test-tool sha256 >actual &&\n+\tgrep 473a0f4c3be8a93681a267e3b1e9a7dcda1185436fe141f7749120a303721813 actual &&\n+\tprintf \"blob 3\\0abc\" | test-tool sha256 >actual &&\n+\tgrep c1cf6e465077930e88dc5136641d402f72a229ddd996f627d60e9639eaba35a6 actual &&\n+\tprintf \"tree 0\\0\" | test-tool sha256 >actual &&\n+\tgrep 6ef19b41225c5369f1c104d45d8d85efa9b057b53b14b4b9b939dd74decc5321 actual\n+'\n+\n test_done\n"},{"id":"362429","messageId":"20181104234458.139223-12-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 11/12] sha256: add an SHA-256 implementation using libgcrypt","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:57Z","receivedAt":"2018-11-04T23:45:22Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Generally, one gets better performance out of cryptographic routines\nwritten in assembly than C, and this is also true for SHA-256. In\naddition, most Linux distributions cannot distribute Git linked against\nOpenSSL for licensing reasons.\n\nMost systems with GnuPG will also have libgcrypt, since it is a\ndependency of GnuPG. libgcrypt is also faster than the SHA1DC\nimplementation for messages of a few KiB and larger.\n\nFor comparison, on a Core i7-6600U, this implementation processes 16 KiB\nchunks at 355 MiB/s while SHA1DC processes equivalent chunks at 337\nMiB/s.\n\nIn addition, libgcrypt is licensed under the LGPL 2.1, which is\ncompatible with the GPL. Add an implementation of SHA-256 that uses\nlibgcrypt.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 13 +++++++++++--\n hash.h | 4 ++++\n sha256/gcrypt.h | 30 ++++++++++++++++++++++++++++++\n 3 files changed, 45 insertions(+), 2 deletions(-)\n create mode 100644 sha256/gcrypt.h\n\ndiff --git a/Makefile b/Makefile\nindex e99b7712f6..5a07e03100 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -179,6 +179,10 @@ all::\n # in one call to the platform's SHA1_Update(). e.g. APPLE_COMMON_CRYPTO\n # wants 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined.\n #\n+# Define BLK_SHA256 to use the built-in SHA-256 routines.\n+#\n+# Define GCRYPT_SHA256 to use the SHA-256 routines in libgcrypt.\n+#\n # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).\n #\n # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).\n@@ -1634,8 +1638,13 @@ endif\n endif\n endif\n \n-LIB_OBJS += sha256/block/sha256.o\n-BASIC_CFLAGS += -DSHA256_BLK\n+ifdef GCRYPT_SHA256\n+\tBASIC_CFLAGS += -DSHA256_GCRYPT\n+\tEXTLIBS += -lgcrypt\n+else\n+\tLIB_OBJS += sha256/block/sha256.o\n+\tBASIC_CFLAGS += -DSHA256_BLK\n+endif\n \n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\ndiff --git a/hash.h b/hash.h\nindex a9bc624020..2ef098052d 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -15,7 +15,11 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n+#if defined(SHA256_GCRYPT)\n+#include \"sha256/gcrypt.h\"\n+#else\n #include \"sha256/block/sha256.h\"\n+#endif\n \n #ifndef platform_SHA_CTX\n /*\ndiff --git a/sha256/gcrypt.h b/sha256/gcrypt.h\nnew file mode 100644\nindex 0000000000..09bd8bb200\n--- /dev/null\n+++ b/sha256/gcrypt.h\n@@ -0,0 +1,30 @@\n+#ifndef SHA256_GCRYPT_H\n+#define SHA256_GCRYPT_H\n+\n+#include \n+\n+#define SHA256_DIGEST_SIZE 32\n+\n+typedef gcry_md_hd_t gcrypt_SHA256_CTX;\n+\n+inline void gcrypt_SHA256_Init(gcrypt_SHA256_CTX *ctx)\n+{\n+\tgcry_md_open(ctx, GCRY_MD_SHA256, 0);\n+}\n+\n+inline void gcrypt_SHA256_Update(gcrypt_SHA256_CTX *ctx, const void *data, size_t len)\n+{\n+\tgcry_md_write(*ctx, data, len);\n+}\n+\n+inline void gcrypt_SHA256_Final(unsigned char *digest, gcrypt_SHA256_CTX *ctx)\n+{\n+\tmemcpy(digest, gcry_md_read(*ctx, GCRY_MD_SHA256), SHA256_DIGEST_SIZE);\n+}\n+\n+#define platform_SHA256_CTX gcrypt_SHA256_CTX\n+#define platform_SHA256_Init gcrypt_SHA256_Init\n+#define platform_SHA256_Update gcrypt_SHA256_Update\n+#define platform_SHA256_Final gcrypt_SHA256_Final\n+\n+#endif\n"},{"id":"362430","messageId":"20181104234458.139223-13-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v5 12/12] hash: add an SHA-256 implementation using OpenSSL","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-04T23:44:58Z","receivedAt":"2018-11-04T23:45:23Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We already have OpenSSL routines available for SHA-1, so add routines\nfor SHA-256 as well.\n\nOn a Core i7-6600U, this SHA-256 implementation compares favorably to\nthe SHA1DC SHA-1 implementation:\n\nSHA-1: 157 MiB/s (64 byte chunks); 337 MiB/s (16 KiB chunks)\nSHA-256: 165 MiB/s (64 byte chunks); 408 MiB/s (16 KiB chunks)\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 7 +++++++\n hash.h | 2 ++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 5a07e03100..36fd3a149b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -183,6 +183,8 @@ all::\n #\n # Define GCRYPT_SHA256 to use the SHA-256 routines in libgcrypt.\n #\n+# Define OPENSSL_SHA256 to use the SHA-256 routines in OpenSSL.\n+#\n # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).\n #\n # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).\n@@ -1638,6 +1640,10 @@ endif\n endif\n endif\n \n+ifdef OPENSSL_SHA256\n+\tEXTLIBS += $(LIB_4_CRYPTO)\n+\tBASIC_CFLAGS += -DSHA256_OPENSSL\n+else\n ifdef GCRYPT_SHA256\n \tBASIC_CFLAGS += -DSHA256_GCRYPT\n \tEXTLIBS += -lgcrypt\n@@ -1645,6 +1651,7 @@ else\n \tLIB_OBJS += sha256/block/sha256.o\n \tBASIC_CFLAGS += -DSHA256_BLK\n endif\n+endif\n \n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\ndiff --git a/hash.h b/hash.h\nindex 2ef098052d..adde708cf2 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -17,6 +17,8 @@\n \n #if defined(SHA256_GCRYPT)\n #include \"sha256/gcrypt.h\"\n+#elif defined(SHA256_OPENSSL)\n+#include \n #else\n #include \"sha256/block/sha256.h\"\n #endif\n"},{"id":"362446","messageId":"xmqqefc0jl8t.fsf@gitster-ct.c.googlers.com","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"Re: [PATCH v5 00/12] Base SHA-256 implementation","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-11-05T02:45:06Z","receivedAt":"2018-11-05T02:45:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> This series provides a functional SHA-256 implementation and wires it\n> up, along with some housekeeping patches to make it suitable for\n> testing.\n>\n> Changes from v4:\n> * Downcase hex constants for consistency.\n> * Remove needless parentheses in return statement.\n> * Remove braces for single statement loops.\n> * Switch to +=.\n> * Add references to rationale for SHA-256.\n> * Remove inclusion of \"git-compat-util.h\" in header.\n\nYou ended up not doing the last one, judging from the interdiff\nbelow. I think it is OK to leave the header in.\n\nThanks, will replace.\n"},{"id":"362476","messageId":"87a7mo7zwn.fsf@evledraar.gmail.com","threadId":"49668","inReplyTo":"20181104234458.139223-2-sandals@crustytoothpaste.net","subject":"Re: [PATCH v5 01/12] sha1-file: rename algorithm to \"sha1\"","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-05T07:21:28Z","receivedAt":"2018-11-05T07:21:34Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Nov 04 2018, brian m. carlson wrote:\n\n> The transition plan anticipates us using a syntax such as \"^{sha1}\" for\n> disambiguation. Since this is a syntax some people will be typing a\n> lot, it makes sense to provide a short, easy-to-type syntax. Omitting\n> the dash doesn't create any ambiguity; however, it does make the syntax\n> shorter and easier to type, especially for touch typists. In addition,\n> the transition plan already uses \"sha1\" in this context.\n\nThe comment for git_hash_algo's \"name\" member in hash.h says:\n\n\t/*\n\t * The name of the algorithm, as appears in the config file and in\n\t * messages.\n\t */\n\tconst char *name;\n\nWhereas this commit message just refers to a doesn't-yet-exist ^{$algo}\nsyntax. The hash-function-transition.txt doc also uses forms like sha1\nor sha256 in config, not sha-1 or sha-256.\n\nI don't have a point I'm leading up to here, other than a question of\nwhether we should be doing something closer to this:\n\ndiff --git a/hash.h b/hash.h\nindex 7c8238bc2e..8ae51ac410 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -67,10 +67,17 @@ typedef void (*git_hash_final_fn)(unsigned char *hash, git_hash_ctx *ctx);\n\n struct git_hash_algo {\n \t/*\n-\t * The name of the algorithm, as appears in the config file and in\n-\t * messages.\n+\t * The short name of the algorithm (e.g. \"sha1\") for use in\n+\t * config files (see hash-function-transition.txt) and the\n+\t * ^{$name} peel syntax.\n \t */\n-\tconst char *name;\n+\tconst char *short_name;\n+\n+\t/*\n+\t * The long name of the algorithm (e.g. \"SHA-1\") for use in\n+\t * messages to users.\n+\t */\n+\tconst char *long_name;\n\n \t/* A four-byte version identifier, used in pack indices. */\n \tuint32_t format_id;\ndiff --git a/sha1-file.c b/sha1-file.c\nindex dd0b6aa873..5ad0526155 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -86,6 +86,7 @@ static void git_hash_unknown_final(unsigned char *hash, git_hash_ctx *ctx)\n\n const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t{\n+\t\tNULL,\n \t\tNULL,\n \t\t0x00000000,\n \t\t0,\n@@ -97,7 +98,8 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\tNULL,\n \t},\n \t{\n-\t\t\"sha-1\",\n+\t\t\"sha1\",\n+\t\t\"SHA-1\",\n \t\t/* \"sha1\", big-endian */\n \t\t0x73686131,\n \t\tGIT_SHA1_RAWSZ,\n"},{"id":"362483","messageId":"87h8gv7nz1.fsf@evledraar.gmail.com","threadId":"49668","inReplyTo":"20181104234458.139223-11-sandals@crustytoothpaste.net","subject":"Re: [PATCH v5 10/12] Add a base implementation of SHA-256 support","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-05T11:39:14Z","receivedAt":"2018-11-05T11:39:19Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Nov 04 2018, brian m. carlson wrote:\n\n> SHA-1 is weak and we need to transition to a new hash function. For\n> some time, we have referred to this new function as NewHash. Recently,\n> we decided to pick SHA-256 as NewHash. The reasons behind the choice of\n> SHA-256 are outlined in the thread starting at [1] and in the commit\n> history for the hash function transition document.\n\nNit: In some contradiction now to what's said in\nhash-function-transition.txt, see 5988eb631a (\"doc\nhash-function-transition: clarify what SHAttered means\", 2018-03-26).\n\n> +\t{\n> +\t\t\"sha256\",\n> +\t\t/* \"s256\", big-endian */\n\nThe existing entry/comment for sha1 is:\n\n\t\t\"sha1\",\n\t\t/* \"sha1\", big-endian */\n\nSo why the sha256/s256 difference in the code/comment? Wondering if I'm\nmissing something and we're using \"s256\" for something.\n\n> const char *empty_tree_oid_hex(void)\n> diff --git a/sha256/block/sha256.c b/sha256/block/sha256.c\n> [...]\n\nI had a question before about whether we see ourselves perma-forking\nthis implementation based off libtomcrypt, as I recall you said yes.\n\nStill, I think it would be better to introduce this in at least two-four\ncommits where the upstream code is added as-is, then trimmed down to\nsize, then adapted to our coding style, and finally we add our own\nutility functions.\n\nIt'll make it easier to forward-port any future upstream changes.\n\n> +\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n> +\t\ttest-tool sha256 >actual &&\n> +\tgrep cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0 actual &&\n> +\tperl -E \"for (1..100000) { print q{abcdefghijklmnopqrstuvwxyz}; }\" | \\\n> +\t\ttest-tool sha256 >actual &&\n\nI've been wanting to make use depend on perl >= 5.10 (previous noises\nabout that on-list), but for now we claim to support >=5.8, which\ndoesn't have the -E switch.\n\nBut most importantly you aren't even using -E features here, and this\nisn't very idoimatic Perl. Instead do, respectively:\n\n perl -e 'print q{aaaaaaaaaa} x 100000'\n perl -e \"print q{abcdefghijklmnopqrstuvwxyz} x 100000\"\n"},{"id":"362626","messageId":"20181107013026.GD890086@genre.crustytoothpaste.net","threadId":"49668","inReplyTo":"87h8gv7nz1.fsf@evledraar.gmail.com","subject":"Re: [PATCH v5 10/12] Add a base implementation of SHA-256 support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-07T01:30:27Z","receivedAt":"2018-11-07T01:30:36Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Nov 05, 2018 at 12:39:14PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> On Sun, Nov 04 2018, brian m. carlson wrote:\n> > +\t{\n> > +\t\t\"sha256\",\n> > +\t\t/* \"s256\", big-endian */\n> \n> The existing entry/comment for sha1 is:\n> \n> \t\t\"sha1\",\n> \t\t/* \"sha1\", big-endian */\n> \n> So why the sha256/s256 difference in the code/comment? Wondering if I'm\n> missing something and we're using \"s256\" for something.\n\nAh, good question. The comment refers to the format_id field which\nfollows this comment. The value is the big-endian representation of\n\"s256\" as a 32-bit value. I picked that over \"sha2\" to avoid confusion\nin case we add SHA-512 in the future, since that's also an SHA-2\nalgorithm.\n\nConfig files and command-line interfaces will use \"sha1\" or \"sha256\",\nand binary formats will use those 32-bit values (\"sha1\" or \"s256\").\n\n> > const char *empty_tree_oid_hex(void)\n> > diff --git a/sha256/block/sha256.c b/sha256/block/sha256.c\n> > [...]\n> \n> I had a question before about whether we see ourselves perma-forking\n> this implementation based off libtomcrypt, as I recall you said yes.\n\nYes.\n\n> Still, I think it would be better to introduce this in at least two-four\n> commits where the upstream code is added as-is, then trimmed down to\n> size, then adapted to our coding style, and finally we add our own\n> utility functions.\n\nAt this point, the only code that's actually used from libtomcrypt is\nthe block transform. The upstream code is split over multiple files in\nmultiple directories and won't compile in our codebase without many\nfiles and a lot of work, so I don't feel good about either including\ncode that doesn't compile or including large numbers of files that don't\nmeet our coding standards (and that may still not compile because of\nplatform issues).\n\n> It'll make it easier to forward-port any future upstream changes.\n\nI don't foresee many, if any, changes to this code. It either\nimplements the specification or it doesn't, and it's objectively easy to\ndetermine which. There's not even an argument to port performance\nimprovements, since almost everyone will be using a crypto library to\nprovide this code because libraries perform so dramatically better.\nI've tried to not make the code perform worse than it did originally,\nbut that's it.\n\nFurthermore, the modified code carries a relatively small amount of\nresemblance to the original, so if we did port changes forward, we'd\nprobably have conflicts.\n\nIt seems like you really want to include the upstream code as a separate\ncommit and I understand where you're coming from with wanting to have\nthis split out into logical commits, but due to the specific nature of\nthis code, I see a lot of downsides and not a lot of upsides.\n\n> > +\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n> > +\t\ttest-tool sha256 >actual &&\n> > +\tgrep cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0 actual &&\n> > +\tperl -E \"for (1..100000) { print q{abcdefghijklmnopqrstuvwxyz}; }\" | \\\n> > +\t\ttest-tool sha256 >actual &&\n> \n> I've been wanting to make use depend on perl >= 5.10 (previous noises\n> about that on-list), but for now we claim to support >=5.8, which\n> doesn't have the -E switch.\n\nGood point. I'll fix that. After having written a lot of one-liners,\nI always write -E, and this was originally a one-liner.\n\n> But most importantly you aren't even using -E features here, and this\n> isn't very idoimatic Perl. Instead do, respectively:\n> \n> perl -e 'print q{aaaaaaaaaa} x 100000'\n> perl -e \"print q{abcdefghijklmnopqrstuvwxyz} x 100000\"\n\nI considered the more idiomatic version originally, but the latter could\nallocate a decent amount of memory in one chunk, and I wanted to avoid\nthat. I think what I'd like to do, actually, is turn on autoflush and\nuse a postfix for, which would be more idiomatic and could potentially\nprovide better testing of the chunking code. I'll add a comment to that\neffect.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"362894","messageId":"878t2153qy.fsf@evledraar.gmail.com","threadId":"49668","inReplyTo":"20181107013026.GD890086@genre.crustytoothpaste.net","subject":"Re: [PATCH v5 10/12] Add a base implementation of SHA-256 support","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-11-10T15:52:37Z","receivedAt":"2018-11-10T15:52:44Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Nov 07 2018, brian m. carlson wrote:\n\n> On Mon, Nov 05, 2018 at 12:39:14PM +0100, Ævar Arnfjörð Bjarmason wrote:\n>> On Sun, Nov 04 2018, brian m. carlson wrote:\n>> > +\t{\n>> > +\t\t\"sha256\",\n>> > +\t\t/* \"s256\", big-endian */\n>>\n>> The existing entry/comment for sha1 is:\n>>\n>> \t\t\"sha1\",\n>> \t\t/* \"sha1\", big-endian */\n>>\n>> So why the sha256/s256 difference in the code/comment? Wondering if I'm\n>> missing something and we're using \"s256\" for something.\n>\n> Ah, good question. The comment refers to the format_id field which\n> follows this comment. The value is the big-endian representation of\n> \"s256\" as a 32-bit value. I picked that over \"sha2\" to avoid confusion\n> in case we add SHA-512 in the future, since that's also an SHA-2\n> algorithm.\n>\n> Config files and command-line interfaces will use \"sha1\" or \"sha256\",\n> and binary formats will use those 32-bit values (\"sha1\" or \"s256\").\n\nOkey.\n\n>> > const char *empty_tree_oid_hex(void)\n>> > diff --git a/sha256/block/sha256.c b/sha256/block/sha256.c\n>> > [...]\n>>\n>> I had a question before about whether we see ourselves perma-forking\n>> this implementation based off libtomcrypt, as I recall you said yes.\n>\n> Yes.\n>\n>> Still, I think it would be better to introduce this in at least two-four\n>> commits where the upstream code is added as-is, then trimmed down to\n>> size, then adapted to our coding style, and finally we add our own\n>> utility functions.\n>\n> At this point, the only code that's actually used from libtomcrypt is\n> the block transform. The upstream code is split over multiple files in\n> multiple directories and won't compile in our codebase without many\n> files and a lot of work, so I don't feel good about either including\n> code that doesn't compile or including large numbers of files that don't\n> meet our coding standards (and that may still not compile because of\n> platform issues).\n>\n>> It'll make it easier to forward-port any future upstream changes.\n>\n> I don't foresee many, if any, changes to this code. It either\n> implements the specification or it doesn't, and it's objectively easy to\n> determine which. There's not even an argument to port performance\n> improvements, since almost everyone will be using a crypto library to\n> provide this code because libraries perform so dramatically better.\n> I've tried to not make the code perform worse than it did originally,\n> but that's it.\n>\n> Furthermore, the modified code carries a relatively small amount of\n> resemblance to the original, so if we did port changes forward, we'd\n> probably have conflicts.\n>\n> It seems like you really want to include the upstream code as a separate\n> commit and I understand where you're coming from with wanting to have\n> this split out into logical commits, but due to the specific nature of\n> this code, I see a lot of downsides and not a lot of upsides.\n\nYeah sorry to keep bringing this up. Your way makes sense, and I'd\nforgotten the details since last time . I'll shut up about it:)\n\n>> > +\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n>> > +\t\ttest-tool sha256 >actual &&\n>> > +\tgrep cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0 actual &&\n>> > +\tperl -E \"for (1..100000) { print q{abcdefghijklmnopqrstuvwxyz}; }\" | \\\n>> > +\t\ttest-tool sha256 >actual &&\n>>\n>> I've been wanting to make use depend on perl >= 5.10 (previous noises\n>> about that on-list), but for now we claim to support >=5.8, which\n>> doesn't have the -E switch.\n>\n> Good point. I'll fix that. After having written a lot of one-liners,\n> I always write -E, and this was originally a one-liner.\n>\n>> But most importantly you aren't even using -E features here, and this\n>> isn't very idoimatic Perl. Instead do, respectively:\n>>\n>> perl -e 'print q{aaaaaaaaaa} x 100000'\n>> perl -e \"print q{abcdefghijklmnopqrstuvwxyz} x 100000\"\n>\n> I considered the more idiomatic version originally, but the latter could\n> allocate a decent amount of memory in one chunk, and I wanted to avoid\n> that.\n\n~2.5MB for the latter, so trivial.\n\n> I think what I'd like to do, actually, is turn on autoflush and\n> use a postfix for, which would be more idiomatic and could potentially\n> provide better testing of the chunking code. I'll add a comment to that\n> effect.\n\nOkey. Maybe better to use syswrite() instead, or maybe print with\nautoflush is more idiomatic, I don't know.\n"},{"id":"363189","messageId":"06e5bc95-b901-163a-5088-888878187ca2@gmail.com","threadId":"49668","inReplyTo":"20181104234458.139223-3-sandals@crustytoothpaste.net","subject":"Re: [PATCH v5 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-11-13T18:42:31Z","receivedAt":"2018-11-13T18:42:37Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 11/4/2018 6:44 PM, brian m. carlson wrote:\n> +int hash_algo_by_name(const char *name)\n> +{\n> +\tint i;\n> +\tif (!name)\n> +\t\treturn GIT_HASH_UNKNOWN;\n> +\tfor (i = 1; i < GIT_HASH_NALGOS; i++)\n> +\t\tif (!strcmp(name, hash_algos[i].name))\n> +\t\t\treturn i;\n> +\treturn GIT_HASH_UNKNOWN;\n> +}\n> +\n\nToday's test coverage report [1] shows this method is not covered in the \ntest suite. Looking at 'pu', it doesn't have any callers.\n\nDo you have a work in progress series that will use this? Could we add a \ntest-tool to exercise this somehow?\n\nThanks,\n-Stolee\n\n[1] \nhttps://public-inbox.org/git/97be3e21-6a8c-9718-5161-37318f6d685f@gmail.com/\n"},{"id":"363190","messageId":"CACsJy8AtWG=3nJuqG07seczqeb8taimzT4fPbFHc-_3Y8FE_5Q@mail.gmail.com","threadId":"49668","inReplyTo":"06e5bc95-b901-163a-5088-888878187ca2@gmail.com","subject":"Re: [PATCH v5 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-13T18:45:41Z","receivedAt":"2018-11-13T18:46:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Nov 13, 2018 at 7:42 PM Derrick Stolee wrote:\n>\n> On 11/4/2018 6:44 PM, brian m. carlson wrote:\n> > +int hash_algo_by_name(const char *name)\n> > +{\n> > + int i;\n> > + if (!name)\n> > + return GIT_HASH_UNKNOWN;\n> > + for (i = 1; i < GIT_HASH_NALGOS; i++)\n> > + if (!strcmp(name, hash_algos[i].name))\n> > + return i;\n> > + return GIT_HASH_UNKNOWN;\n> > +}\n> > +\n>\n> Today's test coverage report [1] shows this method is not covered in the\n> test suite. Looking at 'pu', it doesn't have any callers.\n>\n> Do you have a work in progress series that will use this? Could we add a\n> test-tool to exercise this somehow?\n\nGoing by the function name, it should be used when hash-object or\nother commands learn about --object-format=.\n-- \nDuy\n"},{"id":"363236","messageId":"000ff851-3068-36f0-4fff-1e69cd24cbec@ramsayjones.plus.com","threadId":"49668","inReplyTo":"06e5bc95-b901-163a-5088-888878187ca2@gmail.com","subject":"Re: [PATCH v5 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2018-11-14T00:11:07Z","receivedAt":"2018-11-14T00:11:12Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 13/11/2018 18:42, Derrick Stolee wrote:\n> On 11/4/2018 6:44 PM, brian m. carlson wrote:\n>> +int hash_algo_by_name(const char *name)\n>> +{\n>> +    int i;\n>> +    if (!name)\n>> +        return GIT_HASH_UNKNOWN;\n>> +    for (i = 1; i < GIT_HASH_NALGOS; i++)\n>> +        if (!strcmp(name, hash_algos[i].name))\n>> +            return i;\n>> +    return GIT_HASH_UNKNOWN;\n>> +}\n>> +\n> \n> Today's test coverage report [1] shows this method is not covered in the test suite. Looking at 'pu', it doesn't have any callers.\n> \n> Do you have a work in progress series that will use this? Could we add a test-tool to exercise this somehow?\n\nThere are actually 4 unused external symbols resulting from Brian's\n'bc/sha-256' branch. The new unused externals in 'pu' looks like:\n\n $ diff nsc psc\n 37a38,39\n > hex.o\t- hash_to_hex\n > hex.o\t- hash_to_hex_algop_r\n 48a51\n > parse-options.o\t- optname\n 71a75\n > sha1-file.o\t- for_each_file_in_obj_subdir\n 72a77,78\n > sha1-file.o\t- hash_algo_by_id\n > sha1-file.o\t- hash_algo_by_name\n $ \n\nThe symbols from hex.o and sha1-file.o being the 4 symbols from\nthis branch.\n\nI suspect that upcoming patches will make use of them. ;-)\n\nATB,\nRamsay Jones\n\n\n"},{"id":"363274","messageId":"aa5e6eb3-f823-e79b-df8b-7eed6fe3a2fe@ramsayjones.plus.com","threadId":"49668","inReplyTo":"000ff851-3068-36f0-4fff-1e69cd24cbec@ramsayjones.plus.com","subject":"Re: [PATCH v5 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2018-11-14T00:42:45Z","receivedAt":"2018-11-14T00:42:53Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 14/11/2018 00:11, Ramsay Jones wrote:\n> \n> \n> On 13/11/2018 18:42, Derrick Stolee wrote:\n>> On 11/4/2018 6:44 PM, brian m. carlson wrote:\n>>> +int hash_algo_by_name(const char *name)\n>>> +{\n>>> +    int i;\n>>> +    if (!name)\n>>> +        return GIT_HASH_UNKNOWN;\n>>> +    for (i = 1; i < GIT_HASH_NALGOS; i++)\n>>> +        if (!strcmp(name, hash_algos[i].name))\n>>> +            return i;\n>>> +    return GIT_HASH_UNKNOWN;\n>>> +}\n>>> +\n>>\n>> Today's test coverage report [1] shows this method is not covered in the test suite. Looking at 'pu', it doesn't have any callers.\n>>\n>> Do you have a work in progress series that will use this? Could we add a test-tool to exercise this somehow?\n> \n> There are actually 4 unused external symbols resulting from Brian's\n> 'bc/sha-256' branch. The new unused externals in 'pu' looks like:\n> \n> $ diff nsc psc\n> 37a38,39\n> > hex.o\t- hash_to_hex\n> > hex.o\t- hash_to_hex_algop_r\n> 48a51\n> > parse-options.o\t- optname\n> 71a75\n> > sha1-file.o\t- for_each_file_in_obj_subdir\n> 72a77,78\n> > sha1-file.o\t- hash_algo_by_id\n> > sha1-file.o\t- hash_algo_by_name\n> $ \n> \n> The symbols from hex.o and sha1-file.o being the 4 symbols from\n> this branch.\n> \n> I suspect that upcoming patches will make use of them. ;-)\n\nBTW, if you were puzzling over the 3rd symbol from sha1-file.o\n(which I wasn't counting in the 4 symbols above! ;-) ), then I\nbelieve that is because Jeff's commit 3a2e08245c (\"object-store: \nprovide helpers for loose_objects_cache\", 2018-11-12) effectively\nmoved the only call outside of sha1-file.c (in sha1-name.c) back\ninto sha1-file.c\n\nSo, for_each_file_in_obj_subdir() could now be marked 'static'.\n(whether it should is a different issue).\n\nATB,\nRamsay Jones\n\n"},{"id":"363277","messageId":"20181114005142.GB5059@sigill.intra.peff.net","threadId":"49668","inReplyTo":"aa5e6eb3-f823-e79b-df8b-7eed6fe3a2fe@ramsayjones.plus.com","subject":"Re: [PATCH v5 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2018-11-14T00:51:42Z","receivedAt":"2018-11-14T00:51:46Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Nov 14, 2018 at 12:42:45AM +0000, Ramsay Jones wrote:\n\n> BTW, if you were puzzling over the 3rd symbol from sha1-file.o\n> (which I wasn't counting in the 4 symbols above! ;-) ), then I\n> believe that is because Jeff's commit 3a2e08245c (\"object-store: \n> provide helpers for loose_objects_cache\", 2018-11-12) effectively\n> moved the only call outside of sha1-file.c (in sha1-name.c) back\n> into sha1-file.c\n> \n> So, for_each_file_in_obj_subdir() could now be marked 'static'.\n> (whether it should is a different issue).\n\nI think it would be reasonable to do so. Most code shouldn't have to\ncare about the on-disk hashing structure, so ideally it wouldn't be part\nof the public interface.\n\n-Peff\n"},{"id":"363278","messageId":"20181114010141.GL890086@genre.crustytoothpaste.net","threadId":"49668","inReplyTo":"CACsJy8AtWG=3nJuqG07seczqeb8taimzT4fPbFHc-_3Y8FE_5Q@mail.gmail.com","subject":"Re: [PATCH v5 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T01:01:42Z","receivedAt":"2018-11-14T01:01:51Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Tue, Nov 13, 2018 at 07:45:41PM +0100, Duy Nguyen wrote:\n> On Tue, Nov 13, 2018 at 7:42 PM Derrick Stolee wrote:\n> >\n> > On 11/4/2018 6:44 PM, brian m. carlson wrote:\n> > > +int hash_algo_by_name(const char *name)\n> > > +{\n> > > + int i;\n> > > + if (!name)\n> > > + return GIT_HASH_UNKNOWN;\n> > > + for (i = 1; i < GIT_HASH_NALGOS; i++)\n> > > + if (!strcmp(name, hash_algos[i].name))\n> > > + return i;\n> > > + return GIT_HASH_UNKNOWN;\n> > > +}\n> > > +\n> >\n> > Today's test coverage report [1] shows this method is not covered in the\n> > test suite. Looking at 'pu', it doesn't have any callers.\n> >\n> > Do you have a work in progress series that will use this? Could we add a\n> > test-tool to exercise this somehow?\n> \n> Going by the function name, it should be used when hash-object or\n> other commands learn about --object-format=.\n\nCorrect. I have extensions.objectFormat code that will use this.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"363282","messageId":"20181114021118.GN890086@genre.crustytoothpaste.net","threadId":"49668","inReplyTo":"000ff851-3068-36f0-4fff-1e69cd24cbec@ramsayjones.plus.com","subject":"Re: [PATCH v5 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T02:11:18Z","receivedAt":"2018-11-14T02:11:27Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Wed, Nov 14, 2018 at 12:11:07AM +0000, Ramsay Jones wrote:\n> \n> \n> On 13/11/2018 18:42, Derrick Stolee wrote:\n> > On 11/4/2018 6:44 PM, brian m. carlson wrote:\n> >> +int hash_algo_by_name(const char *name)\n> >> +{\n> >> +    int i;\n> >> +    if (!name)\n> >> +        return GIT_HASH_UNKNOWN;\n> >> +    for (i = 1; i < GIT_HASH_NALGOS; i++)\n> >> +        if (!strcmp(name, hash_algos[i].name))\n> >> +            return i;\n> >> +    return GIT_HASH_UNKNOWN;\n> >> +}\n> >> +\n> > \n> > Today's test coverage report [1] shows this method is not covered in the test suite. Looking at 'pu', it doesn't have any callers.\n> > \n> > Do you have a work in progress series that will use this? Could we add a test-tool to exercise this somehow?\n> \n> There are actually 4 unused external symbols resulting from Brian's\n> 'bc/sha-256' branch. The new unused externals in 'pu' looks like:\n> \n> $ diff nsc psc\n> 37a38,39\n> > hex.o\t- hash_to_hex\n\nI have code that uses this in my object-id-part15 series. I also have\nanother series coming after this one that makes heavy use of it.\n\n> > hex.o\t- hash_to_hex_algop_r\n\nI believe this is because it's inline, since it is indeed used just a\nfew lines below its definition. I'll drop the inline, since it's meant\nto be externally visible.\n\n> > sha1-file.o\t- hash_algo_by_id\n\nThis will be used when I write pack index v3, which will be in my\nobject-id-part15 series.\n\n> > sha1-file.o\t- hash_algo_by_name\n\nThis is used in my object-id-part15 series.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"363292","messageId":"79b436d4-48f9-8e38-0dbe-1831aa3fc208@ramsayjones.plus.com","threadId":"49668","inReplyTo":"20181114021118.GN890086@genre.crustytoothpaste.net","subject":"Re: [PATCH v5 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"Ramsay Jones","fromEmail":"ramsay@ramsayjones.plus.com","sentAt":"2018-11-14T03:53:20Z","receivedAt":"2018-11-14T03:53:25Z","isPatch":true,"sender":{"key":"ramsay@ramsayjones.plus.com","avatar":"https://avatars.githubusercontent.com/u/33702710?v=4"},"body":"\n\nOn 14/11/2018 02:11, brian m. carlson wrote:\n> On Wed, Nov 14, 2018 at 12:11:07AM +0000, Ramsay Jones wrote:\n>>\n>>\n>> On 13/11/2018 18:42, Derrick Stolee wrote:\n>>> On 11/4/2018 6:44 PM, brian m. carlson wrote:\n>>>> +int hash_algo_by_name(const char *name)\n>>>> +{\n>>>> +    int i;\n>>>> +    if (!name)\n>>>> +        return GIT_HASH_UNKNOWN;\n>>>> +    for (i = 1; i < GIT_HASH_NALGOS; i++)\n>>>> +        if (!strcmp(name, hash_algos[i].name))\n>>>> +            return i;\n>>>> +    return GIT_HASH_UNKNOWN;\n>>>> +}\n>>>> +\n>>>\n>>> Today's test coverage report [1] shows this method is not covered in the test suite. Looking at 'pu', it doesn't have any callers.\n>>>\n>>> Do you have a work in progress series that will use this? Could we add a test-tool to exercise this somehow?\n>>\n>> There are actually 4 unused external symbols resulting from Brian's\n>> 'bc/sha-256' branch. The new unused externals in 'pu' looks like:\n>>\n>> $ diff nsc psc\n>> 37a38,39\n>> > hex.o\t- hash_to_hex\n> \n> I have code that uses this in my object-id-part15 series. I also have\n> another series coming after this one that makes heavy use of it.\n> \n>> > hex.o\t- hash_to_hex_algop_r\n> \n> I believe this is because it's inline, since it is indeed used just a\n> few lines below its definition. I'll drop the inline, since it's meant\n> to be externally visible.\n\nNo, this has nothing to do with the 'inline', it is simply not\ncalled outside of hex.c (at present). If you look at the assembler\n(objdump -d hex.o), you will find practically all of the function\ncalls in that file are inlined (even those not marked with 'inline').\n\n[I think the external declaration in cache.h forces the compiler to\nadd the external definition, despite the 'inline'. If you remove the\n'inline' and re-compile and disassemble again, the result is identical.]\n\nThanks for confirming upcoming patches will add uses for all of\nthese functions - I suspected that would be the case.\n\nThanks!\n\nATB,\nRamsay Jones\n\n> \n>> > sha1-file.o\t- hash_algo_by_id\n> \n> This will be used when I write pack index v3, which will be in my\n> object-id-part15 series.\n> \n>> > sha1-file.o\t- hash_algo_by_name\n> \n> This is used in my object-id-part15 series.\n> \n"},{"id":"363293","messageId":"20181114040938.517289-1-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181104234458.139223-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 00/12] Base SHA-256 implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:26Z","receivedAt":"2018-11-14T04:10:10Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This series provides a functional SHA-256 implementation and wires it\nup, along with some housekeeping patches to make it suitable for\ntesting.\n\nChanges from v5:\n* Remove inclusion of \"git-compat-util.h\" in header.\n* Remove \"inline\" from definition of hash_to_hex_algop_r.\n* Switch perl invocations in t0015 to use -e instead of -E.\n* Switch perl invocations in t0015 to use autoflush and postfix for.\n\nChanges from v4:\n* Downcase hex constants for consistency.\n* Remove needless parentheses in return statement.\n* Remove braces for single statement loops.\n* Switch to +=.\n* Add references to rationale for SHA-256.\n\nChanges from v3:\n* Switch to using inline functions instead of macros in many cases.\n* Undefine remaining macros at the top.\n\nChanges from v2:\n* Improve commit messages to include timing and performance information.\n* Improve commit messages to be less ambiguous and more friendly to a\n wider variety of English speakers.\n* Prefer functions taking struct git_hash_algo in hex.c.\n* Port pieces of the block-sha1 implementation over to the block-sha256\n implementation for better compatibility.\n* Drop patch 13 in favor of further discussion about the best way\n forward for versioning commit graph.\n* Rename the test so as to have a different number from other tests.\n* Rebase on master.\n\nChanges from v1:\n* Add a hash_to_hex function mirroring sha1_to_hex, but for\n the_hash_algo.\n* Strip commit message explanation about why we chose SHA-256.\n* Rebase on master\n* Strip leading whitespace from commit message.\n* Improve commit-graph patch to cover new code added since v1.\n* Be more honest about the scope of work involved in porting the SHA-256\n implementation out of libtomcrypt.\n* Revert change to limit hashcmp to 20 bytes.\n\nbrian m. carlson (12):\n sha1-file: rename algorithm to \"sha1\"\n sha1-file: provide functions to look up hash algorithms\n hex: introduce functions to print arbitrary hashes\n cache: make hashcmp and hasheq work with larger hashes\n t: add basic tests for our SHA-1 implementation\n t: make the sha1 test-tool helper generic\n sha1-file: add a constant for hash block size\n t/helper: add a test helper to compute hash speed\n commit-graph: convert to using the_hash_algo\n Add a base implementation of SHA-256 support\n sha256: add an SHA-256 implementation using libgcrypt\n hash: add an SHA-256 implementation using OpenSSL\n\n Makefile | 22 +++\n cache.h | 51 ++++---\n commit-graph.c | 33 ++---\n hash.h | 41 +++++-\n hex.c | 32 +++--\n sha1-file.c | 70 ++++++++-\n sha256/block/sha256.c | 196 ++++++++++++++++++++++++++\n sha256/block/sha256.h | 24 ++++\n sha256/gcrypt.h | 30 ++++\n t/helper/test-hash-speed.c | 61 ++++++++\n t/helper/{test-sha1.c => test-hash.c} | 19 +--\n t/helper/test-sha1.c | 52 +------\n t/helper/test-sha256.c | 7 +\n t/helper/test-tool.c | 2 +\n t/helper/test-tool.h | 4 +\n t/t0015-hash.sh | 55 ++++++++\n 16 files changed, 595 insertions(+), 104 deletions(-)\n create mode 100644 sha256/block/sha256.c\n create mode 100644 sha256/block/sha256.h\n create mode 100644 sha256/gcrypt.h\n create mode 100644 t/helper/test-hash-speed.c\n copy t/helper/{test-sha1.c => test-hash.c} (65%)\n create mode 100644 t/helper/test-sha256.c\n create mode 100755 t/t0015-hash.sh\n\nRange-diff against v5:\n 1: a004a4c982 < -: ---------- :hash-impl\n 2: cf9f7f5620 = 1: fa21b5948c sha1-file: rename algorithm to \"sha1\"\n 3: 0144deaebe = 2: 4e146c92af sha1-file: provide functions to look up hash algorithms\n 4: b74858fb03 ! 3: 10e7893242 hex: introduce functions to print arbitrary hashes\n @@ -64,8 +64,8 @@\n }\n \n -char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n -+inline char *hash_to_hex_algop_r(char *buffer, const unsigned char *hash,\n -+\t\t\t\t\tconst struct git_hash_algo *algop)\n ++char *hash_to_hex_algop_r(char *buffer, const unsigned char *hash,\n ++\t\t\t const struct git_hash_algo *algop)\n {\n \tstatic const char hex[] = \"0123456789abcdef\";\n \tchar *buf = buffer;\n 5: e9703017a4 = 4: 6396a0ff57 cache: make hashcmp and hasheq work with larger hashes\n 6: ab85a834fd ! 5: a51a8b5638 t: add basic tests for our SHA-1 implementation\n @@ -33,7 +33,7 @@\n +\tgrep c12252ceda8be8994d5fa0290a47231c1d16aae3 actual &&\n +\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha1 >actual &&\n +\tgrep 32d10c7b8cf96570ca04ce37f2a19d84240d3a89 actual &&\n -+\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n ++\tperl -e \"$| = 1; print q{aaaaaaaaaa} for 1..100000;\" | \\\n +\t\ttest-tool sha1 >actual &&\n +\tgrep 34aa973cd4c4daa4f61eeb2bdbad27316534016f actual &&\n +\tprintf \"blob 0\\0\" | test-tool sha1 >actual &&\n 7: 962f6d8903 = 6: d50b040b33 t: make the sha1 test-tool helper generic\n 8: 53addf4d58 = 7: cbe15d32ed sha1-file: add a constant for hash block size\n 9: 9ace10faa2 = 8: dbb916ff26 t/helper: add a test helper to compute hash speed\n10: 9adc56d01e = 9: 77a741aa3f commit-graph: convert to using the_hash_algo\n11: 90544c504c ! 10: ff7ec3b1ef Add a base implementation of SHA-256 support\n @@ -413,8 +413,6 @@\n +#ifndef SHA256_BLOCK_SHA256_H\n +#define SHA256_BLOCK_SHA256_H\n +\n -+#include \"git-compat-util.h\"\n -+\n +#define blk_SHA256_BLKSIZE 64\n +\n +struct blk_SHA256_CTX {\n @@ -492,10 +490,11 @@\n +\tgrep f7846f55cf23e14eebeab5b4e1550cad5b509e3348fbc4efa3a1413d393cb650 actual &&\n +\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha256 >actual &&\n +\tgrep 71c480df93d6ae2f1efad1447c66c9525e316218cf51fc8d9ed832f2daf18b73 actual &&\n -+\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n ++\t# Try to exercise the chunking code by turning autoflush on.\n ++\tperl -e \"$| = 1; print q{aaaaaaaaaa} for 1..100000;\" | \\\n +\t\ttest-tool sha256 >actual &&\n +\tgrep cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0 actual &&\n -+\tperl -E \"for (1..100000) { print q{abcdefghijklmnopqrstuvwxyz}; }\" | \\\n ++\tperl -e \"$| = 1; print q{abcdefghijklmnopqrstuvwxyz} for 1..100000;\" | \\\n +\t\ttest-tool sha256 >actual &&\n +\tgrep e406ba321ca712ad35a698bf0af8d61fc4dc40eca6bdcea4697962724ccbde35 actual &&\n +\tprintf \"blob 0\\0\" | test-tool sha256 >actual &&\n12: 467c86e878 = 11: 46f443809b sha256: add an SHA-256 implementation using libgcrypt\n13: 73e4bc17d0 = 12: 6d358417d9 hash: add an SHA-256 implementation using OpenSSL\n"},{"id":"363294","messageId":"20181114040938.517289-2-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 01/12] sha1-file: rename algorithm to \"sha1\"","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:27Z","receivedAt":"2018-11-14T04:10:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"The transition plan anticipates us using a syntax such as \"^{sha1}\" for\ndisambiguation. Since this is a syntax some people will be typing a\nlot, it makes sense to provide a short, easy-to-type syntax. Omitting\nthe dash doesn't create any ambiguity; however, it does make the syntax\nshorter and easier to type, especially for touch typists. In addition,\nthe transition plan already uses \"sha1\" in this context.\n\nRename the name of SHA-1 implementation to \"sha1\".\n\nNote that this change creates no backwards compatibility concerns, since\nwe haven't yet used this field in any configuration settings.\n\nSigned-off-by: brian m. carlson \n---\n sha1-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 2daf7d9935..3b9c042691 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -97,7 +97,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\tNULL,\n \t},\n \t{\n-\t\t\"sha-1\",\n+\t\t\"sha1\",\n \t\t/* \"sha1\", big-endian */\n \t\t0x73686131,\n \t\tGIT_SHA1_RAWSZ,\n"},{"id":"363295","messageId":"20181114040938.517289-3-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:28Z","receivedAt":"2018-11-14T04:10:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"There are several ways we might refer to a hash algorithm: by name, such\nas in the config file; by format ID, such as in a pack; or internally,\nby a pointer to the hash_algos array. Provide functions to look up hash\nalgorithms based on these various forms and return the internal constant\nused for them. If conversion to another form is necessary, this\ninternal constant can be used to look up the proper data in the\nhash_algos array.\n\nSigned-off-by: brian m. carlson \n---\n hash.h | 13 +++++++++++++\n sha1-file.c | 21 +++++++++++++++++++++\n 2 files changed, 34 insertions(+)\n\ndiff --git a/hash.h b/hash.h\nindex 7c8238bc2e..80881eea47 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -98,4 +98,17 @@ struct git_hash_algo {\n };\n extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n \n+/*\n+ * Return a GIT_HASH_* constant based on the name. Returns GIT_HASH_UNKNOWN if\n+ * the name doesn't match a known algorithm.\n+ */\n+int hash_algo_by_name(const char *name);\n+/* Identical, except based on the format ID. */\n+int hash_algo_by_id(uint32_t format_id);\n+/* Identical, except for a pointer to struct git_hash_algo. */\n+static inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n+{\n+\treturn p - hash_algos;\n+}\n+\n #endif\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 3b9c042691..93ed8c8686 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -122,6 +122,27 @@ const char *empty_blob_oid_hex(void)\n \treturn oid_to_hex_r(buf, the_hash_algo->empty_blob);\n }\n \n+int hash_algo_by_name(const char *name)\n+{\n+\tint i;\n+\tif (!name)\n+\t\treturn GIT_HASH_UNKNOWN;\n+\tfor (i = 1; i < GIT_HASH_NALGOS; i++)\n+\t\tif (!strcmp(name, hash_algos[i].name))\n+\t\t\treturn i;\n+\treturn GIT_HASH_UNKNOWN;\n+}\n+\n+int hash_algo_by_id(uint32_t format_id)\n+{\n+\tint i;\n+\tfor (i = 1; i < GIT_HASH_NALGOS; i++)\n+\t\tif (format_id == hash_algos[i].format_id)\n+\t\t\treturn i;\n+\treturn GIT_HASH_UNKNOWN;\n+}\n+\n+\n /*\n * This is meant to hold a *small* number of objects that you would\n * want read_sha1_file() to be able to return, but yet you do not want\n"},{"id":"363296","messageId":"20181114040938.517289-4-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 03/12] hex: introduce functions to print arbitrary hashes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:29Z","receivedAt":"2018-11-14T04:10:20Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Currently, we have functions that turn an arbitrary SHA-1 value or an\nobject ID into hex format, either using a static buffer or with a\nuser-provided buffer. Add variants of these functions that can handle\nan arbitrary hash algorithm, specified by constant. Update the\ndocumentation as well.\n\nWhile we're at it, remove the \"extern\" declaration from this family of\nfunctions, since it's not needed and our style now recommends against\nit.\n\nWe use the variant taking the algorithm structure pointer as the\ninternal variant, since taking an algorithm pointer is the easiest way\nto handle all of the variants in use.\n\nNote that we maintain these functions because there are hashes which\nmust change based on the hash algorithm in use but are not object IDs\n(such as pack checksums).\n\nSigned-off-by: brian m. carlson \n---\n cache.h | 15 +++++++++------\n hex.c | 32 ++++++++++++++++++++++++--------\n 2 files changed, 33 insertions(+), 14 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex ca36b44ee0..cb7268deea 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1365,9 +1365,9 @@ extern int get_oid_hex(const char *hex, struct object_id *sha1);\n extern int hex_to_bytes(unsigned char *binary, const char *hex, size_t len);\n \n /*\n- * Convert a binary sha1 to its hex equivalent. The `_r` variant is reentrant,\n+ * Convert a binary hash to its hex equivalent. The `_r` variant is reentrant,\n * and writes the NUL-terminated output to the buffer `out`, which must be at\n- * least `GIT_SHA1_HEXSZ + 1` bytes, and returns a pointer to out for\n+ * least `GIT_MAX_HEXSZ + 1` bytes, and returns a pointer to out for\n * convenience.\n *\n * The non-`_r` variant returns a static buffer, but uses a ring of 4\n@@ -1375,10 +1375,13 @@ extern int hex_to_bytes(unsigned char *binary, const char *hex, size_t len);\n *\n * printf(\"%s -> %s\", sha1_to_hex(one), sha1_to_hex(two));\n */\n-extern char *sha1_to_hex_r(char *out, const unsigned char *sha1);\n-extern char *oid_to_hex_r(char *out, const struct object_id *oid);\n-extern char *sha1_to_hex(const unsigned char *sha1);\t/* static buffer result! */\n-extern char *oid_to_hex(const struct object_id *oid);\t/* same static buffer as sha1_to_hex */\n+char *hash_to_hex_algop_r(char *buffer, const unsigned char *hash, const struct git_hash_algo *);\n+char *sha1_to_hex_r(char *out, const unsigned char *sha1);\n+char *oid_to_hex_r(char *out, const struct object_id *oid);\n+char *hash_to_hex_algop(const unsigned char *hash, const struct git_hash_algo *);\t/* static buffer result! */\n+char *sha1_to_hex(const unsigned char *sha1);\t\t\t\t\t\t/* same static buffer */\n+char *hash_to_hex(const unsigned char *hash);\t\t\t\t\t\t/* same static buffer */\n+char *oid_to_hex(const struct object_id *oid);\t\t\t\t\t\t/* same static buffer */\n \n /*\n * Parse a 40-character hexadecimal object ID starting from hex, updating the\ndiff --git a/hex.c b/hex.c\nindex 10af1a29e8..7850a8879d 100644\n--- a/hex.c\n+++ b/hex.c\n@@ -73,14 +73,15 @@ int parse_oid_hex(const char *hex, struct object_id *oid, const char **end)\n \treturn ret;\n }\n \n-char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n+char *hash_to_hex_algop_r(char *buffer, const unsigned char *hash,\n+\t\t\t const struct git_hash_algo *algop)\n {\n \tstatic const char hex[] = \"0123456789abcdef\";\n \tchar *buf = buffer;\n \tint i;\n \n-\tfor (i = 0; i < the_hash_algo->rawsz; i++) {\n-\t\tunsigned int val = *sha1++;\n+\tfor (i = 0; i < algop->rawsz; i++) {\n+\t\tunsigned int val = *hash++;\n \t\t*buf++ = hex[val >> 4];\n \t\t*buf++ = hex[val & 0xf];\n \t}\n@@ -89,20 +90,35 @@ char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n \treturn buffer;\n }\n \n-char *oid_to_hex_r(char *buffer, const struct object_id *oid)\n+char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n {\n-\treturn sha1_to_hex_r(buffer, oid->hash);\n+\treturn hash_to_hex_algop_r(buffer, sha1, &hash_algos[GIT_HASH_SHA1]);\n }\n \n-char *sha1_to_hex(const unsigned char *sha1)\n+char *oid_to_hex_r(char *buffer, const struct object_id *oid)\n+{\n+\treturn hash_to_hex_algop_r(buffer, oid->hash, the_hash_algo);\n+}\n+\n+char *hash_to_hex_algop(const unsigned char *hash, const struct git_hash_algo *algop)\n {\n \tstatic int bufno;\n \tstatic char hexbuffer[4][GIT_MAX_HEXSZ + 1];\n \tbufno = (bufno + 1) % ARRAY_SIZE(hexbuffer);\n-\treturn sha1_to_hex_r(hexbuffer[bufno], sha1);\n+\treturn hash_to_hex_algop_r(hexbuffer[bufno], hash, algop);\n+}\n+\n+char *sha1_to_hex(const unsigned char *sha1)\n+{\n+\treturn hash_to_hex_algop(sha1, &hash_algos[GIT_HASH_SHA1]);\n+}\n+\n+char *hash_to_hex(const unsigned char *hash)\n+{\n+\treturn hash_to_hex_algop(hash, the_hash_algo);\n }\n \n char *oid_to_hex(const struct object_id *oid)\n {\n-\treturn sha1_to_hex(oid->hash);\n+\treturn hash_to_hex_algop(oid->hash, the_hash_algo);\n }\n"},{"id":"363297","messageId":"20181114040938.517289-5-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 04/12] cache: make hashcmp and hasheq work with larger hashes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:30Z","receivedAt":"2018-11-14T04:10:20Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"In 183a638b7d (\"hashcmp: assert constant hash size\", 2018-08-23), we\nmodified hashcmp to assert that the hash size was always 20 to help it\noptimize and inline calls to memcmp. In a future series, we replaced\nmany calls to hashcmp and oidcmp with calls to hasheq and oideq to\nimprove inlining further.\n\nHowever, we want to support hash algorithms other than SHA-1, namely\nSHA-256. When doing so, we must handle the case where these values are\n32 bytes long as well as 20. Adjust hashcmp to handle two cases:\n20-byte matches, and maximum-size matches. Therefore, when we include\nSHA-256, we'll automatically handle it properly, while at the same time\nteaching the compiler that there are only two possible options to\nconsider. This will allow the compiler to write the most efficient\npossible code.\n\nCopy similar code into hasheq and perform an identical transformation.\nAt least with GCC 8.2.0, making hasheq defer to hashcmp when there are\ntwo branches prevents the compiler from inlining the comparison, while\nthe code in this patch is inlined properly. Add a comment to avoid an\naccidental performance regression from well-intentioned refactoring.\n\nSigned-off-by: brian m. carlson \n---\n cache.h | 22 ++++++++++++----------\n 1 file changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex cb7268deea..8607878dbc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1028,16 +1028,12 @@ extern const struct object_id null_oid;\n static inline int hashcmp(const unsigned char *sha1, const unsigned char *sha2)\n {\n \t/*\n-\t * This is a temporary optimization hack. By asserting the size here,\n-\t * we let the compiler know that it's always going to be 20, which lets\n-\t * it turn this fixed-size memcmp into a few inline instructions.\n-\t *\n-\t * This will need to be extended or ripped out when we learn about\n-\t * hashes of different sizes.\n+\t * Teach the compiler that there are only two possibilities of hash size\n+\t * here, so that it can optimize for this case as much as possible.\n \t */\n-\tif (the_hash_algo->rawsz != 20)\n-\t\tBUG(\"hash size not yet supported by hashcmp\");\n-\treturn memcmp(sha1, sha2, the_hash_algo->rawsz);\n+\tif (the_hash_algo->rawsz == GIT_MAX_RAWSZ)\n+\t\treturn memcmp(sha1, sha2, GIT_MAX_RAWSZ);\n+\treturn memcmp(sha1, sha2, GIT_SHA1_RAWSZ);\n }\n \n static inline int oidcmp(const struct object_id *oid1, const struct object_id *oid2)\n@@ -1047,7 +1043,13 @@ static inline int oidcmp(const struct object_id *oid1, const struct object_id *o\n \n static inline int hasheq(const unsigned char *sha1, const unsigned char *sha2)\n {\n-\treturn !hashcmp(sha1, sha2);\n+\t/*\n+\t * We write this here instead of deferring to hashcmp so that the\n+\t * compiler can properly inline it and avoid calling memcmp.\n+\t */\n+\tif (the_hash_algo->rawsz == GIT_MAX_RAWSZ)\n+\t\treturn !memcmp(sha1, sha2, GIT_MAX_RAWSZ);\n+\treturn !memcmp(sha1, sha2, GIT_SHA1_RAWSZ);\n }\n \n static inline int oideq(const struct object_id *oid1, const struct object_id *oid2)\n"},{"id":"363298","messageId":"20181114040938.517289-7-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 06/12] t: make the sha1 test-tool helper generic","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:32Z","receivedAt":"2018-11-14T04:10:23Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Since we're going to have multiple hash algorithms to test, it makes\nsense to share as much of the test code as possible. Convert the sha1\nhelper for the test-tool to be generic and move it out into its own\nmodule. This will allow us to share most of this code with our NewHash\nimplementation.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 1 +\n t/helper/{test-sha1.c => test-hash.c} | 19 +++++-----\n t/helper/test-sha1.c | 52 +--------------------------\n t/helper/test-tool.h | 2 ++\n 4 files changed, 14 insertions(+), 60 deletions(-)\n copy t/helper/{test-sha1.c => test-hash.c} (65%)\n\ndiff --git a/Makefile b/Makefile\nindex 016fdcdb81..daf0b198ec 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -724,6 +724,7 @@ TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-dump-untracked-cache.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n+TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-hash.c\nsimilarity index 65%\ncopy from t/helper/test-sha1.c\ncopy to t/helper/test-hash.c\nindex 1ba0675c75..0a31de66f3 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-hash.c\n@@ -1,13 +1,14 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n \n-int cmd__sha1(int ac, const char **av)\n+int cmd_hash_impl(int ac, const char **av, int algo)\n {\n-\tgit_SHA_CTX ctx;\n-\tunsigned char sha1[20];\n+\tgit_hash_ctx ctx;\n+\tunsigned char hash[GIT_MAX_HEXSZ];\n \tunsigned bufsz = 8192;\n \tint binary = 0;\n \tchar *buffer;\n+\tconst struct git_hash_algo *algop = &hash_algos[algo];\n \n \tif (ac == 2) {\n \t\tif (!strcmp(av[1], \"-b\"))\n@@ -26,7 +27,7 @@ int cmd__sha1(int ac, const char **av)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\tgit_SHA1_Init(&ctx);\n+\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -38,20 +39,20 @@ int cmd__sha1(int ac, const char **av)\n \t\t\tif (sz == 0)\n \t\t\t\tbreak;\n \t\t\tif (sz < 0)\n-\t\t\t\tdie_errno(\"test-sha1\");\n+\t\t\t\tdie_errno(\"test-hash\");\n \t\t\tthis_sz += sz;\n \t\t\tcp += sz;\n \t\t\troom -= sz;\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\tgit_SHA1_Update(&ctx, buffer, this_sz);\n+\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\tgit_SHA1_Final(sha1, &ctx);\n+\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n-\t\tfwrite(sha1, 1, 20, stdout);\n+\t\tfwrite(hash, 1, algop->rawsz, stdout);\n \telse\n-\t\tputs(sha1_to_hex(sha1));\n+\t\tputs(hash_to_hex_algop(hash, algop));\n \texit(0);\n }\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-sha1.c\nindex 1ba0675c75..d860c387c3 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-sha1.c\n@@ -3,55 +3,5 @@\n \n int cmd__sha1(int ac, const char **av)\n {\n-\tgit_SHA_CTX ctx;\n-\tunsigned char sha1[20];\n-\tunsigned bufsz = 8192;\n-\tint binary = 0;\n-\tchar *buffer;\n-\n-\tif (ac == 2) {\n-\t\tif (!strcmp(av[1], \"-b\"))\n-\t\t\tbinary = 1;\n-\t\telse\n-\t\t\tbufsz = strtoul(av[1], NULL, 10) * 1024 * 1024;\n-\t}\n-\n-\tif (!bufsz)\n-\t\tbufsz = 8192;\n-\n-\twhile ((buffer = malloc(bufsz)) == NULL) {\n-\t\tfprintf(stderr, \"bufsz %u is too big, halving...\\n\", bufsz);\n-\t\tbufsz /= 2;\n-\t\tif (bufsz < 1024)\n-\t\t\tdie(\"OOPS\");\n-\t}\n-\n-\tgit_SHA1_Init(&ctx);\n-\n-\twhile (1) {\n-\t\tssize_t sz, this_sz;\n-\t\tchar *cp = buffer;\n-\t\tunsigned room = bufsz;\n-\t\tthis_sz = 0;\n-\t\twhile (room) {\n-\t\t\tsz = xread(0, cp, room);\n-\t\t\tif (sz == 0)\n-\t\t\t\tbreak;\n-\t\t\tif (sz < 0)\n-\t\t\t\tdie_errno(\"test-sha1\");\n-\t\t\tthis_sz += sz;\n-\t\t\tcp += sz;\n-\t\t\troom -= sz;\n-\t\t}\n-\t\tif (this_sz == 0)\n-\t\t\tbreak;\n-\t\tgit_SHA1_Update(&ctx, buffer, this_sz);\n-\t}\n-\tgit_SHA1_Final(sha1, &ctx);\n-\n-\tif (binary)\n-\t\tfwrite(sha1, 1, 20, stdout);\n-\telse\n-\t\tputs(sha1_to_hex(sha1));\n-\texit(0);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1);\n }\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 042f12464b..b5b7712a1d 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -51,4 +51,6 @@ int cmd__windows_named_pipe(int argc, const char **argv);\n #endif\n int cmd__write_cache(int argc, const char **argv);\n \n+int cmd_hash_impl(int ac, const char **av, int algo);\n+\n #endif\n"},{"id":"363299","messageId":"20181114040938.517289-6-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 05/12] t: add basic tests for our SHA-1 implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:31Z","receivedAt":"2018-11-14T04:10:23Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We have in the past had some unfortunate endianness issues with some\nSHA-1 implementations we ship, especially on big-endian machines. Add\nan explicit test using the test helper to catch these issues and point\nthem out prominently. This test can also be used as a staging ground\nfor people testing additional algorithms to verify that their\nimplementations are working as expected.\n\nSigned-off-by: brian m. carlson \n---\n t/t0015-hash.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n create mode 100755 t/t0015-hash.sh\n\ndiff --git a/t/t0015-hash.sh b/t/t0015-hash.sh\nnew file mode 100755\nindex 0000000000..884fe5acd1\n--- /dev/null\n+++ b/t/t0015-hash.sh\n@@ -0,0 +1,29 @@\n+#!/bin/sh\n+\n+test_description='test basic hash implementation'\n+. ./test-lib.sh\n+\n+\n+test_expect_success 'test basic SHA-1 hash values' '\n+\ttest-tool sha1 actual &&\n+\tgrep da39a3ee5e6b4b0d3255bfef95601890afd80709 actual &&\n+\tprintf \"a\" | test-tool sha1 >actual &&\n+\tgrep 86f7e437faa5a7fce15d1ddcb9eaeaea377667b8 actual &&\n+\tprintf \"abc\" | test-tool sha1 >actual &&\n+\tgrep a9993e364706816aba3e25717850c26c9cd0d89d actual &&\n+\tprintf \"message digest\" | test-tool sha1 >actual &&\n+\tgrep c12252ceda8be8994d5fa0290a47231c1d16aae3 actual &&\n+\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha1 >actual &&\n+\tgrep 32d10c7b8cf96570ca04ce37f2a19d84240d3a89 actual &&\n+\tperl -e \"$| = 1; print q{aaaaaaaaaa} for 1..100000;\" | \\\n+\t\ttest-tool sha1 >actual &&\n+\tgrep 34aa973cd4c4daa4f61eeb2bdbad27316534016f actual &&\n+\tprintf \"blob 0\\0\" | test-tool sha1 >actual &&\n+\tgrep e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 actual &&\n+\tprintf \"blob 3\\0abc\" | test-tool sha1 >actual &&\n+\tgrep f2ba8f84ab5c1bce84a7b441cb1959cfc7093b7f actual &&\n+\tprintf \"tree 0\\0\" | test-tool sha1 >actual &&\n+\tgrep 4b825dc642cb6eb9a060e54bf8d69288fbee4904 actual\n+'\n+\n+test_done\n"},{"id":"363300","messageId":"20181114040938.517289-8-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 07/12] sha1-file: add a constant for hash block size","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:33Z","receivedAt":"2018-11-14T04:10:25Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"There is one place we need the hash algorithm block size: the HMAC code\nfor push certs. Expose this constant in struct git_hash_algo and expose\nvalues for SHA-1 and for the largest value of any hash.\n\nSigned-off-by: brian m. carlson \n---\n cache.h | 4 ++++\n hash.h | 3 +++\n sha1-file.c | 2 ++\n 3 files changed, 9 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 8607878dbc..62b2f3a5e8 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -45,10 +45,14 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n+/* The block size of SHA-1. */\n+#define GIT_SHA1_BLKSZ 64\n \n /* The length in byte and in hex digits of the largest possible hash value. */\n #define GIT_MAX_RAWSZ GIT_SHA1_RAWSZ\n #define GIT_MAX_HEXSZ GIT_SHA1_HEXSZ\n+/* The largest possible block size for any supported hash. */\n+#define GIT_MAX_BLKSZ GIT_SHA1_BLKSZ\n \n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/hash.h b/hash.h\nindex 80881eea47..1bcf7ab6fd 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -81,6 +81,9 @@ struct git_hash_algo {\n \t/* The length of the hash in hex characters. */\n \tsize_t hexsz;\n \n+\t/* The block size of the hash. */\n+\tsize_t blksz;\n+\n \t/* The hash initialization function. */\n \tgit_hash_init_fn init_fn;\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex 93ed8c8686..c47349a5f8 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -90,6 +90,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t0x00000000,\n \t\t0,\n \t\t0,\n+\t\t0,\n \t\tgit_hash_unknown_init,\n \t\tgit_hash_unknown_update,\n \t\tgit_hash_unknown_final,\n@@ -102,6 +103,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t0x73686131,\n \t\tGIT_SHA1_RAWSZ,\n \t\tGIT_SHA1_HEXSZ,\n+\t\tGIT_SHA1_BLKSZ,\n \t\tgit_hash_sha1_init,\n \t\tgit_hash_sha1_update,\n \t\tgit_hash_sha1_final,\n"},{"id":"363301","messageId":"20181114040938.517289-9-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 08/12] t/helper: add a test helper to compute hash speed","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:34Z","receivedAt":"2018-11-14T04:10:25Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a utility (which is less for the testsuite and more for developers)\nthat can compute hash speeds for whatever hash algorithms are\nimplemented. This allows developers to test their personal systems to\ndetermine the performance characteristics of various algorithms.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 1 +\n t/helper/test-hash-speed.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n 4 files changed, 64 insertions(+)\n create mode 100644 t/helper/test-hash-speed.c\n\ndiff --git a/Makefile b/Makefile\nindex daf0b198ec..c6f06bf50b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -726,6 +726,7 @@ TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n+TEST_BUILTINS_OBJS += test-hash-speed.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\n TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\ndiff --git a/t/helper/test-hash-speed.c b/t/helper/test-hash-speed.c\nnew file mode 100644\nindex 0000000000..432233c7f0\n--- /dev/null\n+++ b/t/helper/test-hash-speed.c\n@@ -0,0 +1,61 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+#define NUM_SECONDS 3\n+\n+static inline void compute_hash(const struct git_hash_algo *algo, git_hash_ctx *ctx, uint8_t *final, const void *p, size_t len)\n+{\n+\talgo->init_fn(ctx);\n+\talgo->update_fn(ctx, p, len);\n+\talgo->final_fn(final, ctx);\n+}\n+\n+int cmd__hash_speed(int ac, const char **av)\n+{\n+\tgit_hash_ctx ctx;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n+\tclock_t initial, start, end;\n+\tunsigned bufsizes[] = { 64, 256, 1024, 8192, 16384 };\n+\tint i;\n+\tvoid *p;\n+\tconst struct git_hash_algo *algo = NULL;\n+\n+\tif (ac == 2) {\n+\t\tfor (i = 1; i < GIT_HASH_NALGOS; i++) {\n+\t\t\tif (!strcmp(av[1], hash_algos[i].name)) {\n+\t\t\t\talgo = &hash_algos[i];\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (!algo)\n+\t\tdie(\"usage: test-tool hash-speed algo_name\");\n+\n+\t/* Use this as an offset to make overflow less likely. */\n+\tinitial = clock();\n+\n+\tprintf(\"algo: %s\\n\", algo->name);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(bufsizes); i++) {\n+\t\tunsigned long j, kb;\n+\t\tdouble kb_per_sec;\n+\t\tp = xcalloc(1, bufsizes[i]);\n+\t\tstart = end = clock() - initial;\n+\t\tfor (j = 0; ((end - start) / CLOCKS_PER_SEC) < NUM_SECONDS; j++) {\n+\t\t\tcompute_hash(algo, &ctx, hash, p, bufsizes[i]);\n+\n+\t\t\t/*\n+\t\t\t * Only check elapsed time every 128 iterations to avoid\n+\t\t\t * dominating the runtime with system calls.\n+\t\t\t */\n+\t\t\tif (!(j & 127))\n+\t\t\t\tend = clock() - initial;\n+\t\t}\n+\t\tkb = j * bufsizes[i];\n+\t\tkb_per_sec = kb / (1024 * ((double)end - start) / CLOCKS_PER_SEC);\n+\t\tprintf(\"size %u: %lu iters; %lu KiB; %0.2f KiB/s\\n\", bufsizes[i], j, kb, kb_per_sec);\n+\t\tfree(p);\n+\t}\n+\n+\texit(0);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex bfb195b1a8..c222d532a2 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -20,6 +20,7 @@ static struct test_cmd cmds[] = {\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"hashmap\", cmd__hashmap },\n+\t{ \"hash-speed\", cmd__hash_speed },\n \t{ \"index-version\", cmd__index_version },\n \t{ \"json-writer\", cmd__json_writer },\n \t{ \"lazy-init-name-hash\", cmd__lazy_init_name_hash },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex b5b7712a1d..70dafb7649 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -16,6 +16,7 @@ int cmd__dump_untracked_cache(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n+int cmd__hash_speed(int argc, const char **argv);\n int cmd__index_version(int argc, const char **argv);\n int cmd__json_writer(int argc, const char **argv);\n int cmd__lazy_init_name_hash(int argc, const char **argv);\n"},{"id":"363302","messageId":"20181114040938.517289-10-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 09/12] commit-graph: convert to using the_hash_algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:35Z","receivedAt":"2018-11-14T04:10:27Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Instead of using hard-coded constants for object sizes, use\nthe_hash_algo to look them up. In addition, use a function call to look\nup the object ID version and produce the correct value. For now, we use\nversion 1, which means to use the default algorithm used in the rest of\nthe repository.\n\nSigned-off-by: brian m. carlson \n---\n commit-graph.c | 33 +++++++++++++++++----------------\n 1 file changed, 17 insertions(+), 16 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 40c855f185..6763d19288 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -23,16 +23,11 @@\n #define GRAPH_CHUNKID_DATA 0x43444154 /* \"CDAT\" */\n #define GRAPH_CHUNKID_LARGEEDGES 0x45444745 /* \"EDGE\" */\n \n-#define GRAPH_DATA_WIDTH 36\n+#define GRAPH_DATA_WIDTH (the_hash_algo->rawsz + 16)\n \n #define GRAPH_VERSION_1 0x1\n #define GRAPH_VERSION GRAPH_VERSION_1\n \n-#define GRAPH_OID_VERSION_SHA1 1\n-#define GRAPH_OID_LEN_SHA1 GIT_SHA1_RAWSZ\n-#define GRAPH_OID_VERSION GRAPH_OID_VERSION_SHA1\n-#define GRAPH_OID_LEN GRAPH_OID_LEN_SHA1\n-\n #define GRAPH_OCTOPUS_EDGES_NEEDED 0x80000000\n #define GRAPH_PARENT_MISSING 0x7fffffff\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n@@ -44,13 +39,18 @@\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n #define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n-\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n+\t\t\t+ GRAPH_FANOUT_SIZE + the_hash_algo->rawsz)\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static uint8_t oid_version(void)\n+{\n+\treturn 1;\n+}\n+\n static struct commit_graph *alloc_commit_graph(void)\n {\n \tstruct commit_graph *g = xcalloc(1, sizeof(*g));\n@@ -125,15 +125,15 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \t}\n \n \thash_version = *(unsigned char*)(data + 5);\n-\tif (hash_version != GRAPH_OID_VERSION) {\n+\tif (hash_version != oid_version()) {\n \t\terror(_(\"hash version %X does not match version %X\"),\n-\t\t hash_version, GRAPH_OID_VERSION);\n+\t\t hash_version, oid_version());\n \t\tgoto cleanup_fail;\n \t}\n \n \tgraph = alloc_commit_graph();\n \n-\tgraph->hash_len = GRAPH_OID_LEN;\n+\tgraph->hash_len = the_hash_algo->rawsz;\n \tgraph->num_chunks = *(unsigned char*)(data + 6);\n \tgraph->graph_fd = fd;\n \tgraph->data = graph_map;\n@@ -149,7 +149,7 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \n \t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n \n-\t\tif (chunk_offset > graph_size - GIT_MAX_RAWSZ) {\n+\t\tif (chunk_offset > graph_size - the_hash_algo->rawsz) {\n \t\t\terror(_(\"improper chunk offset %08x%08x\"), (uint32_t)(chunk_offset >> 32),\n \t\t\t (uint32_t)chunk_offset);\n \t\t\tgoto cleanup_fail;\n@@ -764,6 +764,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n \tstruct progress *progress = NULL;\n+\tconst unsigned hashsz = the_hash_algo->rawsz;\n \n \tif (!commit_graph_compatible(the_repository))\n \t\treturn;\n@@ -909,7 +910,7 @@ void write_commit_graph(const char *obj_dir,\n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, GRAPH_OID_VERSION);\n+\thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, num_chunks);\n \thashwrite_u8(f, 0); /* unused padding byte */\n \n@@ -924,8 +925,8 @@ void write_commit_graph(const char *obj_dir,\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\n \tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n@@ -938,8 +939,8 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\twrite_graph_chunk_oids(f, hashsz, commits.list, commits.nr);\n+\twrite_graph_chunk_data(f, hashsz, commits.list, commits.nr);\n \twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph(the_repository);\n"},{"id":"363303","messageId":"20181114040938.517289-11-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 10/12] Add a base implementation of SHA-256 support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:36Z","receivedAt":"2018-11-14T04:10:28Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"SHA-1 is weak and we need to transition to a new hash function. For\nsome time, we have referred to this new function as NewHash. Recently,\nwe decided to pick SHA-256 as NewHash. The reasons behind the choice of\nSHA-256 are outlined in the thread starting at [1] and in the commit\nhistory for the hash function transition document.\n\nAdd a basic implementation of SHA-256 based off libtomcrypt, which is in\nthe public domain. Optimize it and restructure it to meet our coding\nstandards. Pull in the update and final functions from the SHA-1 block\nimplementation, as we know these function correctly with all compilers.\nThis implementation is slower than SHA-1, but more performant\nimplementations will be introduced in future commits.\n\nWire up SHA-256 in the list of hash algorithms, and add a test that the\nalgorithm works correctly.\n\nNote that with this patch, it is still not possible to switch to using\nSHA-256 in Git. Additional patches are needed to prepare the code to\nhandle a larger hash algorithm and further test fixes are needed.\n\n[1] https://public-inbox.org/git/20180609224913.GC38834@genre.crustytoothpaste.net/\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 4 +\n cache.h | 12 ++-\n hash.h | 19 +++-\n sha1-file.c | 45 ++++++++++\n sha256/block/sha256.c | 196 +++++++++++++++++++++++++++++++++++++++++\n sha256/block/sha256.h | 24 +++++\n t/helper/test-sha256.c | 7 ++\n t/helper/test-tool.c | 1 +\n t/helper/test-tool.h | 1 +\n t/t0015-hash.sh | 26 ++++++\n 10 files changed, 331 insertions(+), 4 deletions(-)\n create mode 100644 sha256/block/sha256.c\n create mode 100644 sha256/block/sha256.h\n create mode 100644 t/helper/test-sha256.c\n\ndiff --git a/Makefile b/Makefile\nindex c6f06bf50b..1302c07f5c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -749,6 +749,7 @@ TEST_BUILTINS_OBJS += test-run-command.o\n TEST_BUILTINS_OBJS += test-scrap-cache-tree.o\n TEST_BUILTINS_OBJS += test-sha1.o\n TEST_BUILTINS_OBJS += test-sha1-array.o\n+TEST_BUILTINS_OBJS += test-sha256.o\n TEST_BUILTINS_OBJS += test-sigchain.o\n TEST_BUILTINS_OBJS += test-strcmp-offset.o\n TEST_BUILTINS_OBJS += test-string-list.o\n@@ -1646,6 +1647,9 @@ endif\n endif\n endif\n \n+LIB_OBJS += sha256/block/sha256.o\n+BASIC_CFLAGS += -DSHA256_BLK\n+\n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\n \tBASIC_CFLAGS += -DSHA1_MAX_BLOCK_SIZE=\"$(SHA1_MAX_BLOCK_SIZE)\"\ndiff --git a/cache.h b/cache.h\nindex 62b2f3a5e8..b0dfe42736 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -48,11 +48,17 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n /* The block size of SHA-1. */\n #define GIT_SHA1_BLKSZ 64\n \n+/* The length in bytes and in hex digits of an object name (SHA-256 value). */\n+#define GIT_SHA256_RAWSZ 32\n+#define GIT_SHA256_HEXSZ (2 * GIT_SHA256_RAWSZ)\n+/* The block size of SHA-256. */\n+#define GIT_SHA256_BLKSZ 64\n+\n /* The length in byte and in hex digits of the largest possible hash value. */\n-#define GIT_MAX_RAWSZ GIT_SHA1_RAWSZ\n-#define GIT_MAX_HEXSZ GIT_SHA1_HEXSZ\n+#define GIT_MAX_RAWSZ GIT_SHA256_RAWSZ\n+#define GIT_MAX_HEXSZ GIT_SHA256_HEXSZ\n /* The largest possible block size for any supported hash. */\n-#define GIT_MAX_BLKSZ GIT_SHA1_BLKSZ\n+#define GIT_MAX_BLKSZ GIT_SHA256_BLKSZ\n \n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/hash.h b/hash.h\nindex 1bcf7ab6fd..a9bc624020 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -15,6 +15,8 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n+#include \"sha256/block/sha256.h\"\n+\n #ifndef platform_SHA_CTX\n /*\n * platform's underlying implementation of SHA-1; could be OpenSSL,\n@@ -34,6 +36,18 @@\n #define git_SHA1_Update\t\tplatform_SHA1_Update\n #define git_SHA1_Final\t\tplatform_SHA1_Final\n \n+#ifndef platform_SHA256_CTX\n+#define platform_SHA256_CTX\tSHA256_CTX\n+#define platform_SHA256_Init\tSHA256_Init\n+#define platform_SHA256_Update\tSHA256_Update\n+#define platform_SHA256_Final\tSHA256_Final\n+#endif\n+\n+#define git_SHA256_CTX\t\tplatform_SHA256_CTX\n+#define git_SHA256_Init\t\tplatform_SHA256_Init\n+#define git_SHA256_Update\tplatform_SHA256_Update\n+#define git_SHA256_Final\tplatform_SHA256_Final\n+\n #ifdef SHA1_MAX_BLOCK_SIZE\n #include \"compat/sha1-chunked.h\"\n #undef git_SHA1_Update\n@@ -52,12 +66,15 @@\n #define GIT_HASH_UNKNOWN 0\n /* SHA-1 */\n #define GIT_HASH_SHA1 1\n+/* SHA-256 */\n+#define GIT_HASH_SHA256 2\n /* Number of algorithms supported (including unknown). */\n-#define GIT_HASH_NALGOS (GIT_HASH_SHA1 + 1)\n+#define GIT_HASH_NALGOS (GIT_HASH_SHA256 + 1)\n \n /* A suitably aligned type for stack allocations of hash contexts. */\n union git_hash_ctx {\n \tgit_SHA_CTX sha1;\n+\tgit_SHA256_CTX sha256;\n };\n typedef union git_hash_ctx git_hash_ctx;\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex c47349a5f8..e7764822ea 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -40,10 +40,20 @@\n #define EMPTY_TREE_SHA1_BIN_LITERAL \\\n \t \"\\x4b\\x82\\x5d\\xc6\\x42\\xcb\\x6e\\xb9\\xa0\\x60\" \\\n \t \"\\xe5\\x4b\\xf8\\xd6\\x92\\x88\\xfb\\xee\\x49\\x04\"\n+#define EMPTY_TREE_SHA256_BIN_LITERAL \\\n+\t\"\\x6e\\xf1\\x9b\\x41\\x22\\x5c\\x53\\x69\\xf1\\xc1\" \\\n+\t\"\\x04\\xd4\\x5d\\x8d\\x85\\xef\\xa9\\xb0\\x57\\xb5\" \\\n+\t\"\\x3b\\x14\\xb4\\xb9\\xb9\\x39\\xdd\\x74\\xde\\xcc\" \\\n+\t\"\\x53\\x21\"\n \n #define EMPTY_BLOB_SHA1_BIN_LITERAL \\\n \t\"\\xe6\\x9d\\xe2\\x9b\\xb2\\xd1\\xd6\\x43\\x4b\\x8b\" \\\n \t\"\\x29\\xae\\x77\\x5a\\xd8\\xc2\\xe4\\x8c\\x53\\x91\"\n+#define EMPTY_BLOB_SHA256_BIN_LITERAL \\\n+\t\"\\x47\\x3a\\x0f\\x4c\\x3b\\xe8\\xa9\\x36\\x81\\xa2\" \\\n+\t\"\\x67\\xe3\\xb1\\xe9\\xa7\\xdc\\xda\\x11\\x85\\x43\" \\\n+\t\"\\x6f\\xe1\\x41\\xf7\\x74\\x91\\x20\\xa3\\x03\\x72\" \\\n+\t\"\\x18\\x13\"\n \n const unsigned char null_sha1[GIT_MAX_RAWSZ];\n const struct object_id null_oid;\n@@ -53,6 +63,12 @@ static const struct object_id empty_tree_oid = {\n static const struct object_id empty_blob_oid = {\n \tEMPTY_BLOB_SHA1_BIN_LITERAL\n };\n+static const struct object_id empty_tree_oid_sha256 = {\n+\tEMPTY_TREE_SHA256_BIN_LITERAL\n+};\n+static const struct object_id empty_blob_oid_sha256 = {\n+\tEMPTY_BLOB_SHA256_BIN_LITERAL\n+};\n \n static void git_hash_sha1_init(git_hash_ctx *ctx)\n {\n@@ -69,6 +85,22 @@ static void git_hash_sha1_final(unsigned char *hash, git_hash_ctx *ctx)\n \tgit_SHA1_Final(hash, &ctx->sha1);\n }\n \n+\n+static void git_hash_sha256_init(git_hash_ctx *ctx)\n+{\n+\tgit_SHA256_Init(&ctx->sha256);\n+}\n+\n+static void git_hash_sha256_update(git_hash_ctx *ctx, const void *data, size_t len)\n+{\n+\tgit_SHA256_Update(&ctx->sha256, data, len);\n+}\n+\n+static void git_hash_sha256_final(unsigned char *hash, git_hash_ctx *ctx)\n+{\n+\tgit_SHA256_Final(hash, &ctx->sha256);\n+}\n+\n static void git_hash_unknown_init(git_hash_ctx *ctx)\n {\n \tBUG(\"trying to init unknown hash\");\n@@ -110,6 +142,19 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t&empty_tree_oid,\n \t\t&empty_blob_oid,\n \t},\n+\t{\n+\t\t\"sha256\",\n+\t\t/* \"s256\", big-endian */\n+\t\t0x73323536,\n+\t\tGIT_SHA256_RAWSZ,\n+\t\tGIT_SHA256_HEXSZ,\n+\t\tGIT_SHA256_BLKSZ,\n+\t\tgit_hash_sha256_init,\n+\t\tgit_hash_sha256_update,\n+\t\tgit_hash_sha256_final,\n+\t\t&empty_tree_oid_sha256,\n+\t\t&empty_blob_oid_sha256,\n+\t}\n };\n \n const char *empty_tree_oid_hex(void)\ndiff --git a/sha256/block/sha256.c b/sha256/block/sha256.c\nnew file mode 100644\nindex 0000000000..37850b4e52\n--- /dev/null\n+++ b/sha256/block/sha256.c\n@@ -0,0 +1,196 @@\n+#include \"git-compat-util.h\"\n+#include \"./sha256.h\"\n+\n+#undef RND\n+#undef BLKSIZE\n+\n+#define BLKSIZE blk_SHA256_BLKSIZE\n+\n+void blk_SHA256_Init(blk_SHA256_CTX *ctx)\n+{\n+\tctx->offset = 0;\n+\tctx->size = 0;\n+\tctx->state[0] = 0x6a09e667ul;\n+\tctx->state[1] = 0xbb67ae85ul;\n+\tctx->state[2] = 0x3c6ef372ul;\n+\tctx->state[3] = 0xa54ff53aul;\n+\tctx->state[4] = 0x510e527ful;\n+\tctx->state[5] = 0x9b05688cul;\n+\tctx->state[6] = 0x1f83d9abul;\n+\tctx->state[7] = 0x5be0cd19ul;\n+}\n+\n+static inline uint32_t ror(uint32_t x, unsigned n)\n+{\n+\treturn (x >> n) | (x << (32 - n));\n+}\n+\n+static inline uint32_t ch(uint32_t x, uint32_t y, uint32_t z)\n+{\n+\treturn z ^ (x & (y ^ z));\n+}\n+\n+static inline uint32_t maj(uint32_t x, uint32_t y, uint32_t z)\n+{\n+\treturn ((x | y) & z) | (x & y);\n+}\n+\n+static inline uint32_t sigma0(uint32_t x)\n+{\n+\treturn ror(x, 2) ^ ror(x, 13) ^ ror(x, 22);\n+}\n+\n+static inline uint32_t sigma1(uint32_t x)\n+{\n+\treturn ror(x, 6) ^ ror(x, 11) ^ ror(x, 25);\n+}\n+\n+static inline uint32_t gamma0(uint32_t x)\n+{\n+\treturn ror(x, 7) ^ ror(x, 18) ^ (x >> 3);\n+}\n+\n+static inline uint32_t gamma1(uint32_t x)\n+{\n+\treturn ror(x, 17) ^ ror(x, 19) ^ (x >> 10);\n+}\n+\n+static void blk_SHA256_Transform(blk_SHA256_CTX *ctx, const unsigned char *buf)\n+{\n+\n+\tuint32_t S[8], W[64], t0, t1;\n+\tint i;\n+\n+\t/* copy state into S */\n+\tfor (i = 0; i < 8; i++)\n+\t\tS[i] = ctx->state[i];\n+\n+\t/* copy the state into 512-bits into W[0..15] */\n+\tfor (i = 0; i < 16; i++, buf += sizeof(uint32_t))\n+\t\tW[i] = get_be32(buf);\n+\n+\t/* fill W[16..63] */\n+\tfor (i = 16; i < 64; i++)\n+\t\tW[i] = gamma1(W[i - 2]) + W[i - 7] + gamma0(W[i - 15]) + W[i - 16];\n+\n+#define RND(a,b,c,d,e,f,g,h,i,ki) \\\n+\tt0 = h + sigma1(e) + ch(e, f, g) + ki + W[i]; \\\n+\tt1 = sigma0(a) + maj(a, b, c); \\\n+\td += t0; \\\n+\th = t0 + t1;\n+\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x71374491);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcf);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba5);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25b);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],5,0x59f111f1);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],6,0x923f82a4);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],7,0xab1c5ed5);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],8,0xd807aa98);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],9,0x12835b01);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],10,0x243185be);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],11,0x550c7dc3);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],12,0x72be5d74);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],13,0x80deb1fe);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],14,0x9bdc06a7);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],15,0xc19bf174);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],16,0xe49b69c1);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],17,0xefbe4786);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],18,0x0fc19dc6);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],19,0x240ca1cc);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],20,0x2de92c6f);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],21,0x4a7484aa);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],22,0x5cb0a9dc);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],23,0x76f988da);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],24,0x983e5152);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],25,0xa831c66d);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],26,0xb00327c8);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],27,0xbf597fc7);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],28,0xc6e00bf3);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],29,0xd5a79147);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],30,0x06ca6351);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],31,0x14292967);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],32,0x27b70a85);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],33,0x2e1b2138);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],34,0x4d2c6dfc);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],35,0x53380d13);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],36,0x650a7354);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],37,0x766a0abb);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],38,0x81c2c92e);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],39,0x92722c85);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],40,0xa2bfe8a1);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],41,0xa81a664b);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],42,0xc24b8b70);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],43,0xc76c51a3);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],44,0xd192e819);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],45,0xd6990624);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],46,0xf40e3585);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],47,0x106aa070);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],48,0x19a4c116);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],49,0x1e376c08);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],50,0x2748774c);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],51,0x34b0bcb5);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],52,0x391c0cb3);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],53,0x4ed8aa4a);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],54,0x5b9cca4f);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],55,0x682e6ff3);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],56,0x748f82ee);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],57,0x78a5636f);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],58,0x84c87814);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],59,0x8cc70208);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],60,0x90befffa);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],61,0xa4506ceb);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],62,0xbef9a3f7);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],63,0xc67178f2);\n+\n+\tfor (i = 0; i < 8; i++)\n+\t\tctx->state[i] += S[i];\n+}\n+\n+void blk_SHA256_Update(blk_SHA256_CTX *ctx, const void *data, size_t len)\n+{\n+\tunsigned int len_buf = ctx->size & 63;\n+\n+\tctx->size += len;\n+\n+\t/* Read the data into buf and process blocks as they get full */\n+\tif (len_buf) {\n+\t\tunsigned int left = 64 - len_buf;\n+\t\tif (len < left)\n+\t\t\tleft = len;\n+\t\tmemcpy(len_buf + ctx->buf, data, left);\n+\t\tlen_buf = (len_buf + left) & 63;\n+\t\tlen -= left;\n+\t\tdata = ((const char *)data + left);\n+\t\tif (len_buf)\n+\t\t\treturn;\n+\t\tblk_SHA256_Transform(ctx, ctx->buf);\n+\t}\n+\twhile (len >= 64) {\n+\t\tblk_SHA256_Transform(ctx, data);\n+\t\tdata = ((const char *)data + 64);\n+\t\tlen -= 64;\n+\t}\n+\tif (len)\n+\t\tmemcpy(ctx->buf, data, len);\n+}\n+\n+void blk_SHA256_Final(unsigned char *digest, blk_SHA256_CTX *ctx)\n+{\n+\tstatic const unsigned char pad[64] = { 0x80 };\n+\tunsigned int padlen[2];\n+\tint i;\n+\n+\t/* Pad with a binary 1 (ie 0x80), then zeroes, then length */\n+\tpadlen[0] = htonl((uint32_t)(ctx->size >> 29));\n+\tpadlen[1] = htonl((uint32_t)(ctx->size << 3));\n+\n+\ti = ctx->size & 63;\n+\tblk_SHA256_Update(ctx, pad, 1 + (63 & (55 - i)));\n+\tblk_SHA256_Update(ctx, padlen, 8);\n+\n+\t/* copy output */\n+\tfor (i = 0; i < 8; i++, digest += sizeof(uint32_t))\n+\t\tput_be32(digest, ctx->state[i]);\n+}\ndiff --git a/sha256/block/sha256.h b/sha256/block/sha256.h\nnew file mode 100644\nindex 0000000000..5099d6421d\n--- /dev/null\n+++ b/sha256/block/sha256.h\n@@ -0,0 +1,24 @@\n+#ifndef SHA256_BLOCK_SHA256_H\n+#define SHA256_BLOCK_SHA256_H\n+\n+#define blk_SHA256_BLKSIZE 64\n+\n+struct blk_SHA256_CTX {\n+\tuint32_t state[8];\n+\tuint64_t size;\n+\tuint32_t offset;\n+\tuint8_t buf[blk_SHA256_BLKSIZE];\n+};\n+\n+typedef struct blk_SHA256_CTX blk_SHA256_CTX;\n+\n+void blk_SHA256_Init(blk_SHA256_CTX *ctx);\n+void blk_SHA256_Update(blk_SHA256_CTX *ctx, const void *data, size_t len);\n+void blk_SHA256_Final(unsigned char *digest, blk_SHA256_CTX *ctx);\n+\n+#define platform_SHA256_CTX blk_SHA256_CTX\n+#define platform_SHA256_Init blk_SHA256_Init\n+#define platform_SHA256_Update blk_SHA256_Update\n+#define platform_SHA256_Final blk_SHA256_Final\n+\n+#endif\ndiff --git a/t/helper/test-sha256.c b/t/helper/test-sha256.c\nnew file mode 100644\nindex 0000000000..0ac6a99d5f\n--- /dev/null\n+++ b/t/helper/test-sha256.c\n@@ -0,0 +1,7 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+int cmd__sha256(int ac, const char **av)\n+{\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c222d532a2..5b137874e1 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -43,6 +43,7 @@ static struct test_cmd cmds[] = {\n \t{ \"scrap-cache-tree\", cmd__scrap_cache_tree },\n \t{ \"sha1\", cmd__sha1 },\n \t{ \"sha1-array\", cmd__sha1_array },\n+\t{ \"sha256\", cmd__sha256 },\n \t{ \"sigchain\", cmd__sigchain },\n \t{ \"strcmp-offset\", cmd__strcmp_offset },\n \t{ \"string-list\", cmd__string_list },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 70dafb7649..ca5c88edb2 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -39,6 +39,7 @@ int cmd__run_command(int argc, const char **argv);\n int cmd__scrap_cache_tree(int argc, const char **argv);\n int cmd__sha1(int argc, const char **argv);\n int cmd__sha1_array(int argc, const char **argv);\n+int cmd__sha256(int argc, const char **argv);\n int cmd__sigchain(int argc, const char **argv);\n int cmd__strcmp_offset(int argc, const char **argv);\n int cmd__string_list(int argc, const char **argv);\ndiff --git a/t/t0015-hash.sh b/t/t0015-hash.sh\nindex 884fe5acd1..291e9061f3 100755\n--- a/t/t0015-hash.sh\n+++ b/t/t0015-hash.sh\n@@ -26,4 +26,30 @@ test_expect_success 'test basic SHA-1 hash values' '\n \tgrep 4b825dc642cb6eb9a060e54bf8d69288fbee4904 actual\n '\n \n+test_expect_success 'test basic SHA-256 hash values' '\n+\ttest-tool sha256 actual &&\n+\tgrep e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 actual &&\n+\tprintf \"a\" | test-tool sha256 >actual &&\n+\tgrep ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb actual &&\n+\tprintf \"abc\" | test-tool sha256 >actual &&\n+\tgrep ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad actual &&\n+\tprintf \"message digest\" | test-tool sha256 >actual &&\n+\tgrep f7846f55cf23e14eebeab5b4e1550cad5b509e3348fbc4efa3a1413d393cb650 actual &&\n+\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha256 >actual &&\n+\tgrep 71c480df93d6ae2f1efad1447c66c9525e316218cf51fc8d9ed832f2daf18b73 actual &&\n+\t# Try to exercise the chunking code by turning autoflush on.\n+\tperl -e \"$| = 1; print q{aaaaaaaaaa} for 1..100000;\" | \\\n+\t\ttest-tool sha256 >actual &&\n+\tgrep cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0 actual &&\n+\tperl -e \"$| = 1; print q{abcdefghijklmnopqrstuvwxyz} for 1..100000;\" | \\\n+\t\ttest-tool sha256 >actual &&\n+\tgrep e406ba321ca712ad35a698bf0af8d61fc4dc40eca6bdcea4697962724ccbde35 actual &&\n+\tprintf \"blob 0\\0\" | test-tool sha256 >actual &&\n+\tgrep 473a0f4c3be8a93681a267e3b1e9a7dcda1185436fe141f7749120a303721813 actual &&\n+\tprintf \"blob 3\\0abc\" | test-tool sha256 >actual &&\n+\tgrep c1cf6e465077930e88dc5136641d402f72a229ddd996f627d60e9639eaba35a6 actual &&\n+\tprintf \"tree 0\\0\" | test-tool sha256 >actual &&\n+\tgrep 6ef19b41225c5369f1c104d45d8d85efa9b057b53b14b4b9b939dd74decc5321 actual\n+'\n+\n test_done\n"},{"id":"363304","messageId":"20181114040938.517289-12-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 11/12] sha256: add an SHA-256 implementation using libgcrypt","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:37Z","receivedAt":"2018-11-14T04:10:30Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Generally, one gets better performance out of cryptographic routines\nwritten in assembly than C, and this is also true for SHA-256. In\naddition, most Linux distributions cannot distribute Git linked against\nOpenSSL for licensing reasons.\n\nMost systems with GnuPG will also have libgcrypt, since it is a\ndependency of GnuPG. libgcrypt is also faster than the SHA1DC\nimplementation for messages of a few KiB and larger.\n\nFor comparison, on a Core i7-6600U, this implementation processes 16 KiB\nchunks at 355 MiB/s while SHA1DC processes equivalent chunks at 337\nMiB/s.\n\nIn addition, libgcrypt is licensed under the LGPL 2.1, which is\ncompatible with the GPL. Add an implementation of SHA-256 that uses\nlibgcrypt.\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 13 +++++++++++--\n hash.h | 4 ++++\n sha256/gcrypt.h | 30 ++++++++++++++++++++++++++++++\n 3 files changed, 45 insertions(+), 2 deletions(-)\n create mode 100644 sha256/gcrypt.h\n\ndiff --git a/Makefile b/Makefile\nindex 1302c07f5c..6c99844aa8 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -179,6 +179,10 @@ all::\n # in one call to the platform's SHA1_Update(). e.g. APPLE_COMMON_CRYPTO\n # wants 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined.\n #\n+# Define BLK_SHA256 to use the built-in SHA-256 routines.\n+#\n+# Define GCRYPT_SHA256 to use the SHA-256 routines in libgcrypt.\n+#\n # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).\n #\n # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).\n@@ -1647,8 +1651,13 @@ endif\n endif\n endif\n \n-LIB_OBJS += sha256/block/sha256.o\n-BASIC_CFLAGS += -DSHA256_BLK\n+ifdef GCRYPT_SHA256\n+\tBASIC_CFLAGS += -DSHA256_GCRYPT\n+\tEXTLIBS += -lgcrypt\n+else\n+\tLIB_OBJS += sha256/block/sha256.o\n+\tBASIC_CFLAGS += -DSHA256_BLK\n+endif\n \n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\ndiff --git a/hash.h b/hash.h\nindex a9bc624020..2ef098052d 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -15,7 +15,11 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n+#if defined(SHA256_GCRYPT)\n+#include \"sha256/gcrypt.h\"\n+#else\n #include \"sha256/block/sha256.h\"\n+#endif\n \n #ifndef platform_SHA_CTX\n /*\ndiff --git a/sha256/gcrypt.h b/sha256/gcrypt.h\nnew file mode 100644\nindex 0000000000..09bd8bb200\n--- /dev/null\n+++ b/sha256/gcrypt.h\n@@ -0,0 +1,30 @@\n+#ifndef SHA256_GCRYPT_H\n+#define SHA256_GCRYPT_H\n+\n+#include \n+\n+#define SHA256_DIGEST_SIZE 32\n+\n+typedef gcry_md_hd_t gcrypt_SHA256_CTX;\n+\n+inline void gcrypt_SHA256_Init(gcrypt_SHA256_CTX *ctx)\n+{\n+\tgcry_md_open(ctx, GCRY_MD_SHA256, 0);\n+}\n+\n+inline void gcrypt_SHA256_Update(gcrypt_SHA256_CTX *ctx, const void *data, size_t len)\n+{\n+\tgcry_md_write(*ctx, data, len);\n+}\n+\n+inline void gcrypt_SHA256_Final(unsigned char *digest, gcrypt_SHA256_CTX *ctx)\n+{\n+\tmemcpy(digest, gcry_md_read(*ctx, GCRY_MD_SHA256), SHA256_DIGEST_SIZE);\n+}\n+\n+#define platform_SHA256_CTX gcrypt_SHA256_CTX\n+#define platform_SHA256_Init gcrypt_SHA256_Init\n+#define platform_SHA256_Update gcrypt_SHA256_Update\n+#define platform_SHA256_Final gcrypt_SHA256_Final\n+\n+#endif\n"},{"id":"363305","messageId":"20181114040938.517289-13-sandals@crustytoothpaste.net","threadId":"49668","inReplyTo":"20181114040938.517289-1-sandals@crustytoothpaste.net","subject":"[PATCH v6 12/12] hash: add an SHA-256 implementation using OpenSSL","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-14T04:09:38Z","receivedAt":"2018-11-14T04:10:32Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We already have OpenSSL routines available for SHA-1, so add routines\nfor SHA-256 as well.\n\nOn a Core i7-6600U, this SHA-256 implementation compares favorably to\nthe SHA1DC SHA-1 implementation:\n\nSHA-1: 157 MiB/s (64 byte chunks); 337 MiB/s (16 KiB chunks)\nSHA-256: 165 MiB/s (64 byte chunks); 408 MiB/s (16 KiB chunks)\n\nSigned-off-by: brian m. carlson \n---\n Makefile | 7 +++++++\n hash.h | 2 ++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 6c99844aa8..6844deb92d 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -183,6 +183,8 @@ all::\n #\n # Define GCRYPT_SHA256 to use the SHA-256 routines in libgcrypt.\n #\n+# Define OPENSSL_SHA256 to use the SHA-256 routines in OpenSSL.\n+#\n # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).\n #\n # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).\n@@ -1651,6 +1653,10 @@ endif\n endif\n endif\n \n+ifdef OPENSSL_SHA256\n+\tEXTLIBS += $(LIB_4_CRYPTO)\n+\tBASIC_CFLAGS += -DSHA256_OPENSSL\n+else\n ifdef GCRYPT_SHA256\n \tBASIC_CFLAGS += -DSHA256_GCRYPT\n \tEXTLIBS += -lgcrypt\n@@ -1658,6 +1664,7 @@ else\n \tLIB_OBJS += sha256/block/sha256.o\n \tBASIC_CFLAGS += -DSHA256_BLK\n endif\n+endif\n \n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\ndiff --git a/hash.h b/hash.h\nindex 2ef098052d..adde708cf2 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -17,6 +17,8 @@\n \n #if defined(SHA256_GCRYPT)\n #include \"sha256/gcrypt.h\"\n+#elif defined(SHA256_OPENSSL)\n+#include \n #else\n #include \"sha256/block/sha256.h\"\n #endif\n"}]}