{"thread":{"id":"49251","subject":"[RFC PATCH 00/12] Base SHA-256 algorithm implementation","startedAt":"2018-08-29T00:59:09Z","lastAt":"2018-09-03T19:11:35Z","messageCount":24,"participants":["brian m. carlson","Ævar Arnfjörð Bjarmason","Derrick Stolee"],"isPatch":true,"patchVersion":1,"patchTotal":12},"messages":[{"id":"356795","messageId":"20180829005857.980820-1-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":null,"subject":"[RFC PATCH 00/12] Base SHA-256 algorithm implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:45Z","receivedAt":"2018-08-29T00:59:09Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This RFC series provides an actual SHA-256 implementation and wires it\nup, along with a few housekeeping patches to make it usable for testing.\n\nAs discussed in some threads, this changes the algorithm name from\n\"sha-1\" to \"sha1\" (and also adds \"sha256\") because it's far easier to\ntype.\n\nI introduced some basic tests for the hash algorithms in use.  Since I\ndid not import the SHA-256 implementation verbatim, I felt it was\nnecessary to ensure that the hash algorithm implementations continued to\nfunction as expected.  My main changes were to adjust the code to use\nour endianness functions, to adopt something closer to our style, and to\nmake use of memcpy and friends for performance reasons.\n\nI opted to place all the implementation code for SHA-256 into one\ndirectory, as opposed to the various directories we have for the SHA-1\nimplementations, mostly for tidiness and ease of use.\n\nI wired up OpenSSL because we already have it and libgcrypt because it\nperforms better than SHA-1.  I didn't provide an implementation for\nSHA-1 with libgcrypt because everybody should be using SHA1DC for\nsecurity.\n\nI didn't provide an implementation based on libnettle because its x86-64\nassembly implementation isn't vectorized and is pretty slow.  Since this\nwas written before I had access to a Mac, Apple Common Crypto hasn't\nbeen wired up, either.  Patches welcome.\n\nIf libgit2 would like to import this SHA-256 implementation, they're\nwelcome to do so under their normal license terms.  If not, that's fine,\ntoo.\n\nbrian m. carlson (12):\n  sha1-file: rename algorithm to \"sha1\"\n  sha1-file: provide functions to look up hash algorithms\n  hex: introduce functions to print arbitrary hashes\n  t: add basic tests for our SHA-1 implementation\n  t: make the sha1 test-tool helper generic\n  sha1-file: add a constant for hash block size\n  t/helper: add a test helper to compute hash speed\n  commit-graph: convert to using the_hash_algo\n  Add a base implementation of SHA-256 support\n  sha256: add an SHA-256 implementation using libgcrypt\n  hash: add an SHA-256 implementation using OpenSSL\n  commit-graph: specify OID version for SHA-256\n\n Makefile                              |  22 ++++\n cache.h                               |  28 ++--\n commit-graph.c                        |  38 +++---\n hash.h                                |  41 +++++-\n hex.c                                 |  32 +++--\n sha1-file.c                           |  70 +++++++++-\n sha256/block/sha256.c                 | 180 ++++++++++++++++++++++++++\n sha256/block/sha256.h                 |  26 ++++\n sha256/gcrypt.h                       |  30 +++++\n t/helper/test-hash-speed.c            |  61 +++++++++\n t/helper/{test-sha1.c => test-hash.c} |  19 +--\n t/helper/test-sha1.c                  |  52 +-------\n t/helper/test-sha256.c                |   7 +\n t/helper/test-tool.c                  |   2 +\n t/helper/test-tool.h                  |   4 +\n t/t0014-hash.sh                       |  54 ++++++++\n 16 files changed, 573 insertions(+), 93 deletions(-)\n create mode 100644 sha256/block/sha256.c\n create mode 100644 sha256/block/sha256.h\n create mode 100644 sha256/gcrypt.h\n create mode 100644 t/helper/test-hash-speed.c\n copy t/helper/{test-sha1.c => test-hash.c} (66%)\n create mode 100644 t/helper/test-sha256.c\n create mode 100755 t/t0014-hash.sh\n\n"},{"id":"356796","messageId":"20180829005857.980820-2-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 01/12] sha1-file: rename algorithm to \"sha1\"","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:46Z","receivedAt":"2018-08-29T00:59:10Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"The transition plan anticipates us using a syntax such as \"^{sha1}\" for\ndisambiguation.  Since this is a syntax some people will be typing a\nlot, it makes sense to provide a short, easy-to-type syntax.  Omitting\nthe dash doesn't create any ambiguity, but it does make it shorter and\neasier to type, especially for touch typists.  In addition, the\ntransition plan already uses \"sha1\" in this context.\n\nRename the name of SHA-1 implementation to \"sha1\".\n\nNote that this change creates no backwards compatibility concerns, since\nwe haven't yet used this field in any serialized data formats.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n sha1-file.c | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 97b7423848..5223e3d1ce 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -97,7 +97,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\tNULL,\n \t},\n \t{\n-\t\t\"sha-1\",\n+\t\t\"sha1\",\n \t\t/* \"sha1\", big-endian */\n \t\t0x73686131,\n \t\tGIT_SHA1_RAWSZ,\n"},{"id":"356797","messageId":"20180829005857.980820-4-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 03/12] hex: introduce functions to print arbitrary hashes","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:48Z","receivedAt":"2018-08-29T00:59:10Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Currently, we have functions that turn an arbitrary SHA-1 value or an\nobject ID into hex format, either using a static buffer or with a\nuser-provided buffer.  Add variants of these functions that can handle\nan arbitrary hash algorithm, specified by constant.  Update the\ndocumentation as well.\n\nWhile we're at it, remove the \"extern\" declaration from this family of\nfunctions, since it's not needed and our style now recommends against\nit.\n\nWe use the variant taking the algorithm structure pointer as the\ninternal variant, since in the future we'll want to replace sha1_to_hex\nwith a hash_to_hex that handles the_hash_algo, and taking an algorithm\npointer is the easiest way to handle all of the variants in use.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n cache.h | 14 ++++++++------\n hex.c   | 32 ++++++++++++++++++++++++--------\n 2 files changed, 32 insertions(+), 14 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 4d014541ab..3cb953445c 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1351,9 +1351,9 @@ extern int get_oid_hex(const char *hex, struct object_id *sha1);\n extern int hex_to_bytes(unsigned char *binary, const char *hex, size_t len);\n \n /*\n- * Convert a binary sha1 to its hex equivalent. The `_r` variant is reentrant,\n+ * Convert a binary hash to its hex equivalent. The `_r` variant is reentrant,\n  * and writes the NUL-terminated output to the buffer `out`, which must be at\n- * least `GIT_SHA1_HEXSZ + 1` bytes, and returns a pointer to out for\n+ * least `GIT_MAX_HEXSZ + 1` bytes, and returns a pointer to out for\n  * convenience.\n  *\n  * The non-`_r` variant returns a static buffer, but uses a ring of 4\n@@ -1361,10 +1361,12 @@ extern int hex_to_bytes(unsigned char *binary, const char *hex, size_t len);\n  *\n  *   printf(\"%s -> %s\", sha1_to_hex(one), sha1_to_hex(two));\n  */\n-extern char *sha1_to_hex_r(char *out, const unsigned char *sha1);\n-extern char *oid_to_hex_r(char *out, const struct object_id *oid);\n-extern char *sha1_to_hex(const unsigned char *sha1);\t/* static buffer result! */\n-extern char *oid_to_hex(const struct object_id *oid);\t/* same static buffer as sha1_to_hex */\n+char *hash_to_hex_algo_r(char *buffer, const unsigned char *hash, int algo);\n+char *sha1_to_hex_r(char *out, const unsigned char *sha1);\n+char *oid_to_hex_r(char *out, const struct object_id *oid);\n+char *hash_to_hex_algo(const unsigned char *hash, int algo);\t/* static buffer result! */\n+char *sha1_to_hex(const unsigned char *sha1);\t\t\t/* same static buffer */\n+char *oid_to_hex(const struct object_id *oid);\t\t\t/* same static buffer */\n \n /*\n  * Parse a 40-character hexadecimal object ID starting from hex, updating the\ndiff --git a/hex.c b/hex.c\nindex 10af1a29e8..870032a868 100644\n--- a/hex.c\n+++ b/hex.c\n@@ -73,14 +73,15 @@ int parse_oid_hex(const char *hex, struct object_id *oid, const char **end)\n \treturn ret;\n }\n \n-char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n+static inline char *hash_to_hex_algop_r(char *buffer, const unsigned char *hash,\n+\t\t\t\t\tconst struct git_hash_algo *algop)\n {\n \tstatic const char hex[] = \"0123456789abcdef\";\n \tchar *buf = buffer;\n \tint i;\n \n-\tfor (i = 0; i < the_hash_algo->rawsz; i++) {\n-\t\tunsigned int val = *sha1++;\n+\tfor (i = 0; i < algop->rawsz; i++) {\n+\t\tunsigned int val = *hash++;\n \t\t*buf++ = hex[val >> 4];\n \t\t*buf++ = hex[val & 0xf];\n \t}\n@@ -89,20 +90,35 @@ char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n \treturn buffer;\n }\n \n-char *oid_to_hex_r(char *buffer, const struct object_id *oid)\n+char *hash_to_hex_algo_r(char *buffer, const unsigned char *hash, int algo)\n {\n-\treturn sha1_to_hex_r(buffer, oid->hash);\n+\treturn hash_to_hex_algop_r(buffer, hash, &hash_algos[algo]);\n }\n \n-char *sha1_to_hex(const unsigned char *sha1)\n+char *sha1_to_hex_r(char *buffer, const unsigned char *sha1)\n+{\n+\treturn hash_to_hex_algo_r(buffer, sha1, GIT_HASH_SHA1);\n+}\n+\n+char *oid_to_hex_r(char *buffer, const struct object_id *oid)\n+{\n+\treturn hash_to_hex_algop_r(buffer, oid->hash, the_hash_algo);\n+}\n+\n+char *hash_to_hex_algo(const unsigned char *hash, int algo)\n {\n \tstatic int bufno;\n \tstatic char hexbuffer[4][GIT_MAX_HEXSZ + 1];\n \tbufno = (bufno + 1) % ARRAY_SIZE(hexbuffer);\n-\treturn sha1_to_hex_r(hexbuffer[bufno], sha1);\n+\treturn hash_to_hex_algo_r(hexbuffer[bufno], hash, algo);\n+}\n+\n+char *sha1_to_hex(const unsigned char *sha1)\n+{\n+\treturn hash_to_hex_algo(sha1, GIT_HASH_SHA1);\n }\n \n char *oid_to_hex(const struct object_id *oid)\n {\n-\treturn sha1_to_hex(oid->hash);\n+\treturn hash_to_hex_algo(oid->hash, hash_algo_by_ptr(the_hash_algo));\n }\n"},{"id":"356798","messageId":"20180829005857.980820-5-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 04/12] t: add basic tests for our SHA-1 implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:49Z","receivedAt":"2018-08-29T00:59:11Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We have in the past had some unfortunate endianness issues with some\nSHA-1 implementations we ship, especially on big-endian machines.  Add\nan explicit test using the test helper to catch these issues and point\nthem out prominently.  This test can also be used as a staging ground\nfor people testing additional algorithms to verify that their\nimplementations are working as expected.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n t/t0014-hash.sh | 29 +++++++++++++++++++++++++++++\n 1 file changed, 29 insertions(+)\n create mode 100755 t/t0014-hash.sh\n\ndiff --git a/t/t0014-hash.sh b/t/t0014-hash.sh\nnew file mode 100755\nindex 0000000000..8e763c2c3d\n--- /dev/null\n+++ b/t/t0014-hash.sh\n@@ -0,0 +1,29 @@\n+#!/bin/sh\n+\n+test_description='test basic hash implementation'\n+. ./test-lib.sh\n+\n+\n+test_expect_success 'test basic SHA-1 hash values' '\n+\ttest-tool sha1 </dev/null >actual &&\n+\tgrep da39a3ee5e6b4b0d3255bfef95601890afd80709 actual &&\n+\tprintf \"a\" | test-tool sha1 >actual &&\n+\tgrep 86f7e437faa5a7fce15d1ddcb9eaeaea377667b8 actual &&\n+\tprintf \"abc\" | test-tool sha1 >actual &&\n+\tgrep a9993e364706816aba3e25717850c26c9cd0d89d actual &&\n+\tprintf \"message digest\" | test-tool sha1 >actual &&\n+\tgrep c12252ceda8be8994d5fa0290a47231c1d16aae3 actual &&\n+\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha1 >actual &&\n+\tgrep 32d10c7b8cf96570ca04ce37f2a19d84240d3a89 actual &&\n+\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n+\t\ttest-tool sha1 >actual &&\n+\tgrep 34aa973cd4c4daa4f61eeb2bdbad27316534016f actual &&\n+\tprintf \"blob 0\\0\" | test-tool sha1 >actual &&\n+\tgrep e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 actual &&\n+\tprintf \"blob 3\\0abc\" | test-tool sha1 >actual &&\n+\tgrep f2ba8f84ab5c1bce84a7b441cb1959cfc7093b7f actual &&\n+\tprintf \"tree 0\\0\" | test-tool sha1 >actual &&\n+\tgrep 4b825dc642cb6eb9a060e54bf8d69288fbee4904 actual\n+'\n+\n+test_done\n"},{"id":"356799","messageId":"20180829005857.980820-6-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 05/12] t: make the sha1 test-tool helper generic","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:50Z","receivedAt":"2018-08-29T00:59:13Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Since we're going to have multiple hash algorithms to test, it makes\nsense to share as much of the test code as possible.  Convert the sha1\nhelper for the test-tool to be generic and move it out into its own\nmodule.  This will allow us to share most of this code with our NewHash\nimplementation.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n Makefile                              |  1 +\n t/helper/{test-sha1.c => test-hash.c} | 19 +++++-----\n t/helper/test-sha1.c                  | 52 +--------------------------\n t/helper/test-tool.h                  |  2 ++\n 4 files changed, 14 insertions(+), 60 deletions(-)\n copy t/helper/{test-sha1.c => test-hash.c} (66%)\n\ndiff --git a/Makefile b/Makefile\nindex 5a969f5830..eb1a080cc9 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -712,6 +712,7 @@ TEST_BUILTINS_OBJS += test-dump-cache-tree.o\n TEST_BUILTINS_OBJS += test-dump-split-index.o\n TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n+TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-hash.c\nsimilarity index 66%\ncopy from t/helper/test-sha1.c\ncopy to t/helper/test-hash.c\nindex 1ba0675c75..9992de2212 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-hash.c\n@@ -1,13 +1,14 @@\n #include \"test-tool.h\"\n #include \"cache.h\"\n \n-int cmd__sha1(int ac, const char **av)\n+int cmd_hash_impl(int ac, const char **av, int algo)\n {\n-\tgit_SHA_CTX ctx;\n-\tunsigned char sha1[20];\n+\tgit_hash_ctx ctx;\n+\tunsigned char hash[GIT_MAX_HEXSZ];\n \tunsigned bufsz = 8192;\n \tint binary = 0;\n \tchar *buffer;\n+\tconst struct git_hash_algo *algop = &hash_algos[algo];\n \n \tif (ac == 2) {\n \t\tif (!strcmp(av[1], \"-b\"))\n@@ -26,7 +27,7 @@ int cmd__sha1(int ac, const char **av)\n \t\t\tdie(\"OOPS\");\n \t}\n \n-\tgit_SHA1_Init(&ctx);\n+\talgop->init_fn(&ctx);\n \n \twhile (1) {\n \t\tssize_t sz, this_sz;\n@@ -38,20 +39,20 @@ int cmd__sha1(int ac, const char **av)\n \t\t\tif (sz == 0)\n \t\t\t\tbreak;\n \t\t\tif (sz < 0)\n-\t\t\t\tdie_errno(\"test-sha1\");\n+\t\t\t\tdie_errno(\"test-hash\");\n \t\t\tthis_sz += sz;\n \t\t\tcp += sz;\n \t\t\troom -= sz;\n \t\t}\n \t\tif (this_sz == 0)\n \t\t\tbreak;\n-\t\tgit_SHA1_Update(&ctx, buffer, this_sz);\n+\t\talgop->update_fn(&ctx, buffer, this_sz);\n \t}\n-\tgit_SHA1_Final(sha1, &ctx);\n+\talgop->final_fn(hash, &ctx);\n \n \tif (binary)\n-\t\tfwrite(sha1, 1, 20, stdout);\n+\t\tfwrite(hash, 1, algop->rawsz, stdout);\n \telse\n-\t\tputs(sha1_to_hex(sha1));\n+\t\tputs(hash_to_hex_algo(hash, algo));\n \texit(0);\n }\ndiff --git a/t/helper/test-sha1.c b/t/helper/test-sha1.c\nindex 1ba0675c75..d860c387c3 100644\n--- a/t/helper/test-sha1.c\n+++ b/t/helper/test-sha1.c\n@@ -3,55 +3,5 @@\n \n int cmd__sha1(int ac, const char **av)\n {\n-\tgit_SHA_CTX ctx;\n-\tunsigned char sha1[20];\n-\tunsigned bufsz = 8192;\n-\tint binary = 0;\n-\tchar *buffer;\n-\n-\tif (ac == 2) {\n-\t\tif (!strcmp(av[1], \"-b\"))\n-\t\t\tbinary = 1;\n-\t\telse\n-\t\t\tbufsz = strtoul(av[1], NULL, 10) * 1024 * 1024;\n-\t}\n-\n-\tif (!bufsz)\n-\t\tbufsz = 8192;\n-\n-\twhile ((buffer = malloc(bufsz)) == NULL) {\n-\t\tfprintf(stderr, \"bufsz %u is too big, halving...\\n\", bufsz);\n-\t\tbufsz /= 2;\n-\t\tif (bufsz < 1024)\n-\t\t\tdie(\"OOPS\");\n-\t}\n-\n-\tgit_SHA1_Init(&ctx);\n-\n-\twhile (1) {\n-\t\tssize_t sz, this_sz;\n-\t\tchar *cp = buffer;\n-\t\tunsigned room = bufsz;\n-\t\tthis_sz = 0;\n-\t\twhile (room) {\n-\t\t\tsz = xread(0, cp, room);\n-\t\t\tif (sz == 0)\n-\t\t\t\tbreak;\n-\t\t\tif (sz < 0)\n-\t\t\t\tdie_errno(\"test-sha1\");\n-\t\t\tthis_sz += sz;\n-\t\t\tcp += sz;\n-\t\t\troom -= sz;\n-\t\t}\n-\t\tif (this_sz == 0)\n-\t\t\tbreak;\n-\t\tgit_SHA1_Update(&ctx, buffer, this_sz);\n-\t}\n-\tgit_SHA1_Final(sha1, &ctx);\n-\n-\tif (binary)\n-\t\tfwrite(sha1, 1, 20, stdout);\n-\telse\n-\t\tputs(sha1_to_hex(sha1));\n-\texit(0);\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA1);\n }\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex e954e8c522..9026a8f608 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -41,4 +41,6 @@ int cmd__urlmatch_normalization(int argc, const char **argv);\n int cmd__wildmatch(int argc, const char **argv);\n int cmd__write_cache(int argc, const char **argv);\n \n+int cmd_hash_impl(int ac, const char **av, int algo);\n+\n #endif\n"},{"id":"356800","messageId":"20180829005857.980820-8-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 07/12] t/helper: add a test helper to compute hash speed","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:52Z","receivedAt":"2018-08-29T00:59:13Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Add a utility (which is less for the testsuite and more for developers)\nthat can compute hash speeds for whatever hash algorithms are\nimplemented.  This allows developers to test their personal systems to\ndetermine the performance characteristics of various algorithms.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n Makefile                   |  1 +\n t/helper/test-hash-speed.c | 61 ++++++++++++++++++++++++++++++++++++++\n t/helper/test-tool.c       |  1 +\n t/helper/test-tool.h       |  1 +\n 4 files changed, 64 insertions(+)\n create mode 100644 t/helper/test-hash-speed.c\n\ndiff --git a/Makefile b/Makefile\nindex eb1a080cc9..e047dea988 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -714,6 +714,7 @@ TEST_BUILTINS_OBJS += test-example-decorate.o\n TEST_BUILTINS_OBJS += test-genrandom.o\n TEST_BUILTINS_OBJS += test-hash.o\n TEST_BUILTINS_OBJS += test-hashmap.o\n+TEST_BUILTINS_OBJS += test-hash-speed.o\n TEST_BUILTINS_OBJS += test-index-version.o\n TEST_BUILTINS_OBJS += test-json-writer.o\n TEST_BUILTINS_OBJS += test-lazy-init-name-hash.o\ndiff --git a/t/helper/test-hash-speed.c b/t/helper/test-hash-speed.c\nnew file mode 100644\nindex 0000000000..432233c7f0\n--- /dev/null\n+++ b/t/helper/test-hash-speed.c\n@@ -0,0 +1,61 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+#define NUM_SECONDS 3\n+\n+static inline void compute_hash(const struct git_hash_algo *algo, git_hash_ctx *ctx, uint8_t *final, const void *p, size_t len)\n+{\n+\talgo->init_fn(ctx);\n+\talgo->update_fn(ctx, p, len);\n+\talgo->final_fn(final, ctx);\n+}\n+\n+int cmd__hash_speed(int ac, const char **av)\n+{\n+\tgit_hash_ctx ctx;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n+\tclock_t initial, start, end;\n+\tunsigned bufsizes[] = { 64, 256, 1024, 8192, 16384 };\n+\tint i;\n+\tvoid *p;\n+\tconst struct git_hash_algo *algo = NULL;\n+\n+\tif (ac == 2) {\n+\t\tfor (i = 1; i < GIT_HASH_NALGOS; i++) {\n+\t\t\tif (!strcmp(av[1], hash_algos[i].name)) {\n+\t\t\t\talgo = &hash_algos[i];\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\t}\n+\tif (!algo)\n+\t\tdie(\"usage: test-tool hash-speed algo_name\");\n+\n+\t/* Use this as an offset to make overflow less likely. */\n+\tinitial = clock();\n+\n+\tprintf(\"algo: %s\\n\", algo->name);\n+\n+\tfor (i = 0; i < ARRAY_SIZE(bufsizes); i++) {\n+\t\tunsigned long j, kb;\n+\t\tdouble kb_per_sec;\n+\t\tp = xcalloc(1, bufsizes[i]);\n+\t\tstart = end = clock() - initial;\n+\t\tfor (j = 0; ((end - start) / CLOCKS_PER_SEC) < NUM_SECONDS; j++) {\n+\t\t\tcompute_hash(algo, &ctx, hash, p, bufsizes[i]);\n+\n+\t\t\t/*\n+\t\t\t * Only check elapsed time every 128 iterations to avoid\n+\t\t\t * dominating the runtime with system calls.\n+\t\t\t */\n+\t\t\tif (!(j & 127))\n+\t\t\t\tend = clock() - initial;\n+\t\t}\n+\t\tkb = j * bufsizes[i];\n+\t\tkb_per_sec = kb / (1024 * ((double)end - start) / CLOCKS_PER_SEC);\n+\t\tprintf(\"size %u: %lu iters; %lu KiB; %0.2f KiB/s\\n\", bufsizes[i], j, kb, kb_per_sec);\n+\t\tfree(p);\n+\t}\n+\n+\texit(0);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex 0edafcfd65..c0d3eecc36 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -18,6 +18,7 @@ static struct test_cmd cmds[] = {\n \t{ \"example-decorate\", cmd__example_decorate },\n \t{ \"genrandom\", cmd__genrandom },\n \t{ \"hashmap\", cmd__hashmap },\n+\t{ \"hash-speed\", cmd__hash_speed },\n \t{ \"index-version\", cmd__index_version },\n \t{ \"json-writer\", cmd__json_writer },\n \t{ \"lazy-init-name-hash\", cmd__lazy_init_name_hash },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex 9026a8f608..abe3a253d5 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -14,6 +14,7 @@ int cmd__dump_split_index(int argc, const char **argv);\n int cmd__example_decorate(int argc, const char **argv);\n int cmd__genrandom(int argc, const char **argv);\n int cmd__hashmap(int argc, const char **argv);\n+int cmd__hash_speed(int argc, const char **argv);\n int cmd__index_version(int argc, const char **argv);\n int cmd__json_writer(int argc, const char **argv);\n int cmd__lazy_init_name_hash(int argc, const char **argv);\n"},{"id":"356801","messageId":"20180829005857.980820-7-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 06/12] sha1-file: add a constant for hash block size","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:51Z","receivedAt":"2018-08-29T00:59:14Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"There is one place we need the hash algorithm block size: the HMAC code\nfor push certs.  Expose this constant in struct git_hash_algo and expose\nvalues for SHA-1 and for the largest value of any hash.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n cache.h     | 4 ++++\n hash.h      | 3 +++\n sha1-file.c | 2 ++\n 3 files changed, 9 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 3cb953445c..c1b5a7a337 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -45,10 +45,14 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n /* The length in bytes and in hex digits of an object name (SHA-1 value). */\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n+/* The block size of SHA-1. */\n+#define GIT_SHA1_BLKSZ 64\n \n /* The length in byte and in hex digits of the largest possible hash value. */\n #define GIT_MAX_RAWSZ GIT_SHA1_RAWSZ\n #define GIT_MAX_HEXSZ GIT_SHA1_HEXSZ\n+/* The largest possible block size for any supported hash. */\n+#define GIT_MAX_BLKSZ GIT_SHA1_BLKSZ\n \n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/hash.h b/hash.h\nindex 90f4344619..46dff69eb3 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -81,6 +81,9 @@ struct git_hash_algo {\n \t/* The length of the hash in hex characters. */\n \tsize_t hexsz;\n \n+\t/* The block size of the hash. */\n+\tsize_t blksz;\n+\n \t/* The hash initialization function. */\n \tgit_hash_init_fn init_fn;\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex f6976b179f..a1ad1b8268 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -90,6 +90,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t0x00000000,\n \t\t0,\n \t\t0,\n+\t\t0,\n \t\tgit_hash_unknown_init,\n \t\tgit_hash_unknown_update,\n \t\tgit_hash_unknown_final,\n@@ -102,6 +103,7 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t0x73686131,\n \t\tGIT_SHA1_RAWSZ,\n \t\tGIT_SHA1_HEXSZ,\n+\t\tGIT_SHA1_BLKSZ,\n \t\tgit_hash_sha1_init,\n \t\tgit_hash_sha1_update,\n \t\tgit_hash_sha1_final,\n"},{"id":"356802","messageId":"20180829005857.980820-9-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 08/12] commit-graph: convert to using the_hash_algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:53Z","receivedAt":"2018-08-29T00:59:15Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Instead of using hard-coded constants for object sizes, use\nthe_hash_algo to look them up.  In addition, use a function call to look\nup the object ID version and produce the correct value.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n commit-graph.c | 31 ++++++++++++++++---------------\n 1 file changed, 16 insertions(+), 15 deletions(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 8a1bec7b8a..29356d84a2 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -25,11 +25,6 @@\n #define GRAPH_VERSION_1 0x1\n #define GRAPH_VERSION GRAPH_VERSION_1\n \n-#define GRAPH_OID_VERSION_SHA1 1\n-#define GRAPH_OID_LEN_SHA1 GIT_SHA1_RAWSZ\n-#define GRAPH_OID_VERSION GRAPH_OID_VERSION_SHA1\n-#define GRAPH_OID_LEN GRAPH_OID_LEN_SHA1\n-\n #define GRAPH_OCTOPUS_EDGES_NEEDED 0x80000000\n #define GRAPH_PARENT_MISSING 0x7fffffff\n #define GRAPH_EDGE_LAST_MASK 0x7fffffff\n@@ -41,13 +36,18 @@\n #define GRAPH_FANOUT_SIZE (4 * 256)\n #define GRAPH_CHUNKLOOKUP_WIDTH 12\n #define GRAPH_MIN_SIZE (GRAPH_HEADER_SIZE + 4 * GRAPH_CHUNKLOOKUP_WIDTH \\\n-\t\t\t+ GRAPH_FANOUT_SIZE + GRAPH_OID_LEN)\n+\t\t\t+ GRAPH_FANOUT_SIZE + the_hash_algo->rawsz)\n \n char *get_commit_graph_filename(const char *obj_dir)\n {\n \treturn xstrfmt(\"%s/info/commit-graph\", obj_dir);\n }\n \n+static uint8_t oid_version(void)\n+{\n+\treturn 1;\n+}\n+\n static struct commit_graph *alloc_commit_graph(void)\n {\n \tstruct commit_graph *g = xcalloc(1, sizeof(*g));\n@@ -100,15 +100,15 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \t}\n \n \thash_version = *(unsigned char*)(data + 5);\n-\tif (hash_version != GRAPH_OID_VERSION) {\n+\tif (hash_version != oid_version()) {\n \t\terror(_(\"hash version %X does not match version %X\"),\n-\t\t      hash_version, GRAPH_OID_VERSION);\n+\t\t      hash_version, oid_version());\n \t\tgoto cleanup_fail;\n \t}\n \n \tgraph = alloc_commit_graph();\n \n-\tgraph->hash_len = GRAPH_OID_LEN;\n+\tgraph->hash_len = the_hash_algo->rawsz;\n \tgraph->num_chunks = *(unsigned char*)(data + 6);\n \tgraph->graph_fd = fd;\n \tgraph->data = graph_map;\n@@ -124,7 +124,7 @@ struct commit_graph *load_commit_graph_one(const char *graph_file)\n \n \t\tchunk_lookup += GRAPH_CHUNKLOOKUP_WIDTH;\n \n-\t\tif (chunk_offset > graph_size - GIT_MAX_RAWSZ) {\n+\t\tif (chunk_offset > graph_size - the_hash_algo->rawsz) {\n \t\t\terror(_(\"improper chunk offset %08x%08x\"), (uint32_t)(chunk_offset >> 32),\n \t\t\t      (uint32_t)chunk_offset);\n \t\t\tgoto cleanup_fail;\n@@ -692,6 +692,7 @@ void write_commit_graph(const char *obj_dir,\n \tint num_chunks;\n \tint num_extra_edges;\n \tstruct commit_list *parent;\n+\tconst unsigned hashsz = the_hash_algo->rawsz;\n \n \toids.nr = 0;\n \toids.alloc = approximate_object_count() / 4;\n@@ -812,7 +813,7 @@ void write_commit_graph(const char *obj_dir,\n \thashwrite_be32(f, GRAPH_SIGNATURE);\n \n \thashwrite_u8(f, GRAPH_VERSION);\n-\thashwrite_u8(f, GRAPH_OID_VERSION);\n+\thashwrite_u8(f, oid_version());\n \thashwrite_u8(f, num_chunks);\n \thashwrite_u8(f, 0); /* unused padding byte */\n \n@@ -827,8 +828,8 @@ void write_commit_graph(const char *obj_dir,\n \n \tchunk_offsets[0] = 8 + (num_chunks + 1) * GRAPH_CHUNKLOOKUP_WIDTH;\n \tchunk_offsets[1] = chunk_offsets[0] + GRAPH_FANOUT_SIZE;\n-\tchunk_offsets[2] = chunk_offsets[1] + GRAPH_OID_LEN * commits.nr;\n-\tchunk_offsets[3] = chunk_offsets[2] + (GRAPH_OID_LEN + 16) * commits.nr;\n+\tchunk_offsets[2] = chunk_offsets[1] + hashsz * commits.nr;\n+\tchunk_offsets[3] = chunk_offsets[2] + (hashsz + 16) * commits.nr;\n \tchunk_offsets[4] = chunk_offsets[3] + 4 * num_extra_edges;\n \n \tfor (i = 0; i <= num_chunks; i++) {\n@@ -841,8 +842,8 @@ void write_commit_graph(const char *obj_dir,\n \t}\n \n \twrite_graph_chunk_fanout(f, commits.list, commits.nr);\n-\twrite_graph_chunk_oids(f, GRAPH_OID_LEN, commits.list, commits.nr);\n-\twrite_graph_chunk_data(f, GRAPH_OID_LEN, commits.list, commits.nr);\n+\twrite_graph_chunk_oids(f, hashsz, commits.list, commits.nr);\n+\twrite_graph_chunk_data(f, hashsz, commits.list, commits.nr);\n \twrite_graph_chunk_large_edges(f, commits.list, commits.nr);\n \n \tclose_commit_graph();\n"},{"id":"356803","messageId":"20180829005857.980820-10-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 09/12] Add a base implementation of SHA-256 support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:54Z","receivedAt":"2018-08-29T00:59:17Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"SHA-1 is weak and we need to transition to a new hash function.  For\nsome time, we have referred to this new function as NewHash.\n\nThe selection criteria for NewHash specify that it should (a) be 256\nbits in length, (b) have high quality implementations available, (c)\nshould match Git's needs in terms of security, and (d) ideally, be fast\nto compute.\n\nSHA-256 has a variety of high quality implementations across various\nlibraries.  It is implemented by every cryptographic library we support\nand is available on every platform and in almost every programming\nlanguage.  It is often highly optimized, since it is commonly used in\nTLS and elsewhere.  Additionally, there are various command line\nutilities that implement it, which is useful for educational and testing\npurposes.\n\nSHA-256 is presently considered secure and has received a reasonable\namount of cryptanalysis in the literature.  It is, admittedly, not\nresistant to length extension attacks, but Git object storage is immune\nto those due to the length field at the beginning.\n\nSHA-256 is somewhat slower to compute than SHA-1 in software.  However,\nsince our default SHA-1 implementation is collision-detecting, a\nreasonable cryptographic library implementation of SHA-256 will actually\nbe faster than SHA-256.  In addition, modern ARM and AMD processors (and\nsome Intel processors) contain instructions for implementing SHA-256 in\nhardware, making it the fastest possible option.\n\nThere are other reasons to select SHA-256.  With signed commits and\ntags, it's possible to use SHA-256 for signatures and therefore have to\nrely on only one hash algorithm for security.\n\nAdd a basic implementation of SHA-256 based off libtomcrypt, which is in\nthe public domain.  Optimize it and tidy it somewhat.\n\nWire up SHA-256 in the list of hash algorithms, and add a test that the\nalgorithm works correctly.\n\nNote that with this patch, it is still not possible to switch to using\nSHA-256 in Git.  Additional patches are needed to prepare the code to\nhandle a larger hash algorithm and further test fixes are needed.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n Makefile               |   4 +\n cache.h                |  12 ++-\n hash.h                 |  19 ++++-\n sha1-file.c            |  45 +++++++++++\n sha256/block/sha256.c  | 180 +++++++++++++++++++++++++++++++++++++++++\n sha256/block/sha256.h  |  26 ++++++\n t/helper/test-sha256.c |   7 ++\n t/helper/test-tool.c   |   1 +\n t/helper/test-tool.h   |   1 +\n t/t0014-hash.sh        |  25 ++++++\n 10 files changed, 316 insertions(+), 4 deletions(-)\n create mode 100644 sha256/block/sha256.c\n create mode 100644 sha256/block/sha256.h\n create mode 100644 t/helper/test-sha256.c\n\ndiff --git a/Makefile b/Makefile\nindex e047dea988..624d852e79 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -733,6 +733,7 @@ TEST_BUILTINS_OBJS += test-run-command.o\n TEST_BUILTINS_OBJS += test-scrap-cache-tree.o\n TEST_BUILTINS_OBJS += test-sha1-array.o\n TEST_BUILTINS_OBJS += test-sha1.o\n+TEST_BUILTINS_OBJS += test-sha256.o\n TEST_BUILTINS_OBJS += test-sigchain.o\n TEST_BUILTINS_OBJS += test-strcmp-offset.o\n TEST_BUILTINS_OBJS += test-string-list.o\n@@ -1623,6 +1624,9 @@ endif\n endif\n endif\n \n+LIB_OBJS += sha256/block/sha256.o\n+BASIC_CFLAGS += -DSHA256_BLK\n+\n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\n \tBASIC_CFLAGS += -DSHA1_MAX_BLOCK_SIZE=\"$(SHA1_MAX_BLOCK_SIZE)\"\ndiff --git a/cache.h b/cache.h\nindex c1b5a7a337..5a35497b34 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -48,11 +48,17 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n /* The block size of SHA-1. */\n #define GIT_SHA1_BLKSZ 64\n \n+/* The length in bytes and in hex digits of an object name (SHA-256 value). */\n+#define GIT_SHA256_RAWSZ 32\n+#define GIT_SHA256_HEXSZ (2 * GIT_SHA256_RAWSZ)\n+/* The block size of SHA-256. */\n+#define GIT_SHA256_BLKSZ 64\n+\n /* The length in byte and in hex digits of the largest possible hash value. */\n-#define GIT_MAX_RAWSZ GIT_SHA1_RAWSZ\n-#define GIT_MAX_HEXSZ GIT_SHA1_HEXSZ\n+#define GIT_MAX_RAWSZ GIT_SHA256_RAWSZ\n+#define GIT_MAX_HEXSZ GIT_SHA256_HEXSZ\n /* The largest possible block size for any supported hash. */\n-#define GIT_MAX_BLKSZ GIT_SHA1_BLKSZ\n+#define GIT_MAX_BLKSZ GIT_SHA256_BLKSZ\n \n struct object_id {\n \tunsigned char hash[GIT_MAX_RAWSZ];\ndiff --git a/hash.h b/hash.h\nindex 46dff69eb3..88d18896d7 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -15,6 +15,8 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n+#include \"sha256/block/sha256.h\"\n+\n #ifndef platform_SHA_CTX\n /*\n  * platform's underlying implementation of SHA-1; could be OpenSSL,\n@@ -34,6 +36,18 @@\n #define git_SHA1_Update\t\tplatform_SHA1_Update\n #define git_SHA1_Final\t\tplatform_SHA1_Final\n \n+#ifndef platform_SHA256_CTX\n+#define platform_SHA256_CTX\tSHA256_CTX\n+#define platform_SHA256_Init\tSHA256_Init\n+#define platform_SHA256_Update\tSHA256_Update\n+#define platform_SHA256_Final\tSHA256_Final\n+#endif\n+\n+#define git_SHA256_CTX\t\tplatform_SHA256_CTX\n+#define git_SHA256_Init\t\tplatform_SHA256_Init\n+#define git_SHA256_Update\tplatform_SHA256_Update\n+#define git_SHA256_Final\tplatform_SHA256_Final\n+\n #ifdef SHA1_MAX_BLOCK_SIZE\n #include \"compat/sha1-chunked.h\"\n #undef git_SHA1_Update\n@@ -52,12 +66,15 @@\n #define GIT_HASH_UNKNOWN 0\n /* SHA-1 */\n #define GIT_HASH_SHA1 1\n+/* SHA-256  */\n+#define GIT_HASH_SHA256 2\n /* Number of algorithms supported (including unknown). */\n-#define GIT_HASH_NALGOS (GIT_HASH_SHA1 + 1)\n+#define GIT_HASH_NALGOS (GIT_HASH_SHA256 + 1)\n \n /* A suitably aligned type for stack allocations of hash contexts. */\n union git_hash_ctx {\n \tgit_SHA_CTX sha1;\n+\tgit_SHA256_CTX sha256;\n };\n typedef union git_hash_ctx git_hash_ctx;\n \ndiff --git a/sha1-file.c b/sha1-file.c\nindex a1ad1b8268..495c890c80 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -40,10 +40,20 @@\n #define EMPTY_TREE_SHA1_BIN_LITERAL \\\n \t \"\\x4b\\x82\\x5d\\xc6\\x42\\xcb\\x6e\\xb9\\xa0\\x60\" \\\n \t \"\\xe5\\x4b\\xf8\\xd6\\x92\\x88\\xfb\\xee\\x49\\x04\"\n+#define EMPTY_TREE_SHA256_BIN_LITERAL \\\n+\t\"\\x6e\\xf1\\x9b\\x41\\x22\\x5c\\x53\\x69\\xf1\\xc1\" \\\n+\t\"\\x04\\xd4\\x5d\\x8d\\x85\\xef\\xa9\\xb0\\x57\\xb5\" \\\n+\t\"\\x3b\\x14\\xb4\\xb9\\xb9\\x39\\xdd\\x74\\xde\\xcc\" \\\n+\t\"\\x53\\x21\"\n \n #define EMPTY_BLOB_SHA1_BIN_LITERAL \\\n \t\"\\xe6\\x9d\\xe2\\x9b\\xb2\\xd1\\xd6\\x43\\x4b\\x8b\" \\\n \t\"\\x29\\xae\\x77\\x5a\\xd8\\xc2\\xe4\\x8c\\x53\\x91\"\n+#define EMPTY_BLOB_SHA256_BIN_LITERAL \\\n+\t\"\\x47\\x3a\\x0f\\x4c\\x3b\\xe8\\xa9\\x36\\x81\\xa2\" \\\n+\t\"\\x67\\xe3\\xb1\\xe9\\xa7\\xdc\\xda\\x11\\x85\\x43\" \\\n+\t\"\\x6f\\xe1\\x41\\xf7\\x74\\x91\\x20\\xa3\\x03\\x72\" \\\n+\t\"\\x18\\x13\"\n \n const unsigned char null_sha1[GIT_MAX_RAWSZ];\n const struct object_id null_oid;\n@@ -53,6 +63,12 @@ static const struct object_id empty_tree_oid = {\n static const struct object_id empty_blob_oid = {\n \tEMPTY_BLOB_SHA1_BIN_LITERAL\n };\n+static const struct object_id empty_tree_oid_sha256 = {\n+\tEMPTY_TREE_SHA256_BIN_LITERAL\n+};\n+static const struct object_id empty_blob_oid_sha256 = {\n+\tEMPTY_BLOB_SHA256_BIN_LITERAL\n+};\n \n static void git_hash_sha1_init(git_hash_ctx *ctx)\n {\n@@ -69,6 +85,22 @@ static void git_hash_sha1_final(unsigned char *hash, git_hash_ctx *ctx)\n \tgit_SHA1_Final(hash, &ctx->sha1);\n }\n \n+\n+static void git_hash_sha256_init(git_hash_ctx *ctx)\n+{\n+\tgit_SHA256_Init(&ctx->sha256);\n+}\n+\n+static void git_hash_sha256_update(git_hash_ctx *ctx, const void *data, size_t len)\n+{\n+\tgit_SHA256_Update(&ctx->sha256, data, len);\n+}\n+\n+static void git_hash_sha256_final(unsigned char *hash, git_hash_ctx *ctx)\n+{\n+\tgit_SHA256_Final(hash, &ctx->sha256);\n+}\n+\n static void git_hash_unknown_init(git_hash_ctx *ctx)\n {\n \tBUG(\"trying to init unknown hash\");\n@@ -110,6 +142,19 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t&empty_tree_oid,\n \t\t&empty_blob_oid,\n \t},\n+\t{\n+\t\t\"sha256\",\n+\t\t/* \"s256\", big-endian */\n+\t\t0x73323536,\n+\t\tGIT_SHA256_RAWSZ,\n+\t\tGIT_SHA256_HEXSZ,\n+\t\tGIT_SHA256_BLKSZ,\n+\t\tgit_hash_sha256_init,\n+\t\tgit_hash_sha256_update,\n+\t\tgit_hash_sha256_final,\n+\t\t&empty_tree_oid_sha256,\n+\t\t&empty_blob_oid_sha256,\n+\t}\n };\n \n const char *empty_tree_oid_hex(void)\ndiff --git a/sha256/block/sha256.c b/sha256/block/sha256.c\nnew file mode 100644\nindex 0000000000..18350c161a\n--- /dev/null\n+++ b/sha256/block/sha256.c\n@@ -0,0 +1,180 @@\n+#include \"git-compat-util.h\"\n+#include \"./sha256.h\"\n+\n+#define BLKSIZE blk_SHA256_BLKSIZE\n+\n+void blk_SHA256_Init(blk_SHA256_CTX *ctx)\n+{\n+\tctx->offset = 0;\n+\tctx->length = 0;\n+\tctx->state[0] = 0x6A09E667UL;\n+\tctx->state[1] = 0xBB67AE85UL;\n+\tctx->state[2] = 0x3C6EF372UL;\n+\tctx->state[3] = 0xA54FF53AUL;\n+\tctx->state[4] = 0x510E527FUL;\n+\tctx->state[5] = 0x9B05688CUL;\n+\tctx->state[6] = 0x1F83D9ABUL;\n+\tctx->state[7] = 0x5BE0CD19UL;\n+}\n+\n+static inline uint32_t ror(uint32_t x, unsigned n)\n+{\n+\treturn (x >> n) | (x << (32 - n));\n+}\n+\n+#define Ch(x,y,z)       (z ^ (x & (y ^ z)))\n+#define Maj(x,y,z)      (((x | y) & z) | (x & y))\n+#define S(x, n)         ror((x),(n))\n+#define R(x, n)         ((x)>>(n))\n+#define Sigma0(x)       (S(x, 2) ^ S(x, 13) ^ S(x, 22))\n+#define Sigma1(x)       (S(x, 6) ^ S(x, 11) ^ S(x, 25))\n+#define Gamma0(x)       (S(x, 7) ^ S(x, 18) ^ R(x, 3))\n+#define Gamma1(x)       (S(x, 17) ^ S(x, 19) ^ R(x, 10))\n+\n+static void blk_SHA256_Transform(blk_SHA256_CTX *ctx, const unsigned char *buf)\n+{\n+\n+\tuint32_t S[8], W[64], t0, t1;\n+\tint i;\n+\n+\t/* copy state into S */\n+\tfor (i = 0; i < 8; i++) {\n+\t\tS[i] = ctx->state[i];\n+\t}\n+\n+\t/* copy the state into 512-bits into W[0..15] */\n+\tfor (i = 0; i < 16; i++, buf += sizeof(uint32_t)) {\n+\t\tW[i] = get_be32(buf);\n+\t}\n+\n+\t/* fill W[16..63] */\n+\tfor (i = 16; i < 64; i++) {\n+\t\tW[i] = Gamma1(W[i - 2]) + W[i - 7] + Gamma0(W[i - 15]) + W[i - 16];\n+\t}\n+\n+#define RND(a,b,c,d,e,f,g,h,i,ki)                    \\\n+\tt0 = h + Sigma1(e) + Ch(e, f, g) + ki + W[i];   \\\n+\tt1 = Sigma0(a) + Maj(a, b, c);                  \\\n+\td += t0;                                        \\\n+\th  = t0 + t1;\n+\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],0,0x428a2f98);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],1,0x71374491);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],2,0xb5c0fbcf);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],3,0xe9b5dba5);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],4,0x3956c25b);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],5,0x59f111f1);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],6,0x923f82a4);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],7,0xab1c5ed5);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],8,0xd807aa98);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],9,0x12835b01);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],10,0x243185be);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],11,0x550c7dc3);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],12,0x72be5d74);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],13,0x80deb1fe);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],14,0x9bdc06a7);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],15,0xc19bf174);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],16,0xe49b69c1);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],17,0xefbe4786);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],18,0x0fc19dc6);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],19,0x240ca1cc);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],20,0x2de92c6f);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],21,0x4a7484aa);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],22,0x5cb0a9dc);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],23,0x76f988da);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],24,0x983e5152);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],25,0xa831c66d);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],26,0xb00327c8);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],27,0xbf597fc7);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],28,0xc6e00bf3);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],29,0xd5a79147);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],30,0x06ca6351);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],31,0x14292967);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],32,0x27b70a85);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],33,0x2e1b2138);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],34,0x4d2c6dfc);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],35,0x53380d13);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],36,0x650a7354);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],37,0x766a0abb);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],38,0x81c2c92e);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],39,0x92722c85);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],40,0xa2bfe8a1);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],41,0xa81a664b);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],42,0xc24b8b70);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],43,0xc76c51a3);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],44,0xd192e819);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],45,0xd6990624);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],46,0xf40e3585);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],47,0x106aa070);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],48,0x19a4c116);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],49,0x1e376c08);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],50,0x2748774c);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],51,0x34b0bcb5);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],52,0x391c0cb3);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],53,0x4ed8aa4a);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],54,0x5b9cca4f);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],55,0x682e6ff3);\n+\tRND(S[0],S[1],S[2],S[3],S[4],S[5],S[6],S[7],56,0x748f82ee);\n+\tRND(S[7],S[0],S[1],S[2],S[3],S[4],S[5],S[6],57,0x78a5636f);\n+\tRND(S[6],S[7],S[0],S[1],S[2],S[3],S[4],S[5],58,0x84c87814);\n+\tRND(S[5],S[6],S[7],S[0],S[1],S[2],S[3],S[4],59,0x8cc70208);\n+\tRND(S[4],S[5],S[6],S[7],S[0],S[1],S[2],S[3],60,0x90befffa);\n+\tRND(S[3],S[4],S[5],S[6],S[7],S[0],S[1],S[2],61,0xa4506ceb);\n+\tRND(S[2],S[3],S[4],S[5],S[6],S[7],S[0],S[1],62,0xbef9a3f7);\n+\tRND(S[1],S[2],S[3],S[4],S[5],S[6],S[7],S[0],63,0xc67178f2);\n+\n+#undef RND\n+\n+\tfor (i = 0; i < 8; i++) {\n+\t\tctx->state[i] = ctx->state[i] + S[i];\n+\t}\n+}\n+\n+#define MIN(x, y) ((x) < (y) ? (x) : (y))\n+void blk_SHA256_Update(blk_SHA256_CTX *ctx, const void *data, size_t len)\n+{\n+\tconst unsigned char *in = data;\n+\tsize_t n;\n+\tctx->length += len;\n+\twhile (len > 0) {\n+\t\tif (!ctx->offset && len >= BLKSIZE) {\n+\t\t\tblk_SHA256_Transform(ctx, in);\n+\t\t\tin += BLKSIZE;\n+\t\t\tlen -= BLKSIZE;\n+\t\t} else {\n+\t\t\tn = MIN(len, (BLKSIZE - ctx->offset));\n+\t\t\tmemcpy(ctx->buf + ctx->offset, in, n);\n+\t\t\tctx->offset += n;\n+\t\t\tin += n;\n+\t\t\tlen -= n;\n+\t\t\tif (ctx->offset == BLKSIZE) {\n+\t\t\t\tblk_SHA256_Transform(ctx, ctx->buf);\n+\t\t\t\tctx->offset = 0;\n+\t\t\t}\n+\t\t}\n+\t}\n+}\n+\n+void blk_SHA256_Final(unsigned char *digest, blk_SHA256_CTX *ctx)\n+{\n+\tconst unsigned trip = BLKSIZE - sizeof(ctx->length);\n+\tint i;\n+\n+\tctx->length <<= 3;\n+\tctx->buf[ctx->offset++] = 0x80;\n+\n+\tif (ctx->offset > trip) {\n+\t\tmemset(ctx->buf + ctx->offset, 0, BLKSIZE - ctx->offset);\n+\t\tblk_SHA256_Transform(ctx, ctx->buf);\n+\t\tctx->offset = 0;\n+\t}\n+\n+\tmemset(ctx->buf + ctx->offset, 0, BLKSIZE - ctx->offset - sizeof(ctx->length));\n+\n+\tput_be64(ctx->buf + trip, ctx->length);\n+\tblk_SHA256_Transform(ctx, ctx->buf);\n+\n+\t/* copy output */\n+\tfor (i = 0; i < 8; i++, digest += sizeof(uint32_t))\n+\t\tput_be32(digest, ctx->state[i]);\n+}\ndiff --git a/sha256/block/sha256.h b/sha256/block/sha256.h\nnew file mode 100644\nindex 0000000000..ad8b178ad3\n--- /dev/null\n+++ b/sha256/block/sha256.h\n@@ -0,0 +1,26 @@\n+#ifndef SHA256_BLOCK_SHA256_H\n+#define SHA256_BLOCK_SHA256_H\n+\n+#include \"git-compat-util.h\"\n+\n+#define blk_SHA256_BLKSIZE 64\n+\n+struct blk_SHA256_CTX {\n+\tuint32_t state[8];\n+\tuint64_t length;\n+\tuint32_t offset;\n+\tuint8_t buf[blk_SHA256_BLKSIZE];\n+};\n+\n+typedef struct blk_SHA256_CTX blk_SHA256_CTX;\n+\n+void blk_SHA256_Init(blk_SHA256_CTX *ctx);\n+void blk_SHA256_Update(blk_SHA256_CTX *ctx, const void *data, size_t len);\n+void blk_SHA256_Final(unsigned char *digest, blk_SHA256_CTX *ctx);\n+\n+#define platform_SHA256_CTX blk_SHA256_CTX\n+#define platform_SHA256_Init blk_SHA256_Init\n+#define platform_SHA256_Update blk_SHA256_Update\n+#define platform_SHA256_Final blk_SHA256_Final\n+\n+#endif\ndiff --git a/t/helper/test-sha256.c b/t/helper/test-sha256.c\nnew file mode 100644\nindex 0000000000..0ac6a99d5f\n--- /dev/null\n+++ b/t/helper/test-sha256.c\n@@ -0,0 +1,7 @@\n+#include \"test-tool.h\"\n+#include \"cache.h\"\n+\n+int cmd__sha256(int ac, const char **av)\n+{\n+\treturn cmd_hash_impl(ac, av, GIT_HASH_SHA256);\n+}\ndiff --git a/t/helper/test-tool.c b/t/helper/test-tool.c\nindex c0d3eecc36..f189c18453 100644\n--- a/t/helper/test-tool.c\n+++ b/t/helper/test-tool.c\n@@ -37,6 +37,7 @@ static struct test_cmd cmds[] = {\n \t{ \"scrap-cache-tree\", cmd__scrap_cache_tree },\n \t{ \"sha1-array\", cmd__sha1_array },\n \t{ \"sha1\", cmd__sha1 },\n+\t{ \"sha256\", cmd__sha256 },\n \t{ \"sigchain\", cmd__sigchain },\n \t{ \"strcmp-offset\", cmd__strcmp_offset },\n \t{ \"string-list\", cmd__string_list },\ndiff --git a/t/helper/test-tool.h b/t/helper/test-tool.h\nindex abe3a253d5..2bc4efb68b 100644\n--- a/t/helper/test-tool.h\n+++ b/t/helper/test-tool.h\n@@ -33,6 +33,7 @@ int cmd__run_command(int argc, const char **argv);\n int cmd__scrap_cache_tree(int argc, const char **argv);\n int cmd__sha1_array(int argc, const char **argv);\n int cmd__sha1(int argc, const char **argv);\n+int cmd__sha256(int argc, const char **argv);\n int cmd__sigchain(int argc, const char **argv);\n int cmd__strcmp_offset(int argc, const char **argv);\n int cmd__string_list(int argc, const char **argv);\ndiff --git a/t/t0014-hash.sh b/t/t0014-hash.sh\nindex 8e763c2c3d..f8e639743f 100755\n--- a/t/t0014-hash.sh\n+++ b/t/t0014-hash.sh\n@@ -26,4 +26,29 @@ test_expect_success 'test basic SHA-1 hash values' '\n \tgrep 4b825dc642cb6eb9a060e54bf8d69288fbee4904 actual\n '\n \n+test_expect_success 'test basic SHA-256 hash values' '\n+\ttest-tool sha256 </dev/null >actual &&\n+\tgrep e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 actual &&\n+\tprintf \"a\" | test-tool sha256 >actual &&\n+\tgrep ca978112ca1bbdcafac231b39a23dc4da786eff8147c4e72b9807785afee48bb actual &&\n+\tprintf \"abc\" | test-tool sha256 >actual &&\n+\tgrep ba7816bf8f01cfea414140de5dae2223b00361a396177a9cb410ff61f20015ad actual &&\n+\tprintf \"message digest\" | test-tool sha256 >actual &&\n+\tgrep f7846f55cf23e14eebeab5b4e1550cad5b509e3348fbc4efa3a1413d393cb650 actual &&\n+\tprintf \"abcdefghijklmnopqrstuvwxyz\" | test-tool sha256 >actual &&\n+\tgrep 71c480df93d6ae2f1efad1447c66c9525e316218cf51fc8d9ed832f2daf18b73 actual &&\n+\tperl -E \"for (1..100000) { print q{aaaaaaaaaa}; }\" | \\\n+\t\ttest-tool sha256 >actual &&\n+\tgrep cdc76e5c9914fb9281a1c7e284d73e67f1809a48a497200e046d39ccc7112cd0 actual &&\n+\tperl -E \"for (1..100000) { print q{abcdefghijklmnopqrstuvwxyz}; }\" | \\\n+\t\ttest-tool sha256 >actual &&\n+\tgrep e406ba321ca712ad35a698bf0af8d61fc4dc40eca6bdcea4697962724ccbde35 actual &&\n+\tprintf \"blob 0\\0\" | test-tool sha256 >actual &&\n+\tgrep 473a0f4c3be8a93681a267e3b1e9a7dcda1185436fe141f7749120a303721813 actual &&\n+\tprintf \"blob 3\\0abc\" | test-tool sha256 >actual &&\n+\tgrep c1cf6e465077930e88dc5136641d402f72a229ddd996f627d60e9639eaba35a6 actual &&\n+\tprintf \"tree 0\\0\" | test-tool sha256 >actual &&\n+\tgrep 6ef19b41225c5369f1c104d45d8d85efa9b057b53b14b4b9b939dd74decc5321 actual\n+'\n+\n test_done\n"},{"id":"356804","messageId":"20180829005857.980820-11-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 10/12] sha256: add an SHA-256 implementation using libgcrypt","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:55Z","receivedAt":"2018-08-29T00:59:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Generally, one gets better performance out of cryptographic routines\nwritten in assembly than C, and this is also true for SHA-256.  In\naddition, most Linux distributions cannot distribute Git linked against\nOpenSSL for licensing reasons.\n\nMost systems with GnuPG will also have libgcrypt, since it is a\ndependency of GnuPG.  libgcrypt is also faster than the SHA1DC\nimplementation for messages of a few KiB and larger. It is licensed\nunder the LGPL 2.1, which is compatible with the GPL.\n\nAdd an implementation of SHA-256 that uses libgcrypt.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n Makefile        | 13 +++++++++++--\n hash.h          |  4 ++++\n sha256/gcrypt.h | 30 ++++++++++++++++++++++++++++++\n 3 files changed, 45 insertions(+), 2 deletions(-)\n create mode 100644 sha256/gcrypt.h\n\ndiff --git a/Makefile b/Makefile\nindex 624d852e79..86867af083 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -179,6 +179,10 @@ all::\n # in one call to the platform's SHA1_Update(). e.g. APPLE_COMMON_CRYPTO\n # wants 'SHA1_MAX_BLOCK_SIZE=1024L*1024L*1024L' defined.\n #\n+# Define BLK_SHA256 to use the built-in SHA-256 routines.\n+#\n+# Define GCRYPT_SHA256 to use the SHA-256 routines in libgcrypt.\n+#\n # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).\n #\n # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).\n@@ -1624,8 +1628,13 @@ endif\n endif\n endif\n \n-LIB_OBJS += sha256/block/sha256.o\n-BASIC_CFLAGS += -DSHA256_BLK\n+ifdef GCRYPT_SHA256\n+\tBASIC_CFLAGS += -DSHA256_GCRYPT\n+\tEXTLIBS += -lgcrypt\n+else\n+\tLIB_OBJS += sha256/block/sha256.o\n+\tBASIC_CFLAGS += -DSHA256_BLK\n+endif\n \n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\ndiff --git a/hash.h b/hash.h\nindex 88d18896d7..9df562f2f6 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -15,7 +15,11 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n+#if defined(SHA256_GCRYPT)\n+#include \"sha256/gcrypt.h\"\n+#else\n #include \"sha256/block/sha256.h\"\n+#endif\n \n #ifndef platform_SHA_CTX\n /*\ndiff --git a/sha256/gcrypt.h b/sha256/gcrypt.h\nnew file mode 100644\nindex 0000000000..09bd8bb200\n--- /dev/null\n+++ b/sha256/gcrypt.h\n@@ -0,0 +1,30 @@\n+#ifndef SHA256_GCRYPT_H\n+#define SHA256_GCRYPT_H\n+\n+#include <gcrypt.h>\n+\n+#define SHA256_DIGEST_SIZE 32\n+\n+typedef gcry_md_hd_t gcrypt_SHA256_CTX;\n+\n+inline void gcrypt_SHA256_Init(gcrypt_SHA256_CTX *ctx)\n+{\n+\tgcry_md_open(ctx, GCRY_MD_SHA256, 0);\n+}\n+\n+inline void gcrypt_SHA256_Update(gcrypt_SHA256_CTX *ctx, const void *data, size_t len)\n+{\n+\tgcry_md_write(*ctx, data, len);\n+}\n+\n+inline void gcrypt_SHA256_Final(unsigned char *digest, gcrypt_SHA256_CTX *ctx)\n+{\n+\tmemcpy(digest, gcry_md_read(*ctx, GCRY_MD_SHA256), SHA256_DIGEST_SIZE);\n+}\n+\n+#define platform_SHA256_CTX gcrypt_SHA256_CTX\n+#define platform_SHA256_Init gcrypt_SHA256_Init\n+#define platform_SHA256_Update gcrypt_SHA256_Update\n+#define platform_SHA256_Final gcrypt_SHA256_Final\n+\n+#endif\n"},{"id":"356805","messageId":"20180829005857.980820-12-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 11/12] hash: add an SHA-256 implementation using OpenSSL","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:56Z","receivedAt":"2018-08-29T00:59:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"We already have OpenSSL routines available for SHA-1, so add routines\nfor SHA-256 as well.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n Makefile | 7 +++++++\n hash.h   | 2 ++\n 2 files changed, 9 insertions(+)\n\ndiff --git a/Makefile b/Makefile\nindex 86867af083..8b7df4dfc5 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -183,6 +183,8 @@ all::\n #\n # Define GCRYPT_SHA256 to use the SHA-256 routines in libgcrypt.\n #\n+# Define OPENSSL_SHA256 to use the SHA-256 routines in OpenSSL.\n+#\n # Define NEEDS_CRYPTO_WITH_SSL if you need -lcrypto when using -lssl (Darwin).\n #\n # Define NEEDS_SSL_WITH_CRYPTO if you need -lssl when using -lcrypto (Darwin).\n@@ -1628,6 +1630,10 @@ endif\n endif\n endif\n \n+ifdef OPENSSL_SHA256\n+\tEXTLIBS += $(LIB_4_CRYPTO)\n+\tBASIC_CFLAGS += -DSHA256_OPENSSL\n+else\n ifdef GCRYPT_SHA256\n \tBASIC_CFLAGS += -DSHA256_GCRYPT\n \tEXTLIBS += -lgcrypt\n@@ -1635,6 +1641,7 @@ else\n \tLIB_OBJS += sha256/block/sha256.o\n \tBASIC_CFLAGS += -DSHA256_BLK\n endif\n+endif\n \n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\ndiff --git a/hash.h b/hash.h\nindex 9df562f2f6..9df06d56b4 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -17,6 +17,8 @@\n \n #if defined(SHA256_GCRYPT)\n #include \"sha256/gcrypt.h\"\n+#elif defined(SHA256_OPENSSL)\n+#include <openssl/sha.h>\n #else\n #include \"sha256/block/sha256.h\"\n #endif\n"},{"id":"356806","messageId":"20180829005857.980820-13-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 12/12] commit-graph: specify OID version for SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:57Z","receivedAt":"2018-08-29T00:59:19Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Since the commit-graph code wants to serialize the hash algorithm into\nthe data store, specify a version number for each supported algorithm.\nNote that we don't use the values of the constants themselves, as they\nare internal and could change in the future.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n commit-graph.c | 9 ++++++++-\n 1 file changed, 8 insertions(+), 1 deletion(-)\n\ndiff --git a/commit-graph.c b/commit-graph.c\nindex 29356d84a2..5b86acdb43 100644\n--- a/commit-graph.c\n+++ b/commit-graph.c\n@@ -45,7 +45,14 @@ char *get_commit_graph_filename(const char *obj_dir)\n \n static uint8_t oid_version(void)\n {\n-\treturn 1;\n+\tswitch (hash_algo_by_ptr(the_hash_algo)) {\n+\t\tcase GIT_HASH_SHA1:\n+\t\t\treturn 1;\n+\t\tcase GIT_HASH_SHA256:\n+\t\t\treturn 2;\n+\t\tdefault:\n+\t\t\tBUG(\"unknown hash algorithm\");\n+\t}\n }\n \n static struct commit_graph *alloc_commit_graph(void)\n"},{"id":"356807","messageId":"20180829005857.980820-3-sandals@crustytoothpaste.net","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 02/12] sha1-file: provide functions to look up hash algorithms","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T00:58:47Z","receivedAt":"2018-08-29T00:59:22Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"There are several ways we might refer to a hash algorithm: by name, such\nas in the config file; by format ID, such as in a pack; or internally,\nby a pointer to the hash_algos array.  Provide functions to look up hash\nalgorithms based on these various forms and return the internal constant\nused for them.  If conversion to another form is necessary, this\ninternal constant can be used to look up the proper data in the\nhash_algos array.\n\nSigned-off-by: brian m. carlson <sandals@crustytoothpaste.net>\n---\n hash.h      | 13 +++++++++++++\n sha1-file.c | 21 +++++++++++++++++++++\n 2 files changed, 34 insertions(+)\n\ndiff --git a/hash.h b/hash.h\nindex 7c8238bc2e..90f4344619 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -98,4 +98,17 @@ struct git_hash_algo {\n };\n extern const struct git_hash_algo hash_algos[GIT_HASH_NALGOS];\n \n+/*\n+ * Return a GIT_HASH_* constant based on the name.  Returns GIT_HASH_UNKNOWN if\n+ * the name doesn't match a known algorithm.\n+ */\n+int hash_algo_by_name(const char *name);\n+/* Identical, except based on the format ID. */\n+int hash_algo_by_id(uint32_t format_id);\n+/* Identical, except for a pointer to struct git_hash_algo. */\n+inline int hash_algo_by_ptr(const struct git_hash_algo *p)\n+{\n+\treturn p - hash_algos;\n+}\n+\n #endif\ndiff --git a/sha1-file.c b/sha1-file.c\nindex 5223e3d1ce..f6976b179f 100644\n--- a/sha1-file.c\n+++ b/sha1-file.c\n@@ -122,6 +122,27 @@ const char *empty_blob_oid_hex(void)\n \treturn oid_to_hex_r(buf, the_hash_algo->empty_blob);\n }\n \n+int hash_algo_by_name(const char *name)\n+{\n+\tint i;\n+\tif (!name)\n+\t\treturn GIT_HASH_UNKNOWN;\n+\tfor (i = 1; i < GIT_HASH_NALGOS; i++)\n+\t\tif (!strcmp(name, hash_algos[i].name))\n+\t\t\treturn i;\n+\treturn GIT_HASH_UNKNOWN;\n+}\n+\n+int hash_algo_by_id(uint32_t format_id)\n+{\n+\tint i;\n+\tfor (i = 1; i < GIT_HASH_NALGOS; i++)\n+\t\tif (format_id == hash_algos[i].format_id)\n+\t\t\treturn i;\n+\treturn GIT_HASH_UNKNOWN;\n+}\n+\n+\n /*\n  * This is meant to hold a *small* number of objects that you would\n  * want read_sha1_file() to be able to return, but yet you do not want\n"},{"id":"356819","messageId":"878t4peeqa.fsf@evledraar.gmail.com","threadId":"49251","inReplyTo":"20180829005857.980820-11-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 10/12] sha256: add an SHA-256 implementation using libgcrypt","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-08-29T08:53:01Z","receivedAt":"2018-08-29T08:53:06Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Aug 29 2018, brian m. carlson wrote:\n\n> Generally, one gets better performance out of cryptographic routines\n> written in assembly than C, and this is also true for SHA-256\n\nIt makes sense to have a libgcrypt implementation...\n\n> In addition, most Linux distributions cannot distribute Git linked\n> against OpenSSL for licensing reasons.\n\n...but I'm curious to know what licensing reasons these are, e.g. Debian\nwho's usually the most strict about these things distributes git linked\nto OpenSSL:\n\n    $ dpkg -S /usr/lib/git-core/git-imap-send; apt policy git 2>/dev/null|grep -F '***'; ldd -r /usr/lib/git-core/git-imap-send|grep ssl; uname -m\n    git: /usr/lib/git-core/git-imap-send\n     *** 1:2.19.0~rc1+next.20180828-1 1001\n            libssl.so.1.0.2 => /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2 (0x00007fd3cc8bb000)\n    x86_64\n    $ dpkg -S /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2\n    libssl1.0.2:amd64: /usr/lib/x86_64-linux-gnu/libssl.so.1.0.2\n    $ apt show libssl1.0.2 2>&1 |grep ssl\n    Package: libssl1.0.2\n    Source: openssl1.0\n    Maintainer: Debian OpenSSL Team <pkg-openssl-devel@lists.alioth.debian.org>\n    Homepage: https://www.openssl.org\n"},{"id":"356821","messageId":"875zztecx3.fsf@evledraar.gmail.com","threadId":"49251","inReplyTo":"20180829005857.980820-10-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 09/12] Add a base implementation of SHA-256 support","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-08-29T09:32:08Z","receivedAt":"2018-08-29T09:32:13Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Aug 29 2018, brian m. carlson wrote:\n\n> SHA-1 is weak and we need to transition to a new hash function.  For\n> some time, we have referred to this new function as NewHash.\n>\n> The selection criteria for NewHash specify that it should (a) be 256\n> bits in length, (b) have high quality implementations available, (c)\n> should match Git's needs in terms of security, and (d) ideally, be fast\n> to compute.\n>\n> SHA-256 has a variety of high quality implementations across various\n> libraries.  It is implemented by every cryptographic library we support\n> and is available on every platform and in almost every programming\n> language.  It is often highly optimized, since it is commonly used in\n> TLS and elsewhere.  Additionally, there are various command line\n> utilities that implement it, which is useful for educational and testing\n> purposes.\n>\n> SHA-256 is presently considered secure and has received a reasonable\n> amount of cryptanalysis in the literature.  It is, admittedly, not\n> resistant to length extension attacks, but Git object storage is immune\n> to those due to the length field at the beginning.\n>\n> SHA-256 is somewhat slower to compute than SHA-1 in software.  However,\n> since our default SHA-1 implementation is collision-detecting, a\n> reasonable cryptographic library implementation of SHA-256 will actually\n> be faster than SHA-256.  In addition, modern ARM and AMD processors (and\n> some Intel processors) contain instructions for implementing SHA-256 in\n> hardware, making it the fastest possible option.\n>\n> There are other reasons to select SHA-256.  With signed commits and\n> tags, it's possible to use SHA-256 for signatures and therefore have to\n> rely on only one hash algorithm for security.\n\nNone of this is wrong, but I think this would be better off as a simple\n\"See Documentation/technical/hash-function-transition.txt for why we're\nswitching to SHA-256\", and to the extent that something is said here\nthat isn't said there it could be a patch to amend that document.\n\n> Add a basic implementation of SHA-256 based off libtomcrypt, which is in\n> the public domain.  Optimize it and tidy it somewhat.\n\nFor future changes & maintenance of this, let's do that in two\nsteps. One where we add the upstream code as-is, and another where the\ntidying / cleanup / git specific stuff is wired, which makes it easy to\naudit upstream as-is v.s. our changes in isolation. Also in the first of\nthose commits, say in the commit message \"add a [libtomcrypt] copy from\nsuch-and-such a URL at such-and-such a version\", so that it's easy to\nreproduce the import & find out how to re-update it.\n\nIs this something we see ourselves perma-forking? Or as with sha1dc are\nwe likely to pull in upstream changes from time-to-time?SHA256 obiously\nisn't under active development, but there's been some churn in the\nupstream code since it was added, and if you're doing some optimizing /\ntidying that's presumably something upstream could benefit from as well,\nas well as just us being nicer open source citizens feeding\ne.g. portability fixes to upstream (since git tends to get ported a\nlot).\n\nSo I wonder if we can't convince them to add a few macros to their code,\nand then do something like what I did in a0103914c2 (\"sha1dc: update\nfrom upstream\", 2017-05-20) for sha1dc allowing us to use their code\nas-is with some defines in the Makefile, which both makes it easier to\nupdate, and sets up a process where our default approach is to submit\nchanges upstream, instead of working on our perma-fork.\n"},{"id":"356822","messageId":"874lfdecoa.fsf@evledraar.gmail.com","threadId":"49251","inReplyTo":"20180829005857.980820-1-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 00/12] Base SHA-256 algorithm implementation","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-08-29T09:37:25Z","receivedAt":"2018-08-29T09:37:30Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Aug 29 2018, brian m. carlson wrote:\n\n> If libgit2 would like to import this SHA-256 implementation, they're\n> welcome to do so under their normal license terms.  If not, that's fine,\n> too.\n\nFor them and anyone else interested in such re-use, can you elaborate on\nwhich parts?\n\nIt seems to me that aside from t/helper/test-hash-speed.c and\nt/t0014-hash.sh everything being added here modifies existing files with\nmany authors, and would thus also need their permission to re-license as\nanything except GPLv2.\n\nOr do you mean whatever fixes/changes you did to libtomcrypt (living in\nsha256/block/ in this series) you consider e.g. LPGL instead of GPL?\n"},{"id":"356827","messageId":"ef3f5ff2-a774-9e9b-d73a-b21630bede53@gmail.com","threadId":"49251","inReplyTo":"20180829005857.980820-9-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 08/12] commit-graph: convert to using the_hash_algo","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-08-29T12:41:36Z","receivedAt":"2018-08-29T12:41:40Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/28/2018 8:58 PM, brian m. carlson wrote:\n> Instead of using hard-coded constants for object sizes, use\n> the_hash_algo to look them up.  In addition, use a function call to look\n> up the object ID version and produce the correct value.\n\nThe C code in this patch looks good to me. The only issue is that I \npredict failure in the 'git commit-graph verify' tests in \nt5318-commit-graph.sh. Squashing in this commit should help (assuming \nthat test_oid works, it doesn't at my current branch):\n\n-->8--\n\nSubject: [PATCH] t5318-commit-graph.sh: use test_oid for HASH_LEN\n\nSigned-off-by: Derrick Stolee <dstolee@microsoft.com>\n---\n  t/t5318-commit-graph.sh | 2 +-\n  1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\nindex 6aee861f78..676c1a9ae0 100755\n--- a/t/t5318-commit-graph.sh\n+++ b/t/t5318-commit-graph.sh\n@@ -333,7 +333,7 @@ test_expect_success 'git commit-graph verify' '\n\n  NUM_COMMITS=9\n  NUM_OCTOPUS_EDGES=2\n-HASH_LEN=20\n+HASH_LEN=\"$(test_oid rawsz)\"\n  GRAPH_BYTE_VERSION=4\n  GRAPH_BYTE_HASH=5\n  GRAPH_BYTE_CHUNK_COUNT=6\n--\n2.19.0.rc1\n"},{"id":"356832","messageId":"7b1aac2e-b768-cfec-2dfe-61f63066ae1a@gmail.com","threadId":"49251","inReplyTo":"20180829005857.980820-10-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 09/12] Add a base implementation of SHA-256 support","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2018-08-29T12:54:56Z","receivedAt":"2018-08-29T12:55:00Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/28/2018 8:58 PM, brian m. carlson wrote:\n> SHA-256 is somewhat slower to compute than SHA-1 in software.  However,\n> since our default SHA-1 implementation is collision-detecting, a\n> reasonable cryptographic library implementation of SHA-256 will actually\n> be faster than SHA-256.\n\nNit: do you mean \"faster than SHA-1DC\"?\n\n"},{"id":"356916","messageId":"20180829233904.GG432229@genre.crustytoothpaste.net","threadId":"49251","inReplyTo":"878t4peeqa.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH 10/12] sha256: add an SHA-256 implementation using libgcrypt","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T23:39:05Z","receivedAt":"2018-08-29T23:43:05Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Wed, Aug 29, 2018 at 10:53:01AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Aug 29 2018, brian m. carlson wrote:\n> \n> > Generally, one gets better performance out of cryptographic routines\n> > written in assembly than C, and this is also true for SHA-256\n> \n> It makes sense to have a libgcrypt implementation...\n> \n> > In addition, most Linux distributions cannot distribute Git linked\n> > against OpenSSL for licensing reasons.\n> \n> ...but I'm curious to know what licensing reasons these are, e.g. Debian\n> who's usually the most strict about these things distributes git linked\n> to OpenSSL:\n\nOn my Debian system, that's linked to libgnutls.\n\nThe reason is section 3 of the GPLv2 (emphasis mine):\n\n  3. You may copy and distribute the Program (or a work based on it,\n  under Section 2) in object code or executable form **under the terms\n  of Sections 1 and 2 above** provided that you also do one of the\n  following:\n\n  [provide source somehow]\n\n  The source code for a work means the preferred form of the work for\n  making modifications to it.  For an executable work, complete source\n  code means all the source code for all modules it contains, plus any\n  associated interface definition files, plus the scripts used to\n  control compilation and installation of the executable.  **However, as\n  a special exception, the source code distributed need not include\n  anything that is normally distributed (in either source or binary\n  form) with the major components (compiler, kernel, and so on) of the\n  operating system on which the executable runs, unless that component\n  itself accompanies the executable.**\n\nBasically, you can only distribute binary versions of Git under the\nterms of the GPLv2, and you have to distribute source for the entire\nthing under those terms.  OpenSSL is licensed incompatibly with the\nGPLv2, so you can't legally comply with that part, but if you use the\nsystem OpenSSL and don't distribute that OpenSSL with Git, you're\nexempt.  This is called the system library exception.\n\nDebian (and Red Hat, and every other Linux distro) ships Git and OpenSSL\nside by side on the same mirrors, and hence \"that component [OpenSSL]\naccompanies the executable.\"  Consequently, they can't take advantage of\nthe exception, and must link it to a GPLv2 compatible library.  Debian\nuses GnuTLS for libcurl, and Red Hat uses NSS.\n\nA more comprehensive explanation of the whole thing is here:\nhttps://people.gnome.org/~markmc/openssl-and-the-gpl.html\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"356919","messageId":"20180829235504.GH432229@genre.crustytoothpaste.net","threadId":"49251","inReplyTo":"875zztecx3.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH 09/12] Add a base implementation of SHA-256 support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-29T23:55:04Z","receivedAt":"2018-08-29T23:55:12Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Wed, Aug 29, 2018 at 11:32:08AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Wed, Aug 29 2018, brian m. carlson wrote:\n> \n> > SHA-1 is weak and we need to transition to a new hash function.  For\n> > some time, we have referred to this new function as NewHash.\n> >\n> > The selection criteria for NewHash specify that it should (a) be 256\n> > bits in length, (b) have high quality implementations available, (c)\n> > should match Git's needs in terms of security, and (d) ideally, be fast\n> > to compute.\n> >\n> > SHA-256 has a variety of high quality implementations across various\n> > libraries.  It is implemented by every cryptographic library we support\n> > and is available on every platform and in almost every programming\n> > language.  It is often highly optimized, since it is commonly used in\n> > TLS and elsewhere.  Additionally, there are various command line\n> > utilities that implement it, which is useful for educational and testing\n> > purposes.\n> >\n> > SHA-256 is presently considered secure and has received a reasonable\n> > amount of cryptanalysis in the literature.  It is, admittedly, not\n> > resistant to length extension attacks, but Git object storage is immune\n> > to those due to the length field at the beginning.\n> >\n> > SHA-256 is somewhat slower to compute than SHA-1 in software.  However,\n> > since our default SHA-1 implementation is collision-detecting, a\n> > reasonable cryptographic library implementation of SHA-256 will actually\n> > be faster than SHA-256.  In addition, modern ARM and AMD processors (and\n> > some Intel processors) contain instructions for implementing SHA-256 in\n> > hardware, making it the fastest possible option.\n> >\n> > There are other reasons to select SHA-256.  With signed commits and\n> > tags, it's possible to use SHA-256 for signatures and therefore have to\n> > rely on only one hash algorithm for security.\n> \n> None of this is wrong, but I think this would be better off as a simple\n> \"See Documentation/technical/hash-function-transition.txt for why we're\n> switching to SHA-256\", and to the extent that something is said here\n> that isn't said there it could be a patch to amend that document.\n\nI can certainly shorten this somewhat.  I wrote this back when there\nwasn't a consensus on hash algorithm and Junio was going to leave it to\nme to make a decision.  I was therefore obligated to provide a coherent\nrationale for that decision.\n\n> > Add a basic implementation of SHA-256 based off libtomcrypt, which is in\n> > the public domain.  Optimize it and tidy it somewhat.\n> \n> For future changes & maintenance of this, let's do that in two\n> steps. One where we add the upstream code as-is, and another where the\n> tidying / cleanup / git specific stuff is wired, which makes it easy to\n> audit upstream as-is v.s. our changes in isolation. Also in the first of\n> those commits, say in the commit message \"add a [libtomcrypt] copy from\n> such-and-such a URL at such-and-such a version\", so that it's easy to\n> reproduce the import & find out how to re-update it.\n\nDoing what you suggest basically means importing a large amount of\nlibtomcrypt into our codebase, since there are a large number of reused\nmacros all over libtomcrypt (such as for processing a generic hash and\nfor memcpy).\n\nThis isn't surprising for a general purpose crypto library, but I did a\ndecent amount to change it, condense it into a small number of files,\nand make it meet our code standards.  The phrase \"somewhat\" may have\nbeen an understatement.\n\nThis is also why I added tests: because I'm human and making a small\nchange in a crypto library can result in wrong output very quickly.\n\n> Is this something we see ourselves perma-forking? Or as with sha1dc are\n> we likely to pull in upstream changes from time-to-time?SHA256 obiously\n> isn't under active development, but there's been some churn in the\n> upstream code since it was added, and if you're doing some optimizing /\n> tidying that's presumably something upstream could benefit from as well,\n> as well as just us being nicer open source citizens feeding\n> e.g. portability fixes to upstream (since git tends to get ported a\n> lot).\n\nThis is a permafork.  We need a basic SHA-256 implementation, and this\none was faster than the one I'd written some time ago.  Similarly to the\nblock-sha1 implementation, I see this as something that we'll be\nshipping forever with little updating.\n\nI expect with the amount of changes we're making, they're unlikely to\nwant our code.  Also, any changes to our code would be under the GPLv2,\nwhich would be unappealing to a public domain library.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"356925","messageId":"20180830022151.GI432229@genre.crustytoothpaste.net","threadId":"49251","inReplyTo":"874lfdecoa.fsf@evledraar.gmail.com","subject":"Re: [RFC PATCH 00/12] Base SHA-256 algorithm implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-30T02:21:51Z","receivedAt":"2018-08-30T02:21:59Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Wed, Aug 29, 2018 at 11:37:25AM +0200, Ævar Arnfjörð Bjarmason wrote:\n\n> It seems to me that aside from t/helper/test-hash-speed.c and\n> t/t0014-hash.sh everything being added here modifies existing files with\n> many authors, and would thus also need their permission to re-license as\n> anything except GPLv2.\n> \n> Or do you mean whatever fixes/changes you did to libtomcrypt (living in\n> sha256/block/ in this series) you consider e.g. LPGL instead of GPL?\n\nYes, that's what I mean, specifically the code in sha256/block.  libgit2\nis GPLv2 with a linking exception, I believe, but either way I'd be fine\nwith it.\n\nIt wasn't my intention to offer code I didn't wholly author, so thanks\nfor clarifying.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"356926","messageId":"20180830023050.GJ432229@genre.crustytoothpaste.net","threadId":"49251","inReplyTo":"ef3f5ff2-a774-9e9b-d73a-b21630bede53@gmail.com","subject":"Re: [RFC PATCH 08/12] commit-graph: convert to using the_hash_algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-30T02:30:50Z","receivedAt":"2018-08-30T02:30:58Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Wed, Aug 29, 2018 at 08:41:36AM -0400, Derrick Stolee wrote:\n> On 8/28/2018 8:58 PM, brian m. carlson wrote:\n> > Instead of using hard-coded constants for object sizes, use\n> > the_hash_algo to look them up.  In addition, use a function call to look\n> > up the object ID version and produce the correct value.\n> \n> The C code in this patch looks good to me. The only issue is that I predict\n> failure in the 'git commit-graph verify' tests in t5318-commit-graph.sh.\n> Squashing in this commit should help (assuming that test_oid works, it\n> doesn't at my current branch):\n\nYeah, this is a separate series not based on the other one.  If I\nfinally submit this after the other series lands, I'll squash that\nchange in.\n\nThanks for the patch.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"356927","messageId":"20180830024109.GK432229@genre.crustytoothpaste.net","threadId":"49251","inReplyTo":"20180830022151.GI432229@genre.crustytoothpaste.net","subject":"Re: [RFC PATCH 00/12] Base SHA-256 algorithm implementation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-08-30T02:41:10Z","receivedAt":"2018-08-30T02:41:18Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Thu, Aug 30, 2018 at 02:21:51AM +0000, brian m. carlson wrote:\n> On Wed, Aug 29, 2018 at 11:37:25AM +0200, Ævar Arnfjörð Bjarmason wrote:\n> \n> > It seems to me that aside from t/helper/test-hash-speed.c and\n> > t/t0014-hash.sh everything being added here modifies existing files with\n> > many authors, and would thus also need their permission to re-license as\n> > anything except GPLv2.\n> > \n> > Or do you mean whatever fixes/changes you did to libtomcrypt (living in\n> > sha256/block/ in this series) you consider e.g. LPGL instead of GPL?\n> \n> Yes, that's what I mean, specifically the code in sha256/block.  libgit2\n> is GPLv2 with a linking exception, I believe, but either way I'd be fine\n> with it.\n> \n> It wasn't my intention to offer code I didn't wholly author, so thanks\n> for clarifying.\n\nI should clarify further: I obviously didn't wholly author the code.  I\ndo, however, have permission to license it under alternative terms,\nsince the original was public domain.  I apologize for my\nmischaracterization of the situation.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"357250","messageId":"20180903191126.GN432229@genre.crustytoothpaste.net","threadId":"49251","inReplyTo":"ef3f5ff2-a774-9e9b-d73a-b21630bede53@gmail.com","subject":"Re: [RFC PATCH 08/12] commit-graph: convert to using the_hash_algo","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-09-03T19:11:26Z","receivedAt":"2018-09-03T19:11:35Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Wed, Aug 29, 2018 at 08:41:36AM -0400, Derrick Stolee wrote:\n> diff --git a/t/t5318-commit-graph.sh b/t/t5318-commit-graph.sh\n> index 6aee861f78..676c1a9ae0 100755\n> --- a/t/t5318-commit-graph.sh\n> +++ b/t/t5318-commit-graph.sh\n> @@ -333,7 +333,7 @@ test_expect_success 'git commit-graph verify' '\n> \n>  NUM_COMMITS=9\n>  NUM_OCTOPUS_EDGES=2\n> -HASH_LEN=20\n> +HASH_LEN=\"$(test_oid rawsz)\"\n>  GRAPH_BYTE_VERSION=4\n>  GRAPH_BYTE_HASH=5\n>  GRAPH_BYTE_CHUNK_COUNT=6\n\nI dropped this at the end of my hash-independent fixes series and I\nslipped in a use of test_oid_init, which is now required.  Thanks for\nthe patch.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"}]}