{"thread":{"id":"47707","subject":"[RFC PATCH 0/2] alternate hash test","startedAt":"2018-01-28T17:07:01Z","lastAt":"2018-02-01T01:25:13Z","messageCount":9,"participants":["brian m. carlson","SZEDER Gábor","Ævar Arnfjörð Bjarmason","Junio C Hamano","Stefan Beller"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"337624","messageId":"20180128170639.216397-1-sandals@crustytoothpaste.net","threadId":"47707","inReplyTo":null,"subject":"[RFC PATCH 0/2] alternate hash test","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-01-28T17:06:37Z","receivedAt":"2018-01-28T17:07:01Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"This series wires up an alternate hash implementation, namely\nBLAKE2b-160.  The goal is to allow us to identify tests which rely on\nthe hash algorithm in use so that we can fix those tests.\n\nFor this test, I picked BLAKE2b-160 for a couple reasons:\n* Debian ships a libb2-1 package which can be used easily (in other\n  words, I was lazy and didn't want to add a crypto implementation just\n  for test purposes);\n* The API of the libb2 package easily allows arbitrary hash lengths, so\n  I didn't have to manage truncation myself;\n* Our codebase isn't yet ready for a hash function larger than 20 bytes,\n  as there's still more work to do on the object_id conversions.\n\nThis isn't an endorsement for or against any particular algorithm\nchoice, just an artifact of the tools that were easily available to me.\nProvoking discussion of which hash to pick for NewHash is explicitly\n*not* a goal for this series.  I'm only interested in the ability to\nidentify and fix tests.\n\nThe first patch does no feature detection and just assumes you have\nlibb2 installed.  For obvious reasons, this series is not meant for\nproduction use.\n\nThis series applies on top of the object_id part 11 series I just sent\nout.\n\nBelow is a list of the prove output indicating which tests failed on my\nsystem with these patches.\n\nTest Summary Report\n-------------------\nt0002-gitfile.sh                                 (Wstat: 256 Tests: 14 Failed: 3)\n  Failed tests:  12-14\n  Non-zero exit status: 1\nt0005-signals.sh                                 (Wstat: 256 Tests: 5 Failed: 2)\n  Failed tests:  4-5\n  Non-zero exit status: 1\nt0000-basic.sh                                   (Wstat: 256 Tests: 78 Failed: 12)\n  Failed tests:  47, 54, 56, 58, 60, 62, 64, 66, 71, 74-76\n  Non-zero exit status: 1\nt1007-hash-object.sh                             (Wstat: 256 Tests: 38 Failed: 15)\n  Failed tests:  6, 8, 10-11, 19-29\n  Non-zero exit status: 1\nt1011-read-tree-sparse-checkout.sh               (Wstat: 256 Tests: 21 Failed: 3)\n  Failed tests:  1-2, 5\n  Non-zero exit status: 1\nt1300-repo-config.sh                             (Wstat: 256 Tests: 146 Failed: 1)\n  Failed test:  144\n  Non-zero exit status: 1\nt1304-default-acl.sh                             (Wstat: 256 Tests: 4 Failed: 1)\n  Failed test:  3\n  Non-zero exit status: 1\nt1405-main-ref-store.sh                          (Wstat: 256 Tests: 17 Failed: 1)\n  Failed test:  10\n  Non-zero exit status: 1\nt1411-reflog-show.sh                             (Wstat: 256 Tests: 18 Failed: 6)\n  Failed tests:  3-5, 7, 10, 13\n  Non-zero exit status: 1\nt1450-fsck.sh                                    (Wstat: 256 Tests: 71 Failed: 3)\n  Failed tests:  18, 26, 59\n  Non-zero exit status: 1\nt1507-rev-parse-upstream.sh                      (Wstat: 256 Tests: 28 Failed: 2)\n  Failed tests:  25-26\n  Non-zero exit status: 1\nt1512-rev-parse-disambiguation.sh                (Wstat: 256 Tests: 33 Failed: 19)\n  Failed tests:  2-5, 7-12, 16, 22, 26-32\n  Non-zero exit status: 1\nt1700-split-index.sh                             (Wstat: 256 Tests: 22 Failed: 9)\n  Failed tests:  1-2, 5-7, 13-16\n  Non-zero exit status: 1\nt2011-checkout-invalid-head.sh                   (Wstat: 256 Tests: 10 Failed: 5)\n  Failed tests:  3, 6-7, 9-10\n  Non-zero exit status: 1\nt2015-checkout-unborn.sh                         (Wstat: 256 Tests: 6 Failed: 2)\n  Failed tests:  3-4\n  Non-zero exit status: 1\nt1013-read-tree-submodule.sh                     (Wstat: 256 Tests: 64 Failed: 15)\n  Failed tests:  1-2, 4-7, 19-20, 22-25, 30, 34-35\n  Non-zero exit status: 1\nt2017-checkout-orphan.sh                         (Wstat: 256 Tests: 13 Failed: 7)\n  Failed tests:  7-13\n  Non-zero exit status: 1\nt2020-checkout-detach.sh                         (Wstat: 256 Tests: 24 Failed: 2)\n  Failed tests:  23-24\n  Non-zero exit status: 1\nt2101-update-index-reupdate.sh                   (Wstat: 256 Tests: 7 Failed: 6)\n  Failed tests:  1-3, 5-7\n  Non-zero exit status: 1\nt2107-update-index-basic.sh                      (Wstat: 256 Tests: 9 Failed: 1)\n  Failed test:  9\n  Non-zero exit status: 1\nt2203-add-intent.sh                              (Wstat: 256 Tests: 16 Failed: 4)\n  Failed tests:  3, 12, 15-16\n  Non-zero exit status: 1\nt3033-merge-toplevel.sh                          (Wstat: 256 Tests: 13 Failed: 11)\n  Failed tests:  3-13\n  Non-zero exit status: 1\nt3103-ls-tree-misc.sh                            (Wstat: 256 Tests: 2 Failed: 1)\n  Failed test:  2\n  Non-zero exit status: 1\nt3201-branch-contains.sh                         (Wstat: 256 Tests: 19 Failed: 1)\n  Failed test:  18\n  Non-zero exit status: 1\nt3301-notes.sh                                   (Wstat: 256 Tests: 134 Failed: 50)\n  Failed tests:  10, 18-23, 42-44, 46, 55-57, 60, 65-78\n                80, 83-84, 86-91, 93-104\n  Non-zero exit status: 1\nt2013-checkout-submodule.sh                      (Wstat: 256 Tests: 70 Failed: 16)\n  Failed tests:  7-8, 10-13, 19, 25-26, 28-31, 36, 40-41\n  Non-zero exit status: 1\nt3306-notes-prune.sh                             (Wstat: 256 Tests: 12 Failed: 8)\n  Failed tests:  1-2, 4-5, 7, 9, 11-12\n  Non-zero exit status: 1\nt3308-notes-merge.sh                             (Wstat: 256 Tests: 19 Failed: 13)\n  Failed tests:  2, 5-6, 10-19\n  Non-zero exit status: 1\nt3309-notes-merge-auto-resolve.sh                (Wstat: 256 Tests: 31 Failed: 27)\n  Failed tests:  3, 5, 7-31\n  Non-zero exit status: 1\nt3310-notes-merge-manual-resolve.sh              (Wstat: 256 Tests: 22 Failed: 16)\n  Failed tests:  2-6, 8-9, 13-21\n  Non-zero exit status: 1\nt3305-notes-fanout.sh                            (Wstat: 256 Tests: 6 Failed: 1)\n  Failed test:  6\n  Non-zero exit status: 1\nt3402-rebase-merge.sh                            (Wstat: 256 Tests: 12 Failed: 4)\n  Failed tests:  8, 10-12\n  Non-zero exit status: 1\nt3311-notes-merge-fanout.sh                      (Wstat: 256 Tests: 24 Failed: 8)\n  Failed tests:  2, 7, 9, 12, 14, 16, 18, 23\n  Non-zero exit status: 1\nt3404-rebase-interactive.sh                      (Wstat: 256 Tests: 98 Failed: 11)\n  Failed tests:  15, 68-71, 89-92, 94, 96\n  Non-zero exit status: 1\nt3421-rebase-topology-linear.sh                  (Wstat: 256 Tests: 76 Failed: 4)\n  Failed tests:  73-76\n  Non-zero exit status: 1\nt3503-cherry-pick-root.sh                        (Wstat: 256 Tests: 6 Failed: 3)\n  Failed tests:  3-5\n  Non-zero exit status: 1\nt3501-revert-cherry-pick.sh                      (Wstat: 256 Tests: 12 Failed: 2)\n  Failed tests:  8-9\n  Non-zero exit status: 1\nt3506-cherry-pick-ff.sh                          (Wstat: 256 Tests: 11 Failed: 2)\n  Failed tests:  10-11\n  Non-zero exit status: 1\nt3702-add-edit.sh                                (Wstat: 256 Tests: 3 Failed: 1)\n  Failed test:  2\n  Non-zero exit status: 1\nt3701-add-interactive.sh                         (Wstat: 256 Tests: 46 Failed: 9)\n  Failed tests:  2, 4-5, 9, 21, 29, 31, 33, 37\n  Non-zero exit status: 1\nt3905-stash-include-untracked.sh                 (Wstat: 256 Tests: 18 Failed: 2)\n  Failed tests:  3, 8\n  Non-zero exit status: 1\nt4002-diff-basic.sh                              (Wstat: 256 Tests: 63 Failed: 14)\n  Failed tests:  45-58\n  Non-zero exit status: 1\nt4007-rename-3.sh                                (Wstat: 256 Tests: 8 Failed: 5)\n  Failed tests:  3-5, 7-8\n  Non-zero exit status: 1\nt4008-diff-break-rewrite.sh                      (Wstat: 256 Tests: 14 Failed: 9)\n  Failed tests:  3-4, 6-7, 9-11, 13-14\n  Non-zero exit status: 1\nt4010-diff-pathspec.sh                           (Wstat: 256 Tests: 17 Failed: 6)\n  Failed tests:  3-6, 9-10\n  Non-zero exit status: 1\nt4011-diff-symlink.sh                            (Wstat: 256 Tests: 8 Failed: 1)\n  Failed test:  8\n  Non-zero exit status: 1\nt4013-diff-various.sh                            (Wstat: 256 Tests: 172 Failed: 131)\n  Failed tests:  6-13, 15, 17, 19, 21, 26-30, 32, 34, 36\n                38, 40, 42-46, 51-79, 81-134, 136-139, 142-148\n                153-154, 158-162, 169-171\n  Non-zero exit status: 1\nt4014-format-patch.sh                            (Wstat: 256 Tests: 142 Failed: 2)\n  Failed tests:  53-54\n  Non-zero exit status: 1\nt4015-diff-whitespace.sh                         (Wstat: 256 Tests: 77 Failed: 11)\n  Failed tests:  1-2, 53, 57-58, 60-65\n  Non-zero exit status: 1\nt4020-diff-external.sh                           (Wstat: 256 Tests: 25 Failed: 2)\n  Failed tests:  18, 23\n  Non-zero exit status: 1\nt4022-diff-rewrite.sh                            (Wstat: 256 Tests: 11 Failed: 2)\n  Failed tests:  3-4\n  Non-zero exit status: 1\nt4029-diff-trailing-space.sh                     (Wstat: 256 Tests: 1 Failed: 1)\n  Failed test:  1\n  Non-zero exit status: 1\nt4030-diff-textconv.sh                           (Wstat: 256 Tests: 19 Failed: 1)\n  Failed test:  19\n  Non-zero exit status: 1\nt4033-diff-patience.sh                           (Wstat: 256 Tests: 4 Failed: 2)\n  Failed tests:  2, 4\n  Non-zero exit status: 1\nt4034-diff-words.sh                              (Wstat: 256 Tests: 41 Failed: 33)\n  Failed tests:  3-8, 10-11, 13-15, 17-20, 22-39\n  Non-zero exit status: 1\nt4039-diff-assume-unchanged.sh                   (Wstat: 256 Tests: 4 Failed: 1)\n  Failed test:  2\n  Non-zero exit status: 1\nt4042-diff-textconv-caching.sh                   (Wstat: 256 Tests: 7 Failed: 5)\n  Failed tests:  2-6\n  Non-zero exit status: 1\nt4044-diff-index-unique-abbrev.sh                (Wstat: 256 Tests: 2 Failed: 2)\n  Failed tests:  1-2\n  Non-zero exit status: 1\nt4045-diff-relative.sh                           (Wstat: 256 Tests: 17 Failed: 8)\n  Failed tests:  2-5, 14-17\n  Non-zero exit status: 1\nt4048-diff-combined-binary.sh                    (Wstat: 256 Tests: 14 Failed: 11)\n  Failed tests:  2-4, 6-8, 10-14\n  Non-zero exit status: 1\nt4050-diff-histogram.sh                          (Wstat: 256 Tests: 3 Failed: 2)\n  Failed tests:  1, 3\n  Non-zero exit status: 1\nt4054-diff-bogus-tree.sh                         (Wstat: 256 Tests: 14 Failed: 2)\n  Failed tests:  3-4\n  Non-zero exit status: 1\nt4060-diff-submodule-option-diff-format.sh       (Wstat: 256 Tests: 47 Failed: 28)\n  Failed tests:  2-3, 6-7, 10-14, 16, 22-23, 26-30, 32-34\n                36-37, 39-41, 43, 46-47\n  Non-zero exit status: 1\nt3903-stash.sh                                   (Wstat: 256 Tests: 90 Failed: 18)\n  Failed tests:  2, 14, 20-21, 24-29, 32, 35, 38, 46, 48\n                57, 62-63\n  Non-zero exit status: 1\nt4151-am-abort.sh                                (Wstat: 256 Tests: 18 Failed: 7)\n  Failed tests:  12-18\n  Non-zero exit status: 1\nt4202-log.sh                                     (Wstat: 256 Tests: 80 Failed: 11)\n  Failed tests:  8, 19-23, 62-63, 78-80\n  Non-zero exit status: 1\nt4205-log-pretty-formats.sh                      (Wstat: 256 Tests: 57 Failed: 2)\n  Failed tests:  51-52\n  Non-zero exit status: 1\nt4208-log-magic-pathspec.sh                      (Wstat: 256 Tests: 17 Failed: 1)\n  Failed test:  9\n  Non-zero exit status: 1\nt4211-line-log.sh                                (Wstat: 256 Tests: 37 Failed: 17)\n  Failed tests:  2-18\n  Non-zero exit status: 1\nt4300-merge-tree.sh                              (Wstat: 256 Tests: 20 Failed: 12)\n  Failed tests:  2, 5, 7, 9-10, 12, 14-15, 17-20\n  Non-zero exit status: 1\nt5150-request-pull.sh                            (Wstat: 256 Tests: 7 Failed: 6)\n  Failed tests:  1, 3-7\n  Non-zero exit status: 1\nt5300-pack-object.sh                             (Wstat: 256 Tests: 38 Failed: 3)\n  Failed tests:  36-38\n  Non-zero exit status: 1\nt5306-pack-nobase.sh                             (Wstat: 256 Tests: 4 Failed: 1)\n  Failed test:  3\n  Non-zero exit status: 1\nt5308-pack-detect-duplicates.sh                  (Wstat: 256 Tests: 5 Failed: 3)\n  Failed tests:  1-2, 4\n  Non-zero exit status: 1\nt5309-pack-delta-cycles.sh                       (Wstat: 256 Tests: 6 Failed: 2)\n  Failed tests:  1-2\n  Non-zero exit status: 1\nt5313-pack-bounds-checks.sh                      (Wstat: 256 Tests: 8 Failed: 1)\n  Failed test:  7\n  Non-zero exit status: 1\nt5504-fetch-receive-strict.sh                    (Wstat: 256 Tests: 12 Failed: 2)\n  Failed tests:  10-11\n  Non-zero exit status: 1\nt5512-ls-remote.sh                               (Wstat: 256 Tests: 26 Failed: 3)\n  Failed tests:  21-22, 24\n  Non-zero exit status: 1\nt5515-fetch-merge-logic.sh                       (Wstat: 256 Tests: 65 Failed: 64)\n  Failed tests:  2-65\n  Non-zero exit status: 1\nt5521-pull-options.sh                            (Wstat: 256 Tests: 18 Failed: 10)\n  Failed tests:  2-11\n  Non-zero exit status: 1\nt5516-fetch-push.sh                              (Wstat: 256 Tests: 89 Failed: 9)\n  Failed tests:  34, 78-83, 88-89\n  Non-zero exit status: 1\nt5520-pull.sh                                    (Wstat: 256 Tests: 64 Failed: 4)\n  Failed tests:  2-3, 6, 58\n  Non-zero exit status: 1\nt6000-rev-list-misc.sh                           (Wstat: 256 Tests: 10 Failed: 1)\n  Failed test:  8\n  Non-zero exit status: 1\nt6024-recursive-merge.sh                         (Wstat: 256 Tests: 6 Failed: 3)\n  Failed tests:  3-5\n  Non-zero exit status: 1\nt6300-for-each-ref.sh                            (Wstat: 256 Tests: 205 Failed: 10)\n  Failed tests:  83, 189-196, 199\n  Non-zero exit status: 1\nt6500-gc.sh                                      (Wstat: 256 Tests: 8 Failed: 3)\n  Failed tests:  6-8\n  Non-zero exit status: 1\nt6030-bisect-porcelain.sh                        (Wstat: 256 Tests: 71 Failed: 18)\n  Failed tests:  19, 22-27, 30, 32, 37, 41, 45-47, 53-56\n  Non-zero exit status: 1\nt7102-reset.sh                                   (Wstat: 256 Tests: 30 Failed: 13)\n  Failed tests:  11, 13-19, 23-24, 26-28\n  Non-zero exit status: 1\nt7106-reset-unborn-branch.sh                     (Wstat: 256 Tests: 7 Failed: 4)\n  Failed tests:  2, 4-5, 7\n  Non-zero exit status: 1\nt7003-filter-branch.sh                           (Wstat: 256 Tests: 42 Failed: 2)\n  Failed tests:  34-35\n  Non-zero exit status: 1\nt7201-co.sh                                      (Wstat: 256 Tests: 36 Failed: 33)\n  Failed tests:  2, 4-25, 27-36\n  Non-zero exit status: 1\nt7063-status-untracked-cache.sh                  (Wstat: 256 Tests: 43 Failed: 14)\n  Failed tests:  5, 7, 9, 11, 13-14, 16-17, 19, 22, 26, 30\n                36, 40\n  Non-zero exit status: 1\nt7112-reset-submodule.sh                         (Wstat: 256 Tests: 78 Failed: 15)\n  Failed tests:  1-2, 4-7, 19-20, 22-25, 30, 34-35\n  Non-zero exit status: 1\nt7507-commit-verbose.sh                          (Wstat: 256 Tests: 44 Failed: 1)\n  Failed test:  2\n  Non-zero exit status: 1\nt7506-status-submodule.sh                        (Wstat: 256 Tests: 40 Failed: 2)\n  Failed tests:  31-32\n  Non-zero exit status: 1\nt7600-merge.sh                                   (Wstat: 256 Tests: 56 Failed: 1)\n  Failed test:  7\n  Non-zero exit status: 1\nt7607-merge-overwrite.sh                         (Wstat: 256 Tests: 18 Failed: 2)\n  Failed tests:  14, 18\n  Non-zero exit status: 1\nt9300-fast-import.sh                             (Wstat: 256 Tests: 183 Failed: 19)\n  Failed tests:  17, 32, 35, 50, 59, 61-66, 68-69, 71-72\n                74, 174, 176, 178\n  Non-zero exit status: 1\nt9500-gitweb-standalone-no-errors.sh             (Wstat: 256 Tests: 118 Failed: 1)\n  Failed test:  118\n  Non-zero exit status: 1\nt9903-bash-prompt.sh                             (Wstat: 256 Tests: 66 Failed: 53)\n  Failed tests:  4, 6-10, 14-34, 36, 39-62, 65\n  Non-zero exit status: 1\n\nbrian m. carlson (2):\n  Base test implementation for short BLAKE2b support\n  Switch default hash algorithm to short BLAKE2b for testing\n\n Makefile                 |  3 +++\n cache.h                  | 16 ++++++++++++++\n hash.h                   |  9 +++++++-\n repository.c             |  2 +-\n setup.c                  |  2 +-\n sha1_file.c              | 33 ++++++++++++++++++++++++++++\n t/helper/test-sblake2b.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++\n t/test-lib.sh            |  4 ++--\n 8 files changed, 120 insertions(+), 5 deletions(-)\n create mode 100644 t/helper/test-sblake2b.c\n\n"},{"id":"337625","messageId":"20180128170639.216397-3-sandals@crustytoothpaste.net","threadId":"47707","inReplyTo":"20180128170639.216397-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 2/2] Switch default hash algorithm to short BLAKE2b for testing","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-01-28T17:06:39Z","receivedAt":"2018-01-28T17:07:04Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Set the default hash algorithm to short (160-bit) BLAKE2b so we can\nidentify tests that rely on a given hash algorithm.\n\nThis is a test commit and should not be used in production.  Widespread\ntest breakage occurs when using this commit.\n---\n repository.c  | 2 +-\n setup.c       | 2 +-\n t/test-lib.sh | 4 ++--\n 3 files changed, 4 insertions(+), 4 deletions(-)\n\ndiff --git a/repository.c b/repository.c\nindex f66fcb1342..95c0ff74d6 100644\n--- a/repository.c\n+++ b/repository.c\n@@ -5,7 +5,7 @@\n \n /* The main repository */\n static struct repository the_repo = {\n-\tNULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SHA1], 0, 0\n+\tNULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, &the_index, &hash_algos[GIT_HASH_SBLAKE2], 0, 0\n };\n struct repository *the_repository = &the_repo;\n \ndiff --git a/setup.c b/setup.c\nindex 8cc34186ce..72eaee9c4e 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -488,7 +488,7 @@ int read_repository_format(struct repository_format *format, const char *path)\n \tmemset(format, 0, sizeof(*format));\n \tformat->version = -1;\n \tformat->is_bare = -1;\n-\tformat->hash_algo = GIT_HASH_SHA1;\n+\tformat->hash_algo = GIT_HASH_SBLAKE2;\n \tstring_list_init(&format->unknown_extensions, 1);\n \tgit_config_from_file(check_repo_format, path, format);\n \treturn format->version;\ndiff --git a/t/test-lib.sh b/t/test-lib.sh\nindex 9a0a21f49a..cd257040ad 100644\n--- a/t/test-lib.sh\n+++ b/t/test-lib.sh\n@@ -183,8 +183,8 @@ _x40=\"$_x35$_x05\"\n # Zero SHA-1\n _z40=0000000000000000000000000000000000000000\n \n-EMPTY_TREE=4b825dc642cb6eb9a060e54bf8d69288fbee4904\n-EMPTY_BLOB=e69de29bb2d1d6434b8b29ae775ad8c2e48c5391\n+EMPTY_TREE=f44422a644bfa5212387098f253a1e89eba94548\n+EMPTY_BLOB=a706650a477f63b9b00eba41272bf36ef5a7dfa2\n \n # Line feed\n LF='\n"},{"id":"337626","messageId":"20180128170639.216397-2-sandals@crustytoothpaste.net","threadId":"47707","inReplyTo":"20180128170639.216397-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 1/2] Base test implementation for short BLAKE2b support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-01-28T17:06:38Z","receivedAt":"2018-01-28T17:07:07Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Test the implementation of a second hash algorithm with \"short\"\n(160-bit) BLAKE2b support.  Simply add an additional algorithm to the\ncodebase without changing the use of any hash function and add a test\nhelper.\n\nBLAKE2b was chosen simply because it provides a readily accessible hash\nalgorithm with arbitrary length support.  A 160-bit (20-byte) hash was\nchosen to allow identification of tests that depend on the hash\nalgorithm in use while not causing buffer overflows, since parts of the\ncodebase still assume a 20-byte hash.\n\nThis is a preliminary commit for test purposes only and should not be\nused in production in any way.\n---\n Makefile                 |  3 +++\n cache.h                  | 16 ++++++++++++++\n hash.h                   |  9 +++++++-\n sha1_file.c              | 33 ++++++++++++++++++++++++++++\n t/helper/test-sblake2b.c | 56 ++++++++++++++++++++++++++++++++++++++++++++++++\n 5 files changed, 116 insertions(+), 1 deletion(-)\n create mode 100644 t/helper/test-sblake2b.c\n\ndiff --git a/Makefile b/Makefile\nindex 1a9b23b679..d46247586b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -677,6 +677,7 @@ TEST_PROGRAMS_NEED_X += test-ref-store\n TEST_PROGRAMS_NEED_X += test-regex\n TEST_PROGRAMS_NEED_X += test-revision-walking\n TEST_PROGRAMS_NEED_X += test-run-command\n+TEST_PROGRAMS_NEED_X += test-sblake2b\n TEST_PROGRAMS_NEED_X += test-scrap-cache-tree\n TEST_PROGRAMS_NEED_X += test-sha1\n TEST_PROGRAMS_NEED_X += test-sha1-array\n@@ -1539,6 +1540,8 @@ endif\n endif\n endif\n \n+EXTLIBS += -lb2\n+\n ifdef SHA1_MAX_BLOCK_SIZE\n \tLIB_OBJS += compat/sha1-chunked.o\n \tBASIC_CFLAGS += -DSHA1_MAX_BLOCK_SIZE=\"$(SHA1_MAX_BLOCK_SIZE)\"\ndiff --git a/cache.h b/cache.h\nindex bfde6f757a..638d832350 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -45,6 +45,10 @@ unsigned long git_deflate_bound(git_zstream *, unsigned long);\n #define GIT_SHA1_RAWSZ 20\n #define GIT_SHA1_HEXSZ (2 * GIT_SHA1_RAWSZ)\n \n+/* The length in bytes and in hex digits of an object name (short BLAKE2b value). */\n+#define GIT_SBLAKE2B_RAWSZ 20\n+#define GIT_SBLAKE2B_HEXSZ (2 * GIT_SBLAKE2B_RAWSZ)\n+\n /* The length in byte and in hex digits of the largest possible hash value. */\n #define GIT_MAX_RAWSZ GIT_SHA1_RAWSZ\n #define GIT_MAX_HEXSZ GIT_SHA1_HEXSZ\n@@ -1013,7 +1017,13 @@ static inline void oidclr(struct object_id *oid)\n #define EMPTY_TREE_SHA1_BIN_LITERAL \\\n \t \"\\x4b\\x82\\x5d\\xc6\\x42\\xcb\\x6e\\xb9\\xa0\\x60\" \\\n \t \"\\xe5\\x4b\\xf8\\xd6\\x92\\x88\\xfb\\xee\\x49\\x04\"\n+#define EMPTY_TREE_SBLAKE2B_HEX \\\n+\t\"f44422a644bfa5212387098f253a1e89eba94548\"\n+#define EMPTY_TREE_SBLAKE2B_BIN_LITERAL \\\n+\t\"\\xf4\\x44\\x22\\xa6\\x44\\xbf\\xa5\\x21\\x23\\x87\" \\\n+\t\"\\x09\\x8f\\x25\\x3a\\x1e\\x89\\xeb\\xa9\\x45\\x48\"\n extern const struct object_id empty_tree_oid;\n+const struct object_id empty_tree_oid_sblake2b;\n #define EMPTY_TREE_SHA1_BIN (empty_tree_oid.hash)\n \n #define EMPTY_BLOB_SHA1_HEX \\\n@@ -1021,7 +1031,13 @@ extern const struct object_id empty_tree_oid;\n #define EMPTY_BLOB_SHA1_BIN_LITERAL \\\n \t\"\\xe6\\x9d\\xe2\\x9b\\xb2\\xd1\\xd6\\x43\\x4b\\x8b\" \\\n \t\"\\x29\\xae\\x77\\x5a\\xd8\\xc2\\xe4\\x8c\\x53\\x91\"\n+#define EMPTY_BLOB_SBLAKE2B_HEX \\\n+\t\"a706650a477f63b9b00eba41272bf36ef5a7dfa2\"\n+#define EMPTY_BLOB_SBLAKE2B_BIN_LITERAL \\\n+\t\"\\xa7\\x06\\x65\\x0a\\x47\\x7f\\x63\\xb9\\xb0\\x0e\" \\\n+\t\"\\xba\\x41\\x27\\x2b\\xf3\\x6e\\xf5\\xa7\\xdf\\xa2\"\n extern const struct object_id empty_blob_oid;\n+const struct object_id empty_blob_oid_sblake2b;\n #define EMPTY_BLOB_SHA1_BIN (empty_blob_oid.hash)\n \n \ndiff --git a/hash.h b/hash.h\nindex 365846a6b5..ef510bd51a 100644\n--- a/hash.h\n+++ b/hash.h\n@@ -15,6 +15,8 @@\n #include \"block-sha1/sha1.h\"\n #endif\n \n+#include <blake2.h>\n+\n #ifndef platform_SHA_CTX\n /*\n  * platform's underlying implementation of SHA-1; could be OpenSSL,\n@@ -40,6 +42,8 @@\n #define git_SHA1_Update\t\tgit_SHA1_Update_Chunked\n #endif\n \n+#define git_BLAKE2B_CTX\t\tblake2b_state\n+\n /*\n  * Note that these constants are suitable for indexing the hash_algos array and\n  * comparing against each other, but are otherwise arbitrary, so they should not\n@@ -52,12 +56,15 @@\n #define GIT_HASH_UNKNOWN 0\n /* SHA-1 */\n #define GIT_HASH_SHA1 1\n+/* 20-byte BLAKE2b (\"short\" BLAKE2b) */\n+#define GIT_HASH_SBLAKE2 2\n /* Number of algorithms supported (including unknown). */\n-#define GIT_HASH_NALGOS (GIT_HASH_SHA1 + 1)\n+#define GIT_HASH_NALGOS (GIT_HASH_SBLAKE2 + 1)\n \n /* A suitably aligned type for stack allocations of hash contexts. */\n union git_hash_ctx {\n \tgit_SHA_CTX sha1;\n+\tgit_BLAKE2B_CTX blake2b;\n };\n typedef union git_hash_ctx git_hash_ctx;\n \ndiff --git a/sha1_file.c b/sha1_file.c\nindex d9e2b1f285..5067b22a30 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -38,6 +38,12 @@ const struct object_id empty_tree_oid = {\n const struct object_id empty_blob_oid = {\n \tEMPTY_BLOB_SHA1_BIN_LITERAL\n };\n+const struct object_id empty_tree_oid_sblake2b = {\n+\tEMPTY_TREE_SBLAKE2B_BIN_LITERAL\n+};\n+const struct object_id empty_blob_oid_sblake2b = {\n+\tEMPTY_BLOB_SBLAKE2B_BIN_LITERAL\n+};\n \n static void git_hash_sha1_init(void *ctx)\n {\n@@ -54,6 +60,21 @@ static void git_hash_sha1_final(unsigned char *hash, void *ctx)\n \tgit_SHA1_Final(hash, (git_SHA_CTX *)ctx);\n }\n \n+static void git_hash_sblake2b_init(void *ctx)\n+{\n+\tblake2b_init((git_BLAKE2B_CTX *)ctx, GIT_SBLAKE2B_RAWSZ);\n+}\n+\n+static void git_hash_sblake2b_update(void *ctx, const void *data, size_t len)\n+{\n+\tblake2b_update((git_BLAKE2B_CTX *)ctx, data, len);\n+}\n+\n+static void git_hash_sblake2b_final(unsigned char *hash, void *ctx)\n+{\n+\tblake2b_final((git_BLAKE2B_CTX *)ctx, hash, GIT_SBLAKE2B_RAWSZ);\n+}\n+\n static void git_hash_unknown_init(void *ctx)\n {\n \tdie(\"trying to init unknown hash\");\n@@ -93,6 +114,18 @@ const struct git_hash_algo hash_algos[GIT_HASH_NALGOS] = {\n \t\t&empty_tree_oid,\n \t\t&empty_blob_oid,\n \t},\n+\t{\n+\t\t\"sblake2b\",\n+\t\t/* \"sb2b\", big-endian */\n+\t\t0x73623262,\n+\t\tGIT_SBLAKE2B_RAWSZ,\n+\t\tGIT_SBLAKE2B_HEXSZ,\n+\t\tgit_hash_sblake2b_init,\n+\t\tgit_hash_sblake2b_update,\n+\t\tgit_hash_sblake2b_final,\n+\t\t&empty_tree_oid_sblake2b,\n+\t\t&empty_blob_oid_sblake2b,\n+\t}\n };\n \n /*\ndiff --git a/t/helper/test-sblake2b.c b/t/helper/test-sblake2b.c\nnew file mode 100644\nindex 0000000000..6623aaa4d5\n--- /dev/null\n+++ b/t/helper/test-sblake2b.c\n@@ -0,0 +1,56 @@\n+#include \"cache.h\"\n+\n+int cmd_main(int ac, const char **av)\n+{\n+\tgit_hash_ctx ctx;\n+\tunsigned char hash[GIT_MAX_RAWSZ];\n+\tunsigned bufsz = 8192;\n+\tint binary = 0;\n+\tchar *buffer;\n+\n+\tif (ac == 2) {\n+\t\tif (!strcmp(av[1], \"-b\"))\n+\t\t\tbinary = 1;\n+\t\telse\n+\t\t\tbufsz = strtoul(av[1], NULL, 10) * 1024 * 1024;\n+\t}\n+\n+\tif (!bufsz)\n+\t\tbufsz = 8192;\n+\n+\twhile ((buffer = malloc(bufsz)) == NULL) {\n+\t\tfprintf(stderr, \"bufsz %u is too big, halving...\\n\", bufsz);\n+\t\tbufsz /= 2;\n+\t\tif (bufsz < 1024)\n+\t\t\tdie(\"OOPS\");\n+\t}\n+\n+\thash_algos[GIT_HASH_SBLAKE2].init_fn(&ctx);\n+\n+\twhile (1) {\n+\t\tssize_t sz, this_sz;\n+\t\tchar *cp = buffer;\n+\t\tunsigned room = bufsz;\n+\t\tthis_sz = 0;\n+\t\twhile (room) {\n+\t\t\tsz = xread(0, cp, room);\n+\t\t\tif (sz == 0)\n+\t\t\t\tbreak;\n+\t\t\tif (sz < 0)\n+\t\t\t\tdie_errno(\"test-sha1\");\n+\t\t\tthis_sz += sz;\n+\t\t\tcp += sz;\n+\t\t\troom -= sz;\n+\t\t}\n+\t\tif (this_sz == 0)\n+\t\t\tbreak;\n+\t\thash_algos[GIT_HASH_SBLAKE2].update_fn(&ctx, buffer, this_sz);\n+\t}\n+\thash_algos[GIT_HASH_SBLAKE2].final_fn(hash, &ctx);\n+\n+\tif (binary)\n+\t\tfwrite(hash, 1, hash_algos[GIT_HASH_SBLAKE2].rawsz, stdout);\n+\telse\n+\t\tputs(sha1_to_hex(hash));\n+\texit(0);\n+}\n"},{"id":"337627","messageId":"20180128183239.11237-1-szeder.dev@gmail.com","threadId":"47707","inReplyTo":"20180128170639.216397-1-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 0/2] alternate hash test","fromName":"SZEDER Gábor","fromEmail":"szeder.dev@gmail.com","sentAt":"2018-01-28T18:32:39Z","receivedAt":"2018-01-28T18:32:51Z","isPatch":true,"sender":{"key":"szeder.dev@gmail.com","avatar":"https://avatars.githubusercontent.com/u/116324?v=4"},"body":"> This series wires up an alternate hash implementation, namely\n> BLAKE2b-160.  The goal is to allow us to identify tests which rely on\n> the hash algorithm in use so that we can fix those tests.\n\n> t9903-bash-prompt.sh                             (Wstat: 256 Tests: 66 Failed: 53)\n>   Failed tests:  4, 6-10, 14-34, 36, 39-62, 65\n>   Non-zero exit status: 1\n\nI didn't recall seeing hardcoded SHA-1s in the bash prompt tests, so\nwent to have a look.  And indeed, these failures are not the test's\nfault, but the result of a segfault in 'git checkout' while trying to\nreturn from an orphan branch in test 4.  The rest of the failures are\nmostly fallout caused by the wrong branch being checked out or the\nleftover index.lock.\n\n$ ./t9903-bash-prompt.sh -v -i\n<...>\nexpecting success: \n      printf \" (unborn)\" >expected &&\n      git checkout --orphan unborn &&\n      test_when_finished \"git checkout master\" &&\n      __git_ps1 >\"$actual\" &&\n      test_cmp expected \"$actual\"\n\nSwitched to a new branch 'unborn'\n./test-lib.sh: line 609: 29342 Segmentation fault      git checkout master\nnot ok 4 - prompt - unborn branch\n\n\n$ gdb --args ../../git checkout master\n<...>\nProgram received signal SIGSEGV, Segmentation fault.\n0x000000000041b636 in merge_working_tree (opts=0x7fffffffd200, \n    old=0x7fffffffd0d0, new=0x7fffffffd1e0,\nwriteout_error=0x7fffffffd0c0)\n    at builtin/checkout.c:525\n525                     init_tree_desc(&trees[0], tree->buffer,\ntree->size);\n(gdb) p tree \n$1 = (struct tree *) 0x0\n\n\nThe root cause is that patch 2/2 misses places where EMPTY_TREE_SHA1_*\nconstants should have been replaced with their EMPTY_TREE_SBLAKE2B_*\ncounterparts.\n\n"},{"id":"337628","messageId":"CACBZZX4TRuPV8j9XV=8gAMirQNTpEJBO8bmE9KBnnmoJ7b_m1g@mail.gmail.com","threadId":"47707","inReplyTo":"20180128170639.216397-1-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 0/2] alternate hash test","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2018-01-28T18:58:10Z","receivedAt":"2018-01-28T18:58:42Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sun, Jan 28, 2018 at 6:06 PM, brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n> For this test, I picked BLAKE2b-160 for a couple reasons:\n> * Debian ships a libb2-1 package which can be used easily (in other\n>   words, I was lazy and didn't want to add a crypto implementation just\n>   for test purposes);\n> [...]\n> This isn't an endorsement for or against any particular algorithm\n> choice, just an artifact of the tools that were easily available to me.\n> Provoking discussion of which hash to pick for NewHash is explicitly\n> *not* a goal for this series.  I'm only interested in the ability to\n> identify and fix tests.\n\nIf the goal is to smoke out hardcoded SHA1s in tests, isn't it easier\nto instrument SHA-1 (e.g. our blk_sha1 copy, or our wrappers) to\npretend that whenever we ask for the hash for STRING to pretend we\nasked for SOME_PREFIX + STRING?\n\nSuch an approach would have the advantage of being more portable\n(easier to run these mock test), and also that if we ever move to\nNewHash we could still test for this, we'd just always set the prefix\nto compilation time(), and could thus guarantee that the hashes would\nchange every time git was built.\n"},{"id":"337629","messageId":"20180128191605.GI431130@genre.crustytoothpaste.net","threadId":"47707","inReplyTo":"CACBZZX4TRuPV8j9XV=8gAMirQNTpEJBO8bmE9KBnnmoJ7b_m1g@mail.gmail.com","subject":"Re: [RFC PATCH 0/2] alternate hash test","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-01-28T19:16:06Z","receivedAt":"2018-01-28T19:16:17Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Jan 28, 2018 at 07:58:10PM +0100, Ævar Arnfjörð Bjarmason wrote:\n> On Sun, Jan 28, 2018 at 6:06 PM, brian m. carlson\n> <sandals@crustytoothpaste.net> wrote:\n> If the goal is to smoke out hardcoded SHA1s in tests, isn't it easier\n> to instrument SHA-1 (e.g. our blk_sha1 copy, or our wrappers) to\n> pretend that whenever we ask for the hash for STRING to pretend we\n> asked for SOME_PREFIX + STRING?\n> \n> Such an approach would have the advantage of being more portable\n> (easier to run these mock test), and also that if we ever move to\n> NewHash we could still test for this, we'd just always set the prefix\n> to compilation time(), and could thus guarantee that the hashes would\n> change every time git was built.\n\nThat's certainly a possibility.  We could simply call the update\nfunction from the init function and prepend a NUL byte or something like\nthat, which would definitely produce different results.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\nhttps://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"337813","messageId":"xmqq4ln3t5o7.fsf@gitster-ct.c.googlers.com","threadId":"47707","inReplyTo":"20180128170639.216397-1-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 0/2] alternate hash test","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2018-01-30T21:09:28Z","receivedAt":"2018-01-30T21:09:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n\n> This series wires up an alternate hash implementation, namely\n> BLAKE2b-160.  The goal is to allow us to identify tests which rely on\n> the hash algorithm in use so that we can fix those tests.\n\nYay.\n\n> Provoking discussion of which hash to pick for NewHash is explicitly\n> *not* a goal for this series.  I'm only interested in the ability to\n> identify and fix tests.\n\nDouble yay.\n"},{"id":"337895","messageId":"CAGZ79kbvWPDKPbe2pzHgvfKS+kToVmFtijsmshzX30uuk3pqkA@mail.gmail.com","threadId":"47707","inReplyTo":"20180128170639.216397-1-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 0/2] alternate hash test","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2018-01-31T01:04:52Z","receivedAt":"2018-01-31T01:04:59Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Sun, Jan 28, 2018 at 9:06 AM, brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n> This series wires up an alternate hash implementation, namely\n> BLAKE2b-160.  The goal is to allow us to identify tests which rely on\n> the hash algorithm in use so that we can fix those tests.\n>\n> For this test, I picked BLAKE2b-160 for a couple reasons:\n> * Debian ships a libb2-1 package which can be used easily (in other\n>   words, I was lazy and didn't want to add a crypto implementation just\n>   for test purposes);\n> * The API of the libb2 package easily allows arbitrary hash lengths, so\n>   I didn't have to manage truncation myself;\n> * Our codebase isn't yet ready for a hash function larger than 20 bytes,\n>   as there's still more work to do on the object_id conversions.\n>\n> This isn't an endorsement for or against any particular algorithm\n> choice, just an artifact of the tools that were easily available to me.\n> Provoking discussion of which hash to pick for NewHash is explicitly\n> *not* a goal for this series.  I'm only interested in the ability to\n> identify and fix tests.\n>\n> The first patch does no feature detection and just assumes you have\n> libb2 installed.  For obvious reasons, this series is not meant for\n> production use.\n\nThanks for writing this, I chose a slightly different approach at\nhttps://public-inbox.org/git/20170728171817.21458-2-sbeller@google.com/\nwhich might be quicker for local testing.\n\nThanks for bringing this discussion back on the list,\nStefan\n"},{"id":"337988","messageId":"20180201012501.GQ431130@genre.crustytoothpaste.net","threadId":"47707","inReplyTo":"CAGZ79kbvWPDKPbe2pzHgvfKS+kToVmFtijsmshzX30uuk3pqkA@mail.gmail.com","subject":"Re: [RFC PATCH 0/2] alternate hash test","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-02-01T01:25:01Z","receivedAt":"2018-02-01T01:25:13Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Tue, Jan 30, 2018 at 05:04:52PM -0800, Stefan Beller wrote:\n> Thanks for writing this, I chose a slightly different approach at\n> https://public-inbox.org/git/20170728171817.21458-2-sbeller@google.com/\n> which might be quicker for local testing.\n\nI'm actually going to rework the patches to be much more similar to\nthat.  I've been convinced that's an easier approach to the series, and\nI like the simplicity.\n\nMy goal is not only to identify failing tests, but to exercise the hash\nalgorithm logic so that we find any hidden dependencies outside of that\ncode.  For example, I noticed some of our Perl scripts have hard-coded\nempty-tree object IDs in them, which obviously will need to be fixed.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\nhttps://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"}]}