git/list[1] front-page[2] threads[3] people[4] search[5] about
 

Re: [RFC PATCH 09/12] Add a base implementation of SHA-256 support

From
brian m. carlson <sandals@crustytoothpaste.net>
Date
Aug 29, 2018, 23:55 UTC
Message-ID
<20180829235504.GH432229@genre.crustytoothpaste.net>
In-Reply-To
<875zztecx3.fsf@evledraar.gmail.com>
On Wed, Aug 29, 2018 at 11:32:08AM +0200, Ævar Arnfjörð Bjarmason wrote:
Show 39 quoted lines
> 
> On Wed, Aug 29 2018, brian m. carlson wrote:
> 
> > SHA-1 is weak and we need to transition to a new hash function.  For
> > some time, we have referred to this new function as NewHash.
> >
> > The selection criteria for NewHash specify that it should (a) be 256
> > bits in length, (b) have high quality implementations available, (c)
> > should match Git's needs in terms of security, and (d) ideally, be fast
> > to compute.
> >
> > SHA-256 has a variety of high quality implementations across various
> > libraries.  It is implemented by every cryptographic library we support
> > and is available on every platform and in almost every programming
> > language.  It is often highly optimized, since it is commonly used in
> > TLS and elsewhere.  Additionally, there are various command line
> > utilities that implement it, which is useful for educational and testing
> > purposes.
> >
> > SHA-256 is presently considered secure and has received a reasonable
> > amount of cryptanalysis in the literature.  It is, admittedly, not
> > resistant to length extension attacks, but Git object storage is immune
> > to those due to the length field at the beginning.
> >
> > SHA-256 is somewhat slower to compute than SHA-1 in software.  However,
> > since our default SHA-1 implementation is collision-detecting, a
> > reasonable cryptographic library implementation of SHA-256 will actually
> > be faster than SHA-256.  In addition, modern ARM and AMD processors (and
> > some Intel processors) contain instructions for implementing SHA-256 in
> > hardware, making it the fastest possible option.
> >
> > There are other reasons to select SHA-256.  With signed commits and
> > tags, it's possible to use SHA-256 for signatures and therefore have to
> > rely on only one hash algorithm for security.
> 
> None of this is wrong, but I think this would be better off as a simple
> "See Documentation/technical/hash-function-transition.txt for why we're
> switching to SHA-256", and to the extent that something is said here
> that isn't said there it could be a patch to amend that document.

I can certainly shorten this somewhat. I wrote this back when there wasn't a consensus on hash algorithm and Junio was going to leave it to me to make a decision. I was therefore obligated to provide a coherent rationale for that decision.

Show 10 quoted lines
> > Add a basic implementation of SHA-256 based off libtomcrypt, which is in
> > the public domain.  Optimize it and tidy it somewhat.
> 
> For future changes & maintenance of this, let's do that in two
> steps. One where we add the upstream code as-is, and another where the
> tidying / cleanup / git specific stuff is wired, which makes it easy to
> audit upstream as-is v.s. our changes in isolation. Also in the first of
> those commits, say in the commit message "add a [libtomcrypt] copy from
> such-and-such a URL at such-and-such a version", so that it's easy to
> reproduce the import & find out how to re-update it.

Doing what you suggest basically means importing a large amount of libtomcrypt into our codebase, since there are a large number of reused macros all over libtomcrypt (such as for processing a generic hash and for memcpy).

This isn't surprising for a general purpose crypto library, but I did a decent amount to change it, condense it into a small number of files, and make it meet our code standards. The phrase "somewhat" may have been an understatement.

This is also why I added tests: because I'm human and making a small change in a crypto library can result in wrong output very quickly.

Show 8 quoted lines
> Is this something we see ourselves perma-forking? Or as with sha1dc are
> we likely to pull in upstream changes from time-to-time?SHA256 obiously
> isn't under active development, but there's been some churn in the
> upstream code since it was added, and if you're doing some optimizing /
> tidying that's presumably something upstream could benefit from as well,
> as well as just us being nicer open source citizens feeding
> e.g. portability fixes to upstream (since git tends to get ported a
> lot).

This is a permafork. We need a basic SHA-256 implementation, and this one was faster than the one I'd written some time ago. Similarly to the block-sha1 implementation, I see this as something that we'll be shipping forever with little updating.

I expect with the amount of changes we're making, they're unlikely to want our code. Also, any changes to our code would be under the GPLv2, which would be unappealing to a public domain library.

-- 
brian m. carlson: Houston, Texas, US
OpenPGP: https://keybase.io/bk2204
Previous: Ævar Arnfjörð BjarmasonNext: Derrick Stolee
Message 14 of 24 in “Base SHA-256 algorithm implementation”
  1. 00/12 Base SHA-256 algorithm implementationbrian m. carlson, Aug 29, 2018
  2. 01/12 sha1-file: rename algorithm to "sha1"brian m. carlson, Aug 29, 2018
  3. 03/12 hex: introduce functions to print arbitrary hashesbrian m. carlson, Aug 29, 2018
  4. 04/12 t: add basic tests for our SHA-1 implementationbrian m. carlson, Aug 29, 2018
  5. 05/12 t: make the sha1 test-tool helper genericbrian m. carlson, Aug 29, 2018
  6. 07/12 t/helper: add a test helper to compute hash speedbrian m. carlson, Aug 29, 2018
  7. 06/12 sha1-file: add a constant for hash block sizebrian m. carlson, Aug 29, 2018
  8. 08/12 commit-graph: convert to using the_hash_algobrian m. carlson, Aug 29, 2018
  9. Derrick StoleeAug 29, 2018
  10. brian m. carlsonAug 30, 2018
  11. brian m. carlsonSep 3, 2018
  12. 09/12 Add a base implementation of SHA-256 supportbrian m. carlson, Aug 29, 2018
  13. Ævar Arnfjörð BjarmasonAug 29, 2018
  14. brian m. carlsonAug 29, 2018
  15. Derrick StoleeAug 29, 2018
  16. 10/12 sha256: add an SHA-256 implementation using libgcryptbrian m. carlson, Aug 29, 2018
  17. Ævar Arnfjörð BjarmasonAug 29, 2018
  18. brian m. carlsonAug 29, 2018
  19. 11/12 hash: add an SHA-256 implementation using OpenSSLbrian m. carlson, Aug 29, 2018
  20. 12/12 commit-graph: specify OID version for SHA-256brian m. carlson, Aug 29, 2018
  21. 02/12 sha1-file: provide functions to look up hash algorithmsbrian m. carlson, Aug 29, 2018
  22. Ævar Arnfjörð BjarmasonAug 29, 2018
  23. brian m. carlsonAug 30, 2018
  24. brian m. carlsonAug 30, 2018

Read the whole thread, see it on lore, or plain text.

$ cat FOOTERMessages come from the public archive at lore.kernel.org/git, fetched every hour. The front page is chosen and written each morning by an AI editor and can be wrong; the threads themselves are the record. About and API. For agents: an MCP server at https://gitlist.dev/mcp, and any thread, story or person page as Markdown by adding .md to its URL (or sending Accept: text/markdown). Details in /llms.txt.