Re: RFC v3: Another proposed hash function transition plan

From: Philip Oakley <philipoakley@iee.org>
Date: Sep 15, 2017, 20:42 UTC
Message-ID: <12CC12FA3A034D6A9B91695BE1A04641@PhilipOakley>
In-Reply-To: <20170914184915.GC78683@aiede.mtv.corp.google.com>

Hi Jonathan,

"Jonathan Nieder" <jrnieder@gmail.com> wrote;

Show 62 quoted lines

> Johannes Schindelin wrote:
>> On Wed, 13 Sep 2017, Jonathan Nieder wrote:
>
>>> As a side note, I am probably misreading, but I found this set of
>>> paragraphs a bit condescending.  It sounds to me like you are saying
>>> "You are making the wrong choice of hash function and everything else
>>> you are describing is irrelevant when compared to that monumental
>>> mistake.  Please stop working on things I don't consider important".
>>> With that reading it is quite demotivating to read.
>>
>> I am sorry you read it that way. I did not feel condescending when I 
>> wrote
>> that mail, I felt annoyed by the side track, and anxious. In my mind, the
>> transition is too important for side tracking, and I worry that we are 
>> not
>> fast enough (imagine what would happen if a better attack was discovered
>> that is not as easily detected as the one we know about?).
>
> Thanks for clarifying.  That makes sense.
>
> [...]
>> As to *my* opinion: after reading https://goo.gl/gh2Mzc (is it really
>> correct that its last update has been on March 6th?), my only concern is
>> really that it still talks about SHA3-256 when I think that the
>> performance benefits of SHA-256 (think: "Git at scale", and also hardware
>> support) really make the latter a better choice.
>>
>> In order to be "ironed out", I think we need to talk about the
>> implementation detail "Translation table". This is important. It needs to
>> be *fast*.
>>
>> Speaking of *fast*, I could imagine that it would make sense to store the
>> SHA-1 objects on disk, still, instead of converting them on the fly. I am
>> not sure whether this is something we need to define in the document,
>> though, as it may very well be premature optimization; Maybe mention that
>> we could do this if necessary?
>>
>> Apart from that, I would *love* to see this document as The Official Plan
>> that I can Show To The Manager so that I can ask to Allocate Time.
>
> Sounds promising!
>
> Thanks much for this feedback.  This is very helpful for knowing what
> v4 of the doc needs.
>
> The discussion of the translation table in [1] didn't make it to the
> doc.  You're right that it needs to.
>
> Caching SHA-1 objects (and the pros and cons involved) makes sense to
> mention in an "ideas for future work" section.
>
> An implementation plan with well-defined pieces for people to take on
> and estimates of how much work each involves may be useful for Showing
> To The Manager.  So I'll include a sketch of that for reviewers to
> poke holes in, too.
>
> Another thing the doc doesn't currently describe is how Git protocol
> would work.  That's worth sketching in a "future work" section as
> well.
>
> Sorry it has been taking so long to get this out.  I think we should
> have something ready to send on Monday.

I had a look at the current doc https://goo.gl/gh2Mzc and thought that the selection of the "NewHash" should be separated out into a section of it's own as a 'separation of concerns', so that the general transition plan only refers to the "NewHash", so as not to accidentally pre-judge that selection.

I did look up the arguments regarding sha2 (sha256) versus sha3-256 and found these two Q&A items

https://security.stackexchange.com/questions/152360/should-we-be-using-sha3-2017

https://security.stackexchange.com/questions/86283/how-does-sha3-keccak-shake-compare-to-sha2-should-i-use-non-shake-parameter

with an onward link to this:
 https://www.imperialviolet.org/2012/10/21/nist.html

"NIST may not have you in mind (21 Oct 2012)"

"A couple of weeks back, NIST announced that Keccak would be SHA-3. Keccak has somewhat disappointing software performance but is a gift to hardware implementations."

which does appear to cover some of the concerns that dscho had noted, and speed does appear to be a core Git selling point.

It would be worth at least covering these trade offs in the "select a NewHash" section of the document, as at the end of the day it will be a political judgement about what the future might hold regarding the contenders.

What may also be worth noting is the fall back plan should the chosen NewHash be the first to fail, perhaps spectacularly, as having a ready plan could support the choice at risk.

Show 6 quoted lines

>
> Thanks,
> Jonathan
>
> [1] 
> https://public-inbox.org/git/CAJo=hJtoX9=AyLHHpUJS7fueV9ciZ_MNpnEPHUz8Whui6g9F0A@mail.gmail.com/

-- Philip