Re: Git and SHA-1 security (again)
- From
brian m. carlson <sandals@crustytoothpaste.net>
- Date
- Jul 17, 2016, 22:04 UTC
- Message-ID
- <20160717220417.GE6644@vauxhall.crustytoothpaste.net>
- In-Reply-To
- <20160717162349.GB11276@thunk.org>
On Sun, Jul 17, 2016 at 12:23:49PM -0400, Theodore Ts'o wrote:
Show 9 quoted lines
> On Sun, Jul 17, 2016 at 03:42:34PM +0000, brian m. carlson wrote: > > As I said, I'm not planning on multiple hash support at first, but it > > doesn't appear impossible if we go this route. We might still have to > > rewrite objects, but we can verify signatures over the legacy SHA-1 > > objects by forcing them into the old-style object format. > > How hard would it be to make the on-disk format be multihash, even if > there is no support for anything other than a single hash, at least > for now? That way we won't have to rewrite the objects twice.
Other than the amount of work to change reading from the on-disk format, nothing prevents us from doing that, although I would recommend storing the object database with the tag prefix if we do so (i.e., instead of .git/objects/17, writing .git/objects/111417). That future-proofs us for when we change the hash.
I will say that the pack format will likely require some changes, because it assumes things are 4-byte aligned. It also assumes you can use the object ID in the mmaped pack directly (4-byte aligned), which you can no longer do. We have some cases where we cast that memory directly to struct object_id, which will no longer be valid, and even if we add the two prefix bytes to struct object_id, that doesn't guarantee that struct won't be aligned differently.
We could require that the pack format have two NUL bytes before the hash, which would force it to be aligned. We'd still have to make the Git protocol negotiate the new extension and fail gracefully if the version is too old. We could do this by requiring a pack version 5, which would simply cause older Gits to report errors.
It's a lot of work, and it's definitely a flag day. That's why I had planned to only do it with a new hash format: it would impact only people who were moving to the new hash. It also means that we get to work out any problems with the design at that point and not be committed to a design that might be inadequate. This is a place where I don't want to mess up.
Show 5 quoted lines
> Personally, so long as the newer versions of the tree are secured, I > wouldn't mind if the older commits stayed using SHA1 only. The newer > commits are the ones that are most important and security-critical > anyway. It seems like the main reason to rewrite all of the objects > is to simplify the initial rollout of a newer hash algorithm, no?
The reason is that we can't have an unambiguous parse of the current objects if two hash algorithms are in use. tree objects don't use a hex encoding of hashes; they use a binary encoding. It's therefore possible to create an ambiguous tree representation.
So when we look at a new hash, we need to provide an unambiguous way to know what hash is in use. The two choices are to either require all object use the new hash, or to extend the objects to include the hash. Until a couple days ago, I had planned to do the former. I had not even considered using a multihash approach due to the complexity.
-- brian m. carlson / brian with sandals: Houston, Texas, US +1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only OpenPGP: https://keybase.io/bk2204