{"thread":{"id":"42863","subject":"Git and SHA-1 security (again)","startedAt":"2016-07-16T13:48:57Z","lastAt":"2016-08-22T22:01:23Z","messageCount":52,"participants":["Herczeg Zsolt","brian m. carlson","Johannes Schindelin","Duy Nguyen","Theodore Ts'o","Ævar Arnfjörð Bjarmason","Junio C Hamano","David Lang","Jonathan Nieder","Stefan Beller","Philip Oakley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"291597","messageId":"CAPp-Vrb_n6z39RLHZ4AeUaBFiJfL3_xX8Utfq7+bTgzZrza58Q@mail.gmail.com","threadId":"42863","inReplyTo":null,"subject":"Git and SHA-1 security (again)","fromName":"Herczeg Zsolt","fromEmail":"zsolt94@gmail.com","sentAt":"2016-07-16T13:48:49Z","receivedAt":"2016-07-16T13:48:57Z","isPatch":false,"sender":{"key":"zsolt94@gmail.com","avatar":"https://gravatar.com/avatar/6a29b7ff538efff0a47e635c4bd50ad9f17b3744caf13aa053bec0127a29fce3?d=mp&s=160"},"body":"Dear List Members, Git Developers,\n\nI would like to discuss an old topic from 2006. I understand it was\nalready discussed. The only reason i'm sending this e-mail is to talk\nabout a possible solution which didn't show up on this list before.\n\nI think we all understand that SHA-1 is broken. It still works perfect\nas a storage key, but it's not cryptographically secure anymore. Git\nis not moving away from SHA-1 because it would break too many\nprojects, and cryptographic security is not needed but git if you have\nyour own repository.\n\nHowever I would like to show some big problems caused by SHA-1:\n - Git signed tags and signed commits are cryptographically insecure,\nthey're useless at the moment.\n - Git Torrent (https://github.com/cjb/GitTorrent) is also\ncryptographically broken, however it would be an awesome experiment.\n - Linus said: \"You only need to know the SHA-1 of the top of your\ntree, and if you know that, you can trust your tree.\" That's not true\nanymore. You have to trust your computer, you servers, your git\nprovider in a way that no-one can maliciously modify your data.\n\nI understand that git is perfect for a work flow, where you have your\nvery own repository and you double-check any commits or diffs you\naccepting to it. But that's not everybody's work flow. For example: if\nI want to blindly trust my college, I could just include all commits\nhe signed without review. Currently I can't do that. There are\nworkarounds of course: signing the e-mail he sends me, or signing the\nentire git repository's tarball, etc... But that's not the right way\nto do things.\n\nAs a final thought on this, I would like to say: Git is a great tool,\nbut it can be so much better with a safe hash.\n\n\nI would like to propose a solution for changing git's hash algorithm:\nIt would be a breaking change, bit I think it can be done pretty\npainless. (If you read the discussion back in 2006 the problems of\nmoving are clear.)\n\nIn git, every data has to have one and only one key - so a hybrid hash\nis a no-go. That means changing hash algo involves re-hashing every\ndata in a git repository, but it's not that bad. On a git clone, we\nactually re-hash everything to check integrity. Changing all the keys\nshouldn't be worth than that.\n\nBut - and that's the main idea i'm writing here - changing the storage\nkeys does not mean you should drop your old hashes out. If you change\nthe git data structure in a way, that it can keep multiple hashes for\nthe same \"link\" in each objects (trees, commits, etc) you can keep the\nold ones right next to the new one. If you want to look up the\nreferenced object, you must use the newest hash - which is the key.\nBut if you want to verify some old hash, it's still possible! Just\nlook up the objects by the new key, remove all the newer generation\nkeys, and verify the old hash on that.\n\nA storage structure like this would allow a very great flexibility:\n - You can change your hash algorithm in the future. If SHA-256\nbecomes broken, it's not a problem. Just re-hash the storage, and\nappend the new hashes the git objects.\n - You can still verify your old hashes after a hash change - removing\nthe new hashes from the objects before hashing should give you back\nthe old objects, thus giving you the same hash as before.\n - That makes possible for signed tags, and commits to keep their\nvalidity after hash change! With a clever-enough new format, you can\neven keep the validity of current hashes and signs. (To be able to do\nthat, you should be able to calculate back the current format from the\nnew format.)\n\nMoving git forward to a format like this would solve the weak-key\nproblem in git forever. You would be able to configure your key algo\non a per repository basis, you - and git - can do the daily work on\nthe newest hashes, while still carrying the old hashes and signatures,\nin case you ever want to verify them. That would allow repositories to\ngracefully change hashes in case they need to, and to only\ncompatibility limitation is that you must use a new enough git to\nunderstand the new storage format.\n\nWhat are your thoughts on this approach? Will git ever reach a release\nwith exchangeable hash algorithm? Or should someone look for\nalternatives if there's a need for cryptographic security?\n\nThank you for your time reading this.\n\nReferences:\nSHA-256 discussion in 2006:\nhttp://www.gelato.unsw.edu.au/archives/git/0608/26446.html\nDiscussion about git signatures in 2014\nhttps://www.mail-archive.com/git%40vger.kernel.org/msg61087.html\nLinus's talk on git\nhttps://www.youtube.com/watch?v=4XpnKHJAok8&t=56m20s\n\nKind regards,\nZsolt Herczeg\n"},{"id":"291603","messageId":"20160716201313.GA298717@vauxhall.crustytoothpaste.net","threadId":"42863","inReplyTo":"CAPp-Vrb_n6z39RLHZ4AeUaBFiJfL3_xX8Utfq7+bTgzZrza58Q@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2016-07-16T20:13:14Z","receivedAt":"2016-07-16T20:13:29Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sat, Jul 16, 2016 at 03:48:49PM +0200, Herczeg Zsolt wrote:\n> But - and that's the main idea i'm writing here - changing the storage\n> keys does not mean you should drop your old hashes out. If you change\n> the git data structure in a way, that it can keep multiple hashes for\n> the same \"link\" in each objects (trees, commits, etc) you can keep the\n> old ones right next to the new one. If you want to look up the\n> referenced object, you must use the newest hash - which is the key.\n> But if you want to verify some old hash, it's still possible! Just\n> look up the objects by the new key, remove all the newer generation\n> keys, and verify the old hash on that.\n> \n> A storage structure like this would allow a very great flexibility:\n>  - You can change your hash algorithm in the future. If SHA-256\n> becomes broken, it's not a problem. Just re-hash the storage, and\n> append the new hashes the git objects.\n>  - You can still verify your old hashes after a hash change - removing\n> the new hashes from the objects before hashing should give you back\n> the old objects, thus giving you the same hash as before.\n>  - That makes possible for signed tags, and commits to keep their\n> validity after hash change! With a clever-enough new format, you can\n> even keep the validity of current hashes and signs. (To be able to do\n> that, you should be able to calculate back the current format from the\n> new format.)\n> \n> Moving git forward to a format like this would solve the weak-key\n> problem in git forever. You would be able to configure your key algo\n> on a per repository basis, you - and git - can do the daily work on\n> the newest hashes, while still carrying the old hashes and signatures,\n> in case you ever want to verify them. That would allow repositories to\n> gracefully change hashes in case they need to, and to only\n> compatibility limitation is that you must use a new enough git to\n> understand the new storage format.\n> \n> What are your thoughts on this approach? Will git ever reach a release\n> with exchangeable hash algorithm? Or should someone look for\n> alternatives if there's a need for cryptographic security?\n\nI'm working on adding new hash algorithm support in Git.  However, it\nrequires a significant refactor of the code base.  My current plan is\nnot to implement side-by-side data, for a couple reasons.\n\nOne is that it requires significantly more work to implement and\ncomplicates the code.  It's also incompatible with all the refactoring\nI've done already.\n\nThe second is that it requires that Git have the ability to store\nmultiple hashes at once, which is very expensive in terms of memory.\nMoving from a 160-bit hash to a 256-bit hash (my current plan is\nSHA3-256) requires 1.6× the memory.  Storing both requires 2.6× the\nmemory.  If you add a third hash, it requires even more.  Memory is\noften a constraint with using Git.\n\nThe current plan is to use git-fast-import and git-fast-export to handle\nthat conversion process, and then maybe provide wrappers to make it more\ntransparent.\n\nCurrently the process of the refactor is ongoing, but it is a free time\nactivity for me.\n\nIf you'd like to follow the progress roughly, you can do so by checking\nthe output of the following commands:\n\n  git grep 'unsigned char.*20' | wc -l\n  git grep 'struct object_id' | wc -l\n\nYou are also welcome to contribute, of course.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"291607","messageId":"CAPp-Vrb5MQw-S25sTmdeatrFv5bv2pLG_mS5=9-ew8J_GNi+kQ@mail.gmail.com","threadId":"42863","inReplyTo":"20160716201313.GA298717@vauxhall.crustytoothpaste.net","subject":"Re: Git and SHA-1 security (again)","fromName":"Herczeg Zsolt","fromEmail":"zsolt94@gmail.com","sentAt":"2016-07-16T21:46:06Z","receivedAt":"2016-07-16T21:46:17Z","isPatch":false,"sender":{"key":"zsolt94@gmail.com","avatar":"https://gravatar.com/avatar/6a29b7ff538efff0a47e635c4bd50ad9f17b3744caf13aa053bec0127a29fce3?d=mp&s=160"},"body":"Dear Brian,\n\nThank you for your response. It very good to hear that changing the\nhash is on the git project's list. I haven't found any official\ncommunication on that topic since 2006.\nI'll look into the contributions guide and the source codes, to check\nif I can contribute to this transition. If you have any documentation\nor other related info, please point me towards it.\n\nThanks,\nZsolt Herczeg\n\n\n2016-07-16 22:13 GMT+02:00 brian m. carlson <sandals@crustytoothpaste.net>:\n> On Sat, Jul 16, 2016 at 03:48:49PM +0200, Herczeg Zsolt wrote:\n>> But - and that's the main idea i'm writing here - changing the storage\n>> keys does not mean you should drop your old hashes out. If you change\n>> the git data structure in a way, that it can keep multiple hashes for\n>> the same \"link\" in each objects (trees, commits, etc) you can keep the\n>> old ones right next to the new one. If you want to look up the\n>> referenced object, you must use the newest hash - which is the key.\n>> But if you want to verify some old hash, it's still possible! Just\n>> look up the objects by the new key, remove all the newer generation\n>> keys, and verify the old hash on that.\n>>\n>> A storage structure like this would allow a very great flexibility:\n>>  - You can change your hash algorithm in the future. If SHA-256\n>> becomes broken, it's not a problem. Just re-hash the storage, and\n>> append the new hashes the git objects.\n>>  - You can still verify your old hashes after a hash change - removing\n>> the new hashes from the objects before hashing should give you back\n>> the old objects, thus giving you the same hash as before.\n>>  - That makes possible for signed tags, and commits to keep their\n>> validity after hash change! With a clever-enough new format, you can\n>> even keep the validity of current hashes and signs. (To be able to do\n>> that, you should be able to calculate back the current format from the\n>> new format.)\n>>\n>> Moving git forward to a format like this would solve the weak-key\n>> problem in git forever. You would be able to configure your key algo\n>> on a per repository basis, you - and git - can do the daily work on\n>> the newest hashes, while still carrying the old hashes and signatures,\n>> in case you ever want to verify them. That would allow repositories to\n>> gracefully change hashes in case they need to, and to only\n>> compatibility limitation is that you must use a new enough git to\n>> understand the new storage format.\n>>\n>> What are your thoughts on this approach? Will git ever reach a release\n>> with exchangeable hash algorithm? Or should someone look for\n>> alternatives if there's a need for cryptographic security?\n>\n> I'm working on adding new hash algorithm support in Git.  However, it\n> requires a significant refactor of the code base.  My current plan is\n> not to implement side-by-side data, for a couple reasons.\n>\n> One is that it requires significantly more work to implement and\n> complicates the code.  It's also incompatible with all the refactoring\n> I've done already.\n>\n> The second is that it requires that Git have the ability to store\n> multiple hashes at once, which is very expensive in terms of memory.\n> Moving from a 160-bit hash to a 256-bit hash (my current plan is\n> SHA3-256) requires 1.6× the memory.  Storing both requires 2.6× the\n> memory.  If you add a third hash, it requires even more.  Memory is\n> often a constraint with using Git.\n>\n> The current plan is to use git-fast-import and git-fast-export to handle\n> that conversion process, and then maybe provide wrappers to make it more\n> transparent.\n>\n> Currently the process of the refactor is ongoing, but it is a free time\n> activity for me.\n>\n> If you'd like to follow the progress roughly, you can do so by checking\n> the output of the following commands:\n>\n>   git grep 'unsigned char.*20' | wc -l\n>   git grep 'struct object_id' | wc -l\n>\n> You are also welcome to contribute, of course.\n> --\n> brian m. carlson / brian with sandals: Houston, Texas, US\n> +1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\n> OpenPGP: https://keybase.io/bk2204\n"},{"id":"291608","messageId":"20160716220305.GA409690@vauxhall.crustytoothpaste.net","threadId":"42863","inReplyTo":"CAPp-Vrb5MQw-S25sTmdeatrFv5bv2pLG_mS5=9-ew8J_GNi+kQ@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2016-07-16T22:03:05Z","receivedAt":"2016-07-16T22:03:16Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sat, Jul 16, 2016 at 11:46:06PM +0200, Herczeg Zsolt wrote:\n> Dear Brian,\n> \n> Thank you for your response. It very good to hear that changing the\n> hash is on the git project's list. I haven't found any official\n> communication on that topic since 2006.\n\nThere's been some recent discussion on the list about it.  It is less on\nthe Git project's list and more on my personal list.  It's my hope that\nJunio and other contributors will decide to accept my patches when they\nare ready.  Also, the plan is to keep SHA-1 available, probably as the\ndefault, for backwards compatibility.\n\n> I'll look into the contributions guide and the source codes, to check\n> if I can contribute to this transition. If you have any documentation\n> or other related info, please point me towards it.\n\nThe major work at this point is turning instances of unsigned char [20]\ninto struct object_id, as well as converting hardcoded 20 and 40 (and\nderivative values) to GIT_SHA1_RAWSZ and GIT_SHA1_HEXSZ.  This work\nallows us to make as little code as possible know about the size of the\nhash, as well as generally being easier to maintain.\n\nYou can look at the bc/cocci branch which was recently merged into next.\n(It doesn't exist independently outside of next, so you'll have to\nsearch through the history).  That work is what in my branches is called\nobject-id-part4.  I'm currently working on getting to the point of\nconverting get_tree_entry to use struct object_id, which is what will\nbecome my object-id-part5.\n\nI recommend if you're planning on doing some of this work that you try\nto avoid areas which are under work by other developers, especially the\nrefs code, which is undergoing massive changes.  Other people will\nappreciate it.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"291617","messageId":"alpine.DEB.2.20.1607170949360.28832@virtualbox","threadId":"42863","inReplyTo":"20160716201313.GA298717@vauxhall.crustytoothpaste.net","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-17T08:01:38Z","receivedAt":"2016-07-17T08:02:25Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brian,\n\nOn Sat, 16 Jul 2016, brian m. carlson wrote:\n\n> My current plan is not to implement side-by-side data, for a couple\n> reasons.\n\nI am as guilty as the next person to have use the \"deafbee(This is my\ncommit, 2007-08-21)\" format to refer to older commits. So I do have\nconcerns about rewriting history when switching to a new hash.\n\nI understand the technical challenges, of course.\n\nOut of curiosity: have you considered something like padding the SHA-1s\nwith, say 0xa1, to the size of the new hash and using that padding to\ndistinguish between old vs new hash?\n\nI guess that it would also possible to introduce an opt-in \"legacy mapper\"\nwhich would generate a mapping locally of all objects' SHA-1 to whatever\nnew hash you choose. Generating it locally would side-step the security\nissues of the SHA-1 algorithm. We would need to teach Git to pick that\nmapping up if available and use it, of course.\n\nHowever, that latter solution would do nothing to address the problem of\nexisting GPG-signed commits.\n\nCiao,\nDscho\n"},{"id":"291628","messageId":"20160717142157.GA6644@vauxhall.crustytoothpaste.net","threadId":"42863","inReplyTo":"alpine.DEB.2.20.1607170949360.28832@virtualbox","subject":"Re: Git and SHA-1 security (again)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2016-07-17T14:21:57Z","receivedAt":"2016-07-17T14:22:11Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Jul 17, 2016 at 10:01:38AM +0200, Johannes Schindelin wrote:\n> Out of curiosity: have you considered something like padding the SHA-1s\n> with, say 0xa1, to the size of the new hash and using that padding to\n> distinguish between old vs new hash?\n\nI'm going to end up having to do something similar because of the issue\nof submodules.  Submodules may still be SHA-1, while the main repo may\nbe a newer hash.  I was going to zero-pad, however.  I was also, at\nleast at first, going to force a separate .git dir for those, to avoid\nhaving to try to store two separate types of objects in the same repo.\n\nThe other limitation with this is that it isn't immediately obvious what\nhash is in use just because it has a certain length.  For example, I\nplan on implementing SHA3-256, but it's also possible I might add\nBLAKE2b-256 for people for whom SHA3-256 is too slow.  There's no way to\ndistinguish between those two algorithms.  Thus allowing multiple hashes\nin the same repo won't work without a format byte.\n\nWhat I might do, however, is add multihash-style format information to\nthe on-disk format for non-SHA-1 repos.  Then SHA-1 compatibility could\ncome in a future iteration.  That would be compatible with the existing\nrefactor.\n\n> I guess that it would also possible to introduce an opt-in \"legacy mapper\"\n> which would generate a mapping locally of all objects' SHA-1 to whatever\n> new hash you choose. Generating it locally would side-step the security\n> issues of the SHA-1 algorithm. We would need to teach Git to pick that\n> mapping up if available and use it, of course.\n\nI think that might be easier.  Considering the number of tests that\nhard-code object names, I might need that for the testsuite.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"291629","messageId":"CACsJy8C+2=qv5Vu=tGeDTK_Q+XSAv3qEJw0nrHbEWU7psDf=Cg@mail.gmail.com","threadId":"42863","inReplyTo":"20160717142157.GA6644@vauxhall.crustytoothpaste.net","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-17T15:19:02Z","receivedAt":"2016-07-17T15:19:40Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jul 17, 2016 at 4:21 PM, brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n> On Sun, Jul 17, 2016 at 10:01:38AM +0200, Johannes Schindelin wrote:\n>> Out of curiosity: have you considered something like padding the SHA-1s\n>> with, say 0xa1, to the size of the new hash and using that padding to\n>> distinguish between old vs new hash?\n>\n> I'm going to end up having to do something similar because of the issue\n> of submodules.  Submodules may still be SHA-1, while the main repo may\n> be a newer hash.  I was going to zero-pad, however.  I was also, at\n> least at first, going to force a separate .git dir for those, to avoid\n> having to try to store two separate types of objects in the same repo.\n\nIf it's just the external hash representation, can we go with a prefix\n<algo><colon> to identify the hash algorithm? For example\nsha256:1234... is SHA-256 while 1235... by default is SHA-1 (but we\ncould switch the default to SHA-256 via config file later SHA-1 is\ndead and nobody wants to type sha256: every time). It catches\nincorrect hash algorithm references.\n-- \nDuy\n"},{"id":"291630","messageId":"20160717154234.GC6644@vauxhall.crustytoothpaste.net","threadId":"42863","inReplyTo":"CACsJy8C+2=qv5Vu=tGeDTK_Q+XSAv3qEJw0nrHbEWU7psDf=Cg@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2016-07-17T15:42:34Z","receivedAt":"2016-07-17T15:42:48Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Jul 17, 2016 at 05:19:02PM +0200, Duy Nguyen wrote:\n> On Sun, Jul 17, 2016 at 4:21 PM, brian m. carlson\n> <sandals@crustytoothpaste.net> wrote:\n> > On Sun, Jul 17, 2016 at 10:01:38AM +0200, Johannes Schindelin wrote:\n> >> Out of curiosity: have you considered something like padding the SHA-1s\n> >> with, say 0xa1, to the size of the new hash and using that padding to\n> >> distinguish between old vs new hash?\n> >\n> > I'm going to end up having to do something similar because of the issue\n> > of submodules.  Submodules may still be SHA-1, while the main repo may\n> > be a newer hash.  I was going to zero-pad, however.  I was also, at\n> > least at first, going to force a separate .git dir for those, to avoid\n> > having to try to store two separate types of objects in the same repo.\n> \n> If it's just the external hash representation, can we go with a prefix\n> <algo><colon> to identify the hash algorithm? For example\n> sha256:1234... is SHA-256 while 1235... by default is SHA-1 (but we\n> could switch the default to SHA-256 via config file later SHA-1 is\n> dead and nobody wants to type sha256: every time). It catches\n> incorrect hash algorithm references.\n\nI'd make it such that the default is that of the repo.  If the current\nrepo is generating SHA-256, say, then 473a0f4 refers to the empty blob.\nIf you want to refer to an SHA-1 object, then you write sha-1:e69de29.\n\nOn disk, multihash[0] seems like the right way to go.  We'd serialize\nreferences to the SHA-1 and SHA-256 empty blobs as\n1114e69de29bb2d1d6434b8b29ae775ad8c2e48c5391 and\n1220473a0f4c3be8a93681a267e3b1e9a7dcda1185436fe141f7749120a303721813\nrespectively.  This makes parsing significantly easier.  On disk, we\ncould write them into the object database as 1114e6/9de2… and\n122047/3a0f….\n\nWe could implement the default hash algorithm as extensions.hash and the\non-disk format (which would be a requirement for extensions.hash) as\nextensions.explicitHash.\n\nAs I said, I'm not planning on multiple hash support at first, but it\ndoesn't appear impossible if we go this route.  We might still have to\nrewrite objects, but we can verify signatures over the legacy SHA-1\nobjects by forcing them into the old-style object format.\n\n[0] https://github.com/jbenet/multihash\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"291631","messageId":"20160717162349.GB11276@thunk.org","threadId":"42863","inReplyTo":"20160717154234.GC6644@vauxhall.crustytoothpaste.net","subject":"Re: Git and SHA-1 security (again)","fromName":"Theodore Ts'o","fromEmail":"tytso@mit.edu","sentAt":"2016-07-17T16:23:49Z","receivedAt":"2016-07-17T16:24:10Z","isPatch":false,"sender":{"key":"tytso@mit.edu","avatar":"https://avatars.githubusercontent.com/u/51416?v=4"},"body":"On Sun, Jul 17, 2016 at 03:42:34PM +0000, brian m. carlson wrote:\n> As I said, I'm not planning on multiple hash support at first, but it\n> doesn't appear impossible if we go this route.  We might still have to\n> rewrite objects, but we can verify signatures over the legacy SHA-1\n> objects by forcing them into the old-style object format.\n\nHow hard would it be to make the on-disk format be multihash, even if\nthere is no support for anything other than a single hash, at least\nfor now?  That way we won't have to rewrite the objects twice.\n\nPersonally, so long as the newer versions of the tree are secured, I\nwouldn't mind if the older commits stayed using SHA1 only.  The newer\ncommits are the ones that are most important and security-critical\nanyway.  It seems like the main reason to rewrite all of the objects\nis to simplify the initial rollout of a newer hash algorithm, no?\n\n\t\t     \t   \t\t   - Ted\n"},{"id":"291638","messageId":"20160717220417.GE6644@vauxhall.crustytoothpaste.net","threadId":"42863","inReplyTo":"20160717162349.GB11276@thunk.org","subject":"Re: Git and SHA-1 security (again)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2016-07-17T22:04:17Z","receivedAt":"2016-07-17T22:04:53Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Sun, Jul 17, 2016 at 12:23:49PM -0400, Theodore Ts'o wrote:\n> On Sun, Jul 17, 2016 at 03:42:34PM +0000, brian m. carlson wrote:\n> > As I said, I'm not planning on multiple hash support at first, but it\n> > doesn't appear impossible if we go this route.  We might still have to\n> > rewrite objects, but we can verify signatures over the legacy SHA-1\n> > objects by forcing them into the old-style object format.\n> \n> How hard would it be to make the on-disk format be multihash, even if\n> there is no support for anything other than a single hash, at least\n> for now?  That way we won't have to rewrite the objects twice.\n\nOther than the amount of work to change reading from the on-disk format,\nnothing prevents us from doing that, although I would recommend storing\nthe object database with the tag prefix if we do so (i.e., instead of\n.git/objects/17, writing .git/objects/111417).  That future-proofs us\nfor when we change the hash.\n\nI will say that the pack format will likely require some changes,\nbecause it assumes things are 4-byte aligned.  It also assumes you can\nuse the object ID in the mmaped pack directly (4-byte aligned), which\nyou can no longer do.  We have some cases where we cast that memory\ndirectly to struct object_id, which will no longer be valid, and even if\nwe add the two prefix bytes to struct object_id, that doesn't guarantee\nthat struct won't be aligned differently.\n\nWe could require that the pack format have two NUL bytes before the\nhash, which would force it to be aligned.  We'd still have to make the\nGit protocol negotiate the new extension and fail gracefully if the\nversion is too old.  We could do this by requiring a pack version 5,\nwhich would simply cause older Gits to report errors.\n\nIt's a lot of work, and it's definitely a flag day.  That's why I had\nplanned to only do it with a new hash format: it would impact only\npeople who were moving to the new hash.  It also means that we get to\nwork out any problems with the design at that point and not be committed\nto a design that might be inadequate.  This is a place where I don't\nwant to mess up.\n\n> Personally, so long as the newer versions of the tree are secured, I\n> wouldn't mind if the older commits stayed using SHA1 only.  The newer\n> commits are the ones that are most important and security-critical\n> anyway.  It seems like the main reason to rewrite all of the objects\n> is to simplify the initial rollout of a newer hash algorithm, no?\n\nThe reason is that we can't have an unambiguous parse of the current\nobjects if two hash algorithms are in use.  tree objects don't use a hex\nencoding of hashes; they use a binary encoding.  It's therefore possible\nto create an ambiguous tree representation.\n\nSo when we look at a new hash, we need to provide an unambiguous way to\nknow what hash is in use.  The two choices are to either require all\nobject use the new hash, or to extend the objects to include the hash.\nUntil a couple days ago, I had planned to do the former.  I had not even\nconsidered using a multihash approach due to the complexity.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"291641","messageId":"CAPp-VrazVQqA3iEpbUYP=kscdm+LrEMesoDid3s5AD5zfU5Z+w@mail.gmail.com","threadId":"42863","inReplyTo":"1468804249.2037.0@smtp.gmail.com","subject":"Fwd: Git and SHA-1 security (again)","fromName":"Herczeg Zsolt","fromEmail":"zsolt94@gmail.com","sentAt":"2016-07-18T01:18:32Z","receivedAt":"2016-07-18T01:18:38Z","isPatch":false,"sender":{"key":"zsolt94@gmail.com","avatar":"https://gravatar.com/avatar/6a29b7ff538efff0a47e635c4bd50ad9f17b3744caf13aa053bec0127a29fce3?d=mp&s=160"},"body":"Do you think the multi-hash approach worth the added complexity? It'll\nbreak a lot of things. I mean almost everything. All git algorithms\nrely on the \"same hash => same content\" \"same content => same hash\"\nstatements.\n\nI think converting is a much better option. Use a single-hash storage,\nand convert everything to that on import/clone/pull. I would only\nintroduce a a two-way mapping table (old-hash <=> new-hash) to help\nthe users. In the normal workflow, everything can go with new-hashes\nonly.\n\nThat leaves most algorithms and code intact, and introduces a very few\nnew cases:\n\nOn import:\nIf the imported data does not match your selected new-hash format, add\nit's hash to the lookup table, than convert it to your selected\nformat, and handle it as such.\n\nIf a user references an old hash:\nWe can look-up that table forward, and find the referenced object in\nstorage. We can handle it from now as normal.\n\nOn an old signature verify:\nLook-up the table forward, find the object by knew key. Look-up\nbackwards for all referenced objects, reconstruct the \"old-format\" and\nverify the hash on that.\n\nIf you double-check your hashes as you build the mapping, you can even\ntrust it, which makes the lookups and verifys very fast. You can\nintroduce as many hash mapping tables as you want, so you can not only\nsupport old-to-new has transition, but there can be as many different\nhashes in the world as you want. Your only rule is \"you reference your\nwork in your current format\", but you can look-up any references which\nwas valid at the moment it was made.\n\n(There are slight issues with this aproach if we \"convert than\nconvert\". As for example, when you import from A (sha1) repo to B\n(sha2) repo it's perfectl. But when you import the same commits from B\nto C (sha3), you might loose sha1 references. That could be considered\nnormal if we wan't to keep-it-simple-stupid, only support a few\nhashes, always going forward. Or you may add an extra \"list\" field to\nobjects, which could show what type of hashes you have to keep in\nlookup-tables for that particular object. Or, you can even include a\nlist of old hashes in the object itself, which should make it to the\nlookup table on import.)\n\nAnyway, I think a single-hash storage, and the ability to hide any old\nhashes from most of the internal algorithms is a key point in making\ntransition. If we want to provide multi-hash interface to users, than\nwe should look for \"wrapper\" solutions, that translates multi-hash\nuser needs to a single-hash backend.\n\nZsolt\n"},{"id":"291644","messageId":"alpine.DEB.2.20.1607180853300.28832@virtualbox","threadId":"42863","inReplyTo":"20160717142157.GA6644@vauxhall.crustytoothpaste.net","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-18T07:00:06Z","receivedAt":"2016-07-18T07:00:36Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brian,\n\nOn Sun, 17 Jul 2016, brian m. carlson wrote:\n\n> On Sun, Jul 17, 2016 at 10:01:38AM +0200, Johannes Schindelin wrote:\n> > Out of curiosity: have you considered something like padding the SHA-1s\n> > with, say 0xa1, to the size of the new hash and using that padding to\n> > distinguish between old vs new hash?\n> \n> I'm going to end up having to do something similar because of the issue\n> of submodules.  Submodules may still be SHA-1, while the main repo may\n> be a newer hash.  I was going to zero-pad, however.\n\nI thought about zero-padding, but there are plenty of\nis_null_sha1()/is_null_oid() calls around. Of course, I assumed\nleft-padding. But you may have thought of right-padding instead? That\nwould make short name handling much easier, too.\n\nFWIW it never crossed my mind to allow different same-sized hash\nalgorithms. So I never thought we'd need a way to distinguish, say,\nBLAKE2b-256 from SHA-256.\n\nIs there a good reason to add the maintenance burden of several 256-bit\nhash algorithms, apart from speed (which in my mind should decide which\none to use, always, rather than letting the user choose)? It would also\ncomplicate transport even further, let alone subtree merges from\ndifferently-hashed repositories.\n\nCiao,\nDscho\n"},{"id":"291645","messageId":"alpine.DEB.2.20.1607180905320.28832@virtualbox","threadId":"42863","inReplyTo":"1468804249.2037.0@smtp.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-18T07:12:29Z","receivedAt":"2016-07-18T07:12:55Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Zsolt,\n\nOn Mon, 18 Jul 2016, Herczeg Zsolt wrote:\n\n> I think converting is a much better option. Use a single-hash storage, and\n> convert everything to that on import/clone/pull.\n\nThat ignores two very important issues that I already had mentioned:\n\n- existing references, both in-repository, e.g. in commit messages\n  referring to earlier commits, as well as out-of-repository, e.g.\n  referring to commits in mails, blog posts, etc\n\n- GPG-signed commits\n\nThose issues cannot just be hand-waved away.\n\nThe \"convert everything\" strategy also ignores the problem of interacting\nwith servers and collaborators. Think of hosting repositories,\nrediscovering forgotten work trees, and of the \"D\" in DSCM.\n\nCiao,\nJohannes\n"},{"id":"291652","messageId":"CAPp-Vran2GZFTyJHb2qxgh3uRpM0ar7K2+VbbLcBK74_7aaxVw@mail.gmail.com","threadId":"42863","inReplyTo":"alpine.DEB.2.20.1607180905320.28832@virtualbox","subject":"Re: Git and SHA-1 security (again)","fromName":"Herczeg Zsolt","fromEmail":"zsolt94@gmail.com","sentAt":"2016-07-18T15:09:16Z","receivedAt":"2016-07-18T15:09:23Z","isPatch":false,"sender":{"key":"zsolt94@gmail.com","avatar":"https://gravatar.com/avatar/6a29b7ff538efff0a47e635c4bd50ad9f17b3744caf13aa053bec0127a29fce3?d=mp&s=160"},"body":">> I think converting is a much better option. Use a single-hash storage, and\n>> convert everything to that on import/clone/pull.\n>\n> That ignores two very important issues that I already had mentioned:\n\nThat's not true. If you double-check the next part of my message, you\nI just showed that an automatic two-way mapping could solve these\nproblems! (I even give briefs explanation how to handle referencing\nand signature verification in those cases.)\n\nMy point is not to throw out old hashes and break signatures. My point\nis to convert the data storage, and use mapping to resolve problems\nwith those old hashes and signatures. A single-hash data storage is\nobviously way easier to handle, than a multi-hash mass. (See Linus's\nold e-mail: multiple hashes [=meaning database keys] for the same\ncontent is a complete nonsense in git-speak)\n\n> The \"convert everything\" strategy also ignores the problem of interacting\n> with servers and collaborators. Think of hosting repositories,\n> rediscovering forgotten work trees, and of the \"D\" in DSCM.\n\nThat's not an issue when we're working with a single repository. It's\nreasonable to ask for all git clients of the same repository, to\nsupport the same hash. Yes, you have the need to configure the hash\nalgo on a per-repository basis but that's all. For importing and\nco-working between different repositories, it's a bit harder, problem,\nbut it's possible to handle the conversions correctly.\n"},{"id":"291653","messageId":"alpine.DEB.2.20.1607181750470.3472@virtualbox","threadId":"42863","inReplyTo":"CAPp-Vran2GZFTyJHb2qxgh3uRpM0ar7K2+VbbLcBK74_7aaxVw@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-18T15:57:37Z","receivedAt":"2016-07-18T15:58:10Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Zsolt,\n\nOn Mon, 18 Jul 2016, Herczeg Zsolt wrote:\n\n> >> I think converting is a much better option. Use a single-hash\n> >> storage, and convert everything to that on import/clone/pull.\n> >\n> > That ignores two very important issues that I already had mentioned:\n> \n> That's not true. If you double-check the next part of my message, you I\n> just showed that an automatic two-way mapping could solve these\n> problems! (I even give briefs explanation how to handle referencing and\n> signature verification in those cases.)\n> \n> My point is not to throw out old hashes and break signatures. My point\n> is to convert the data storage, and use mapping to resolve problems\n> with those old hashes and signatures.\n\nIf you convert the data storage, then the SHA-1s listed in the commit\nobjects will have to be rewritten, and then the GPG signature will not\nmatch anymore.\n\nCall e.g. `git cat-file commit 44cc742a8ca17b9c279be4cc195a93a6ef7a320e`\nto see the anatomy of a gpg-signed commit object.\n\nCiao,\nJohannes\n"},{"id":"291654","messageId":"CACsJy8Ba=c+-WV2TsY768_fTDO2KesS1b6BK7kdykNY6gkh=UQ@mail.gmail.com","threadId":"42863","inReplyTo":"alpine.DEB.2.20.1607181750470.3472@virtualbox","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-18T16:05:23Z","receivedAt":"2016-07-18T16:06:01Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jul 18, 2016 at 5:57 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> Hi Zsolt,\n>\n> On Mon, 18 Jul 2016, Herczeg Zsolt wrote:\n>\n>> >> I think converting is a much better option. Use a single-hash\n>> >> storage, and convert everything to that on import/clone/pull.\n>> >\n>> > That ignores two very important issues that I already had mentioned:\n>>\n>> That's not true. If you double-check the next part of my message, you I\n>> just showed that an automatic two-way mapping could solve these\n>> problems! (I even give briefs explanation how to handle referencing and\n>> signature verification in those cases.)\n>>\n>> My point is not to throw out old hashes and break signatures. My point\n>> is to convert the data storage, and use mapping to resolve problems\n>> with those old hashes and signatures.\n>\n> If you convert the data storage, then the SHA-1s listed in the commit\n> objects will have to be rewritten, and then the GPG signature will not\n> match anymore.\n\nBut we can recreate SHA-1 from the same content and verify GPG, right?\nI know it's super expensive, but it feels safer to not carry SHA-1\naround when it's not secure anymore (I recall something about\nexploiting the weakest link when you have both sha1 and sha256 in the\nobject content). Rehashing would be done locally and is better\ncontrolled.\n-- \nDuy\n"},{"id":"291656","messageId":"CAPp-Vra8c7eFegysKrM-hZKNmD6B21XqVkE8HB0QcG_m8hu+Ow@mail.gmail.com","threadId":"42863","inReplyTo":"alpine.DEB.2.20.1607181750470.3472@virtualbox","subject":"Re: Git and SHA-1 security (again)","fromName":"Herczeg Zsolt","fromEmail":"zsolt94@gmail.com","sentAt":"2016-07-18T16:12:57Z","receivedAt":"2016-07-18T16:13:47Z","isPatch":false,"sender":{"key":"zsolt94@gmail.com","avatar":"https://gravatar.com/avatar/6a29b7ff538efff0a47e635c4bd50ad9f17b3744caf13aa053bec0127a29fce3?d=mp&s=160"},"body":"Hi Johannes,\n\n>> My point is not to throw out old hashes and break signatures. My point\n>> is to convert the data storage, and use mapping to resolve problems\n>> with those old hashes and signatures.\n>\n> If you convert the data storage, then the SHA-1s listed in the commit\n> objects will have to be rewritten, and then the GPG signature will not\n> match anymore.\n>\n> Call e.g. `git cat-file commit 44cc742a8ca17b9c279be4cc195a93a6ef7a320e`\n> to see the anatomy of a gpg-signed commit object.\n>\n\nYes and no. That's the reason you need the two-way lookup table. If\nyou need to verify a commit which was signed as SHA-1, you must use\nthe lookup table in reverse. This way you can reconstruct the original\ncommit structure, which than can be verified. Of course it's work to\ndo so but you only need to develop the new signature verification\nalgorithm. You save much more on the other side where you don't have\nto rework all the other algorithms to multi-hash.\n\nAnother interesting point is that multi-hash storage, actively hurts\nsignature security! (Duy just mentoined that while I'm writing.) A\nsigned commit (or tag) is just as secure as the least secure hash it\nrefers (directly or indirectly). Let's imagine that you make a new a\ncommit, and there is on old file in the tree somewhere. That's a weak\npoint: cause it has SHA-1 hash, someone can replace it (and thus\nchange your commits content.\n\nI would clearly mark any signature wether it's SHA-1 or SHA2 (or\nanything else) based, and strictly allow that hash in all the trees\nand objects while verifying that commit. If it's not the same\nhash-type as the storage-key, than use the lookup table for conversion\nbefore check. (This has some interesting side-effects, but it's all\nabout good implementation).\n"},{"id":"291657","messageId":"CACBZZX5wjhYosupW=-n7UbbXAQug+sgSNS+jm_uUn6QwC8p6Aw@mail.gmail.com","threadId":"42863","inReplyTo":"CAPp-Vrb_n6z39RLHZ4AeUaBFiJfL3_xX8Utfq7+bTgzZrza58Q@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2016-07-18T16:51:23Z","receivedAt":"2016-07-18T16:51:49Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Sat, Jul 16, 2016 at 3:48 PM, Herczeg Zsolt <zsolt94@gmail.com> wrote:\n> I would like to discuss an old topic from 2006. I understand it was\n> already discussed. The only reason i'm sending this e-mail is to talk\n> about a possible solution which didn't show up on this list before.\n\nYou mention the 2006 discussion, but I wonder if you've read the more\nrecent discussion from April on the subject.\n\n> I think we all understand that SHA-1 is broken. It still works perfect\n> as a storage key, but it's not cryptographically secure anymore. Git\n> is not moving away from SHA-1 because it would break too many\n> projects, and cryptographic security is not needed but git if you have\n> your own repository.\n>\n> However I would like to show some big problems caused by SHA-1:\n>  - Git signed tags and signed commits are cryptographically insecure,\n> they're useless at the moment.\n>  - Git Torrent (https://github.com/cjb/GitTorrent) is also\n> cryptographically broken, however it would be an awesome experiment.\n>  - Linus said: \"You only need to know the SHA-1 of the top of your\n> tree, and if you know that, you can trust your tree.\" That's not true\n> anymore. You have to trust your computer, you servers, your git\n> provider in a way that no-one can maliciously modify your data.\n\nIn particular, as far as I know and as Theodore Ts'o's post describes\nbetter than I could[1], you seem to be confusing preimage attacks with\ncollision attacks, and then concluding that because SHA1 is vulnerable\nto collision attacks that use-cases that would need a preimage attack\nto be compromised (which as far is I can tell, includes all your\nexamples) are also \"broken\".\n\n1. http://thread.gmane.org/gmane.comp.version-control.git/291305/focus=291511\n"},{"id":"291658","messageId":"CACsJy8AH9Q6rOgvcWGsLGPaP96koGA=k0PYgXP6F3RZ=XAwaSw@mail.gmail.com","threadId":"42863","inReplyTo":"20160717142157.GA6644@vauxhall.crustytoothpaste.net","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-18T16:51:43Z","receivedAt":"2016-07-18T16:52:47Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Sun, Jul 17, 2016 at 4:21 PM, brian m. carlson\n<sandals@crustytoothpaste.net> wrote:\n> I'm going to end up having to do something similar because of the issue\n> of submodules.  Submodules may still be SHA-1, while the main repo may\n> be a newer hash.\n\nOr even the other way around, main repo is one with sha1 while\nsubmodule is on sha256. I wonder if we should address this separately\n(and even in parallel with sha256 support), making submodules work\nwith an any external VCS system (that supports some basic operations\nwe define).\n-- \nDuy\n"},{"id":"291661","messageId":"CAPp-Vra5ri7V03SSkU6=UnWrsq-jk0XcNpQuzGH8kQxE-TkPjA@mail.gmail.com","threadId":"42863","inReplyTo":"CACBZZX5wjhYosupW=-n7UbbXAQug+sgSNS+jm_uUn6QwC8p6Aw@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Herczeg Zsolt","fromEmail":"zsolt94@gmail.com","sentAt":"2016-07-18T17:48:12Z","receivedAt":"2016-07-18T17:48:33Z","isPatch":false,"sender":{"key":"zsolt94@gmail.com","avatar":"https://gravatar.com/avatar/6a29b7ff538efff0a47e635c4bd50ad9f17b3744caf13aa053bec0127a29fce3?d=mp&s=160"},"body":"> In particular, as far as I know and as Theodore Ts'o's post describes\n> better than I could[1], you seem to be confusing preimage attacks with\n> collision attacks, and then concluding that because SHA1 is vulnerable\n> to collision attacks that use-cases that would need a preimage attack\n> to be compromised (which as far is I can tell, includes all your\n> examples) are also \"broken\".\n\nI understand the differences between the collision and preimage\nattacks. A collision attack is not that bad for git in a typical\nuse-case. But I think that it's important to note that there are many\nuse-cases which do need a hash safe from collision attack. Some\nexamples:\n\nYou maintain a repository with gittorrent with signed commits Others\ncan use these signatures to verify it's original. Let's say you\ninclude some safe file (potentially binary) from a third-party\ncontributor. That would be fine if the hash algo is safe. Currently\nthere is the possibility that you received a (safe) file which was\nmade to collide with another malicious one. Once you committed (and\nsigned) that file, the attacker joins the gittorrent network and\nstarts to distribute the malicious file. Suddenly most of your clients\npulling are infected however your signature is correct.\n\nOr, you would like to make a continuous delivery system, where you use\nsigned tags. The delivery happens only when signature is right, and\nthe signer is responsible for it. Your colleague makes a collision,\npushes the good-file. You make all the tests, everything is fine, sign\nand push and wait for the delivery to happen. Your colleague changes\nthe file on the server. The delivery makes a huge mass, and you're\nfired.\n\nOr, let's say you use a service like github, which is nice enough to\nmake a repository for you, with .gitignore, licenses and everything.\nLikely, you'll never change dose files. Let's say that service made\none of those initial files to collide something bad. That means, they\ncan \"infect\" anyone, who is pulling your repo.\n\nDo you need more hypothetical stories? There are a lot. Of course they\nneed a lot of work, and they're unlikely to happen. But it's possible.\nIf you need trust, and gpg signatures that means you need ultimate\ntrust. What's the point in making GPG signatures anyway if you cannot\nultimately trust them? You could just as well say: well that's\nrepository is only reachable by trustworthy persons, everything here\nis just fine and really made by the person named in the \"author\nfield\".\n"},{"id":"291662","messageId":"xmqq4m7mltbw.fsf@gitster.mtv.corp.google.com","threadId":"42863","inReplyTo":"20160717220417.GE6644@vauxhall.crustytoothpaste.net","subject":"Re: Git and SHA-1 security (again)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-18T18:00:35Z","receivedAt":"2016-07-18T18:01:15Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n\n> I will say that the pack format will likely require some changes,\n> because it assumes ...\n> The reason is that we can't have an unambiguous parse of the current\n> objects if two hash algorithms are in use....\n> So when we look at a new hash, we need to provide an unambiguous way to\n> know what hash is in use.  The two choices are to either require all\n> object use the new hash, or to extend the objects to include the hash.\n> Until a couple days ago, I had planned to do the former.  I had not even\n> considered using a multihash approach due to the complexity.\n\nObjects in Git identify themselves, but once you introduce the\nsecond hash function (as opposed to replacing the hash function to a\nnew one), you would allow people to call the same object by two\nnames.  That has interesting implications.\n\nLet's say you have a blob at path F in a top-level tree object and\ncreate a commit.  You have three objects in total, the tree knows\nthe blob as one name based on SHA-1 and the commit knows the tree as\none name based on SHA-1.  The same contents of the blob and the tree\ncould have different names based on SHA-256 in the future Git.\n\nLet's further say you have a future Git and clone from the above\nrepository with three objects.  You get a pack stream, containing\nthe data for one commit, tree and blob each.  These objects do not\ncarry their own name as extra pieces of information.  You only get\ntheir contents, and it is up to you to name them by hashing.  .idx\nfiles are created by running index-pack while receiving the pack\ndata stream.  You _somehow_ need to know that these three objects\nneed to be hashed with SHA-1, even though you are SHA-256 capable,\nbecause otherwise the object name recorded in the tree object for\nthe blob would not match what your .idx file would call the blob\ndata.  Also the object name recorded in the ref to point at the\ncommit would not match the commit object's object name, unless you\nhash with SHA-1.  It is a possibility to always hash these objects\ntwice and record _both_ hashes in the updated .idx file; after all,\n.idx files are strictly local matter.\n\nNow let's further say that you update the file F in the working\ntree, and do \"git commit -a\" with updated version of Git.  What\nshould happen?  Assuming that we are trying to migrate to a\ndifferent hashing algorithm over time, we would want to create a new\nblob under object name based on SHA-256, add that to the index and\nwrite a new tree out, named by hashing with SHA-256.  We then record\nthat longer-named tree in a commit whose parent commit is still\nnamed with SHA-1 based hash, and the new commit in turn is named by\nhashing with SHA-256.\n\nThen you push the result back.  Let's assume by now the place you\ncloned from is also SHA-256 capable.  You look at the tips of refs\nat your clone-source and discover that you would need to only send\nthe new commit, its tree and the updated blob.  You send data in\nthese three objects.  The receiving end would now need to do the\nsame \"magically choose hash to make sure the new blob gets the name\nthat is recorded in the new tree (and the new tree the new commit)\"\nthing.  The same discussion applies if somebody else clones from you\nat this point.  The objects introduced by the second commit all need\nto be hashed with the new hash to be named, while the other objects\nneed to be hashed with the old hash.\n\nContinuing this thought process, I do not see a good way to allow us\nto wean ourselves off of the old hash, unless we _break_ the pack\nstream format so that each object in the pack carries not just the\ndata but also the hash algorithm to be used to _name_ it, so that\nnew objects will never be referred to using the old hash.\n\nIt matters performance-wise that the weaning process go as quickly\nas possible, once the system becomes capable of new hash algorighm,\nbecause during the transition period, we'd have to suffer the full\ntree-diff becoming inefficient (Note: don't limit your thinking to\njust \"git diff\" and \"git log\"; the same inefficiency hits \"git\ncheckout\" to switch branches and \"git merge\" to walk three trees in\nparallel), because we cannot skip descending into subdirectories\nbased on the tree object name being equal, which guarantees that\neverything under the hierarchy is equal.\n\n\n"},{"id":"291678","messageId":"CACBZZX4N37Ty2yZosHA+PexscLc_Y4_QDZNktVM8bza_5cGu+g@mail.gmail.com","threadId":"42863","inReplyTo":"CAPp-Vra5ri7V03SSkU6=UnWrsq-jk0XcNpQuzGH8kQxE-TkPjA@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2016-07-18T20:02:21Z","receivedAt":"2016-07-18T20:02:58Z","isPatch":false,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"On Mon, Jul 18, 2016 at 7:48 PM, Herczeg Zsolt <zsolt94@gmail.com> wrote:\n>> In particular, as far as I know and as Theodore Ts'o's post describes\n>> better than I could[1], you seem to be confusing preimage attacks with\n>> collision attacks, and then concluding that because SHA1 is vulnerable\n>> to collision attacks that use-cases that would need a preimage attack\n>> to be compromised (which as far is I can tell, includes all your\n>> examples) are also \"broken\".\n>\n> I understand the differences between the collision and preimage\n> attacks.\n\nFair enough. The rest of your E-Mail certainly shows that you do, and\nI didn't know enough anything about GitTorrent and this case where\nit's vulnerable to collission attacks.\n\nBut I didn't get that impression from your initial E-Mail which\noutright said said:\n\n    Git signed tags and signed commits are cryptographically\n    insecure, they're useless at the moment.\n\nIt's important that those of us who *do* understand the difference\nbetween collision and preimage attacks carefully phrase things, least\nthey turn into FUD.\n\nYour initial E-Mail does *not* make it sound like you're just talking\nabout the cases where someone's provided you with a crafted blob that\nyou've been tricked into signing, but rather makes it sound like\nsigned tags & commits are just categorically broken, even for preimage\nattacks, which is not the case.\n\nThe reality of the current situation is that it's largely mitigated in\npractice because:\n\na) it's hard to hand someone a crafted blob to begin with for reasons\nthat have nothing to do with SHA-1 (they'll go \"wtf is this garbage?\")\n\nb) even in that case it's *very* hard to come up with two colliding\nblobs that are *useful* for some nefarious purpose, e.g. a program A\nthat looks normal being replaced by an evil program B with the same\nSHA-1.\n"},{"id":"291680","messageId":"alpine.DEB.2.02.1607181258160.25425@nftneq.ynat.uz","threadId":"42863","inReplyTo":"CAPp-Vra5ri7V03SSkU6=UnWrsq-jk0XcNpQuzGH8kQxE-TkPjA@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2016-07-18T20:01:54Z","receivedAt":"2016-07-18T20:07:30Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Mon, 18 Jul 2016, Herczeg Zsolt wrote:\n\n>> In particular, as far as I know and as Theodore Ts'o's post describes\n>> better than I could[1], you seem to be confusing preimage attacks with\n>> collision attacks, and then concluding that because SHA1 is vulnerable\n>> to collision attacks that use-cases that would need a preimage attack\n>> to be compromised (which as far is I can tell, includes all your\n>> examples) are also \"broken\".\n>\n> I understand the differences between the collision and preimage\n> attacks. A collision attack is not that bad for git in a typical\n> use-case. But I think that it's important to note that there are many\n> use-cases which do need a hash safe from collision attack. Some\n> examples:\n>\n> You maintain a repository with gittorrent with signed commits Others\n> can use these signatures to verify it's original. Let's say you\n> include some safe file (potentially binary) from a third-party\n> contributor. That would be fine if the hash algo is safe. Currently\n> there is the possibility that you received a (safe) file which was\n> made to collide with another malicious one. Once you committed (and\n> signed) that file, the attacker joins the gittorrent network and\n> starts to distribute the malicious file. Suddenly most of your clients\n> pulling are infected however your signature is correct.\n>\n> Or, you would like to make a continuous delivery system, where you use\n> signed tags. The delivery happens only when signature is right, and\n> the signer is responsible for it. Your colleague makes a collision,\n> pushes the good-file. You make all the tests, everything is fine, sign\n> and push and wait for the delivery to happen. Your colleague changes\n> the file on the server. The delivery makes a huge mass, and you're\n> fired.\n>\n> Or, let's say you use a service like github, which is nice enough to\n> make a repository for you, with .gitignore, licenses and everything.\n> Likely, you'll never change dose files. Let's say that service made\n> one of those initial files to collide something bad. That means, they\n> can \"infect\" anyone, who is pulling your repo.\n>\n> Do you need more hypothetical stories? There are a lot. Of course they\n> need a lot of work, and they're unlikely to happen. But it's possible.\n> If you need trust, and gpg signatures that means you need ultimate\n> trust. What's the point in making GPG signatures anyway if you cannot\n> ultimately trust them? You could just as well say: well that's\n> repository is only reachable by trustworthy persons, everything here\n> is just fine and really made by the person named in the \"author\n> field\".\n\nAll of your examples are actually preimage attacks. If the bad guy can tamper \nwith the both the 'safe' and 'malicious' versions of the file, they don't \nactually need the malicious version, they can attack you through the one you \nthink is 'safe'\n\nThe 'collision' attack isn't that there is some increased chance of a random \nfile colliding with your safe file, it's that if you are manipulating the \ncontents of both files, you can create two that collide. This won't hurt a Git \nrepository unless one of these manipulated files is able to be introduced as a \nlegitimate part of the repo you are dealing with.\n\nDavid Lang\n"},{"id":"291684","messageId":"xmqq7fciis45.fsf@gitster.mtv.corp.google.com","threadId":"42863","inReplyTo":"CACBZZX4N37Ty2yZosHA+PexscLc_Y4_QDZNktVM8bza_5cGu+g@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-18T20:55:06Z","receivedAt":"2016-07-18T20:55:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:\n\n> The reality of the current situation is that it's largely mitigated in\n> practice because:\n>\n> a) it's hard to hand someone a crafted blob to begin with for reasons\n> that have nothing to do with SHA-1 (they'll go \"wtf is this garbage?\")\n>\n> b) even in that case it's *very* hard to come up with two colliding\n> blobs that are *useful* for some nefarious purpose, e.g. a program A\n> that looks normal being replaced by an evil program B with the same\n> SHA-1.\n\nThanks.  That's a nice rephrasing of\n\n  http://public-inbox.org/git/Pine.LNX.4.58.0504291221250.18901%40ppc970.osdl.org/\n\nwhere Linus explains SHA-1 is not the security, and the real\nsecurity is in distribution.\n"},{"id":"291687","messageId":"20160718212626.GD29326@google.com","threadId":"42863","inReplyTo":"xmqq4m7mltbw.fsf@gitster.mtv.corp.google.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2016-07-18T21:26:26Z","receivedAt":"2016-07-18T21:26:50Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Junio C Hamano wrote:\n\n> Continuing this thought process, I do not see a good way to allow us\n> to wean ourselves off of the old hash, unless we _break_ the pack\n> stream format so that each object in the pack carries not just the\n> data but also the hash algorithm to be used to _name_ it, so that\n> new objects will never be referred to using the old hash.\n\nTaking a step further: I don't think that any backward-compatible\nformat change would address the security concerns with sufficiently\nold hashing algorithms.\n\nAs long as my favorite repository is allowed to contain objects\nidentified by SHA-1, my adversary can exploit a SHA-1 collision using\nsigned tags referring (possibly indirectly) to backdated objects.  The\nGit object format does not include a proof of commit date, so I cannot\nguarantee \"Only old objects are named by SHA-1\".\n\nThere is a way to get a backward-compatible *user experience* without\nthe format change being backward-compatible, though.  Name all objects\nin the repository using FuturisticHash.  Also store enough information\nto recover the old hashes, either in objects as a new field or in a\nside table.\n\nIf the old hash is broken, signatures using the old hash cannot be\ntrusted.  An adversary could generate a collision to retroactively\nchange the meaning of an existing signature.  To maintain the meaning\nof old signatures, someone has to record the new names of all involved\nobjects, either before the state of the art in breaking the old hash\nadvances far enough or using a copy of the repository from before the\nstate of the art had advanced --- in effect you need new signatures to\nmaintain the meaning of old signatures.  This could happen as part of\nthe process of updating a repository to use a new hash.\n\nE.g.\n\n\tobject a787a87b98a7s98798a798b7a98b798a7b98a7b987a9b87a9b87a98b79a87b98a7b98a7b987a987987a878a78a\n\tsha1tag object 04b871796dc0420f8e7561a895b52484b701d51a\n\t type commit\n\t tag signedtag\n\t tagger C O Mitter <committer@example.com> 1465981006 +0000\n\n\t signed tag\n\n\t signed tag message body\n\t -----BEGIN PGP SIGNATURE-----\n\t Version: GnuPG v1\n\n\t iQEcBAABAgAGBQJXYRhOAAoJEGEJLoW3InGJklkIAIcnhL7RwEb/+QeX9enkXhxn\n\t rxfdqrvWd1K80sl2TOt8Bg/NYwrUBw/RWJ+sg/hhHp4WtvE1HDGHlkEz3y11Lkuh\n\t 8tSxS3qKTxXUGozyPGuE90sJfExhZlW4knIQ1wt/yWqM+33E9pN4hzPqLwyrdods\n\t q8FWEqPPUbSJXoMbRPw04S5jrLtZSsUWbRYjmJCHzlhSfFWW4eFd37uquIaLUBS0\n\t rkC3Jrx7420jkIpgFcTI2s60uhSQLzgcCwdA2ukSYIRnjg/zDkj8+3h/GaROJ72x\n\t lZyI6HWixKJkWw8lE9aAOD9TmTW9sFJwcVAzmAuFX2kUreDUKMZduGcoRYGpD7E=\n\t =jpXa\n\t -----END PGP SIGNATURE-----\n\t-----BEGIN PGP SIGNATURE----\n\t...\n\t-----END PGP SIGNATURE\n\nThis example uses a signature to attest that mapping\n04b871796dc0420f8e7561a895b52484b701d51a->a787a87b98a7s98798a798b7a98b798a7b98a7b987a9b87a9b87a98b79a87b98a7b98a7b987a987987a878a78a\nis correct.  A more straightforward approach would be for the\nconversion process to produce an out-of-band signed mapping list to\nmake the sha1tag usable without such a signature.\n\nSummary:\n * Git's properties depend on using a single hash function throughout\n   a repository.  I don't think we should change that.\n\n * A safe and mostly painless migration to a stronger hash function is\n   possible using a signed assertion (for example generated by the\n   conversion process) of the mapping from old object names to new\n   object names.\n\n * Dealing with multiple such signed mappings (for example due to\n   separate conversion of repositories based on linux.git) is left as\n   an exercise to the reader.\n\nHope that helps,\nJonathan\n"},{"id":"291689","messageId":"CAPp-VrZoJ3JwUTZ9Bjv2NwLUE3ia9fNcanbujFB5P+Kw8hAzoQ@mail.gmail.com","threadId":"42863","inReplyTo":"xmqq7fciis45.fsf@gitster.mtv.corp.google.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Herczeg Zsolt","fromEmail":"zsolt94@gmail.com","sentAt":"2016-07-18T21:28:20Z","receivedAt":"2016-07-18T21:28:25Z","isPatch":false,"sender":{"key":"zsolt94@gmail.com","avatar":"https://gravatar.com/avatar/6a29b7ff538efff0a47e635c4bd50ad9f17b3744caf13aa053bec0127a29fce3?d=mp&s=160"},"body":">> The reality of the current situation is that it's largely mitigated in\n>> practice because:\n>>\n>> a) it's hard to hand someone a crafted blob to begin with for reasons\n>> that have nothing to do with SHA-1 (they'll go \"wtf is this garbage?\")\n>>\n>> b) even in that case it's *very* hard to come up with two colliding\n>> blobs that are *useful* for some nefarious purpose, e.g. a program A\n>> that looks normal being replaced by an evil program B with the same\n>> SHA-1.\n>\n> Thanks.  That's a nice rephrasing of\n>\n>   http://public-inbox.org/git/Pine.LNX.4.58.0504291221250.18901%40ppc970.osdl.org/\n>\n> where Linus explains SHA-1 is not the security, and the real\n> security is in distribution.\n\nIf the real security is in the distribution, than why git supports\nsigned commits and objects?\n\nThe security of the signatures do depend on the hash. Saying the hash\nis not a security feature and offering GPG signing based on that hash\nis a damn big lie. You can change the hash algorithm to a secure one,\nor change the signing method to be independent of the hash algorithm,\nor you can stop offering signatures at all, but something has to be\ndone here.\n"},{"id":"291694","messageId":"20160718224450.GF6644@vauxhall.crustytoothpaste.net","threadId":"42863","inReplyTo":"alpine.DEB.2.20.1607180853300.28832@virtualbox","subject":"Re: Git and SHA-1 security (again)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2016-07-18T22:44:50Z","receivedAt":"2016-07-18T22:45:03Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Jul 18, 2016 at 09:00:06AM +0200, Johannes Schindelin wrote:\n> Hi Brian,\n> \n> On Sun, 17 Jul 2016, brian m. carlson wrote:\n> \n> > On Sun, Jul 17, 2016 at 10:01:38AM +0200, Johannes Schindelin wrote:\n> > > Out of curiosity: have you considered something like padding the SHA-1s\n> > > with, say 0xa1, to the size of the new hash and using that padding to\n> > > distinguish between old vs new hash?\n> > \n> > I'm going to end up having to do something similar because of the issue\n> > of submodules.  Submodules may still be SHA-1, while the main repo may\n> > be a newer hash.  I was going to zero-pad, however.\n> \n> I thought about zero-padding, but there are plenty of\n> is_null_sha1()/is_null_oid() calls around. Of course, I assumed\n> left-padding. But you may have thought of right-padding instead? That\n> would make short name handling much easier, too.\n\nI was going to right-pad.\n\n> FWIW it never crossed my mind to allow different same-sized hash\n> algorithms. So I never thought we'd need a way to distinguish, say,\n> BLAKE2b-256 from SHA-256.\n> \n> Is there a good reason to add the maintenance burden of several 256-bit\n> hash algorithms, apart from speed (which in my mind should decide which\n> one to use, always, rather than letting the user choose)? It would also\n> complicate transport even further, let alone subtree merges from\n> differently-hashed repositories.\n\nThere are really three candidates:\n\n* SHA-256 (the SHA-2 algorithm): While this looks good right now,\n  cryptanalysis is advancing.  This is not a good choice for a long-term\n  solution.\n* SHA3-256 (the SHA-3 algorithm): This is the conservative choice.  It's\n  also faster than SHA-256 on 64-bit systems.  It has a very\n  conservative security margin and is a good long-term choice.\n* BLAKE2b-256: This is the blisteringly fast choice.  It outperforms\n  SHA-1 and even MD5 on 64-bit systems.  This algorithm was designed so\n  that nobody would have a reason to use an insecure algorithm.  It will\n  probably be secure for some time, but maybe not as long as SHA3-256.\n\nI'm only considering 256-bit hashes, because anything longer won't fit\non an 80-column terminal in hex form.\n\nThe reason I had considered implementing both SHA3-256 and BLAKE2b-256\nis that I want there to be no reason not to upgrade.  People who need a\nFIPS-approved algorithm or want a long-term, conservative choice should\nuse SHA3-256.  People who want even better performance than current Git\nwould use BLAKE2b-256.\n\nPerformance comparison (my implementations):\nSHA-1:     437 MiB/s\nSHA-256:   196 MiB/s\nSHA3-256:  273 MiB/s\nBLAKE2b:   649 MiB/s\n\nI hadn't thought about subtree merges, though.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"291697","messageId":"20160718230324.GG6644@vauxhall.crustytoothpaste.net","threadId":"42863","inReplyTo":"xmqq4m7mltbw.fsf@gitster.mtv.corp.google.com","subject":"Re: Git and SHA-1 security (again)","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2016-07-18T23:03:25Z","receivedAt":"2016-07-18T23:03:49Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Jul 18, 2016 at 11:00:35AM -0700, Junio C Hamano wrote:\n> Continuing this thought process, I do not see a good way to allow us\n> to wean ourselves off of the old hash, unless we _break_ the pack\n> stream format so that each object in the pack carries not just the\n> data but also the hash algorithm to be used to _name_ it, so that\n> new objects will never be referred to using the old hash.\n\nI think for this reason, I'm going to propose the following approach\nwhen we get there:\n\n* We serialize the hash in the object formats, using multihash or\n  something similar.  This means that it is minimally painful if we ever\n  need to change in the future[0].\n* Each repository carries exactly one hash algorithm, except for\n  submodule data.  If we don't do this, then some people will never\n  switch because the submodules they depend on haven't.\n* If people on the new format need to refer to submodule commits using\n  SHA-1, then they have to use a prefix on the hash form; otherwise,\n  they can use the raw hash value (without any multihash prefix).\n* git fsck verifies one consistent algorithm (excepting submodule\n  references).\n\nThis preserves the security benefits, avoids future-proofing problems,\nand minimizes performance impacts due to naming like you mentioned.\n\n[0] We are practically limited to 256-bit hashes because anything longer\nwill wrap on an 80-column terminal when in hex form.\n-- \nbrian m. carlson / brian with sandals: Houston, Texas, US\n+1 832 623 2791 | https://www.crustytoothpaste.net/~bmc | My opinion only\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"291708","messageId":"alpine.DEB.2.20.1607190910370.3472@virtualbox","threadId":"42863","inReplyTo":"CACsJy8Ba=c+-WV2TsY768_fTDO2KesS1b6BK7kdykNY6gkh=UQ@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-19T07:18:50Z","receivedAt":"2016-07-19T07:19:32Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Mon, 18 Jul 2016, Duy Nguyen wrote:\n\n> On Mon, Jul 18, 2016 at 5:57 PM, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n> >\n> > On Mon, 18 Jul 2016, Herczeg Zsolt wrote:\n> >\n> >> >> I think converting is a much better option. Use a single-hash\n> >> >> storage, and convert everything to that on import/clone/pull.\n> >> >\n> >> > That ignores two very important issues that I already had mentioned:\n> >>\n> >> That's not true. If you double-check the next part of my message, you I\n> >> just showed that an automatic two-way mapping could solve these\n> >> problems! (I even give briefs explanation how to handle referencing and\n> >> signature verification in those cases.)\n> >>\n> >> My point is not to throw out old hashes and break signatures. My point\n> >> is to convert the data storage, and use mapping to resolve problems\n> >> with those old hashes and signatures.\n> >\n> > If you convert the data storage, then the SHA-1s listed in the commit\n> > objects will have to be rewritten, and then the GPG signature will not\n> > match anymore.\n> \n> But we can recreate SHA-1 from the same content and verify GPG, right?\n> I know it's super expensive, but it feels safer to not carry SHA-1\n> around when it's not secure anymore (I recall something about\n> exploiting the weakest link when you have both sha1 and sha256 in the\n> object content). Rehashing would be done locally and is better\n> controlled.\n\nYou could. But how would you determine whether to recreate the commit\nobject from a SHA-1-ified version of the commit buffer? Fall back if the\noriginal did not match the signature? That would pose at least these two\nproblems:\n\n1. The point of a signature is trust. If all of a sudden the signature\ndoes not match what is supposedly signed, that trust is broken.\n\n2. The point of going to a stronger hash is to increase the trust. If\nany developer could decide to sign the SHA-1-ified version of any future\ncommit, and Git validating it, it would be even worse than not switching\nto a new hash: it would leave us open to collision attacks *and* pretend\nthat we prevented such attacks.\n\nThe more I think about it, the more I am convinced that we have no choice\nbut allow mixed hashes (i.e. both 160-bit SHA-1 and 256-bit new hash,\nwhatever we settle on). Otherwise there would be no reliable and\ntrustworthy upgrade path.\n\nBut maybe there is a clever strategy I failed to think of?\n\nCiao,\nDscho\n"},{"id":"291709","messageId":"alpine.DEB.2.20.1607190920020.3472@virtualbox","threadId":"42863","inReplyTo":"CAPp-Vra8c7eFegysKrM-hZKNmD6B21XqVkE8HB0QcG_m8hu+Ow@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-19T07:21:46Z","receivedAt":"2016-07-19T07:22:21Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Zsolt,\n\nOn Mon, 18 Jul 2016, Herczeg Zsolt wrote:\n\n> >> My point is not to throw out old hashes and break signatures. My point\n> >> is to convert the data storage, and use mapping to resolve problems\n> >> with those old hashes and signatures.\n> >\n> > If you convert the data storage, then the SHA-1s listed in the commit\n> > objects will have to be rewritten, and then the GPG signature will not\n> > match anymore.\n> >\n> > Call e.g. `git cat-file commit 44cc742a8ca17b9c279be4cc195a93a6ef7a320e`\n> > to see the anatomy of a gpg-signed commit object.\n> >\n> \n> Yes and no. That's the reason you need the two-way lookup table. If\n> you need to verify a commit which was signed as SHA-1, you must use\n> the lookup table in reverse.\n\nThat pretends that it is both easy and trustworthy to know when (and how)\nto recreate the SHA-1-ified version of the commit object.\n\nNeither is the case, though.\n\nCiao,\nJohannes\n"},{"id":"291710","messageId":"alpine.DEB.2.20.1607190926420.3472@virtualbox","threadId":"42863","inReplyTo":"CACsJy8AH9Q6rOgvcWGsLGPaP96koGA=k0PYgXP6F3RZ=XAwaSw@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-19T07:31:31Z","receivedAt":"2016-07-19T07:36:05Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Mon, 18 Jul 2016, Duy Nguyen wrote:\n\n> On Sun, Jul 17, 2016 at 4:21 PM, brian m. carlson\n> <sandals@crustytoothpaste.net> wrote:\n> > I'm going to end up having to do something similar because of the issue\n> > of submodules.  Submodules may still be SHA-1, while the main repo may\n> > be a newer hash.\n> \n> Or even the other way around, main repo is one with sha1 while\n> submodule is on sha256. I wonder if we should address this separately\n> (and even in parallel with sha256 support), making submodules work\n> with an any external VCS system (that supports some basic operations\n> we define).\n\nIt is safe to assume that any project using a submodule with a more secure\nhash would require Git tooling capable of said hash. It would hence make\nno sense to use SHA-1 for the super project.\n\nSo I do not believe that we have to support the use case of a SHA-1-based\nproject using SHA-256-based submodules.\n\nCiao,\nDscho\n"},{"id":"291711","messageId":"alpine.DEB.2.02.1607190045240.2015@nftneq.ynat.uz","threadId":"42863","inReplyTo":"alpine.DEB.2.20.1607190926420.3472@virtualbox","subject":"Re: Git and SHA-1 security (again)","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2016-07-19T07:46:50Z","receivedAt":"2016-07-19T07:47:10Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Tue, 19 Jul 2016, Johannes Schindelin wrote:\n\n> Hi Duy,\n>\n> On Mon, 18 Jul 2016, Duy Nguyen wrote:\n>\n>> On Sun, Jul 17, 2016 at 4:21 PM, brian m. carlson\n>> <sandals@crustytoothpaste.net> wrote:\n>>> I'm going to end up having to do something similar because of the issue\n>>> of submodules.  Submodules may still be SHA-1, while the main repo may\n>>> be a newer hash.\n>>\n>> Or even the other way around, main repo is one with sha1 while\n>> submodule is on sha256. I wonder if we should address this separately\n>> (and even in parallel with sha256 support), making submodules work\n>> with an any external VCS system (that supports some basic operations\n>> we define).\n>\n> It is safe to assume that any project using a submodule with a more secure\n> hash would require Git tooling capable of said hash. It would hence make\n> no sense to use SHA-1 for the super project.\n>\n> So I do not believe that we have to support the use case of a SHA-1-based\n> project using SHA-256-based submodules.\n\nthey have different upstreams, what if the upstream of the submodule has \nupgraded and is using signed commits of the sha-256 but the upstream of the \nparent hasn't and is using signed commits of sha1?\n\nDavid Lang\n"},{"id":"291724","messageId":"CACsJy8CSUu=AemQ-7uxth_2M=ko_KDGsdObwYdiE=L4OMKcVZw@mail.gmail.com","threadId":"42863","inReplyTo":"alpine.DEB.2.20.1607190910370.3472@virtualbox","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-19T15:31:59Z","receivedAt":"2016-07-19T15:32:33Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 19, 2016 at 9:18 AM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n>> But we can recreate SHA-1 from the same content and verify GPG, right?\n>> I know it's super expensive, but it feels safer to not carry SHA-1\n>> around when it's not secure anymore (I recall something about\n>> exploiting the weakest link when you have both sha1 and sha256 in the\n>> object content). Rehashing would be done locally and is better\n>> controlled.\n>\n> You could. But how would you determine whether to recreate the commit\n> object from a SHA-1-ified version of the commit buffer? Fall back if the\n> original did not match the signature?\n\nAny repo would have a cut point when they move to sha256 (or whatever\nnew hash), if we can record this somewhere (e.g. as a tag or a bunch\nof tags, or some dummy commits to mark the heads of the repo) then we\nonly verify gpg signatures _in_ the repository before this point.\n\n> That would pose at least these two problems:\n>\n> 1. The point of a signature is trust. If all of a sudden the signature\n> does not match what is supposedly signed, that trust is broken.\n>\n> 2. The point of going to a stronger hash is to increase the trust. If\n> any developer could decide to sign the SHA-1-ified version of any future\n> commit, and Git validating it, it would be even worse than not switching\n> to a new hash: it would leave us open to collision attacks *and* pretend\n> that we prevented such attacks.\n\nGPG signatures are still valid on the old repo (we will keep old repos\naround forever, I suppose). And because they sign on the \"weak\" hash,\nsha1, at some point they will be broken (but until then we can still\nregenerate sha1 and verify locally). When sha1 is broken, GPG\nsignatures of the past can't be trusted anymore.\n\nIf people care enough about the past, they should re-sign (at least\nfor tags). Commits can be re-signed by the person who does the\nconversion. Yes you have to trust that person. Sort of a painful fresh\nstart, with hopefully better security.\n-- \nDuy\n"},{"id":"291727","messageId":"CACsJy8CR_fkYL5UYbV1MqiTSe3gTqWfOrA1NOHTw09vZn7Y-Aw@mail.gmail.com","threadId":"42863","inReplyTo":"CACsJy8AH9Q6rOgvcWGsLGPaP96koGA=k0PYgXP6F3RZ=XAwaSw@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-19T16:07:35Z","receivedAt":"2016-07-19T16:08:12Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Jul 18, 2016 at 6:51 PM, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Sun, Jul 17, 2016 at 4:21 PM, brian m. carlson\n> <sandals@crustytoothpaste.net> wrote:\n>> I'm going to end up having to do something similar because of the issue\n>> of submodules.  Submodules may still be SHA-1, while the main repo may\n>> be a newer hash.\n>\n> Or even the other way around, main repo is one with sha1 while\n> submodule is on sha256. I wonder if we should address this separately\n> (and even in parallel with sha256 support), making submodules work\n> with an any external VCS system (that supports some basic operations\n> we define).\n\nPost-shower thoughts. In a tree object, a submodule entry consists of\nperm (S_IFGITLINK), hash (which is the external hash) and path. We\ncould fill the \"hash\" part with all zero (invalid, signature of new\nsubmodule hash format), then append \"/<hashtype>:<external hash>\" to\nthe \"path\" part. This way we don't have to update tree object or index\nformat. And I suspect the \"path\" part is available everywhere we need\nto handle submodules already, so extracting the external hash should\nbe possible...\n-- \nDuy\n"},{"id":"291732","messageId":"xmqqk2ghh80x.fsf@gitster.mtv.corp.google.com","threadId":"42863","inReplyTo":"CACsJy8CR_fkYL5UYbV1MqiTSe3gTqWfOrA1NOHTw09vZn7Y-Aw@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-19T17:06:38Z","receivedAt":"2016-07-19T17:06:45Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n> Post-shower thoughts. In a tree object, a submodule entry consists of\n> perm (S_IFGITLINK), hash (which is the external hash) and path. We\n> could fill the \"hash\" part with all zero (invalid, signature of new\n> submodule hash format), then append \"/<hashtype>:<external hash>\" to\n> the \"path\" part. This way we don't have to update tree object or index\n> format. And I suspect the \"path\" part is available everywhere we need\n> to handle submodules already, so extracting the external hash should\n> be possible...\n\nEven though that single operation might be possible, do not go\nthere.  A \"pathname\" identifies a \"path\", not its contents, and\n\"appending crap after path\" breaks the data model badly.  Also other\nthings like merge, checkout and diff would break by butchering\nordering the entries in tree objects.\n"},{"id":"291734","messageId":"CACsJy8DyybW5kTWZ2nJ4GN=S46M9rU0EapOrR6PjSV=b1ZFrBw@mail.gmail.com","threadId":"42863","inReplyTo":"xmqqk2ghh80x.fsf@gitster.mtv.corp.google.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-19T17:27:02Z","receivedAt":"2016-07-19T17:27:49Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 19, 2016 at 7:06 PM, Junio C Hamano <gitster@pobox.com> wrote:\n> Duy Nguyen <pclouds@gmail.com> writes:\n>\n>> Post-shower thoughts. In a tree object, a submodule entry consists of\n>> perm (S_IFGITLINK), hash (which is the external hash) and path. We\n>> could fill the \"hash\" part with all zero (invalid, signature of new\n>> submodule hash format), then append \"/<hashtype>:<external hash>\" to\n>> the \"path\" part. This way we don't have to update tree object or index\n>> format. And I suspect the \"path\" part is available everywhere we need\n>> to handle submodules already, so extracting the external hash should\n>> be possible...\n>\n> Even though that single operation might be possible, do not go\n> there.  A \"pathname\" identifies a \"path\", not its contents, and\n> \"appending crap after path\" breaks the data model badly.  Also other\n> things like merge, checkout and diff would break by butchering\n> ordering the entries in tree objects.\n\nI thought about that but I thought all those operations required\nspecial treatment for submodules anyway. But I forgot about d/f\nconflicts so yeah it's a bad idea. We still have some invalid \"mode\"\ncombination that can be used as S_IFGITLINK2, then we can have\nvariable length hash field in the entry.\n-- \nDuy\n"},{"id":"291735","messageId":"alpine.DEB.2.02.1607191032270.25425@nftneq.ynat.uz","threadId":"42863","inReplyTo":"CACsJy8CSUu=AemQ-7uxth_2M=ko_KDGsdObwYdiE=L4OMKcVZw@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2016-07-19T17:34:34Z","receivedAt":"2016-07-19T17:34:52Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Tue, 19 Jul 2016, Duy Nguyen wrote:\n\n> On Tue, Jul 19, 2016 at 9:18 AM, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n>>> But we can recreate SHA-1 from the same content and verify GPG, right?\n>>> I know it's super expensive, but it feels safer to not carry SHA-1\n>>> around when it's not secure anymore (I recall something about\n>>> exploiting the weakest link when you have both sha1 and sha256 in the\n>>> object content). Rehashing would be done locally and is better\n>>> controlled.\n>>\n>> You could. But how would you determine whether to recreate the commit\n>> object from a SHA-1-ified version of the commit buffer? Fall back if the\n>> original did not match the signature?\n>\n> Any repo would have a cut point when they move to sha256 (or whatever\n> new hash), if we can record this somewhere (e.g. as a tag or a bunch\n> of tags, or some dummy commits to mark the heads of the repo) then we\n> only verify gpg signatures _in_ the repository before this point.\n\nremember that a repo doesn't have a single 'now', each branch has it's own head, \nand you can easily go back to prior points and branch off from there.\n\nSince timestamps in repos can't be trusted (different people's clocks may not be \nin sync), how would you define this cutoff point?\n\nDavid Lang\n"},{"id":"291736","messageId":"CACsJy8Bvqt9r2dRtRfx1C-3Fp16z3SJ=hp0i7-itwFwfEfZwTw@mail.gmail.com","threadId":"42863","inReplyTo":"alpine.DEB.2.02.1607191032270.25425@nftneq.ynat.uz","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-19T17:43:49Z","receivedAt":"2016-07-19T17:44:24Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 19, 2016 at 7:34 PM, David Lang <david@lang.hm> wrote:\n> On Tue, 19 Jul 2016, Duy Nguyen wrote:\n>\n>> On Tue, Jul 19, 2016 at 9:18 AM, Johannes Schindelin\n>> <Johannes.Schindelin@gmx.de> wrote:\n>>>>\n>>>> But we can recreate SHA-1 from the same content and verify GPG, right?\n>>>> I know it's super expensive, but it feels safer to not carry SHA-1\n>>>> around when it's not secure anymore (I recall something about\n>>>> exploiting the weakest link when you have both sha1 and sha256 in the\n>>>> object content). Rehashing would be done locally and is better\n>>>> controlled.\n>>>\n>>>\n>>> You could. But how would you determine whether to recreate the commit\n>>> object from a SHA-1-ified version of the commit buffer? Fall back if the\n>>> original did not match the signature?\n>>\n>>\n>> Any repo would have a cut point when they move to sha256 (or whatever\n>> new hash), if we can record this somewhere (e.g. as a tag or a bunch\n>> of tags, or some dummy commits to mark the heads of the repo) then we\n>> only verify gpg signatures _in_ the repository before this point.\n>\n>\n> remember that a repo doesn't have a single 'now', each branch has it's own\n> head, and you can easily go back to prior points and branch off from there.\n>\n> Since timestamps in repos can't be trusted (different people's clocks may\n> not be in sync), how would you define this cutoff point?\n\nThe set of all heads at the time the conversion happens (maybe plus\nall the real tags). We can make an octopus merge commit to cover all\nthe heads, then it can be the reference point.\n-- \nDuy\n"},{"id":"291738","messageId":"alpine.DEB.2.02.1607191057170.25425@nftneq.ynat.uz","threadId":"42863","inReplyTo":"CACsJy8Bvqt9r2dRtRfx1C-3Fp16z3SJ=hp0i7-itwFwfEfZwTw@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"David Lang","fromEmail":"david@lang.hm","sentAt":"2016-07-19T17:59:20Z","receivedAt":"2016-07-19T17:59:37Z","isPatch":false,"sender":{"key":"david@lang.hm","avatar":null},"body":"On Tue, 19 Jul 2016, Duy Nguyen wrote:\n\n> On Tue, Jul 19, 2016 at 7:34 PM, David Lang <david@lang.hm> wrote:\n>> On Tue, 19 Jul 2016, Duy Nguyen wrote:\n>>\n>>> On Tue, Jul 19, 2016 at 9:18 AM, Johannes Schindelin\n>>> <Johannes.Schindelin@gmx.de> wrote:\n>>>>>\n>>>>> But we can recreate SHA-1 from the same content and verify GPG, right?\n>>>>> I know it's super expensive, but it feels safer to not carry SHA-1\n>>>>> around when it's not secure anymore (I recall something about\n>>>>> exploiting the weakest link when you have both sha1 and sha256 in the\n>>>>> object content). Rehashing would be done locally and is better\n>>>>> controlled.\n>>>>\n>>>>\n>>>> You could. But how would you determine whether to recreate the commit\n>>>> object from a SHA-1-ified version of the commit buffer? Fall back if the\n>>>> original did not match the signature?\n>>>\n>>>\n>>> Any repo would have a cut point when they move to sha256 (or whatever\n>>> new hash), if we can record this somewhere (e.g. as a tag or a bunch\n>>> of tags, or some dummy commits to mark the heads of the repo) then we\n>>> only verify gpg signatures _in_ the repository before this point.\n>>\n>>\n>> remember that a repo doesn't have a single 'now', each branch has it's own\n>> head, and you can easily go back to prior points and branch off from there.\n>>\n>> Since timestamps in repos can't be trusted (different people's clocks may\n>> not be in sync), how would you define this cutoff point?\n>\n> The set of all heads at the time the conversion happens (maybe plus\n> all the real tags). We can make an octopus merge commit to cover all\n> the heads, then it can be the reference point.\n\nso to make sure I'm understanding this, anything not reachable from that merge \nmust be the new hash, correct? Including forks, merges, etc that happen from \nearlier points in the history.\n\nDavid Lang\n"},{"id":"291740","messageId":"CACsJy8D1RtwVF4ZtRHV2Z=huTqRBp8Du5GMZq9qxwXDZezBF2g@mail.gmail.com","threadId":"42863","inReplyTo":"alpine.DEB.2.02.1607191057170.25425@nftneq.ynat.uz","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-19T18:04:45Z","receivedAt":"2016-07-19T18:05:39Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 19, 2016 at 7:59 PM, David Lang <david@lang.hm> wrote:\n> On Tue, 19 Jul 2016, Duy Nguyen wrote:\n>\n>> On Tue, Jul 19, 2016 at 7:34 PM, David Lang <david@lang.hm> wrote:\n>>>\n>>> On Tue, 19 Jul 2016, Duy Nguyen wrote:\n>>>\n>>>> On Tue, Jul 19, 2016 at 9:18 AM, Johannes Schindelin\n>>>> <Johannes.Schindelin@gmx.de> wrote:\n>>>>>>\n>>>>>>\n>>>>>> But we can recreate SHA-1 from the same content and verify GPG, right?\n>>>>>> I know it's super expensive, but it feels safer to not carry SHA-1\n>>>>>> around when it's not secure anymore (I recall something about\n>>>>>> exploiting the weakest link when you have both sha1 and sha256 in the\n>>>>>> object content). Rehashing would be done locally and is better\n>>>>>> controlled.\n>>>>>\n>>>>>\n>>>>>\n>>>>> You could. But how would you determine whether to recreate the commit\n>>>>> object from a SHA-1-ified version of the commit buffer? Fall back if\n>>>>> the\n>>>>> original did not match the signature?\n>>>>\n>>>>\n>>>>\n>>>> Any repo would have a cut point when they move to sha256 (or whatever\n>>>> new hash), if we can record this somewhere (e.g. as a tag or a bunch\n>>>> of tags, or some dummy commits to mark the heads of the repo) then we\n>>>> only verify gpg signatures _in_ the repository before this point.\n>>>\n>>>\n>>>\n>>> remember that a repo doesn't have a single 'now', each branch has it's\n>>> own\n>>> head, and you can easily go back to prior points and branch off from\n>>> there.\n>>>\n>>> Since timestamps in repos can't be trusted (different people's clocks may\n>>> not be in sync), how would you define this cutoff point?\n>>\n>>\n>> The set of all heads at the time the conversion happens (maybe plus\n>> all the real tags). We can make an octopus merge commit to cover all\n>> the heads, then it can be the reference point.\n>\n>\n> so to make sure I'm understanding this, anything not reachable from that\n> merge must be the new hash, correct? Including forks, merges, etc that\n> happen from earlier points in the history.\n\nYes everything except that merge and everything reachable from it, the\nwhole old clone, basically.\n-- \nDuy\n"},{"id":"291747","messageId":"xmqqpoq9fotg.fsf@gitster.mtv.corp.google.com","threadId":"42863","inReplyTo":"CACsJy8DyybW5kTWZ2nJ4GN=S46M9rU0EapOrR6PjSV=b1ZFrBw@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-19T18:46:51Z","receivedAt":"2016-07-19T18:46:59Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Duy Nguyen <pclouds@gmail.com> writes:\n\n>> Even though that single operation might be possible, do not go\n>> there.  A \"pathname\" identifies a \"path\", not its contents, and\n>> \"appending crap after path\" breaks the data model badly.\n>\n> I thought about that but I thought all those operations required\n> special treatment for submodules anyway.\n\nOperatins requiring special treatment does not make it right to\nbreak the data model anyway, so...\n"},{"id":"291751","messageId":"CAPp-VrZ1YjUyPYwgH2CD1fohU1R+uSizyVE6QwMXHcztvcQqSQ@mail.gmail.com","threadId":"42863","inReplyTo":"CACsJy8D1RtwVF4ZtRHV2Z=huTqRBp8Du5GMZq9qxwXDZezBF2g@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Herczeg Zsolt","fromEmail":"zsolt94@gmail.com","sentAt":"2016-07-19T18:58:14Z","receivedAt":"2016-07-19T18:58:34Z","isPatch":false,"sender":{"key":"zsolt94@gmail.com","avatar":"https://gravatar.com/avatar/6a29b7ff538efff0a47e635c4bd50ad9f17b3744caf13aa053bec0127a29fce3?d=mp&s=160"},"body":"2016-07-19 20:04 GMT+02:00 Duy Nguyen <pclouds@gmail.com>:\n> On Tue, Jul 19, 2016 at 7:59 PM, David Lang <david@lang.hm> wrote:\n>> On Tue, 19 Jul 2016, Duy Nguyen wrote:\n>>\n>>> On Tue, Jul 19, 2016 at 7:34 PM, David Lang <david@lang.hm> wrote:\n>>>>\n>>>> On Tue, 19 Jul 2016, Duy Nguyen wrote:\n>>>>\n>>>>> On Tue, Jul 19, 2016 at 9:18 AM, Johannes Schindelin\n>>>>> <Johannes.Schindelin@gmx.de> wrote:\n>>>>>>>\n>>>>>>>\n>>>>>>> But we can recreate SHA-1 from the same content and verify GPG, right?\n>>>>>>> I know it's super expensive, but it feels safer to not carry SHA-1\n>>>>>>> around when it's not secure anymore (I recall something about\n>>>>>>> exploiting the weakest link when you have both sha1 and sha256 in the\n>>>>>>> object content). Rehashing would be done locally and is better\n>>>>>>> controlled.\n>>>>>>\n>>>>>>\n>>>>>>\n>>>>>> You could. But how would you determine whether to recreate the commit\n>>>>>> object from a SHA-1-ified version of the commit buffer? Fall back if\n>>>>>> the\n>>>>>> original did not match the signature?\n>>>>>\n>>>>>\n>>>>>\n>>>>> Any repo would have a cut point when they move to sha256 (or whatever\n>>>>> new hash), if we can record this somewhere (e.g. as a tag or a bunch\n>>>>> of tags, or some dummy commits to mark the heads of the repo) then we\n>>>>> only verify gpg signatures _in_ the repository before this point.\n>>>>\n>>>>\n>>>>\n>>>> remember that a repo doesn't have a single 'now', each branch has it's\n>>>> own\n>>>> head, and you can easily go back to prior points and branch off from\n>>>> there.\n>>>>\n>>>> Since timestamps in repos can't be trusted (different people's clocks may\n>>>> not be in sync), how would you define this cutoff point?\n>>>\n>>>\n>>> The set of all heads at the time the conversion happens (maybe plus\n>>> all the real tags). We can make an octopus merge commit to cover all\n>>> the heads, then it can be the reference point.\n>>\n>>\n>> so to make sure I'm understanding this, anything not reachable from that\n>> merge must be the new hash, correct? Including forks, merges, etc that\n>> happen from earlier points in the history.\n>\n> Yes everything except that merge and everything reachable from it, the\n> whole old clone, basically.\n\nIt could work, but does it worth it?\n\n1) If you use multihash, you should assume that anything with SHA1\ncould be manipulated. That means you can \"inject\" something later to\nthat \"old clone\" anyway.\n2) Even if the content is re-hashed, it's hard to understand for a\nuser where the trust comes from. The user should decide weather he\ntrust (or not) the person who signed that octopus breakpoint.\n\nEven without git you can achieve this security: Get the complete old\nrepository, make a signed tarball of it. If anytime later you want to\ncheck that signatures, you can just use that tarball. I don't think\nit's worth the trouble to create a native method for something which\nis rare, and can be worked around easily. It's actually easier for a\nuser to understand the \"trust relation\" when using this workaround.\n\nReferring to that signed-tarball approach, you may just as well drop\nall signature data on conversion... As long as you can look up the\nreferences to old hashes easily, I think it's usable enough.\n"},{"id":"291799","messageId":"alpine.DEB.2.20.1607201428030.14111@virtualbox","threadId":"42863","inReplyTo":"CACsJy8D1RtwVF4ZtRHV2Z=huTqRBp8Du5GMZq9qxwXDZezBF2g@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-20T12:28:07Z","receivedAt":"2016-07-20T12:29:15Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Duy,\n\nOn Tue, 19 Jul 2016, Duy Nguyen wrote:\n\n> On Tue, Jul 19, 2016 at 7:59 PM, David Lang <david@lang.hm> wrote:\n> > On Tue, 19 Jul 2016, Duy Nguyen wrote:\n> >\n> >> On Tue, Jul 19, 2016 at 7:34 PM, David Lang <david@lang.hm> wrote:\n> >>>\n> >>> On Tue, 19 Jul 2016, Duy Nguyen wrote:\n> >>>\n> >>>> On Tue, Jul 19, 2016 at 9:18 AM, Johannes Schindelin\n> >>>> <Johannes.Schindelin@gmx.de> wrote:\n> >>>>>>\n> >>>>>>\n> >>>>>> But we can recreate SHA-1 from the same content and verify GPG,\n> >>>>>> right?  I know it's super expensive, but it feels safer to not\n> >>>>>> carry SHA-1 around when it's not secure anymore (I recall\n> >>>>>> something about exploiting the weakest link when you have both\n> >>>>>> sha1 and sha256 in the object content). Rehashing would be done\n> >>>>>> locally and is better controlled.\n> >>>>>\n> >>>>> You could. But how would you determine whether to recreate the\n> >>>>> commit object from a SHA-1-ified version of the commit buffer?\n> >>>>> Fall back if the original did not match the signature?\n> >>>>\n> >>>> Any repo would have a cut point when they move to sha256 (or\n> >>>> whatever new hash), if we can record this somewhere (e.g. as a tag\n> >>>> or a bunch of tags, or some dummy commits to mark the heads of the\n> >>>> repo) then we only verify gpg signatures _in_ the repository before\n> >>>> this point.\n> >>>\n> >>> remember that a repo doesn't have a single 'now', each branch has\n> >>> it's own head, and you can easily go back to prior points and branch\n> >>> off from there.\n> >>>\n> >>> Since timestamps in repos can't be trusted (different people's\n> >>> clocks may not be in sync), how would you define this cutoff point?\n> >>\n> >> The set of all heads at the time the conversion happens (maybe plus\n> >> all the real tags). We can make an octopus merge commit to cover all\n> >> the heads, then it can be the reference point.\n> >\n> > so to make sure I'm understanding this, anything not reachable from\n> > that merge must be the new hash, correct? Including forks, merges, etc\n> > that happen from earlier points in the history.\n> \n> Yes everything except that merge and everything reachable from it, the\n> whole old clone, basically.\n\nBut that strategy *still* ignores the distributed nature of Git. Just\nbecause *you* make that merge at a certain point does not necessarily mean\nthat I make it at that point, too.\n\nAny approach that tries to have one single point of conversion will most\nlikely fall short of a solution.\n\nTo be honest, I am less concerned about the GPG-signed commits (after all,\nafter switching to a more secure hash algorithm, a maintainer could\ncross-sign all signed commits, or only the branch tips or tags, as new\ntags, to reinstitute trust).\n\nI am much more concerned about references to commits, both inside and\noutside the repository. That is, if I read anywhere on the internet about\nGit having added support for `git add --chmod=+x <file>` in 4e55ed3 (add:\nadd --chmod=+x / --chmod=-x options, 2016-05-31), I want to find that\ncommit by that reference.\n\nAnd I am of course concerned what should happen if a user wants to fetch\nfrom, or push to, a SHA-1-hashed remote repository into, or from, a\nSHA-256-hashed local one.\n\nCiao,\nDscho\n"},{"id":"291811","messageId":"CACsJy8A3ZTfRRNuYXYD-nzXm6KXHu8cHU6fzfysJxfW0EggKNA@mail.gmail.com","threadId":"42863","inReplyTo":"alpine.DEB.2.20.1607201428030.14111@virtualbox","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-20T14:44:07Z","receivedAt":"2016-07-20T14:45:00Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Wed, Jul 20, 2016 at 2:28 PM, Johannes Schindelin\n<Johannes.Schindelin@gmx.de> wrote:\n> But that strategy *still* ignores the distributed nature of Git. Just\n> because *you* make that merge at a certain point does not necessarily mean\n> that I make it at that point, too.\n>\n> Any approach that tries to have one single point of conversion will most\n> likely fall short of a solution.\n\nOK I see the difference in our views now. To me an sha256 repo would\nsee an sha1 repo as a _foreign_ DVCS, pretty much like git sees\nmercurial now. So a transition from sha1 to sha256 is not that\ndifferent from cvs -> svn -> a dvcs bubble -> git.\n\n> To be honest, I am less concerned about the GPG-signed commits (after all,\n> after switching to a more secure hash algorithm, a maintainer could\n> cross-sign all signed commits, or only the branch tips or tags, as new\n> tags, to reinstitute trust).\n>\n> I am much more concerned about references to commits, both inside and\n> outside the repository. That is, if I read anywhere on the internet about\n> Git having added support for `git add --chmod=+x <file>` in 4e55ed3 (add:\n> add --chmod=+x / --chmod=-x options, 2016-05-31), I want to find that\n> commit by that reference.\n>\n> And I am of course concerned what should happen if a user wants to fetch\n> from, or push to, a SHA-1-hashed remote repository into, or from, a\n> SHA-256-hashed local one.\n\nto follow the above, in my view, interaction with sha1 repos go\nthrough some conversion bridges like what we have with hg and svn. I\ndon't know if we are going this route. It's certainly simpler and\npeople already have experiences (from previous migration) to prepare\nfor it.\n-- \nDuy\n"},{"id":"291812","messageId":"CACsJy8CgDCROJURW9ERTRX1HwjwEDE5jXwFqg2P9+JhL2v_Knw@mail.gmail.com","threadId":"42863","inReplyTo":"CAPp-VrZ1YjUyPYwgH2CD1fohU1R+uSizyVE6QwMXHcztvcQqSQ@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2016-07-20T14:48:14Z","receivedAt":"2016-07-20T14:48:53Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Tue, Jul 19, 2016 at 8:58 PM, Herczeg Zsolt <zsolt94@gmail.com> wrote:\n> 2016-07-19 20:04 GMT+02:00 Duy Nguyen <pclouds@gmail.com>:\n>> On Tue, Jul 19, 2016 at 7:59 PM, David Lang <david@lang.hm> wrote:\n>>> On Tue, 19 Jul 2016, Duy Nguyen wrote:\n>>>\n>>>> On Tue, Jul 19, 2016 at 7:34 PM, David Lang <david@lang.hm> wrote:\n>>>>>\n>>>>> On Tue, 19 Jul 2016, Duy Nguyen wrote:\n>>>>>\n>>>>>> On Tue, Jul 19, 2016 at 9:18 AM, Johannes Schindelin\n>>>>>> <Johannes.Schindelin@gmx.de> wrote:\n>>>>>>>>\n>>>>>>>>\n>>>>>>>> But we can recreate SHA-1 from the same content and verify GPG, right?\n>>>>>>>> I know it's super expensive, but it feels safer to not carry SHA-1\n>>>>>>>> around when it's not secure anymore (I recall something about\n>>>>>>>> exploiting the weakest link when you have both sha1 and sha256 in the\n>>>>>>>> object content). Rehashing would be done locally and is better\n>>>>>>>> controlled.\n>>>>>>>\n>>>>>>>\n>>>>>>>\n>>>>>>> You could. But how would you determine whether to recreate the commit\n>>>>>>> object from a SHA-1-ified version of the commit buffer? Fall back if\n>>>>>>> the\n>>>>>>> original did not match the signature?\n>>>>>>\n>>>>>>\n>>>>>>\n>>>>>> Any repo would have a cut point when they move to sha256 (or whatever\n>>>>>> new hash), if we can record this somewhere (e.g. as a tag or a bunch\n>>>>>> of tags, or some dummy commits to mark the heads of the repo) then we\n>>>>>> only verify gpg signatures _in_ the repository before this point.\n>>>>>\n>>>>>\n>>>>>\n>>>>> remember that a repo doesn't have a single 'now', each branch has it's\n>>>>> own\n>>>>> head, and you can easily go back to prior points and branch off from\n>>>>> there.\n>>>>>\n>>>>> Since timestamps in repos can't be trusted (different people's clocks may\n>>>>> not be in sync), how would you define this cutoff point?\n>>>>\n>>>>\n>>>> The set of all heads at the time the conversion happens (maybe plus\n>>>> all the real tags). We can make an octopus merge commit to cover all\n>>>> the heads, then it can be the reference point.\n>>>\n>>>\n>>> so to make sure I'm understanding this, anything not reachable from that\n>>> merge must be the new hash, correct? Including forks, merges, etc that\n>>> happen from earlier points in the history.\n>>\n>> Yes everything except that merge and everything reachable from it, the\n>> whole old clone, basically.\n>\n> It could work, but does it worth it?\n>\n> 1) If you use multihash, you should assume that anything with SHA1\n> could be manipulated. That means you can \"inject\" something later to\n> that \"old clone\" anyway.\n\nNo it's not multihash. The repo only uses sha256, but by substituting\nit with sha1 using the same dag, we can recreate the exact same sha1\nrepo (up to the conversion point). This is mostly to avoid people\ninjecting something because _you_ generate the repo locally.\n\n> 2) Even if the content is re-hashed, it's hard to understand for a\n> user where the trust comes from. The user should decide weather he\n> trust (or not) the person who signed that octopus breakpoint.\n>\n> Even without git you can achieve this security: Get the complete old\n> repository, make a signed tarball of it. If anytime later you want to\n> check that signatures, you can just use that tarball. I don't think\n> it's worth the trouble to create a native method for something which\n> is rare, and can be worked around easily. It's actually easier for a\n> user to understand the \"trust relation\" when using this workaround.\n>\n> Referring to that signed-tarball approach, you may just as well drop\n> all signature data on conversion... As long as you can look up the\n> references to old hashes easily, I think it's usable enough.\n\nIt's more or less the signed-tarball approach in my view, except that\nyou recreate that tarball dynamically with your sha256 repo (so this\ntarball is \"signed\" with sha256).\n-- \nDuy\n"},{"id":"291834","messageId":"CAGZ79kYzhPayXKq5LrddKTuO7d5=7_AZVvr7rJ4ztAzBFv-7eg@mail.gmail.com","threadId":"42863","inReplyTo":"CACsJy8A3ZTfRRNuYXYD-nzXm6KXHu8cHU6fzfysJxfW0EggKNA@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2016-07-20T17:10:02Z","receivedAt":"2016-07-20T17:10:11Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jul 20, 2016 at 7:44 AM, Duy Nguyen <pclouds@gmail.com> wrote:\n> On Wed, Jul 20, 2016 at 2:28 PM, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n>> But that strategy *still* ignores the distributed nature of Git. Just\n>> because *you* make that merge at a certain point does not necessarily mean\n>> that I make it at that point, too.\n>>\n>> Any approach that tries to have one single point of conversion will most\n>> likely fall short of a solution.\n>\n> OK I see the difference in our views now. To me an sha256 repo would\n> see an sha1 repo as a _foreign_ DVCS, pretty much like git sees\n> mercurial now. So a transition from sha1 to sha256 is not that\n> different from cvs -> svn -> a dvcs bubble -> git.\n>\n>> To be honest, I am less concerned about the GPG-signed commits (after all,\n>> after switching to a more secure hash algorithm, a maintainer could\n>> cross-sign all signed commits, or only the branch tips or tags, as new\n>> tags, to reinstitute trust).\n>>\n>> I am much more concerned about references to commits, both inside and\n>> outside the repository. That is, if I read anywhere on the internet about\n>> Git having added support for `git add --chmod=+x <file>` in 4e55ed3 (add:\n>> add --chmod=+x / --chmod=-x options, 2016-05-31), I want to find that\n>> commit by that reference.\n>>\n>> And I am of course concerned what should happen if a user wants to fetch\n>> from, or push to, a SHA-1-hashed remote repository into, or from, a\n>> SHA-256-hashed local one.\n>\n> to follow the above, in my view, interaction with sha1 repos go\n> through some conversion bridges like what we have with hg and svn. I\n> don't know if we are going this route. It's certainly simpler and\n> people already have experiences (from previous migration) to prepare\n> for it.\n\nWhen treating the SHA1 version as a foreign dvcs and the SHA256\nas the real deal, we could introduce \"pointer objects\", and during the\nconversion\ncreate a 4e55ed3 pointer that points to the SHA256 commit of (add:\nadd --chmod=+x / --chmod=-x options, 2016-05-31). Ideally we would\nnot even expose this sort of object a lot, e.g. git show <pointer> would just\nredirect automatically. Instead of a new class of \"pointer objects\" we could\nalso solve this via a lot of refs. (refs/old-sha1/4e55ed3 pointing to\nthe converted\ncommit; Though we would need to accept partial refs names then :/)\n\n> --\n> Duy\n> --\n> To unsubscribe from this list: send the line \"unsubscribe git\" in\n> the body of a message to majordomo@vger.kernel.org\n> More majordomo info at  http://vger.kernel.org/majordomo-info.html\n"},{"id":"291850","messageId":"xmqq1t2odsaz.fsf@gitster.mtv.corp.google.com","threadId":"42863","inReplyTo":"CAGZ79kYzhPayXKq5LrddKTuO7d5=7_AZVvr7rJ4ztAzBFv-7eg@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-20T19:26:44Z","receivedAt":"2016-07-20T19:26:53Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller <sbeller@google.com> writes:\n\n>> to follow the above, in my view, interaction with sha1 repos go\n>> through some conversion bridges like what we have with hg and svn. I\n>> don't know if we are going this route. It's certainly simpler and\n>> people already have experiences (from previous migration) to prepare\n>> for it.\n>\n> When treating the SHA1 version as a foreign dvcs and the SHA256\n> as the real deal, we could introduce \"pointer objects\", and during the\n> conversion\n> create a 4e55ed3 pointer that points to the SHA256 commit of (add:\n> add --chmod=+x / --chmod=-x options, 2016-05-31).\n\nHmmm.  If you are designing this \"pointer objects\" to be extensible\nenough to cover other foreign vcs (i.e.e.g. you make it to be\ncapable of mapping Subversion's r24323 to a matching commit in the\nconverted result), I would think it may be a very useful thing to\nhave, but I think it is pretty much orthogonal to the discussion in\nthis topic.  IOW, that can happen with or without change of the hash\nfunction.\n\nAnd looking at it that way, I am not sure if such a mapping feature\nshould require adding a new type of \"object\".\n"},{"id":"291911","messageId":"alpine.DEB.2.20.1607211451060.14111@virtualbox","threadId":"42863","inReplyTo":"xmqq4m7mltbw.fsf@gitster.mtv.corp.google.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-21T12:53:33Z","receivedAt":"2016-07-21T12:54:54Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 18 Jul 2016, Junio C Hamano wrote:\n\n> \"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n> \n> > I will say that the pack format will likely require some changes,\n> > because it assumes ...  The reason is that we can't have an\n> > unambiguous parse of the current objects if two hash algorithms are in\n> > use....  So when we look at a new hash, we need to provide an\n> > unambiguous way to know what hash is in use.  The two choices are to\n> > either require all object use the new hash, or to extend the objects\n> > to include the hash.  Until a couple days ago, I had planned to do the\n> > former.  I had not even considered using a multihash approach due to\n> > the complexity.\n> \n> Objects in Git identify themselves, but once you introduce the second\n> hash function (as opposed to replacing the hash function to a new one),\n> you would allow people to call the same object by two names.  That has\n> interesting implications.\n> \n> [...]\n\nSo essentially you are saying that the multi-hash approach has too many\nnegative implications, right? At least that is what I understand.\n\nLooks more and more like we do need to convert repositories wholesale, and\nkeep a two-way mapping for talking to remote repositories.\n\nWould you concur?\n\nCiao,\nDscho\n"},{"id":"291913","messageId":"alpine.DEB.2.20.1607211454061.14111@virtualbox","threadId":"42863","inReplyTo":"20160718230324.GG6644@vauxhall.crustytoothpaste.net","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-21T13:19:29Z","receivedAt":"2016-07-21T13:21:01Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brian,\n\nOn Mon, 18 Jul 2016, brian m. carlson wrote:\n\n> On Mon, Jul 18, 2016 at 11:00:35AM -0700, Junio C Hamano wrote:\n> > Continuing this thought process, I do not see a good way to allow us\n> > to wean ourselves off of the old hash, unless we _break_ the pack\n> > stream format so that each object in the pack carries not just the\n> > data but also the hash algorithm to be used to _name_ it, so that\n> > new objects will never be referred to using the old hash.\n> \n> I think for this reason, I'm going to propose the following approach\n> when we get there:\n> \n> * We serialize the hash in the object formats, using multihash or\n>   something similar.  This means that it is minimally painful if we ever\n>   need to change in the future[0].\n\nThis adds a lot of redundancy, though, and has an adverse performance\nimpact, no?\n\nCould we not simply require packs to identify the used hash *once*, and\nuse a single hash algorithm per repository?\n\nThat would mean that we would have to re-hash packs on-the-fly if, say,\ntalking to a SHA-1 remote from a SHA-256 local repository.\n\n> * Each repository carries exactly one hash algorithm, except for\n>   submodule data.  If we don't do this, then some people will never\n>   switch because the submodules they depend on haven't.\n\nIf we re-hash transparently, we could get away with SHA-256 even for\nsubmodules.\n\n> * If people on the new format need to refer to submodule commits using\n>   SHA-1, then they have to use a prefix on the hash form; otherwise,\n>   they can use the raw hash value (without any multihash prefix).\n> * git fsck verifies one consistent algorithm (excepting submodule\n>   references).\n> \n> This preserves the security benefits, avoids future-proofing problems,\n> and minimizes performance impacts due to naming like you mentioned.\n> \n> [0] We are practically limited to 256-bit hashes because anything longer\n> will wrap on an 80-column terminal when in hex form.\n\nWe are not really bound by the 80-column limit when choosing a hash\nalgorithm. We typically refer to a commit by a shorter name, and the\n80-column limit applies only to Git's own source code.\n\nCiao,\nDscho\n"},{"id":"291914","messageId":"alpine.DEB.2.20.1607211520170.14111@virtualbox","threadId":"42863","inReplyTo":"20160718224450.GF6644@vauxhall.crustytoothpaste.net","subject":"Re: Git and SHA-1 security (again)","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2016-07-21T14:13:39Z","receivedAt":"2016-07-21T14:15:06Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Brian,\n\nOn Mon, 18 Jul 2016, brian m. carlson wrote:\n\n> On Mon, Jul 18, 2016 at 09:00:06AM +0200, Johannes Schindelin wrote:\n> \n> > FWIW it never crossed my mind to allow different same-sized hash\n> > algorithms. So I never thought we'd need a way to distinguish, say,\n> > BLAKE2b-256 from SHA-256.\n> > \n> > Is there a good reason to add the maintenance burden of several 256-bit\n> > hash algorithms, apart from speed (which in my mind should decide which\n> > one to use, always, rather than letting the user choose)? It would also\n> > complicate transport even further, let alone subtree merges from\n> > differently-hashed repositories.\n> \n> There are really three candidates:\n> \n> * SHA-256 (the SHA-2 algorithm): While this looks good right now,\n>   cryptanalysis is advancing.  This is not a good choice for a long-term\n>   solution.\n> * SHA3-256 (the SHA-3 algorithm): This is the conservative choice.  It's\n>   also faster than SHA-256 on 64-bit systems.  It has a very\n>   conservative security margin and is a good long-term choice.\n> * BLAKE2b-256: This is the blisteringly fast choice.  It outperforms\n>   SHA-1 and even MD5 on 64-bit systems.  This algorithm was designed so\n>   that nobody would have a reason to use an insecure algorithm.  It will\n>   probably be secure for some time, but maybe not as long as SHA3-256.\n> \n> I'm only considering 256-bit hashes, because anything longer won't fit\n> on an 80-column terminal in hex form.\n> \n> The reason I had considered implementing both SHA3-256 and BLAKE2b-256\n> is that I want there to be no reason not to upgrade.  People who need a\n> FIPS-approved algorithm or want a long-term, conservative choice should\n> use SHA3-256.  People who want even better performance than current Git\n> would use BLAKE2b-256.\n> \n> Performance comparison (my implementations):\n> SHA-1:     437 MiB/s\n> SHA-256:   196 MiB/s\n> SHA3-256:  273 MiB/s\n> BLAKE2b:   649 MiB/s\n\nThose are impressive numbers on BLAKE2b. However, Keccak was chosen as\nSHA-3 because it can be implemented in hardware more efficiently than\nBLAKE (and hence, probably, also BLAKE2). Given that there are already SSE\ninstructions implementing SHA-1/SHA-256 on some CPUs [*1*], I would not be\nsurprised if SHA3 would also see some hardware support.\n\nSo speed seems less of a concern to me. We are talking about a multi-year\nroadmap, after all.\n\nAnd given the complications for public repository hosters, I would like to\nsettle for a single 256-bit hash. That'll be challenging enough.\n\nCiao,\nDscho\n\nFootnote *1*: https://en.wikipedia.org/wiki/Intel_SHA_extensions\n"},{"id":"291968","messageId":"xmqqvazxbr4h.fsf@gitster.mtv.corp.google.com","threadId":"42863","inReplyTo":"alpine.DEB.2.20.1607211451060.14111@virtualbox","subject":"Re: Git and SHA-1 security (again)","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2016-07-22T15:59:42Z","receivedAt":"2016-07-22T15:59:54Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Johannes Schindelin <Johannes.Schindelin@gmx.de> writes:\n\n> Hi Junio,\n>\n> On Mon, 18 Jul 2016, Junio C Hamano wrote:\n>\n>> \"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n>> \n>> > I will say that the pack format will likely require some changes,\n>> > because it assumes ...  The reason is that we can't have an\n>> > unambiguous parse of the current objects if two hash algorithms are in\n>> > use....  So when we look at a new hash, we need to provide an\n>> > unambiguous way to know what hash is in use.  The two choices are to\n>> > either require all object use the new hash, or to extend the objects\n>> > to include the hash.  Until a couple days ago, I had planned to do the\n>> > former.  I had not even considered using a multihash approach due to\n>> > the complexity.\n>> \n>> Objects in Git identify themselves, but once you introduce the second\n>> hash function (as opposed to replacing the hash function to a new one),\n>> you would allow people to call the same object by two names.  That has\n>> interesting implications.\n>> \n>> [...]\n>\n> So essentially you are saying that the multi-hash approach has too many\n> negative implications, right? At least that is what I understand.\n>\n> Looks more and more like we do need to convert repositories wholesale, and\n> keep a two-way mapping for talking to remote repositories.\n>\n> Would you concur?\n\nNot necessarily.\n\nThat was me thinking aloud, listing some issues that I would imagine\nto be tricky to solve, without even attempting to be exhaustive,\nthat I expect to see solved in a good end-result implementation.\nFor example, \"I do not see a nice way to solve X myself without\ndoing Y\" in the message you are responding to does not necessarily\nmean there is no good solution to X (just \"I do not think of any\noffhand\"), and it does not mean I think it is terrible that we have\nto do Y to solve X.\n\n\n"},{"id":"299874","messageId":"C651AA234E5A461BBA55E2D8D6E4DC85@PhilipOakley","threadId":"42863","inReplyTo":"CACsJy8A3ZTfRRNuYXYD-nzXm6KXHu8cHU6fzfysJxfW0EggKNA@mail.gmail.com","subject":"Re: Git and SHA-1 security (again)","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2016-08-22T22:01:00Z","receivedAt":"2016-08-22T22:01:23Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Sorry if I'm dropping in at the wrong point (this is one I'd bookmarked)..\n\n\n\nFrom: \"Duy Nguyen\" <pclouds@gmail.com>\nSent: Wednesday, July 20, 2016 3:44 PM\n> On Wed, Jul 20, 2016 at 2:28 PM, Johannes Schindelin\n> <Johannes.Schindelin@gmx.de> wrote:\n>> But that strategy *still* ignores the distributed nature of Git. Just\n>> because *you* make that merge at a certain point does not necessarily \n>> mean\n>> that I make it at that point, too.\n>>\n>> Any approach that tries to have one single point of conversion will most\n>> likely fall short of a solution.\n>\n> OK I see the difference in our views now. To me an sha256 repo would\n> see an sha1 repo as a _foreign_ DVCS, pretty much like git sees\n> mercurial now. So a transition from sha1 to sha256 is not that\n> different from cvs -> svn -> a dvcs bubble -> git.\n>\n\nI think that within Git, that it is possible to have inter-workability (for \nthose parts that negotiate) between instances with different views about the \navailability of two hash types. Fetch/push negotiation is a normal part of \nworking with a remote.\n\n>> To be honest, I am less concerned about the GPG-signed commits (after \n>> all,\n>> after switching to a more secure hash algorithm, a maintainer could\n>> cross-sign all signed commits, or only the branch tips or tags, as new\n>> tags, to reinstitute trust).\n>>\n>> I am much more concerned about references to commits, both inside and\n>> outside the repository. That is, if I read anywhere on the internet about\n>> Git having added support for `git add --chmod=+x <file>` in 4e55ed3 (add:\n>> add --chmod=+x / --chmod=-x options, 2016-05-31), I want to find that\n>> commit by that reference.\n>>\n>> And I am of course concerned what should happen if a user wants to fetch\n>> from, or push to, a SHA-1-hashed remote repository into, or from, a\n>> SHA-256-hashed local one.\n>\n> to follow the above, in my view, interaction with sha1 repos go\n> through some conversion bridges like what we have with hg and svn. I\n> don't know if we are going this route. It's certainly simpler and\n> people already have experiences (from previous migration) to prepare\n> for it.\n> -- \n\nThe main thought was that rather than worrying about which advanced hash to \npick (with all the arguments that entails), rather it is worth reducing the \nproblem space to create a 'toy problem', to look at the interaction issues.\n\nFor the toy problem view we'd keep the current oid length (so that the \ntransmission formats don't change size), however we swap the old-new to make \nsha1 the new hash and use an older shorter hash (e.g. md5) to investigate \nthe transition from a short to long hash.\n\nKeeping it as a 'toy problem' avoids folks having too much invested in the \nnew hash choice, rather the interworking can be more easily sorted, and some \nissue can be punted on (e.g. the choice of salt to extend the md5 to the \nsha1, and collisions therein).\n\n--\nPhilip\n\n"}]}