{"thread":{"id":"59082","subject":"Gitorious should use CRC128 / 256 / 512 instead of SHA-1","startedAt":"2023-01-13T13:17:38Z","lastAt":"2023-01-13T17:52:03Z","messageCount":26,"participants":["Hans Petter Selasky","Konstantin Khomoutov","rsbecker@nexbridge.com","Konstantin Ryabitsev","Philip Oakley"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"470253","messageId":"39dd1a00-786b-acf5-8a40-2425f7dab6cc@selasky.org","threadId":"59082","inReplyTo":null,"subject":"Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T12:59:44Z","receivedAt":"2023-01-13T13:17:38Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"Hi,\n\nCurrently GIT only supports cryptographic hashes for its commit tags.\n\nThat means:\n\n1) It's very difficult to edit the history without also recomputing the \nhash tags for all commits after the needed change-point, which then \nmeans references to a repository is broken.\n\n2) Only a single bit error in the main repository can break everything!\n\n3) Illicit contents may be present in binary blobs, which in the future \nmay be need to be removed without warrant and the only way to do that is \nby rebasing and force pushing, which will break \"everything\". It can be \neverything from child-porn to expired distribution licenses.\n\nMany people think that bit errors cannot happen because the memory uses \nECC and the file system uses cryptographic hashes to verify the \nintegrity of the data. But what many people forget about is that when \ncopying data from memory to disk, typically using a DMA channel data is \ncopied w/o any kind of integrity protection, because the integrity \nprotection is not end-to-end. The integrity protection is only per-link.\n\nTherefore I propose the following changes to GIT.\n\n1) Use a CRC128 / 256 or 512 non-cryptographic based hashing algorithm \nas default.\n\n2) Add support for a CRC fixup field, which usually is zero, but when \nmerges are needed, it can be non-zero, to allow the hash-tag-value to \nremain the same! This also allows for easy conversion of existing GIT \nrepositories to the new scheme.\n\n3) All git objects should be uncompressed.\n\nCRC-XXX can easily be used to correct multiple bit errors without any \nperformance overhead.\n\nPlease CC me. I'm not subscribed to this list.\n\n--HPS\n"},{"id":"470255","messageId":"20230113133059.snyjblh3sz2wzcnd@carbon","threadId":"59082","inReplyTo":"39dd1a00-786b-acf5-8a40-2425f7dab6cc@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Konstantin Khomoutov","fromEmail":"kostix@bswap.ru","sentAt":"2023-01-13T13:30:59Z","receivedAt":"2023-01-13T13:38:17Z","isPatch":false,"sender":{"key":"kostix@bswap.ru","avatar":null},"body":"On Fri, Jan 13, 2023 at 01:59:44PM +0100, Hans Petter Selasky wrote:\n\n> Currently GIT only supports cryptographic hashes for its commit tags.\n[...]\n\nhttps://github.com/git/git/blob/9bf691b78cf906751e65d65ba0c6ffdcd9a5a12c/Documentation/technical/hash-function-transition.txt\n\nIt's not clear why are you referring to Gitorious in your mail's subject and\nthen talk about Git.\n\n"},{"id":"470256","messageId":"446984f6-0d2e-04da-11a3-8b1481fac953@selasky.org","threadId":"59082","inReplyTo":"20230113133059.snyjblh3sz2wzcnd@carbon","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T13:39:37Z","receivedAt":"2023-01-13T13:46:34Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 14:30, Konstantin Khomoutov wrote:\n> On Fri, Jan 13, 2023 at 01:59:44PM +0100, Hans Petter Selasky wrote:\n> \n>> Currently GIT only supports cryptographic hashes for its commit tags.\n> [...]\n> \n> https://github.com/git/git/blob/9bf691b78cf906751e65d65ba0c6ffdcd9a5a12c/Documentation/technical/hash-function-transition.txt\n> \n> It's not clear why are you referring to Gitorious in your mail's subject and\n> then talk about Git.\n> \n\nHi,\n\nI thought that Git was short for Gitorious? My bad.\n\nThe document you refer to really highlights my concerns, that a strong \ncryptographic hash algorithm is the highway to hell.\n\nDo _not_ use a cryptographic hash for Git. Use plain good old CRC hashes.\n\nJust imagine the consequences of finding child porn inside a 10-year old \nfirmware binary blob in the Linux kernel. Will you just ignore it, or \nwill you fix it?\n\nThat's why I say, that it must be possible to forge the hashes by default.\n\n--HPS\n"},{"id":"470259","messageId":"009701d9275a$678416b0$368c4410$@nexbridge.com","threadId":"59082","inReplyTo":"446984f6-0d2e-04da-11a3-8b1481fac953@selasky.org","subject":"RE: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2023-01-13T14:21:56Z","receivedAt":"2023-01-13T14:30:56Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On January 13, 2023 8:40 AM, Hans Petter Selasky wrote:\n>On 1/13/23 14:30, Konstantin Khomoutov wrote:\n>> On Fri, Jan 13, 2023 at 01:59:44PM +0100, Hans Petter Selasky wrote:\n>>\n>>> Currently GIT only supports cryptographic hashes for its commit tags.\n>> [...]\n>>\n>> https://github.com/git/git/blob/9bf691b78cf906751e65d65ba0c6ffdcd9a5a1\n>> 2c/Documentation/technical/hash-function-transition.txt\n>>\n>> It's not clear why are you referring to Gitorious in your mail's\n>> subject and then talk about Git.\n>>\n>\n>Hi,\n>\n>I thought that Git was short for Gitorious? My bad.\n>\n>The document you refer to really highlights my concerns, that a strong\n>cryptographic hash algorithm is the highway to hell.\n>\n>Do _not_ use a cryptographic hash for Git. Use plain good old CRC hashes.\n>\n>Just imagine the consequences of finding child porn inside a 10-year old firmware\n>binary blob in the Linux kernel. Will you just ignore it, or will you fix it?\n>\n>That's why I say, that it must be possible to forge the hashes by default.\n\nI do not understand the goal of this request. If it is possible to forge hashes, then nothing in a git repository can ever be trusted. Signed content will no longer be verifiable. The whole Merkel Tree representing the commit history becomes easily corruptible by hackers and no upstream remote repository can ever be trusted - or someone's own if someone targets a repo with malware that rewrites hashes. Imagine a scenario when malware replaces a blob in a repo and then forges the hash to pretend that the replacement never occurred. Using git as a supply chain audit trail becomes impossible. This is a potential vector for ransomware invading the git ecosystem. This seems like a really fatal path to take for the product.\n\nThe advantage of how git functions is that it is possible to mirror or clone repositories, protecting from hardware errors. Repositories exist in distributed form, so there may be hundreds or thousands of copies in case someone's copy is corrupted by a disk or memory write error - so that takes hash reconstruction out of the requirement set. If the git architecture was based on a central repository model only, then this might be a reasonable request, but that is not how git works. If, for instance, a main GitHub repo is somehow corrupted, it can be repaired by a push --force or a clone from a different instance.\n\nUnless I am missing your point.\n--Randall\n\n"},{"id":"470260","messageId":"8a8fbe42-7809-f3e7-b233-6bef790254e1@selasky.org","threadId":"59082","inReplyTo":"009701d9275a$678416b0$368c4410$@nexbridge.com","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T14:42:48Z","receivedAt":"2023-01-13T14:57:23Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 15:21, rsbecker@nexbridge.com wrote:\n> On January 13, 2023 8:40 AM, Hans Petter Selasky wrote:\n>> On 1/13/23 14:30, Konstantin Khomoutov wrote:\n>>> On Fri, Jan 13, 2023 at 01:59:44PM +0100, Hans Petter Selasky wrote:\n>>>\n>>>> Currently GIT only supports cryptographic hashes for its commit tags.\n>>> [...]\n>>>\n>>> https://github.com/git/git/blob/9bf691b78cf906751e65d65ba0c6ffdcd9a5a1\n>>> 2c/Documentation/technical/hash-function-transition.txt\n>>>\n>>> It's not clear why are you referring to Gitorious in your mail's\n>>> subject and then talk about Git.\n>>>\n>>\n>> Hi,\n>>\n>> I thought that Git was short for Gitorious? My bad.\n>>\n>> The document you refer to really highlights my concerns, that a strong\n>> cryptographic hash algorithm is the highway to hell.\n>>\n>> Do _not_ use a cryptographic hash for Git. Use plain good old CRC hashes.\n>>\n>> Just imagine the consequences of finding child porn inside a 10-year old firmware\n>> binary blob in the Linux kernel. Will you just ignore it, or will you fix it?\n>>\n>> That's why I say, that it must be possible to forge the hashes by default.\n> \n\nHi,\n\n> I do not understand the goal of this request. If it is possible to forge hashes, then nothing in a git repository can ever be trusted. Signed content will no longer be verifiable. The whole Merkel Tree representing the commit history becomes easily corruptible by hackers and no upstream remote repository can ever be trusted - or someone's own if someone targets a repo with malware that rewrites hashes. Imagine a scenario when malware replaces a blob in a repo and then forges the hash to pretend that the replacement never occurred. Using git as a supply chain audit trail becomes impossible. This is a potential vector for ransomware invading the git ecosystem. This seems like a really fatal path to take for the product.\n\nIf a hacker replaces a blob, everyone on the project will see it, \nbecause such changes typically generate a commit e-mail. And then an \naction will be made to revoke the access of that hacker. Now a clever \nhacker wouldn't do that. A clever hacker would just flip one bit \nsomewhere in a random blob, looking like a hardware fault, and then \nforce the project to rewind to backups every day, because the repository \ncan no longer be verified.\n\n> The advantage of how git functions is that it is possible to mirror or clone repositories, protecting from hardware errors. Repositories exist in distributed form, so there may be hundreds or thousands of copies in case someone's copy is corrupted by a disk or memory write error - so that takes hash reconstruction out of the requirement set. If the git architecture was based on a central repository model only, then this might be a reasonable request, but that is not how git works. If, for instance, a main GitHub repo is somehow corrupted, it can be repaired by a push --force or a clone from a different instance.\n> \n\nThere is no advantage from protecting from hardware errors, unless you \ncan recover from them! Cryptographic hash algorithms are not suitable to \nrecover bits. They only tell data is OK or NOK, and if there is no \nbackup, you loose it! It is no solution for big repositories to rewind \nto backups just because of bit-flips. Such problems should be fixed w/o \nthe need to roll-back, because that stops the entire production!\n\n > it can be repaired by a push --force\n\nHobby projects can do that, but not big projects like FreeBSD and the \nLinux kernel.\n\n> Unless I am missing your point.\n\nYes, a little bit :-)\n\n--HPS\n\n"},{"id":"470261","messageId":"ba5f35d2-1c94-9216-6676-d845c73ad6c3@selasky.org","threadId":"59082","inReplyTo":"009701d9275a$678416b0$368c4410$@nexbridge.com","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T15:15:32Z","receivedAt":"2023-01-13T15:24:00Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 15:21, rsbecker@nexbridge.com wrote:\n> Signed content will no longer be verifiable. The whole Merkel Tree representing the commit history becomes easily corruptible by hackers\n\nHi,\n\nAs a long time open sourcer and hacker, I'm totally against signing \nsoftware. Is the GIT project going to build the new infrastructure for \nJohn-Deers new tractor firmware adventure? It is totally against the \nvalues of open source craftmanship.\n\nI don't think any of you crypto-enthusiasts understand how propritary \ncompanies use signed software to keep their power intact.\n\nThat's also an argument for using a non-crypto hash.\n\n--HPS\n"},{"id":"470262","messageId":"20230113153932.gsnt4t26tlzotw65@meerkat.local","threadId":"59082","inReplyTo":"446984f6-0d2e-04da-11a3-8b1481fac953@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2023-01-13T15:39:32Z","receivedAt":"2023-01-13T15:48:17Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Jan 13, 2023 at 02:39:37PM +0100, Hans Petter Selasky wrote:\n> Just imagine the consequences of finding child porn inside a 10-year old\n> firmware binary blob in the Linux kernel. Will you just ignore it, or will\n> you fix it?\n\nHow do you expect something like this would happen? A much more likely\nscenario would be someone contributing a binary blob that doesn't actually\nallow redistribution, and therefore would need to be purged from the\nrepository.\n\nWhen something like this happens, everyone is given a heads-up, the history is\nrewritten, and everyone moves on. It's a fairly routine procedure -- ask\nanyone who's ever committed an API key into their repo.\n\nGit supports history edits and everyone lives with it just fine -- I think you\nare under the impression that git is some kind of globally distributed\nblockchain where any history edit requires a consensus fork. It's not at all\nthe case.\n\n-K\n"},{"id":"470263","messageId":"20230113154516.jxm2cer4sogatayp@meerkat.local","threadId":"59082","inReplyTo":"8a8fbe42-7809-f3e7-b233-6bef790254e1@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2023-01-13T15:45:16Z","receivedAt":"2023-01-13T15:53:12Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Jan 13, 2023 at 03:42:48PM +0100, Hans Petter Selasky wrote:\n> > I do not understand the goal of this request. If it is possible to forge\n> > hashes, then nothing in a git repository can ever be trusted. Signed\n> > content will no longer be verifiable. The whole Merkel Tree representing\n> > the commit history becomes easily corruptible by hackers and no upstream\n> > remote repository can ever be trusted - or someone's own if someone\n> > targets a repo with malware that rewrites hashes. Imagine a scenario when\n> > malware replaces a blob in a repo and then forges the hash to pretend that\n> > the replacement never occurred. Using git as a supply chain audit trail\n> > becomes impossible. This is a potential vector for ransomware invading the\n> > git ecosystem. This seems like a really fatal path to take for the\n> > product.\n> \n\n> If a hacker replaces a blob, everyone on the project will see it, because\n> such changes typically generate a commit e-mail.\n\nI don't think you have a very clear picture of how git works.\n\n> And then an action will be made to revoke the access of that hacker. Now a\n> clever hacker wouldn't do that. A clever hacker would just flip one bit\n> somewhere in a random blob, looking like a hardware fault, and then force\n> the project to rewind to backups every day, because the repository can no\n> longer be verified.\n\nThat's not how it works at all. If there is a corrupted object, the admins of\nthe repository just put the correct object into place either from a backup or\nfrom another copy of the repository. There is no rewinding required.\n\n\n> There is no advantage from protecting from hardware errors, unless you can\n> recover from them! Cryptographic hash algorithms are not suitable to recover\n> bits. They only tell data is OK or NOK, and if there is no backup, you loose\n> it!\n\nThis is true about all digital media.\n\n> It is no solution for big repositories to rewind to backups just because\n> of bit-flips. Such problems should be fixed w/o the need to roll-back,\n> because that stops the entire production!\n\nNo it doesn't.\n\n> > it can be repaired by a push --force\n> \n> Hobby projects can do that, but not big projects like FreeBSD and the Linux\n> kernel.\n\nSure they can, but not due to missing objects (a corrupted object is just a\nmissing object). If, for some reason, Linus ever needs to remove something\nfrom linux.git, he will do it and just give a heads-up why and for what\nreason.\n\nI think you're misunderstanding some of the core principles of git.\n\n-K\n"},{"id":"470264","messageId":"6061d012-13b7-ca4b-5556-70875b65c887@selasky.org","threadId":"59082","inReplyTo":"20230113154516.jxm2cer4sogatayp@meerkat.local","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T15:50:04Z","receivedAt":"2023-01-13T15:59:11Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 16:45, Konstantin Ryabitsev wrote:\n> I think you're misunderstanding some of the core principles of git.\n\nMaybe, I'm usually commandering git via the terminal.\n\nBut if you say you can already edit stuff, why does the commit hash need \nto be cryptographic? I don't get that part. Yeah, I think of git commits \nlike blockchain.\n\n--HPS\n"},{"id":"470265","messageId":"d087568b-919e-61f8-c203-e59a2e0572c6@selasky.org","threadId":"59082","inReplyTo":"20230113154516.jxm2cer4sogatayp@meerkat.local","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T15:54:39Z","receivedAt":"2023-01-13T16:05:18Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 16:45, Konstantin Ryabitsev wrote:\n>   If, for some reason, Linus ever needs to remove something\n> from linux.git, he will do it and just give a heads-up why and for what\n> reason.\n\nThis gotta be a joke.\n\nThere are 46K forks of Linus Torvalds Linux kernel on GitHUB, and if \nLinus Torvalds one day decides to do a forced push, it will for sure be \na disaster!\n\n--HPS\n"},{"id":"470266","messageId":"00b201d92767$a0403ee0$e0c0bca0$@nexbridge.com","threadId":"59082","inReplyTo":"6061d012-13b7-ca4b-5556-70875b65c887@selasky.org","subject":"RE: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"","fromEmail":"rsbecker@nexbridge.com","sentAt":"2023-01-13T15:56:35Z","receivedAt":"2023-01-13T16:05:47Z","isPatch":false,"sender":{"key":"randall.becker@nexbridge.ca","avatar":"https://avatars.githubusercontent.com/u/28956764?v=4"},"body":"On January 13, 2023 10:50 AM, Hans Petter Selasky wrote:\n>On 1/13/23 16:45, Konstantin Ryabitsev wrote:\n>> I think you're misunderstanding some of the core principles of git.\n>\n>Maybe, I'm usually commandering git via the terminal.\n>\n>But if you say you can already edit stuff, why does the commit hash need to be\n>cryptographic? I don't get that part. Yeah, I think of git commits like blockchain.\n\ngit is using SHA1/SHA256 (which happen to be coincidentally cryptographic) as message digests with a very low probability of collisions when the hashes are computed. There is never a situation, implied by cryptography, where there is a decode of a git hash.  In order to make git a blockchain, you would need to implement central signing authorities, which would require a fork if the signature mechanism changes. The signature mechanism (SSH, GPG) is distinct from hash computation in git's trees, but depends on hash integrity.\n\n"},{"id":"470267","messageId":"20230113160218.d3nsoxpbrxrrszhz@meerkat.local","threadId":"59082","inReplyTo":"d087568b-919e-61f8-c203-e59a2e0572c6@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2023-01-13T16:02:18Z","receivedAt":"2023-01-13T16:10:04Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Jan 13, 2023 at 04:54:39PM +0100, Hans Petter Selasky wrote:\n> On 1/13/23 16:45, Konstantin Ryabitsev wrote:\n> >   If, for some reason, Linus ever needs to remove something\n> > from linux.git, he will do it and just give a heads-up why and for what\n> > reason.\n> \n> This gotta be a joke.\n> \n> There are 46K forks of Linus Torvalds Linux kernel on GitHUB, and if Linus\n> Torvalds one day decides to do a forced push, it will for sure be a\n> disaster!\n\nNo it won't, and I speak from some position of authority on this subject (I'm\nresponsible for git.kernel.org).\n\nIf Linus has to alter the history of linux.git, it will for sure be an\nextraordinary event -- it's never happened yet.  However, it will be widely\npublicised, the reasons for it will be made clear, and everyone will just\naccept it and move on.\n\nGit history edits occur all the time. Most tooling expects this to\noccasionally happen and deals with it correctly.\n\n-K\n"},{"id":"470268","messageId":"078526fc-7f2d-c745-a292-ff78dc06e407@selasky.org","threadId":"59082","inReplyTo":"00b201d92767$a0403ee0$e0c0bca0$@nexbridge.com","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T16:02:23Z","receivedAt":"2023-01-13T16:10:06Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 16:56, rsbecker@nexbridge.com wrote:\n> git is using SHA1/SHA256 (which happen to be coincidentally cryptographic) as message digests with a very low probability of collisions when the hashes are computed. There is never a situation, implied by cryptography, where there is a decode of a git hash.  In order to make git a blockchain, you would need to implement central signing authorities, which would require a fork if the signature mechanism changes. The signature mechanism (SSH, GPG) is distinct from hash computation in git's trees, but depends on hash integrity.\n\nI see.\n\nBut at the same time any unique enough hash, identifies a specific piece \nof code or checkout, even though it is not under a specific signing \nauthority. And that is the problem, that authorities may distribute \nallowed-only-hashes for their hardware ...\n\n--HPS\n"},{"id":"470269","messageId":"a2e6fdc3-fbb0-821c-078f-1ad4e55dc8e3@selasky.org","threadId":"59082","inReplyTo":"20230113160218.d3nsoxpbrxrrszhz@meerkat.local","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T16:06:57Z","receivedAt":"2023-01-13T16:13:16Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 17:02, Konstantin Ryabitsev wrote:\n> On Fri, Jan 13, 2023 at 04:54:39PM +0100, Hans Petter Selasky wrote:\n>> On 1/13/23 16:45, Konstantin Ryabitsev wrote:\n>>>    If, for some reason, Linus ever needs to remove something\n>>> from linux.git, he will do it and just give a heads-up why and for what\n>>> reason.\n>>\n>> This gotta be a joke.\n>>\n>> There are 46K forks of Linus Torvalds Linux kernel on GitHUB, and if Linus\n>> Torvalds one day decides to do a forced push, it will for sure be a\n>> disaster!\n> \n> No it won't, and I speak from some position of authority on this subject (I'm\n> responsible for git.kernel.org).\n> \n> If Linus has to alter the history of linux.git, it will for sure be an\n> extraordinary event -- it's never happened yet.  However, it will be widely\n> publicised, the reasons for it will be made clear, and everyone will just\n> accept it and move on.\n> \n> Git history edits occur all the time. Most tooling expects this to\n> occasionally happen and deals with it correctly.\n> \n\nOK, if you say so. Though in my mind 46K rebases of millions of commits \nseem a lot overhead.\n\nHowever, if history can be edited anyway, why do you need the \ncryptographic hash algorithm. Why not use a non-cryptographic one?\n\nWhat's the point? Only so that one party can stay in control?\n\n--HPS\n\n"},{"id":"470270","messageId":"20230113153004.l3crds3db74ffnis@carbon","threadId":"59082","inReplyTo":"446984f6-0d2e-04da-11a3-8b1481fac953@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Konstantin Khomoutov","fromEmail":"kostix@bswap.ru","sentAt":"2023-01-13T15:30:04Z","receivedAt":"2023-01-13T16:21:24Z","isPatch":false,"sender":{"key":"kostix@bswap.ru","avatar":null},"body":"On Fri, Jan 13, 2023 at 02:39:37PM +0100, Hans Petter Selasky wrote:\n\n[...]\n> > It's not clear why are you referring to Gitorious in your mail's subject and\n> > then talk about Git.\n[...]\n> I thought that Git was short for Gitorious? My bad.\n\nNo, unless you're late to the Git party ;-)\nOld-timers do remember Gitorious as a software project [1] which is closely\nrelated to Git but was a totally separate project.\n\n  1. https://en.wikipedia.org/wiki/Gitorious\n\n"},{"id":"470273","messageId":"5971b434-6409-8fd6-130f-f5b871a10f6d@selasky.org","threadId":"59082","inReplyTo":"a2e6fdc3-fbb0-821c-078f-1ad4e55dc8e3@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T16:18:40Z","receivedAt":"2023-01-13T16:26:08Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 17:06, Hans Petter Selasky wrote:\n> What's the point? Only so that one party can stay in control?\n\nLet me phrase it like this:\n\nYou clearly believe in the zero-trust principle. I don't.\n\nWhy can't git support both beliefs, and it can be configurable somehow then?\n\n--HPS\n"},{"id":"470274","messageId":"20230113162721.qwl2asjo542cxe3c@meerkat.local","threadId":"59082","inReplyTo":"a2e6fdc3-fbb0-821c-078f-1ad4e55dc8e3@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2023-01-13T16:27:21Z","receivedAt":"2023-01-13T16:33:09Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Jan 13, 2023 at 05:06:57PM +0100, Hans Petter Selasky wrote:\n> OK, if you say so. Though in my mind 46K rebases of millions of commits seem\n> a lot overhead.\n\nNot to discourage you, but you seem to be making statements without a good\nunderstanding of how git works. If there is a history rewrite (even one that\nfor some reason goes back millions of commits) all hash calculations will\nhappen exactly once -- on the system of the person who's rewriting the\nhistory. After they push it, it's just a bunch of objects that everyone else\nmerely downloads.\n\n> However, if history can be edited anyway, why do you need the cryptographic\n> hash algorithm. Why not use a non-cryptographic one?\n\nThe answer is, unhelpfully, \"because that's how git works.\" Every commit is a\nstandalone object that references the previous commit, plus includes hashes of\nall trees, and those include hashes of all blobs. SHA-1 was picked because of\nits speed and the fact that it guarantees an extremely low potential for\ncollisions (even better with SHA256). As a side-effect, it's easy to calculate\nthe integrity of the entire tree, including its history, by verifying its\nhashes (this is what git fsck does).\n\nHashes aren't really \"cryptographic\" anyway (they just happen to be used all\nover the place in cryptography). It's really just a one-way function to reduce\ncontent of arbitrary size to a set of bytes of a determined size (and give a\nrelatively high assurance of it being collision-free).\n\n-K\n"},{"id":"470275","messageId":"95797daa-4d5d-73d3-fec5-6b25707182fd@selasky.org","threadId":"59082","inReplyTo":"20230113162721.qwl2asjo542cxe3c@meerkat.local","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T16:30:20Z","receivedAt":"2023-01-13T16:36:22Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 17:27, Konstantin Ryabitsev wrote:\n> The answer is, unhelpfully, \"because that's how git works.\" Every commit is a\n> standalone object that references the previous commit, plus includes hashes of\n> all trees, and those include hashes of all blobs. SHA-1 was picked because of\n> its speed and the fact that it guarantees an extremely low potential for\n> collisions (even better with SHA256). As a side-effect, it's easy to calculate\n> the integrity of the entire tree, including its history, by verifying its\n> hashes (this is what git fsck does).\n\nSame thing can be said for CRC-XXX. Just some magic CPU instructions and \nwe're good. You don't even need a library.\n\n--HPS\n"},{"id":"470276","messageId":"038b4ea4-2a10-9b5a-14e4-e987a68e93b0@selasky.org","threadId":"59082","inReplyTo":"20230113162721.qwl2asjo542cxe3c@meerkat.local","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T16:35:57Z","receivedAt":"2023-01-13T16:39:02Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 17:27, Konstantin Ryabitsev wrote:\n> Not to discourage you, but you seem to be making statements without a good\n> understanding of how git works. If there is a history rewrite (even one that\n> for some reason goes back millions of commits) all hash calculations will\n> happen exactly once -- on the system of the person who's rewriting the\n> history. After they push it, it's just a bunch of objects that everyone else\n> merely downloads.\n\nIf you used CRC, you would not need that, because CRC calculations are \n\"concatenatable\", while SHA-1's are not. CRC would just need the first \nand the last hash, and then you would apply the \"difference\".\n\n--HPS\n"},{"id":"470277","messageId":"20230113163619.4ab5oyqyjrthxrwv@meerkat.local","threadId":"59082","inReplyTo":"5971b434-6409-8fd6-130f-f5b871a10f6d@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2023-01-13T16:36:19Z","receivedAt":"2023-01-13T16:39:21Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Jan 13, 2023 at 05:18:40PM +0100, Hans Petter Selasky wrote:\n> On 1/13/23 17:06, Hans Petter Selasky wrote:\n> > What's the point? Only so that one party can stay in control?\n> \n> Let me phrase it like this:\n> \n> You clearly believe in the zero-trust principle. I don't.\n\nI'm not sure what you mean here, but git is certainly not zero-trust. When you\nclone linux.git from git.kernel.org, you're very much trusting that:\n\n- I (or members of my team) didn't mess with the repository\n- Linus (or someone who hacked his laptop) didn't mess with the repository\n\nGit is tamper-evident, not tamper-proof, so by definition it cannot be\nzero-trust.\n\n> Why can't git support both beliefs, and it can be configurable somehow then?\n\nWell, git is literally built on the concept of unique hashes. It's not\npossible to make this bit configurable, as it would be a totally different\nproject with entirely different internals.\n\nNot saying such framework doesn't have a reason to exist, but it's not\nsomething that can be built on top of git.\n\n-K\n"},{"id":"470278","messageId":"20230113164131.swif4buln2py7tyl@meerkat.local","threadId":"59082","inReplyTo":"038b4ea4-2a10-9b5a-14e4-e987a68e93b0@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2023-01-13T16:41:31Z","receivedAt":"2023-01-13T16:43:50Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Jan 13, 2023 at 05:35:57PM +0100, Hans Petter Selasky wrote:\n> > Not to discourage you, but you seem to be making statements without a good\n> > understanding of how git works. If there is a history rewrite (even one that\n> > for some reason goes back millions of commits) all hash calculations will\n> > happen exactly once -- on the system of the person who's rewriting the\n> > history. After they push it, it's just a bunch of objects that everyone else\n> > merely downloads.\n> \n> If you used CRC, you would not need that, because CRC calculations are\n> \"concatenatable\", while SHA-1's are not. CRC would just need the first and\n> the last hash, and then you would apply the \"difference\".\n\nIt doesn't matter how it works behind the scenes as long as the produced hash\nis not unique (and CRC gives you no assurance of being unique). Git is built\non the concept that every object has a unique hash. If this is no longer true,\nthen it's literally no longer git, but is something else.\n\nSince we're discussing this on the git list, it's not really a discussion\nworth having here.\n\n-K\n"},{"id":"470279","messageId":"7a51b925-cb0a-4b48-fc14-171006f73298@selasky.org","threadId":"59082","inReplyTo":"20230113163619.4ab5oyqyjrthxrwv@meerkat.local","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T16:44:03Z","receivedAt":"2023-01-13T16:46:20Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 17:36, Konstantin Ryabitsev wrote:\n> I'm not sure what you mean here, but git is certainly not zero-trust. When you\n> clone linux.git from git.kernel.org, you're very much trusting that:\n> \n> - I (or members of my team) didn't mess with the repository\n> - Linus (or someone who hacked his laptop) didn't mess with the repository\n> \n> Git is tamper-evident, not tamper-proof, so by definition it cannot be\n> zero-trust.\n\nHi,\n\nBy using a cryptographic hash algorithm, the goal is to avoid tampering \nyou say, like tampering on the internet, ISP, cache node and so on. To \nme that's clearly a zero-trust thought. You don't trust the guy(s) that \nput down the infrastructure, neither those that provide that local cache \nfor the GIT repository, only the master repository. SHA-1 gives a \ncertain confidence, that if you checkout XXXXXXX, then you get a likely \nexpected result with reduced possibility of tampering.\n\nAnyone could intercept a CRC protected blob and re-compute the hash and \nsend it on. But not a SHA-1 one.\n\nI on the other hand trust the guys that put down the internet and are \nproviding the cache nodes for GIT.\n\nIt's two different world views.\n\n--HPS\n"},{"id":"470280","messageId":"af36e6cd-f649-5ef5-ecdb-19555833b5d4@selasky.org","threadId":"59082","inReplyTo":"20230113164131.swif4buln2py7tyl@meerkat.local","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T16:45:45Z","receivedAt":"2023-01-13T16:49:10Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 17:41, Konstantin Ryabitsev wrote:\n> It doesn't matter how it works behind the scenes as long as the produced hash\n> is not unique (and CRC gives you no assurance of being unique). Git is built\n> on the concept that every object has a unique hash. If this is no longer true,\n> then it's literally no longer git, but is something else.\n\nThat's why I say you need a fixup field, in case of collisions. CRC is \nused plenty all over the place and has good entropy.\n\n--HPS\n"},{"id":"470281","messageId":"20230113164953.o63hu5pgetci4sbb@meerkat.local","threadId":"59082","inReplyTo":"7a51b925-cb0a-4b48-fc14-171006f73298@selasky.org","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Konstantin Ryabitsev","fromEmail":"konstantin@linuxfoundation.org","sentAt":"2023-01-13T16:49:53Z","receivedAt":"2023-01-13T16:53:57Z","isPatch":false,"sender":{"key":"konstantin@linuxfoundation.org","avatar":"https://gravatar.com/avatar/7cb8827c6de56e1bd2dea16508c6708aa43feed3bf3813bcdacecdf96ceadd79?d=mp&s=160"},"body":"On Fri, Jan 13, 2023 at 05:44:03PM +0100, Hans Petter Selasky wrote:\n> By using a cryptographic hash algorithm, the goal is to avoid tampering you\n> say, like tampering on the internet, ISP, cache node and so on. To me that's\n> clearly a zero-trust thought. You don't trust the guy(s) that put down the\n> infrastructure, neither those that provide that local cache for the GIT\n> repository, only the master repository. SHA-1 gives a certain confidence,\n> that if you checkout XXXXXXX, then you get a likely expected result with\n> reduced possibility of tampering.\n> \n> Anyone could intercept a CRC protected blob and re-compute the hash and send\n> it on. But not a SHA-1 one.\n> \n> I on the other hand trust the guys that put down the internet and are\n> providing the cache nodes for GIT.\n\nI admit, I never trust the \"guys who put down the internet,\" so that's a very\nscary scenario to me (and I would say to pretty much everyone else on this\nlist).\n\n> It's two different world views.\n\nIndeed, werenotalike.gif :)\n\n-K\n"},{"id":"470283","messageId":"144284f0-7b8d-0b1d-22c1-e3a9ea200507@selasky.org","threadId":"59082","inReplyTo":"20230113164953.o63hu5pgetci4sbb@meerkat.local","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Hans Petter Selasky","fromEmail":"hps@selasky.org","sentAt":"2023-01-13T16:51:14Z","receivedAt":"2023-01-13T16:54:41Z","isPatch":false,"sender":{"key":"hps@selasky.org","avatar":"https://gravatar.com/avatar/574493343874307c721bb8826c487ad2f070c55437d734d9d7130ae98f50ea25?d=mp&s=160"},"body":"On 1/13/23 17:49, Konstantin Ryabitsev wrote:\n>> It's two different world views.\n> Indeed, werenotalike.gif 😄\n\nOK, I have no problem about that.\n\nThanks for the discussion.\n\n--HPS\n"},{"id":"470291","messageId":"5c696705-08f8-3ca4-530d-c2c12abc4593@iee.email","threadId":"59082","inReplyTo":"009701d9275a$678416b0$368c4410$@nexbridge.com","subject":"Re: Gitorious should use CRC128 / 256 / 512 instead of SHA-1","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.email","sentAt":"2023-01-13T17:44:17Z","receivedAt":"2023-01-13T17:52:03Z","isPatch":false,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"On 13/01/2023 14:21, rsbecker@nexbridge.com wrote:\n> On January 13, 2023 8:40 AM, Hans Petter Selasky wrote:\n>> On 1/13/23 14:30, Konstantin Khomoutov wrote:\n>>> On Fri, Jan 13, 2023 at 01:59:44PM +0100, Hans Petter Selasky wrote:\n>>>\n>>>> Currently GIT only supports cryptographic hashes for its commit tags.\n>>> [...]\n>>>\n>>> https://github.com/git/git/blob/9bf691b78cf906751e65d65ba0c6ffdcd9a5a1\n>>> 2c/Documentation/technical/hash-function-transition.txt\n>>>\n>>> It's not clear why are you referring to Gitorious in your mail's\n>>> subject and then talk about Git.\n>>>\n>> Hi,\n>>\n>> I thought that Git was short for Gitorious? My bad.\n>>\n>> The document you refer to really highlights my concerns, that a strong\n>> cryptographic hash algorithm is the highway to hell.\n>>\n>> Do _not_ use a cryptographic hash for Git. Use plain good old CRC hashes.\n>>\n>> Just imagine the consequences of finding child porn inside a 10-year old firmware\n>> binary blob in the Linux kernel. Will you just ignore it, or will you fix it?\n>>\n>> That's why I say, that it must be possible to forge the hashes by default.\n> I do not understand the goal of this request. \n\nI'd agree about the core need for 'absolute' integrity checking.\n\nHowever we have been here before, but without a way forward.\n\nIt was the \"Subject: [TOPIC 3/17] Obliterate\" at Git Contributor Summit,\nLos Angeles (April 5, 2020).\nhttps://lore.kernel.org/git/5B2FEA46-A12F-4DE7-A184-E8856EF66248@jramsay.com.au/\n\nDiscussion at\nhttps://docs.google.com/document/d/15a_MPnKaEPbC92a4jhprlHvkyirDh2CtTtgOxNbnIbA/edit#heading=h.wljwyo3r1m6l\n\nThe core need I think HPS is referring to is that need to 'obliterate'\nsome blob (which contains the en-mass data), and perhaps some trees,\ncommits and tags, which may also hold objectionable meta data, at least\nfrom reference repositories, and at the same time authenticate (if\nthat's the right term) the list of such obliterated objects.\n\nIt will be a difficult task to carefully cut the fog of misdirection and\nscares in this arena.\n\nIt's one of those problem statements whose answer is \"42\".\n\n> If it is possible to forge hashes, then nothing in a git repository can ever be trusted. Signed content will no longer be verifiable. The whole Merkel Tree representing the commit history becomes easily corruptible by hackers and no upstream remote repository can ever be trusted - or someone's own if someone targets a repo with malware that rewrites hashes. Imagine a scenario when malware replaces a blob in a repo and then forges the hash to pretend that the replacement never occurred. Using git as a supply chain audit trail becomes impossible. This is a potential vector for ransomware invading the git ecosystem. This seems like a really fatal path to take for the product.\n\nThe supply chain audit is (would be) a real problem if the presence of a\nspecific hash is a punishable criminal offence. I suspect it already is\nin some jurisdictions.\n\n>\n> The advantage of how git functions is that it is possible to mirror or clone repositories, protecting from hardware errors. Repositories exist in distributed form, so there may be hundreds or thousands of copies in case someone's copy is corrupted by a disk or memory write error - so that takes hash reconstruction out of the requirement set. If the git architecture was based on a central repository model only, then this might be a reasonable request, but that is not how git works.\n\nThe law works in mysterious ways it's wonderful ways to demonstrate ;-)\nPossession of certain artefacts can be a problem, so it is something\nthat is worth careful consideration. We shouldn't let the 'distribution\nof criminal artefacts' be something 'guaranteed' by Git, despite careful\nusers.\n\n>  If, for instance, a main GitHub repo is somehow corrupted, it can be repaired by a push --force or a clone from a different instance.\n>\n> Unless I am missing your point.\n> --Randall\n>\n\nThe forced replacement of 'redacted' material is already a problem in\nother domains. We should be able to manage a redaction list for a\nrepository that needs it.\n\nAll that said, CRC isn't any sort of solution!\n\n--\nPhilip\n"}]}