{"thread":{"id":"59926","subject":"SHA256 support not experimental, or?","startedAt":"2023-06-28T16:28:57Z","lastAt":"2023-07-31T16:45:01Z","messageCount":21,"participants":["Adam Majer","brian m. carlson","Junio C Hamano","Patrick Steinhardt","Son Luong Ngoc"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"478962","messageId":"2f5de416-04ba-c23d-1e0b-83bb655829a7@zombino.com","threadId":"59926","inReplyTo":null,"subject":"SHA256 support not experimental, or?","fromName":"Adam Majer","fromEmail":"adamm@zombino.com","sentAt":"2023-06-28T16:28:28Z","receivedAt":"2023-06-28T16:28:57Z","isPatch":false,"sender":{"key":"adamm@zombino.com","avatar":"https://avatars.githubusercontent.com/u/1211498?v=4"},"body":"Hi all,\n\nIs sha256 still considered experimental or can it be assumed to be stable?\n\nThe usecase here is we are planning on moving to sha256 repositories \nmostly due to integrity guarantees, hypothetical or otherwise. What is \nimportant is not the initial interop challenges with sha1 repos, but \nwhether the on-disk format will remain compatible with future versions \nof git. At minimum, the on-disk format would be converted by some future \nversion(s) of git into another one and not be an end-of-the-road because \nit was \"experimental\" where dataloss is an implied risk.\n\nAttached is a patch that removes the scary text, if indeed sha256 should \nbe viewed as stable.\n\nCheers,\n- Adam\n\n---\n Documentation/git.txt | 4 ++--\n Documentation/object-format-disclaimer.txt | 8 ++------\n 2 files changed, 4 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex f0cafa2290..7c150a473c 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -553,8 +553,8 @@ double-quotes and respecting backslash escapes. E.g., the value\n \tIf this variable is set, the default hash algorithm for new\n \trepositories will be set to this value. This value is\n \tignored when cloning and the setting of the remote repository\n-\tis always used. The default is \"sha1\". THIS VARIABLE IS\n-\tEXPERIMENTAL! See `--object-format` in linkgit:git-init[1].\n+\tis always used. The default is \"sha1\".\n+ See `--object-format` in linkgit:git-init[1].\n \n Git Commits\n ~~~~~~~~~~~\ndiff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt\nindex 4cb106f0d1..dccee9c400 100644\n--- a/Documentation/object-format-disclaimer.txt\n+++ b/Documentation/object-format-disclaimer.txt\n@@ -1,6 +1,2 @@\n-THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still\n-in an early stage. A SHA-256 repository will in general not be able to\n-share work with \"regular\" SHA-1 repositories. It should be assumed\n-that, e.g., Git internal file formats in relation to SHA-256\n-repositories may change in backwards-incompatible ways. Only use\n-`--object-format=sha256` for testing purposes.\n+Note: SHA-256 repository will in general not be able to\n+share work with \"regular\" SHA-1 repositories.\n-- \n2.41.0\n\n"},{"id":"479001","messageId":"ZJzlezhHk4HpPmRk@tapette.crustytoothpaste.net","threadId":"59926","inReplyTo":"2f5de416-04ba-c23d-1e0b-83bb655829a7@zombino.com","subject":"Re: SHA256 support not experimental, or?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-06-29T01:59:23Z","receivedAt":"2023-06-29T01:59:39Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-06-28 at 16:28:28, Adam Majer wrote:\n> Hi all,\n> \n> Is sha256 still considered experimental or can it be assumed to be stable?\n> \n> The usecase here is we are planning on moving to sha256 repositories mostly\n> due to integrity guarantees, hypothetical or otherwise. What is important is\n> not the initial interop challenges with sha1 repos, but whether the on-disk\n> format will remain compatible with future versions of git. At minimum, the\n> on-disk format would be converted by some future version(s) of git into\n> another one and not be an end-of-the-road because it was \"experimental\"\n> where dataloss is an implied risk.\n\nI have no intention of changing things at this point. I think it should\nbe viewed as stable by now, and I'd support this patch, although to get\nit picked up it will need a commit message and a sign-off.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"479004","messageId":"xmqqmt0iajww.fsf@gitster.g","threadId":"59926","inReplyTo":"2f5de416-04ba-c23d-1e0b-83bb655829a7@zombino.com","subject":"Re: SHA256 support not experimental, or?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-29T05:59:11Z","receivedAt":"2023-06-29T05:59:41Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adam Majer writes:\n\n> Is sha256 still considered experimental or can it be assumed to be stable?\n\nI do not think we would officially label SHA-256 support as \"stable\"\nuntil we have good interoperability with SHA-1 repositories, but the\nexpectation is that we will make reasonable effort to keep migration\npath for the current SHA-256 repositories, even if it turns out that\nits on-disk format need to be updated, to keep the end-user data safe.\n\nSo while \"no-longer-experimental\" patch is probably a bit premature,\nthe warning in flashing red letters to caution against any use other\nthan testing may want to be toned down.\n\nThanks.\n\n"},{"id":"479008","messageId":"c8a8f298-f328-f24e-35ee-5b97cafca721@zombino.com","threadId":"59926","inReplyTo":"ZJzlezhHk4HpPmRk@tapette.crustytoothpaste.net","subject":"Re: SHA256 support not experimental, or?","fromName":"Adam Majer","fromEmail":"adamm@zombino.com","sentAt":"2023-06-29T10:42:30Z","receivedAt":"2023-06-29T10:49:26Z","isPatch":false,"sender":{"key":"adamm@zombino.com","avatar":"https://avatars.githubusercontent.com/u/1211498?v=4"},"body":"On 6/29/23 03:59, brian m. carlson wrote:\n> I have no intention of changing things at this point. I think it should\n> be viewed as stable by now, and I'd support this patch, although to get\n> it picked up it will need a commit message and a sign-off.\n\nSounds good. Patch follows.\n\n- Adam\n\nFrom 90be51143e741053390810720ba4a639c3b0b74c Mon Sep 17 00:00:00 2001\nFrom: Adam Majer \nDate: Wed, 28 Jun 2023 14:46:02 +0200\nSubject: [PATCH] doc: sha256 is no longer experimental\n\nThe purpose of this patch is to remove scary wording that basically\nstops people using sha256 repositories not because of interoperability\nissues with sha1 repositories, but from fear that their work will\nsuddenly become incompatible in some future version of git.\n\nWe should be clear that currently sha256 repositories will not work with\nsha1 repositories but stop the scary words.\n\nSigned-off-by: Adam Majer \n---\n Documentation/git.txt | 4 ++--\n Documentation/object-format-disclaimer.txt | 8 ++------\n 2 files changed, 4 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex f0cafa2290..666dbdb55c 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -553,8 +553,8 @@ double-quotes and respecting backslash escapes. E.g., the value\n \tIf this variable is set, the default hash algorithm for new\n \trepositories will be set to this value. This value is\n \tignored when cloning and the setting of the remote repository\n-\tis always used. The default is \"sha1\". THIS VARIABLE IS\n-\tEXPERIMENTAL! See `--object-format` in linkgit:git-init[1].\n+\tis always used. The default is \"sha1\". See `--object-format`\n+\tin linkgit:git-init[1].\n \n Git Commits\n ~~~~~~~~~~~\ndiff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt\nindex 4cb106f0d1..1e976688be 100644\n--- a/Documentation/object-format-disclaimer.txt\n+++ b/Documentation/object-format-disclaimer.txt\n@@ -1,6 +1,2 @@\n-THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still\n-in an early stage. A SHA-256 repository will in general not be able to\n-share work with \"regular\" SHA-1 repositories. It should be assumed\n-that, e.g., Git internal file formats in relation to SHA-256\n-repositories may change in backwards-incompatible ways. Only use\n-`--object-format=sha256` for testing purposes.\n+Note: SHA-256 repositories currently will not be able to share work\n+with \"regular\" SHA-1 repositories.\n-- \n2.41.0\n\n"},{"id":"479009","messageId":"65148753-7071-d5d5-3b4e-bad020e6ab63@zombino.com","threadId":"59926","inReplyTo":"xmqqmt0iajww.fsf@gitster.g","subject":"Re: SHA256 support not experimental, or?","fromName":"Adam Majer","fromEmail":"adamm@zombino.com","sentAt":"2023-06-29T10:53:16Z","receivedAt":"2023-06-29T10:54:23Z","isPatch":false,"sender":{"key":"adamm@zombino.com","avatar":"https://avatars.githubusercontent.com/u/1211498?v=4"},"body":"On 6/29/23 07:59, Junio C Hamano wrote:\n> Adam Majer writes:\n> \n>> Is sha256 still considered experimental or can it be assumed to be stable?\n> \n> I do not think we would officially label SHA-256 support as \"stable\"\n> until we have good interoperability with SHA-1 repositories, but the\n> expectation is that we will make reasonable effort to keep migration\n> path for the current SHA-256 repositories, even if it turns out that\n> its on-disk format need to be updated, to keep the end-user data safe.\n\nThat could be a different definition of stable. But I'm satisfied that \ncurrent sha256 repositories will not end up incompatible with some \nfuture version of git without migration path (talking about on-disk format).\n\nSo maybe my question should be reworded to \"is sha256 still considered \nearly stage, for testing purposes only with possible data-loss or can it \nbe relied on for actual long lived repositories?\"\n\n\n> So while \"no-longer-experimental\" patch is probably a bit premature,\n> the warning in flashing red letters to caution against any use other\n> than testing may want to be toned down.\n\nAgreed. I think it should be clear that SHA256 and SHA1 repositories \ncannot share data at this point. The scary wording should be removed \nthough, as currently it sounds like \"data loss incoming and it's your \nfault\" if one chooses sha256\n\n- Adam\n"},{"id":"479030","messageId":"xmqqpm5e7zs6.fsf@gitster.g","threadId":"59926","inReplyTo":"65148753-7071-d5d5-3b4e-bad020e6ab63@zombino.com","subject":"Re: SHA256 support not experimental, or?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-29T20:56:57Z","receivedAt":"2023-06-29T20:57:03Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adam Majer writes:\n\n> So maybe my question should be reworded to \"is sha256 still considered\n> early stage, for testing purposes only with possible data-loss or can\n> it be relied on for actual long lived repositories?\"\n\nMy understanding is that they are in a happy place where they are\njust as usable as SHA-1 based repositories have been. As we have\nwell-worked out interoperability design but no implementation, it\nmay have to change once we discover something missing in the design,\nthough. But without such clarification, you already know the answer\nto the above question in the message you are responding to. Having\na migration path means \"possible data-los\" is not in the picture.\n\n> The scary wording should be removed\n> though, as currently it sounds like \"data loss incoming and it's your\n> fault\" if one chooses sha256\n\nGood.\n\nTHanks.\n\n"},{"id":"479032","messageId":"ZJ303bm+VAvp5nyV@tapette.crustytoothpaste.net","threadId":"59926","inReplyTo":"xmqqmt0iajww.fsf@gitster.g","subject":"Re: SHA256 support not experimental, or?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-06-29T21:17:17Z","receivedAt":"2023-06-29T21:17:42Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-06-29 at 05:59:11, Junio C Hamano wrote:\n> Adam Majer writes:\n> \n> > Is sha256 still considered experimental or can it be assumed to be stable?\n> \n> I do not think we would officially label SHA-256 support as \"stable\"\n> until we have good interoperability with SHA-1 repositories, but the\n> expectation is that we will make reasonable effort to keep migration\n> path for the current SHA-256 repositories, even if it turns out that\n> its on-disk format need to be updated, to keep the end-user data safe.\n\nI don't think that's a good position to have. I'm not working on\ninterop more than incidentally at the moment, and to my knowledge,\nnobody else is, either. Absent me having substantially more free time\nor having my employer pay me to work on it, it is probably not\nhappening.\n\nWe desperately do want people to move away from SHA-1 to SHA-256, and as\nsoon as there's tooling and forges to do so, we should encourage them to\ndo so. Just because people can't interop existing SHA-1 repositories\ndoesn't mean people can't or shouldn't build new SHA-256 repositories.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"479034","messageId":"xmqqa5wh9adg.fsf@gitster.g","threadId":"59926","inReplyTo":"ZJ303bm+VAvp5nyV@tapette.crustytoothpaste.net","subject":"Re: SHA256 support not experimental, or?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-29T22:22:51Z","receivedAt":"2023-06-29T22:23:00Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> On 2023-06-29 at 05:59:11, Junio C Hamano wrote:\n>> Adam Majer writes:\n>> \n>> > Is sha256 still considered experimental or can it be assumed to be stable?\n>> \n>> I do not think we would officially label SHA-256 support as \"stable\"\n>> until we have good interoperability with SHA-1 repositories, but the\n>> expectation is that we will make reasonable effort to keep migration\n>> path for the current SHA-256 repositories, even if it turns out that\n>> its on-disk format need to be updated, to keep the end-user data safe.\n>\n> I don't think that's a good position to have.\n> We desperately do want people to move away from SHA-1 to SHA-256, and as\n> soon as there's tooling and forges to do so, we should encourage them to\n> do so.\n\nI agree that it is good to ensure that SHA-256 support is good\nenough to start new projects with.\n\n> Just because people can't interop existing SHA-1 repositories\n> doesn't mean people can't or shouldn't build new SHA-256 repositories.\n\nTrue, and our messaging should avoid scaring them away from doing\nso. But isn't the lack of interoperability one of the reasons why\nGitHub and Gitlab do not yet offer choice of the hash? There\ncertainly is a chicken-and-egg problem here.\n\n"},{"id":"479040","messageId":"ZJ4uKYIZMxi3DHo3@tapette.crustytoothpaste.net","threadId":"59926","inReplyTo":"xmqqa5wh9adg.fsf@gitster.g","subject":"Re: SHA256 support not experimental, or?","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-06-30T01:21:45Z","receivedAt":"2023-06-30T01:21:54Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-06-29 at 22:22:51, Junio C Hamano wrote:\n> True, and our messaging should avoid scaring them away from doing\n> so. But isn't the lack of interoperability one of the reasons why\n> GitHub and Gitlab do not yet offer choice of the hash? There\n> certainly is a chicken-and-egg problem here.\n\nThere are a lot of necessary changes for a forge to adopt SHA-256. For\nexample, at GitHub, we have a single null OID constant in some code that\nhas to be addressed, libgit2 has to be taught about SHA-256 or removed,\nand UI changes need to be done to accommodate the larger IDs. I'm\nsure that GitLab has very similar situations, as do all of the other\nforges. After all, think about the extensive number of patches that\nwent into Git itself to get us there. Everyone has made all of those\nsame assumptions in their forges.\n\nI'm certain that whether or not interoperability were available would\nnot influence the forges' desire to support SHA-256. It's simply a lot\nof work to fix all of those spots that need it and requires a lot of\ncommunication and discussions across teams, all of which takes time.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"479045","messageId":"vt3cizczmwbcpgktwrkr3jbiwhee37rt7m243hnkzxik7gt4m2@d2upsqoxtlgc","threadId":"59926","inReplyTo":"ZJ4uKYIZMxi3DHo3@tapette.crustytoothpaste.net","subject":"Re: SHA256 support not experimental, or?","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-06-30T09:31:27Z","receivedAt":"2023-06-30T09:32:40Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Jun 30, 2023 at 01:21:45AM +0000, brian m. carlson wrote:\n> On 2023-06-29 at 22:22:51, Junio C Hamano wrote:\n> > True, and our messaging should avoid scaring them away from doing\n> > so. But isn't the lack of interoperability one of the reasons why\n> > GitHub and Gitlab do not yet offer choice of the hash? There\n> > certainly is a chicken-and-egg problem here.\n> \n> There are a lot of necessary changes for a forge to adopt SHA-256. For\n> example, at GitHub, we have a single null OID constant in some code that\n> has to be addressed, libgit2 has to be taught about SHA-256 or removed,\n> and UI changes need to be done to accommodate the larger IDs. I'm\n> sure that GitLab has very similar situations, as do all of the other\n> forges. After all, think about the extensive number of patches that\n> went into Git itself to get us there. Everyone has made all of those\n> same assumptions in their forges.\n\nIndeed, supporting SHA256 is a major effort on our side at GitLab. Most\nof the work isn't really adapting our production code, but it's rather\nthat tons of tests were written with seed repositories and hardcoded\nobject hashes. Converting all of that isn't all that hard in the general\ncase, but it's a tedious job.\n\nIn the Gitaly team we have already started to put significant time into\nthis problem and are slowly chipping away at it. We are at a state where\nmost of our codebase works with SHA256 alright, and we in fact continue\ndown that road as a low-priority side project where we convert a handful\nof tests every release.\n\n> I'm certain that whether or not interoperability were available would\n> not influence the forges' desire to support SHA-256. It's simply a lot\n> of work to fix all of those spots that need it and requires a lot of\n> communication and discussions across teams, all of which takes time.\n\nTrue as well. Even though Gitaly will likely be SHA256-ready in the not\ntoo distant future, that doesn't mean that GitLab as a whole is. The\nfrontend will need investments as well, and there's likely a long tail\nof other stuff that needs to be done that I ain't yet got on my radar\nright now.\n\nIn any case I'm fully supportive of relaxing the current warning. Except\nfor the recently discussed edge case where cloning empty repositories\ndidn't create a SHA256 repository I have found the SHA256 code to be\nstable and working as advertised. We should caution people that many\nservices will not work with SHA256 yet though.\n\nPatrick\n"},{"id":"479048","messageId":"5880fe56-aa98-64ce-4d91-ca078d3a7354@zombino.com","threadId":"59926","inReplyTo":"vt3cizczmwbcpgktwrkr3jbiwhee37rt7m243hnkzxik7gt4m2@d2upsqoxtlgc","subject":"Re: SHA256 support not experimental, or?","fromName":"Adam Majer","fromEmail":"adamm@zombino.com","sentAt":"2023-06-30T11:25:06Z","receivedAt":"2023-06-30T11:25:14Z","isPatch":false,"sender":{"key":"adamm@zombino.com","avatar":"https://avatars.githubusercontent.com/u/1211498?v=4"},"body":"On 6/30/23 11:31, Patrick Steinhardt wrote:\n> Indeed, supporting SHA256 is a major effort on our side at GitLab. Most\n> of the work isn't really adapting our production code, but it's rather\n> that tons of tests were written with seed repositories and hardcoded\n> object hashes. Converting all of that isn't all that hard in the general\n> case, but it's a tedious job.\n\nHi!\n\nThis actually reminds me of a funny story from my side.\n\nEarlier this year, I was testing various frontends and how they would \nhandle SHA256 repositories. All of them failed, not surprising. I even \nmanaged to lock myself out of Gitlab by importing a SHA256 private repo \ninto my home project -- every time this project became visible, it would \nresult in Error 500 from the UI. Today (few weeks ago), this appears to \nbe fixed -- the UI is just broken, so you can't see anything in sha256 \nrepository, but at least I was able to delete the project.\n\nThe repository was correctly imported and I could clone from gitlab, so \nthe problem is mostly \"just\" UI. :-)\n\nThe most likely frontend we'll use for our internal project is Gitea. \nThe sha256 support is in progress\n\nhttps://github.com/go-gitea/gitea/pull/23894\n\n From the size of this patch, you can see how ingrained SHA1 assumption \nwas. Most of the patch is just to remove the hardcoded elements, \nincluding hardcoded SHA1 empty-tree hashes and assumption that 20 bytes \nis enough to hold a hash. And I didn't even add sha256 test cases...\n\nBut I have to say that in at least one occasion, people are bringing up \nthe experimental nature of git's sha256 support (per current wording) as \nreason not to make their tools sha256 compliant.\n\n> In any case I'm fully supportive of relaxing the current warning. Except\n> for the recently discussed edge case where cloning empty repositories\n> didn't create a SHA256 repository I have found the SHA256 code to be\n> stable and working as advertised. We should caution people that many\n> services will not work with SHA256 yet though.\n\nThat is exactly true. But this is also chicken-egg problem. Services are \nnot adapted for sha256 repositories because there is simply no demand \nfor them. Only when people will start using sha256 repos, will there be \nsome demand generated.\n\n- Adam\n"},{"id":"479049","messageId":"cxakwc2qnqgnh3hg7oufj5qoizu7ltw5qqbq44aafxr5syssm3@4e6eziwdt7jr","threadId":"59926","inReplyTo":"5880fe56-aa98-64ce-4d91-ca078d3a7354@zombino.com","subject":"Re: SHA256 support not experimental, or?","fromName":"Patrick Steinhardt","fromEmail":"ps@pks.im","sentAt":"2023-06-30T11:38:11Z","receivedAt":"2023-06-30T11:38:37Z","isPatch":false,"sender":{"key":"ps@pks.im","avatar":"https://avatars.githubusercontent.com/u/4056630?v=4"},"body":"On Fri, Jun 30, 2023 at 01:25:06PM +0200, Adam Majer wrote:\n> On 6/30/23 11:31, Patrick Steinhardt wrote:\n> > Indeed, supporting SHA256 is a major effort on our side at GitLab. Most\n> > of the work isn't really adapting our production code, but it's rather\n> > that tons of tests were written with seed repositories and hardcoded\n> > object hashes. Converting all of that isn't all that hard in the general\n> > case, but it's a tedious job.\n> \n> Hi!\n> \n> This actually reminds me of a funny story from my side.\n> \n> Earlier this year, I was testing various frontends and how they would handle\n> SHA256 repositories. All of them failed, not surprising. I even managed to\n> lock myself out of Gitlab by importing a SHA256 private repo into my home\n> project -- every time this project became visible, it would result in Error\n> 500 from the UI. Today (few weeks ago), this appears to be fixed -- the UI\n> is just broken, so you can't see anything in sha256 repository, but at least\n> I was able to delete the project.\n\nYeah, thinks gradually start to work. It's kind of satisfying to see how\nmore and more things start to fall into place.\n\n> The repository was correctly imported and I could clone from gitlab, so the\n> problem is mostly \"just\" UI. :-)\n\nThe UI is a significantly broken right now, mostly because the request\nrouting logic still has a maximum object ID length of 40 characters\nhardcoded. So indeed, most of the stuff in the UI doesn't work unless\nyou do a few changes in the frontend. I should probably just create the\nmerge request to fix these as I already have those changes available\nlocally anyway.\n\nBut there's other parts that are in the Gitaly backend that don't yet\nwork. There's some RPCs that parse object IDs, but still use the\nhardcoded SHA1 hash. Updating them is trivial, but as mentioned updating\ntheir tests is tedious work.\n\n> The most likely frontend we'll use for our internal project is Gitea. The\n> sha256 support is in progress\n> \n> https://github.com/go-gitea/gitea/pull/23894\n> \n> From the size of this patch, you can see how ingrained SHA1 assumption was.\n> Most of the patch is just to remove the hardcoded elements, including\n> hardcoded SHA1 empty-tree hashes and assumption that 20 bytes is enough to\n> hold a hash. And I didn't even add sha256 test cases...\n\nI guess most projects that started a long time a go made the same error\nof taking SHA1 for granted, so they didn't bother writing neither the\nproduction code nor the tests with swapping out the object format in\nmind. I guess we've learned our lesson here, which also means that the\nnext transition (if there ever will be one) should go a lot faster as\nthe codebases should be prepared then.\n\n> But I have to say that in at least one occasion, people are bringing up the\n> experimental nature of git's sha256 support (per current wording) as reason\n> not to make their tools sha256 compliant.\n\nYeah, it's this chicken-and-egg problem. Things are experimental as most\ntools ain't got support, but because most things ain't got support we\nnever get any testers and thus are stuck in that state.\n\n> > In any case I'm fully supportive of relaxing the current warning. Except\n> > for the recently discussed edge case where cloning empty repositories\n> > didn't create a SHA256 repository I have found the SHA256 code to be\n> > stable and working as advertised. We should caution people that many\n> > services will not work with SHA256 yet though.\n> \n> That is exactly true. But this is also chicken-egg problem. Services are not\n> adapted for sha256 repositories because there is simply no demand for them.\n> Only when people will start using sha256 repos, will there be some demand\n> generated.\n\nYup, and that is why I have been pushing for SHA256 support internally\nat GitLab for quite a while -- our efforts here started almost exactly a\nyear ago, but has gained more steam in recent months.\n\nPatrick\n"},{"id":"479050","messageId":"CAL3xRKfQLj7Zufy5fMMs=ykeexBn7duqFH1jZ84+WRKKOaEpFA@mail.gmail.com","threadId":"59926","inReplyTo":"5880fe56-aa98-64ce-4d91-ca078d3a7354@zombino.com","subject":"Re: SHA256 support not experimental, or?","fromName":"Son Luong Ngoc","fromEmail":"sluongng@gmail.com","sentAt":"2023-06-30T12:20:14Z","receivedAt":"2023-06-30T12:20:30Z","isPatch":false,"sender":{"key":"sluongng@gmail.com","avatar":"https://avatars.githubusercontent.com/u/26684313?v=4"},"body":"Hi,\n\n> On 30 Jun 2023, at 13:25, Adam Majer wrote:...\n> On 6/30/23 11:31, Patrick Steinhardt wrote:\n...\n> > In any case I'm fully supportive of relaxing the current warning. Except\n> > for the recently discussed edge case where cloning empty repositories\n> > didn't create a SHA256 repository I have found the SHA256 code to be\n> > stable and working as advertised. We should caution people that many\n> > services will not work with SHA256 yet though.\n>\n> That is exactly true. But this is also chicken-egg problem. Services are not adapted for sha256 repositories because there is simply no demand for them. Only when people will start using sha256 repos, will there be some demand generated.\n\nFWIW, in the Bazel ecosystem where SHA256 is very popular, there has\nbeen an increasing appetite for FUSE file system to lazily fetch contents\nof a git repository.\n\nBuild tools such as Bazel would often need to hash the content of the\nsource files to build a dependency graph. And in a FUSE setup, it would\nbe ideal if the FUSE server could supply the hash via an xattr, so that\nFUSE client does not need to fetch the whole file content and only the\nmetadata.\n\nMost tools in this space (Bazel, Buck2) are using SHA256 and are exploring\nfaster hash such as Blake3, Aegis, KangarooTwelve for larger file\nsupport. As these matured build tools gains popularity, so will the usage\nof SHA256 (and newer hash algorithm).\n\nAnother point I think might help motivate different forges to\nmove would be switching from the object's hash to digest (hash and\nfile size). The additional file size information would help tremendously\nin predicting compute resources when serving files of a repository.\n\nSo I think Git would simply need a bit more time for these related\necosystems to reach a critical mass and help fuel the transition to a\n.\n\n> - Adam\n\nRegards,\nSon Luong.\n\nReferences:\n\n- https://buck2.build/docs/rfcs/drafts/digest-kinds/#use-cases\n- https://github.com/bazelbuild/bazel/pull/18784\n"},{"id":"479055","messageId":"xmqqilb47vci.fsf@gitster.g","threadId":"59926","inReplyTo":"CAL3xRKfQLj7Zufy5fMMs=ykeexBn7duqFH1jZ84+WRKKOaEpFA@mail.gmail.com","subject":"Re: SHA256 support not experimental, or?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-06-30T16:45:01Z","receivedAt":"2023-06-30T16:45:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Son Luong Ngoc writes:\n\n> Build tools such as Bazel would often need to hash the content of the\n> source files to build a dependency graph. And in a FUSE setup, it would\n> be ideal if the FUSE server could supply the hash via an xattr, so that\n> FUSE client does not need to fetch the whole file content and only the\n> metadata.\n\nThis is unrelated tangent, but the implementation of virtual\nfilesystem on top of Git's object store will be able to give such\nSHA-256 hash only by computing the hash itself, if the \"hash the\ncontent of the source files\" has to be exactly SHA-256. Using Git\nrepository that uses SHA-256 would *not* help.\n\n $ git init --object-format sha256\n $ echo hello | git hash-object --stdin\n 2cf8d83d9ee29543b34a87727421fdecb7e3f3a183d337639025de576db9ebb4\n $ echo hello | sha256sum\n 5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03 -\n\nThis is because the object name used by Git is not the hash of the\ncontent. It is a hash of an object header (object type and byte\ncount) followed by its contents.\n\n $ printf \"blob 6\\0hello\\n\" | sha256sum\n 2cf8d83d9ee29543b34a87727421fdecb7e3f3a183d337639025de576db9ebb4 -\n\nThe build systems can choose to tell FUSE server to expose the Git\nobject names via xattr, but if it needs to see if some contents (not\nin FUSE) it has on hand is the same as what is stored in the FUSE\nserver, it needs to use the \"slightly modified SHA-256\" that matches\nwhat Git uses. It would still be using some hash that has the same\nstrength as underlying SHA-256, but it is *not* SHA-256.\n\n"},{"id":"479670","messageId":"ZLlNtbAbVcYH7eFb@adams","threadId":"59926","inReplyTo":"2f5de416-04ba-c23d-1e0b-83bb655829a7@zombino.com","subject":"Re: SHA256 support not experimental, or?","fromName":"Adam Majer","fromEmail":"adamm@zombino.com","sentAt":"2023-07-20T15:07:54Z","receivedAt":"2023-07-20T15:08:01Z","isPatch":false,"sender":{"key":"adamm@zombino.com","avatar":"https://avatars.githubusercontent.com/u/1211498?v=4"},"body":"I'll try again with inline patch. I think it wasn't picked up since it\nwas mime encoded by the mail client..\n\n- Adam\n\n\nFrom 90be51143e741053390810720ba4a639c3b0b74c Mon Sep 17 00:00:00 2001\nFrom: Adam Majer \nDate: Wed, 28 Jun 2023 14:46:02 +0200\nSubject: [PATCH] doc: sha256 is no longer experimental\n\nThe purpose of this patch is to remove scary wording that basically\nstops people using sha256 repositories not because of interoperability\nissues with sha1 repositories, but from fear that their work will\nsuddenly become incompatible in some future version of git.\n\nWe should be clear that currently sha256 repositories will not work with\nsha1 repositories but stop the scary words.\n\nSigned-off-by: Adam Majer \n---\n Documentation/git.txt | 4 ++--\n Documentation/object-format-disclaimer.txt | 8 ++------\n 2 files changed, 4 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex f0cafa2290..666dbdb55c 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -553,8 +553,8 @@ double-quotes and respecting backslash escapes. E.g., the value\n \tIf this variable is set, the default hash algorithm for new\n \trepositories will be set to this value. This value is\n \tignored when cloning and the setting of the remote repository\n-\tis always used. The default is \"sha1\". THIS VARIABLE IS\n-\tEXPERIMENTAL! See `--object-format` in linkgit:git-init[1].\n+\tis always used. The default is \"sha1\". See `--object-format`\n+\tin linkgit:git-init[1].\n \n Git Commits\n ~~~~~~~~~~~\ndiff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt\nindex 4cb106f0d1..1e976688be 100644\n--- a/Documentation/object-format-disclaimer.txt\n+++ b/Documentation/object-format-disclaimer.txt\n@@ -1,6 +1,2 @@\n-THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still\n-in an early stage. A SHA-256 repository will in general not be able to\n-share work with \"regular\" SHA-1 repositories. It should be assumed\n-that, e.g., Git internal file formats in relation to SHA-256\n-repositories may change in backwards-incompatible ways. Only use\n-`--object-format=sha256` for testing purposes.\n+Note: SHA-256 repositories currently will not be able to share work\n+with \"regular\" SHA-1 repositories.\n-- \n2.41.0\n\n"},{"id":"479675","messageId":"xmqqr0p230rj.fsf@gitster.g","threadId":"59926","inReplyTo":"ZLlNtbAbVcYH7eFb@adams","subject":"Re: SHA256 support not experimental, or?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-20T18:18:08Z","receivedAt":"2023-07-20T18:18:16Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adam Majer writes:\n\n> I'll try again with inline patch. I think it wasn't picked up since it\n> was mime encoded by the mail client..\n>\n> - Adam\n>\n>\n> From 90be51143e741053390810720ba4a639c3b0b74c Mon Sep 17 00:00:00 2001\n\nRemove all the above lines (including the \"From \"). If you want to add a note that should not be recorded in\nthe message of the resulting commit, write it _after_ the three-dash\nline after your sign-off.\n\n> From: Adam Majer \n> Date: Wed, 28 Jun 2023 14:46:02 +0200\n> Subject: [PATCH] doc: sha256 is no longer experimental\n\nIt is not technically incorrect to have these three lines here, but\nwhen you are presenting your own work, it is preferrable to do\nwithout them. The \"From:\" address line and \"Subject:\" text line do\nnot have to be here---most people should be able to make the\ncorresponding e-mail headers to have the value they want to use,\nand while the above \"Date:\" might be the time you wrote the commit,\nit is way earlier than the time the contents of the commit was\npresented for consideration to the general public, which is recorded\nin the e-mail header of the message you are sending.\n\nSo, the body of the message usually should start from here (below).\n\nIn general, please follow [[describe-changes]] part of the\nDocumentation/SubmittingPatches document, and also \"git log\n--no-merges\" of recent contributions by others. \"The purpose of\nthis patch is\" is not how we usually talk about our work.\n\n> The purpose of this patch is to remove scary wording that basically\n> stops people using sha256 repositories not because of interoperability\n> issues with sha1 repositories, but from fear that their work will\n> suddenly become incompatible in some future version of git.\n>\n> We should be clear that currently sha256 repositories will not work with\n> sha1 repositories but stop the scary words.\n>\n> Signed-off-by: Adam Majer \n> ---\n> Documentation/git.txt | 4 ++--\n> Documentation/object-format-disclaimer.txt | 8 ++------\n> 2 files changed, 4 insertions(+), 8 deletions(-)\n>\n> diff --git a/Documentation/git.txt b/Documentation/git.txt\n> index f0cafa2290..666dbdb55c 100644\n> --- a/Documentation/git.txt\n> +++ b/Documentation/git.txt\n> @@ -553,8 +553,8 @@ double-quotes and respecting backslash escapes. E.g., the value\n> \tIf this variable is set, the default hash algorithm for new\n> \trepositories will be set to this value. This value is\n> \tignored when cloning and the setting of the remote repository\n> -\tis always used. The default is \"sha1\". THIS VARIABLE IS\n> -\tEXPERIMENTAL! See `--object-format` in linkgit:git-init[1].\n> +\tis always used. The default is \"sha1\". See `--object-format`\n> +\tin linkgit:git-init[1].\n\nThis side looks OK (just removing the single sentence).\n\n> Git Commits\n> ~~~~~~~~~~~\n> diff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt\n> index 4cb106f0d1..1e976688be 100644\n> --- a/Documentation/object-format-disclaimer.txt\n> +++ b/Documentation/object-format-disclaimer.txt\n> @@ -1,6 +1,2 @@\n> -THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still\n> -in an early stage. A SHA-256 repository will in general not be able to\n> -share work with \"regular\" SHA-1 repositories. It should be assumed\n> -that, e.g., Git internal file formats in relation to SHA-256\n> -repositories may change in backwards-incompatible ways. Only use\n> -`--object-format=sha256` for testing purposes.\n> +Note: SHA-256 repositories currently will not be able to share work\n> +with \"regular\" SHA-1 repositories.\n\nThe original did not have this problem because it had enough\nsurrounding context, but the updated text now risks getting misread\nas if there are \"regular\" and \"special\" SHA-1 repositories, the\nlatter of which might work better with SHA-256.\n\nAnd the message about SHA-256's non-experimental status can probably\nbe a lot stronger, after the discussion we had recently. How about\nsaying something like:\n\n Note: there is no interoperability between SHA-256 repositories\n and SHA-1 repositories right now. We historically warned that\n SHA-256 repositories may need backward incompatible changes\n later when we introduce such interoperability features, but at\n this point we do not expect that we need to make such a change\n when we do so, and the users can expect that their SHA-256\n repositories they create with today's Git will be usable by\n future versions of Git without losing information.\n\nwhich would probably be much closer to what you wanted to hear?\n\nThanks.\n"},{"id":"479902","messageId":"xmqqedkuei7e.fsf@gitster.g","threadId":"59926","inReplyTo":"xmqqr0p230rj.fsf@gitster.g","subject":"Re: SHA256 support not experimental, or?","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-26T16:44:05Z","receivedAt":"2023-07-26T16:44:32Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Adam Majer writes:\n>\n>> I'll try again with inline patch.\n>>\n>> From 90be51143e741053390810720ba4a639c3b0b74c Mon Sep 17 00:00:00 2001\n>\n> Remove all the above lines (including the \"From ...\n>> Signed-off-by: Adam Majer \n>> ---\n>> Documentation/git.txt | 4 ++--\n>> Documentation/object-format-disclaimer.txt | 8 ++------\n>> 2 files changed, 4 insertions(+), 8 deletions(-)\n>> ...\n> This side looks OK (just removing the single sentence).\n>\n>> Git Commits\n>> ~~~~~~~~~~~\n>> diff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt\n>> index 4cb106f0d1..1e976688be 100644\n>> --- a/Documentation/object-format-disclaimer.txt\n>> +++ b/Documentation/object-format-disclaimer.txt\n>> @@ -1,6 +1,2 @@\n>> ...\n>\n> The original did not have this problem because it had enough\n> surrounding context, but the updated text now risks getting misread\n> as if there are \"regular\" and \"special\" SHA-1 repositories, the\n> latter of which might work better with SHA-256.\n>\n> And the message about SHA-256's non-experimental status can probably\n> be a lot stronger, after the discussion we had recently. How about\n> saying something like:\n>\n> Note: there is no interoperability between SHA-256 repositories\n> and SHA-1 repositories right now. We historically warned that\n> SHA-256 repositories may need backward incompatible changes\n> later when we introduce such interoperability features, but at\n> this point we do not expect that we need to make such a change\n> when we do so, and the users can expect that their SHA-256\n> repositories they create with today's Git will be usable by\n> future versions of Git without losing information.\n>\n> which would probably be much closer to what you wanted to hear?\n\nIt has been a week. Any news on this topic?\n\nThanks.\n"},{"id":"479994","messageId":"d8ba032f-51bc-0bab-fd24-25dea0d56966@zombino.com","threadId":"59926","inReplyTo":"xmqqr0p230rj.fsf@gitster.g","subject":"Re: SHA256 support not experimental, or?","fromName":"Adam Majer","fromEmail":"adamm@zombino.com","sentAt":"2023-07-31T13:38:14Z","receivedAt":"2023-07-31T13:38:22Z","isPatch":false,"sender":{"key":"adamm@zombino.com","avatar":"https://avatars.githubusercontent.com/u/1211498?v=4"},"body":"On 7/20/23 20:18, Junio C Hamano wrote:\n> Adam Majer writes:\n> \n>> From 90be51143e741053390810720ba4a639c3b0b74c Mon Sep 17 00:00:00 2001\n> \n> Remove all the above lines (including the \"From name>\"). If you want to add a note that should not be recorded in\n> the message of the resulting commit, write it _after_ the three-dash\n> line after your sign-off.\n\nWill do. I think the problem was `git format-patch` and then basically \npasting that inline instead of using it for basis of an email.\n\nI will try again.\n\n> So, the body of the message usually should start from here (below).\n\n+1\n\n> In general, please follow [[describe-changes]] part of the\n> Documentation/SubmittingPatches document, and also \"git log\n> --no-merges\" of recent contributions by others. \"The purpose of\n> this patch is\" is not how we usually talk about our work.\n\n+1\n\n> And the message about SHA-256's non-experimental status can probably\n> be a lot stronger, after the discussion we had recently. How about\n> saying something like:\n> \n> Note: there is no interoperability between SHA-256 repositories\n> and SHA-1 repositories right now. We historically warned that\n> SHA-256 repositories may need backward incompatible changes\n> later when we introduce such interoperability features, but at\n> this point we do not expect that we need to make such a change\n> when we do so, and the users can expect that their SHA-256\n> repositories they create with today's Git will be usable by\n> future versions of Git without losing information.\n> \n> which would probably be much closer to what you wanted to hear?\n\nThanks, I've included additional context now, rebased on top of next \nbranch and will attach it as reply to this message.\n\n- Adam\n"},{"id":"479995","messageId":"ZMe6KmzZGVubYpvO@adams","threadId":"59926","inReplyTo":"d8ba032f-51bc-0bab-fd24-25dea0d56966@zombino.com","subject":"[PATCH] doc: sha256 is no longer experimental","fromName":"Adam Majer","fromEmail":"adamm@zombino.com","sentAt":"2023-07-31T13:42:02Z","receivedAt":"2023-07-31T13:42:10Z","isPatch":true,"sender":{"key":"adamm@zombino.com","avatar":"https://avatars.githubusercontent.com/u/1211498?v=4"},"body":"Remove scary wording that basically stops people using sha256\nrepositories not because of interoperability issues with sha1\nrepositories, but from fear that their work will suddenly become\nincompatible in some future version of git.\n\nWe should be clear that currently sha256 repositories will not work with\nsha1 repositories but stop the scary words.\n\nSigned-off-by: Adam Majer \n---\n Documentation/git.txt | 4 ++--\n Documentation/object-format-disclaimer.txt | 15 +++++++++------\n 2 files changed, 11 insertions(+), 8 deletions(-)\n\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex f0cafa2290..11228956cd 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -553,8 +553,8 @@ double-quotes and respecting backslash escapes. E.g., the value\n \tIf this variable is set, the default hash algorithm for new\n \trepositories will be set to this value. This value is\n \tignored when cloning and the setting of the remote repository\n-\tis always used. The default is \"sha1\". THIS VARIABLE IS\n-\tEXPERIMENTAL! See `--object-format` in linkgit:git-init[1].\n+\tis always used. The default is \"sha1\".\n+\tSee `--object-format` in linkgit:git-init[1].\n \n Git Commits\n ~~~~~~~~~~~\ndiff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt\nindex 4cb106f0d1..359f393ec9 100644\n--- a/Documentation/object-format-disclaimer.txt\n+++ b/Documentation/object-format-disclaimer.txt\n@@ -1,6 +1,9 @@\n-THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still\n-in an early stage. A SHA-256 repository will in general not be able to\n-share work with \"regular\" SHA-1 repositories. It should be assumed\n-that, e.g., Git internal file formats in relation to SHA-256\n-repositories may change in backwards-incompatible ways. Only use\n-`--object-format=sha256` for testing purposes.\n+Note: At present, there is no interoperability between SHA-256\n+repositories and SHA-1 repositories.\n+\n+Historically, we warned that SHA-256 repositories may later need\n+backward incompatible changes when we introduce such interoperability\n+features. Today, we only expect compatible changes. Furthermore, if such\n+changes prove to be necessary, it can expected that SHA-256 repositories\n+created with today's Git will be usable by future versions of Git\n+without data loss.\n-- \n2.41.0\n\n"},{"id":"480001","messageId":"xmqqr0ooyss9.fsf@gitster.g","threadId":"59926","inReplyTo":"ZMe6KmzZGVubYpvO@adams","subject":"Re: [PATCH] doc: sha256 is no longer experimental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-07-31T16:01:10Z","receivedAt":"2023-07-31T16:01:24Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Adam Majer writes:\n\n> +Note: At present, there is no interoperability between SHA-256\n> +repositories and SHA-1 repositories.\n> +\n> +Historically, we warned that SHA-256 repositories may later need\n> +backward incompatible changes when we introduce such interoperability\n> +features. Today, we only expect compatible changes. Furthermore, if such\n> +changes prove to be necessary, it can expected that SHA-256 repositories\n\n\"can BE expected\" (will tweak locally while queueing; no need to resend).\n\n> +created with today's Git will be usable by future versions of Git\n> +without data loss.\n\nWill queue. Thanks!\n"},{"id":"480003","messageId":"27BFEA76-A6F5-418C-B18C-1206704849F7@zombino.com","threadId":"59926","inReplyTo":"xmqqr0ooyss9.fsf@gitster.g","subject":"Re: [PATCH] doc: sha256 is no longer experimental","fromName":"Adam Majer","fromEmail":"adamm@zombino.com","sentAt":"2023-07-31T16:44:45Z","receivedAt":"2023-07-31T16:45:01Z","isPatch":true,"sender":{"key":"adamm@zombino.com","avatar":"https://avatars.githubusercontent.com/u/1211498?v=4"},"body":"\n\nOn July 31, 2023 6:01:10 p.m. GMT+02:00, Junio C Hamano wrote:\n>Adam Majer writes:\n>> +changes prove to be necessary, it can expected that SHA-256 repositories\n>\n>\"can BE expected\" (will tweak locally while queueing; no need to resend).\n\n+1\n\n>\n>Will queue. Thanks!\n\nThanks!\n\n-Adam\n"}]}