{"thread":{"id":"54003","subject":"[PATCH] Documentation: mark `--object-format=sha256` as experimental","startedAt":"2020-08-06T20:24:13Z","lastAt":"2020-08-16T10:01:41Z","messageCount":8,"participants":["Martin Ågren","brian m. carlson","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":null},"messages":[{"id":"403093","messageId":"20200806202358.2265705-1-martin.agren@gmail.com","threadId":"54003","inReplyTo":null,"subject":"[PATCH] Documentation: mark `--object-format=sha256` as experimental","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-06T20:23:58Z","receivedAt":"2020-08-06T20:24:13Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"After eff45daab8 (\"repository: enable SHA-256 support by default\",\n2020-07-29), vanilla builds of Git enable the user to run, e.g.,\n\n git init --object-format=sha256\n\nand hack away. This can be a good way to gain experience with the\nSHA-256 world, e.g., to find bugs that\n\n GIT_TEST_DEFAULT_HASH=sha256 make test\n\ndoesn't spot.\n\nBut it really is a separate world: Such SHA-256 repos will live entirely\nseparate from the (by now fairly large) set of SHA-1 repos. Interacting\nacross the border is possible in principle, e.g., through \"diff + apply\"\n(or \"format-patch + am\"), but even that has its limitations: Applying a\nSHA-256 diff in a SHA-1 repo works in the simple case, but if you need\nto resort to `-3`, you're out of luck.\n\nSimilarly, \"push + pull\" should work, but you really will be operating\nmostly offset from the rest of the world. That might be ok by the time\nyou initialize your repository, and it might be ok for several months\nafter that, but there might come a day when you're starting to regret\nyour use of `git init --object-format=sha256` and have dug yourself into\na fairly deep hole.\n\nWorkflows aside, let's consider a more technical aspect. Pack index\nfiles (pack-*.idx) exist in two flavours: v1 and v2. The hash transition\ndocument foresees a v3, which we do not yet support (and for all we\nknow, the final v3 might end up different from the one sketched in the\nhash transition document).\n\nWhen the test suite is run with SHA-1 as the default hash algo, it\ncreates and consumes v2 pack files. But with SHA-256, we use an\nundocumented, hybrid format where the header looks like v2, but where\nthe payload is not only \"not correct SHA1\", but where even the data\nsizes are different. The trailing checksum is different, meaning no-one\n(except us!) should/would try to interpret this file as a proper v2 pack\nindex.\n\nWe could certainly (re)define v2 to match our SHA-256 behavior, but we\ndo foresee v3 for a reason. And that would still just fix this specific\nissue. And even when everything around SHA-256 is well-defined and we\nhave SHA-1--SHA-256 interoperability, there's a risk, at least\ninitially, that somewhere we'd be permeating buggy data that we'd then\nfeel responsible for and need to be able to handle for a long time to\ncome.\n\nIn short: we need some time and leeway.\n\nWherever `--object-format` is mentioned in our documentation, let's make\nit clear that using it with \"sha256\" is experimental. If we later need\nto explain why we can't handle data we generated back in 2020, we can\nalways point to this paragraph we're adding here.\n\nBy \"include::\"-ing a small blurb, we should be able to be consistent\nthroughout the documentation and can eventually gradually tone down the\nseverity of this text. One day, we might even use it to start phasing\nout `--object-format=sha1`, but let's not get ahead of ourselves...\n\nThere's also `extensions.objectFormat`, but it's only mentioned three\ntimes. Twice where we're adding this new disclaimer and in the third\nspot we already have a \"do not edit\" warning. From there, interested\nreaders should eventually find this new one that we're adding here.\n\nBecause `GIT_DEFAULT_HASH` provides another entry point to this\nfunctionality, document the experimental nature of it too.\n\nSigned-off-by: Martin Ågren \n---\n Documentation/git-index-pack.txt | 2 ++\n Documentation/git-init.txt | 2 ++\n Documentation/git-show-index.txt | 2 ++\n Documentation/git.txt | 3 ++-\n Documentation/object-format-disclaimer.txt | 6 ++++++\n 5 files changed, 14 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/object-format-disclaimer.txt\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex 9316d9a80b..139285dd0f 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -100,6 +100,8 @@ OPTIONS\n \tvalue is set or outside a repository.\n +\n This option cannot be used with --stdin.\n++\n+include::object-format-disclaimer.txt[]\n \n NOTES\n -----\ndiff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\nindex ddfe265da5..f35f70f13d 100644\n--- a/Documentation/git-init.txt\n+++ b/Documentation/git-init.txt\n@@ -53,6 +53,8 @@ current working directory.\n \n Specify the given object format (hash algorithm) for the repository. The valid\n values are 'sha1' and (if enabled) 'sha256'. 'sha1' is the default.\n++\n+include::object-format-disclaimer.txt[]\n \n --template=::\n \ndiff --git a/Documentation/git-show-index.txt b/Documentation/git-show-index.txt\nindex 39b1d8eaa1..e49318a5a0 100644\n--- a/Documentation/git-show-index.txt\n+++ b/Documentation/git-show-index.txt\n@@ -44,6 +44,8 @@ OPTIONS\n \tvalid values are 'sha1' and (if enabled) 'sha256'. The default is the\n \talgorithm for the current repository (set by `extensions.objectFormat`), or\n \t'sha1' if no value is set or outside a repository..\n++\n+include::object-format-disclaimer.txt[]\n \n GIT\n ---\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 3e50065198..e0a2690ba2 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -497,7 +497,8 @@ double-quotes and respecting backslash escapes. E.g., the value\n \tIf this variable is set, the default hash algorithm for new\n \trepositories will be set to this value. This value is currently\n \tignored when cloning; the setting of the remote repository\n-\tis used instead. The default is \"sha1\".\n+\tis used instead. The default is \"sha1\". THIS VARIABLE IS\n+\tEXPERIMENTAL! See `--object-format` in linkgit:git-init[1].\n \n Git Commits\n ~~~~~~~~~~~\ndiff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt\nnew file mode 100644\nindex 0000000000..4cb106f0d1\n--- /dev/null\n+++ b/Documentation/object-format-disclaimer.txt\n@@ -0,0 +1,6 @@\n+THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still\n+in an early stage. A SHA-256 repository will in general not be able to\n+share work with \"regular\" SHA-1 repositories. It should be assumed\n+that, e.g., Git internal file formats in relation to SHA-256\n+repositories may change in backwards-incompatible ways. Only use\n+`--object-format=sha256` for testing purposes.\n-- \n2.28.0.199.g4234a9100e\n\n"},{"id":"403103","messageId":"20200806230837.GA8085@camp.crustytoothpaste.net","threadId":"54003","inReplyTo":"20200806202358.2265705-1-martin.agren@gmail.com","subject":"Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-08-06T23:08:37Z","receivedAt":"2020-08-06T23:09:22Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-08-06 at 20:23:58, Martin Ågren wrote:\n> After eff45daab8 (\"repository: enable SHA-256 support by default\",\n> 2020-07-29), vanilla builds of Git enable the user to run, e.g.,\n> \n> git init --object-format=sha256\n> \n> and hack away. This can be a good way to gain experience with the\n> SHA-256 world, e.g., to find bugs that\n> \n> GIT_TEST_DEFAULT_HASH=sha256 make test\n> \n> doesn't spot.\n> \n> But it really is a separate world: Such SHA-256 repos will live entirely\n> separate from the (by now fairly large) set of SHA-1 repos. Interacting\n> across the border is possible in principle, e.g., through \"diff + apply\"\n> (or \"format-patch + am\"), but even that has its limitations: Applying a\n> SHA-256 diff in a SHA-1 repo works in the simple case, but if you need\n> to resort to `-3`, you're out of luck.\n> \n> Similarly, \"push + pull\" should work, but you really will be operating\n> mostly offset from the rest of the world. That might be ok by the time\n> you initialize your repository, and it might be ok for several months\n> after that, but there might come a day when you're starting to regret\n> your use of `git init --object-format=sha256` and have dug yourself into\n> a fairly deep hole.\n\nI do agree that they don't interoperate right now, and that we'd like it\nto in the future. But there are definitely people who can use SHA-256\nsupport for new projects without problems. I'm aware of certain\ngovernment agencies who very much do not want to use SHA-1 at all (and\nat some point will be legally prohibited from doing so), and they will\nbe completely fine with the status quo. Some of those same\norganizations are unhappy about prohibited algorithms even being linked\ninto the binaries they use. These folks can use a suitably new version\nof Git everywhere and not care about the lack of backwards\ncompatibility.\n\nI am, of course, in favor of abandoning SHA-1 as fast as practically\npossible, but I understand that backwards compatibility is obviously a\nconcern.\n\n> Workflows aside, let's consider a more technical aspect. Pack index\n> files (pack-*.idx) exist in two flavours: v1 and v2. The hash transition\n> document foresees a v3, which we do not yet support (and for all we\n> know, the final v3 might end up different from the one sketched in the\n> hash transition document).\n> \n> When the test suite is run with SHA-1 as the default hash algo, it\n> creates and consumes v2 pack files. But with SHA-256, we use an\n> undocumented, hybrid format where the header looks like v2, but where\n> the payload is not only \"not correct SHA1\", but where even the data\n> sizes are different. The trailing checksum is different, meaning no-one\n> (except us!) should/would try to interpret this file as a proper v2 pack\n> index.\n>\n> We could certainly (re)define v2 to match our SHA-256 behavior, but we\n> do foresee v3 for a reason. And that would still just fix this specific\n> issue. And even when everything around SHA-256 is well-defined and we\n> have SHA-1--SHA-256 interoperability, there's a risk, at least\n> initially, that somewhere we'd be permeating buggy data that we'd then\n> feel responsible for and need to be able to handle for a long time to\n> come.\n\nThese are valid index v1 and v2 files, just with a different hash\nalgorithm. v3 is there for the point where we do interoperate and need\nto store hash values of multiple algorithms at once. There's little to\nno benefit to v3 if you don't need multiple algorithm support, other\nthan the fact that they declare the algorithms in them.\n\nThis is no different than saying that our commit or tree objects are in\na different form; they are syntactically identical, just with a\ndifferent hash algorithm. That's how everything is in the .git\ndirectory.\n\n> diff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt\n> new file mode 100644\n> index 0000000000..4cb106f0d1\n> --- /dev/null\n> +++ b/Documentation/object-format-disclaimer.txt\n> @@ -0,0 +1,6 @@\n> +THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still\n> +in an early stage. A SHA-256 repository will in general not be able to\n> +share work with \"regular\" SHA-1 repositories. It should be assumed\n> +that, e.g., Git internal file formats in relation to SHA-256\n> +repositories may change in backwards-incompatible ways. Only use\n> +`--object-format=sha256` for testing purposes.\n\nI'm fine with marking the functionality experimental for a few releases,\nsince it is possible we have bugs people haven't found, and adding a\nnote about interoperability after that point, since I think that's a\nfair and valuable issue. I think if we go a few releases without any\nmajor issues, we can change this to the following:\n\n Note that a SHA-256 repository cannot yet share work with \"regular\"\n SHA-1 repositories. Many tools do not yet understand SHA-256\n repositories, so users may wish to take this into account when\n creating new repositories.\n-- \nbrian m. carlson: Houston, Texas, US\n"},{"id":"403130","messageId":"CAN0heSptiJL9d86ZeNPMUaZeTA68juwTyf3K-uWR=K-vt=1Hrg@mail.gmail.com","threadId":"54003","inReplyTo":"20200806230837.GA8085@camp.crustytoothpaste.net","subject":"Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-07T14:08:10Z","receivedAt":"2020-08-07T14:08:34Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 7 Aug 2020 at 01:08, brian m. carlson\n wrote:\n>\n> On 2020-08-06 at 20:23:58, Martin Ågren wrote:\n> > After eff45daab8 (\"repository: enable SHA-256 support by default\",\n> > 2020-07-29), vanilla builds of Git enable the user to run, e.g.,\n> >\n> > git init --object-format=sha256\n> >\n> > and hack away. [...]\n[...]\n> > Similarly, \"push + pull\" should work, but you really will be operating\n> > mostly offset from the rest of the world. That might be ok by the time\n> > you initialize your repository, and it might be ok for several months\n> > after that, but there might come a day when you're starting to regret\n> > your use of `git init --object-format=sha256` and have dug yourself into\n> > a fairly deep hole.\n>\n> I do agree that they don't interoperate right now, and that we'd like it\n> to in the future. But there are definitely people who can use SHA-256\n> support for new projects without problems. I'm aware of certain\n> government agencies who very much do not want to use SHA-1 at all (and\n> at some point will be legally prohibited from doing so), and they will\n> be completely fine with the status quo. Some of those same\n> organizations are unhappy about prohibited algorithms even being linked\n> into the binaries they use. These folks can use a suitably new version\n> of Git everywhere and not care about the lack of backwards\n> compatibility.\n>\n> I am, of course, in favor of abandoning SHA-1 as fast as practically\n> possible, but I understand that backwards compatibility is obviously a\n> concern.\n\nYeah, I'd prefer them to know that they are early adopters and that they\nshould be prepared for a situation where there's some incompatibility\nacross versions. I don't just mean \"I can't read my old SHA-1 data any\nmore\", I mean \"I used Git v2.29.0 to create a SHA-256 repo and now Git\nv3.15.0 won't read it\". (Or v2.31.0?)\n\nI've followed the work on the commit graph functionality and file format\nmostly from the sidelines. It's been lots of good work with lots of good\noutcome, but there also seems to have been (of course) a few\nincompatibilities, bugs and \"argh, if only we'd have done it like this\nfrom the beginning\". I'd assume the effort -- and potential for bugs and\n\"ooh, we should have done it that way\" -- for SHA-256--SHA-1\ninteroperability to be larger than what's been put into the commit graph\nso far.\n\n> > Workflows aside, let's consider a more technical aspect. Pack index\n> > files (pack-*.idx) exist in two flavours: v1 and v2. [...]\n[...]\n> > We could certainly (re)define v2 to match our SHA-256 behavior, but we\n> > do foresee v3 for a reason. And that would still just fix this specific\n> > issue. And even when everything around SHA-256 is well-defined and we\n> > have SHA-1--SHA-256 interoperability, there's a risk, at least\n> > initially, that somewhere we'd be permeating buggy data that we'd then\n> > feel responsible for and need to be able to handle for a long time to\n> > come.\n>\n> These are valid index v1 and v2 files, just with a different hash\n> algorithm.\n\nI claim that they are not valid, precisely because they use a different\nhash algorithm.\n\n> v3 is there for the point where we do interoperate and need\n> to store hash values of multiple algorithms at once. There's little to\n> no benefit to v3 if you don't need multiple algorithm support, other\n> than the fact that they declare the algorithms in them.\n\nOne additional benefit: they'd correspond to a specification. :-)\n\n> This is no different than saying that our commit or tree objects are in\n> a different form; they are syntactically identical, just with a\n> different hash algorithm. That's how everything is in the .git\n> directory.\n\nFor objects, I could perhaps accept that the format outlined in the hash\ntransition document is the specification.\n\nThat document says that pack indices \"use a new v3 format that supports\nmultiple hash functions.\" It then goes on to draft such a format. (Maybe\nit's a specification, but until there exists at least one\nimplementation, I'd rather see it as a draft.) No mention of v2 pack\nindices with SHA-256 data, neither in that document, nor anywhere else\nin our documentation that I could find.\n\nThe \"v2 but with SHA-256\" packfile index format we're producing contains\nlots of 32 B SHA-256s instead of 20 B SHA-1s, ok, that much can be\nguessed in one try. The index file ends with a 32 B SHA-256, after\nreferencing the 32 B packfile SHA-256. Ok, maybe that could also be\nguessed. If we're committed to maintaining that format, we should put it\ndown in writing. And if we're not committed to it, we should make that\nclear.\n\nThe hash transition document foresees a packfile index format v3.\nNotably, it uses a _20_ B SHA-256 checksum and references a _20_ B\nSHA-256 packfile SHA-256. In light of that, are we certain that the \"v2\nwith SHA-256\" format outlined above is not a maintenance burden? Or that\nif there is any kind of cost, that it's worth it? Or, for that matter,\nthat guessing the details of \"v2 but with SHA-256\" is trivial?\n\nI fully respect the effort that has gone into making the test suite\nrun with 32 B SHA-256 instead of 20 B SHA-1. But do we really intend to\nsupport for many years to come the new file formats that such a test run\nproduces and consumes?\n\nBundles v3, yeah I guess so. Thanks for making that move!\n\nPack index \"v2 but with SHA-256\", maybe. At the very least, we should\nset down our feet consciously.\n\n> > +THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still\n> > +in an early stage. A SHA-256 repository will in general not be able to\n> > +share work with \"regular\" SHA-1 repositories. It should be assumed\n> > +that, e.g., Git internal file formats in relation to SHA-256\n> > +repositories may change in backwards-incompatible ways. Only use\n> > +`--object-format=sha256` for testing purposes.\n>\n> I'm fine with marking the functionality experimental for a few releases,\n> since it is possible we have bugs people haven't found, and adding a\n> note about interoperability after that point, since I think that's a\n> fair and valuable issue. I think if we go a few releases without any\n> major issues, we can change this to the following:\n>\n> Note that a SHA-256 repository cannot yet share work with \"regular\"\n> SHA-1 repositories. Many tools do not yet understand SHA-256\n> repositories, so users may wish to take this into account when\n> creating new repositories.\n\nWith respect, I think that's too aggressive. By that time, we may\nconclude that, e.g., the \"v2 pack indices with SHA-256\" file handling is\nrobust. But I'd be surprised if using `git init --object-format=sha256`\nin June 2021 won't cause *some* extra work for users or ourselves\nfurther down the line compared to using a regular SHA-1 `git init`.\nPushing to a SHA-1 hosting service will become *possible* at some point,\nbut maybe it won't be *efficient enough to be practical in the real\nworld* until some time after that.\n\nAll those other, *new* file formats outlined in the hash\ntransition document won't exist at that time (at least not in master).\n\nNow would probably be a good time to update the hash transition\ndocuments, first of all to tick off what we've already done, and second,\nto reassess the rest. Quoting:\n\n The first user-visible change is the introduction of the objectFormat\n extension (without compatObjectFormat). This requires:\n - implementing the loose-object-idx\n - teaching fsck about this mode of operation\n - using the hash function API (vtable) when computing object names\n - signing objects and verifying signatures\n - rejecting attempts to fetch from or push to an incompatible\n repository\n\nI don't think we're there yet. Maybe, e.g., the new loose-object-idx\nisn't strictly needed, in which case this part of the plan could be\nupdated. (Or maybe who wrote the above thought there'd be some value in\nknowing that *all* SHA-256 repos *always* have loose-object-idx tables\nto save us from some file-discovery dancing?) We do say elsewhere in the\ndocument that\n\n Alongside the packfile, a SHA-256 repository stores a bidirectional\n mapping between SHA-256 and SHA-1 object names.\n\nSo at the time we do not seem to be producing correct, proper,\nas-specified (or at least as-drafted) SHA-256 repositories. Or maybe in\n2030, we can stop insisting on such a mapping, because everyone uses\nSHA-256 anyway, so then maybe it shouldn't be mandatory now, either.\n\n\"Signing objects\" is a bit vague, but under \"Signed Commits\", I see:\n\n [...]\n\n This means commits can be signed\n 1. using SHA-1 only, as in existing signed commit objects\n 2. using both SHA-1 and SHA-256, by using both gpgsig-sha256 and gpgsig\n fields.\n 3. using only SHA-256, by only using the gpgsig-sha256 field.\n\nRight now, we can do either 1 or 3. Maybe that's enough. I do think\nthere's a bug in git-replace where we'll only remove the last signature,\nbut as we'll currently only create one signature, that's perhaps \"ok\".\nI still believe we should think hard before saying (even if we only say\nso by omission) that, e.g., file structures are known-good and will be\nsupported for a long time to come.\n\nMartin\n"},{"id":"403150","messageId":"xmqqr1sifaeu.fsf@gitster.c.googlers.com","threadId":"54003","inReplyTo":"CAN0heSptiJL9d86ZeNPMUaZeTA68juwTyf3K-uWR=K-vt=1Hrg@mail.gmail.com","subject":"Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-07T20:50:17Z","receivedAt":"2020-08-07T20:50:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Ågren writes:\n\n>> I'm fine with marking the functionality experimental for a few releases,\n>> since it is possible we have bugs people haven't found, and adding a\n>> note about interoperability after that point, since I think that's a\n>> fair and valuable issue. I think if we go a few releases without any\n>> major issues, we can change this to the following:\n>>\n>> Note that a SHA-256 repository cannot yet share work with \"regular\"\n>> SHA-1 repositories. Many tools do not yet understand SHA-256\n>> repositories, so users may wish to take this into account when\n>> creating new repositories.\n>\n> With respect, I think that's too aggressive. By that time, we may\n> conclude that, e.g., the \"v2 pack indices with SHA-256\" file handling is\n> robust. But I'd be surprised if using `git init --object-format=sha256`\n> in June 2021 won't cause *some* extra work for users or ourselves\n> further down the line compared to using a regular SHA-1 `git init`.\n> Pushing to a SHA-1 hosting service will become *possible* at some point,\n> but maybe it won't be *efficient enough to be practical in the real\n> world* until some time after that.\n\nIOW, you question \"if we go a few releases without any major issues\"\npart? I tend to agree that for a large change like this, a few\nreleases may not be sufficiently long time for a feature that is\nmarked as experimental in big flashing red letters to get exercised\nenough to get major issues noticed.\n\n> All those other, *new* file formats outlined in the hash\n> transition document won't exist at that time (at least not in master).\n>\n> Now would probably be a good time to update the hash transition\n> documents, first of all to tick off what we've already done, and second,\n> to reassess the rest.\n\nYes, it is a good idea to stop and see where in the overall large\npicture we currently are.\n\nThanks.\n"},{"id":"403171","messageId":"CAN0heSqyBzW_+vWSAxV9O1XAJKmQgrhCms7mSa+hFFx35uU05w@mail.gmail.com","threadId":"54003","inReplyTo":"xmqqr1sifaeu.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-07T22:15:13Z","receivedAt":"2020-08-07T22:15:30Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 7 Aug 2020 at 22:50, Junio C Hamano wrote:\n>\n> Martin Ågren writes:\n>\n> >> I'm fine with marking the functionality experimental for a few releases,\n> >> since it is possible we have bugs people haven't found, and adding a\n> >> note about interoperability after that point, since I think that's a\n> >> fair and valuable issue. I think if we go a few releases without any\n> >> major issues, we can change this to the following:\n> >>\n> >> Note that a SHA-256 repository cannot yet share work with \"regular\"\n> >> SHA-1 repositories. Many tools do not yet understand SHA-256\n> >> repositories, so users may wish to take this into account when\n> >> creating new repositories.\n> >\n> > With respect, I think that's too aggressive. By that time, we may\n> > conclude that, e.g., the \"v2 pack indices with SHA-256\" file handling is\n> > robust. But I'd be surprised if using `git init --object-format=sha256`\n> > in June 2021 won't cause *some* extra work for users or ourselves\n> > further down the line compared to using a regular SHA-1 `git init`.\n> > Pushing to a SHA-1 hosting service will become *possible* at some point,\n> > but maybe it won't be *efficient enough to be practical in the real\n> > world* until some time after that.\n>\n> IOW, you question \"if we go a few releases without any major issues\"\n> part? I tend to agree that for a large change like this, a few\n> releases may not be sufficiently long time for a feature that is\n> marked as experimental in big flashing red letters to get exercised\n> enough to get major issues noticed.\n\nYeah, thanks for summarizing what I failed to express using so many\nwords.\n\nI'm fully open to the idea that some people want to leave SHA-1 behind\nand that they can do it today, in some \"local\" sense. If those people\nare fully aware that they are guinea pigs, it might actually be ok for\nus to subject them to a few rounds of \"oops, Git v2.32.0 produces data\nthat v2.34.0 and later will barf on\". Or at least it would be on our\ntable whether we wanted to be that cavalier.\n\nOnce SHA-256 repos as such are no longer experimental, I fear that we\ncan only buy ourselves that leeway by introducing fiftyeleven different\nconfig flags for \"please produce auxiliary files X even if you don't\nactually use them\", \"please do use X, and I'm fully expecting to trip on\nthem if you decide to tweak them in backwards-incompatible ways\", and so\non. The alternative to buying such leeway might be to establish, pretty\nearly on, a respectable set of things we support \"for compatibility\nreasons\".\n\n> > Now would probably be a good time to update the hash transition\n> > documents, first of all to tick off what we've already done, and second,\n> > to reassess the rest.\n>\n> Yes, it is a good idea to stop and see where in the overall large\n> picture we currently are.\n\nMartin\n"},{"id":"403592","messageId":"xmqqmu2y1e0t.fsf@gitster.c.googlers.com","threadId":"54003","inReplyTo":"CAN0heSqyBzW_+vWSAxV9O1XAJKmQgrhCms7mSa+hFFx35uU05w@mail.gmail.com","subject":"Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-13T20:34:10Z","receivedAt":"2020-08-13T20:34:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Ågren writes:\n\n>> IOW, you question \"if we go a few releases without any major issues\"\n>> part? I tend to agree that for a large change like this, a few\n>> releases may not be sufficiently long time for a feature that is\n>> marked as experimental in big flashing red letters to get exercised\n>> enough to get major issues noticed.\n>\n> Yeah, thanks for summarizing what I failed to express using so many\n> words.\n>\n> I'm fully open to the idea that some people want to leave SHA-1 behind\n> and that they can do it today, in some \"local\" sense. If those people\n> are fully aware that they are guinea pigs, it might actually be ok for\n> us to subject them to a few rounds of \"oops, Git v2.32.0 produces data\n> that v2.34.0 and later will barf on\". Or at least it would be on our\n> table whether we wanted to be that cavalier.\n>\n> Once SHA-256 repos as such are no longer experimental, I fear that we\n> can only buy ourselves that leeway by introducing fiftyeleven different\n> config flags for \"please produce auxiliary files X even if you don't\n> actually use them\", \"please do use X, and I'm fully expecting to trip on\n> them if you decide to tweak them in backwards-incompatible ways\", and so\n> on. The alternative to buying such leeway might be to establish, pretty\n> early on, a respectable set of things we support \"for compatibility\n> reasons\".\n\nOK, so can we resolve this one way or the other and move on?\n\nFor now, I'd vote for applying this warning patch, but with or\nwithout such warning, it is more important to iron out those details\nwe fear might have to change.\n\nThanks.\n\n"},{"id":"403604","messageId":"20200814005151.GI8085@camp.crustytoothpaste.net","threadId":"54003","inReplyTo":"xmqqmu2y1e0t.fsf@gitster.c.googlers.com","subject":"Re: [PATCH] Documentation: mark `--object-format=sha256` as experimental","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-08-14T00:51:51Z","receivedAt":"2020-08-14T00:52:00Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-08-13 at 20:34:10, Junio C Hamano wrote:\n> Martin Ågren writes:\n> \n> >> IOW, you question \"if we go a few releases without any major issues\"\n> >> part? I tend to agree that for a large change like this, a few\n> >> releases may not be sufficiently long time for a feature that is\n> >> marked as experimental in big flashing red letters to get exercised\n> >> enough to get major issues noticed.\n> >\n> > Yeah, thanks for summarizing what I failed to express using so many\n> > words.\n> >\n> > I'm fully open to the idea that some people want to leave SHA-1 behind\n> > and that they can do it today, in some \"local\" sense. If those people\n> > are fully aware that they are guinea pigs, it might actually be ok for\n> > us to subject them to a few rounds of \"oops, Git v2.32.0 produces data\n> > that v2.34.0 and later will barf on\". Or at least it would be on our\n> > table whether we wanted to be that cavalier.\n> >\n> > Once SHA-256 repos as such are no longer experimental, I fear that we\n> > can only buy ourselves that leeway by introducing fiftyeleven different\n> > config flags for \"please produce auxiliary files X even if you don't\n> > actually use them\", \"please do use X, and I'm fully expecting to trip on\n> > them if you decide to tweak them in backwards-incompatible ways\", and so\n> > on. The alternative to buying such leeway might be to establish, pretty\n> > early on, a respectable set of things we support \"for compatibility\n> > reasons\".\n> \n> OK, so can we resolve this one way or the other and move on?\n> \n> For now, I'd vote for applying this warning patch, but with or\n> without such warning, it is more important to iron out those details\n> we fear might have to change.\n\nI'm fine with applying this patch.\n\nAs for changes, I don't think there's any changes we need to make for a\nstage 4 implementation. It works and it passes the testsuite. I've\nverified the block and gcrypt SHA-256 implementations produce identical\nresults, which was my major worry. Other than the philosophical\ndisagreement over whether index v1 and v2 should support SHA-256, I\ndon't think there's any points of contention.\n\nWhen we add support for SHA-1 interoperability, then we'll need pack\nindex v3, proper multi-pack index support, and the loose object index.\nThose are being written at the moment, and I think it's fine to add a\ntool to generate the necessary files once that code is in place (which,\nit's looking like, can just be a shell script that pipes every loose\nobject to git hash-object and rebuilds the pack indexes). Support for\ninteroperability is an additional extension, so we don't have any\ncompatibility concerns and we can generate all of those files based on\nthat option.\n\nI don't plan to enable support for extensions.compatObjectFormat without\nall of the required pieces in place. There won't be any incremental\nsteps there.\n-- \nbrian m. carlson: Houston, Texas, US\n"},{"id":"403793","messageId":"20200816100118.30558-1-martin.agren@gmail.com","threadId":"54003","inReplyTo":"20200814005151.GI8085@camp.crustytoothpaste.net","subject":"[PATCH v2] Documentation: mark `--object-format=sha256` as experimental","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-16T10:01:18Z","receivedAt":"2020-08-16T10:01:41Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"After eff45daab8 (\"repository: enable SHA-256 support by default\",\n2020-07-29), vanilla builds of Git enable the user to run, e.g.,\n\n git init --object-format=sha256\n\nand hack away. This can be a good way to gain experience with the\nSHA-256 world, e.g., to find bugs that\n\n GIT_TEST_DEFAULT_HASH=sha256 make test\n\ndoesn't spot.\n\nBut it really is a separate world: Such SHA-256 repos will live entirely\nseparate from the (by now fairly large) set of SHA-1 repos. Interacting\nacross the border is possible in principle, e.g., through \"diff + apply\"\n(or \"format-patch + am\"), but even that has its limitations: Applying a\nSHA-256 diff in a SHA-1 repo works in the simple case, but if you need\nto resort to `-3`, you're out of luck.\n\nSimilarly, \"push + pull\" should work, but you really will be operating\nmostly offset from the rest of the world. That might be ok by the time\nyou initialize your repository, and it might be ok for several months\nafter that, but there might come a day when you're starting to regret\nyour use of `git init --object-format=sha256` and have dug yourself into\na fairly deep hole.\n\nThere are currently topics in flight to document our data formats and\nprotocols regarding SHA-256 and in some cases (midx and commit-graph),\nwe're considering adjusting how the file formats indicate which object\nformat to use.\n\nWherever `--object-format` is mentioned in our documentation, let's make\nit clear that using it with \"sha256\" is experimental. If we later need\nto explain why we can't handle data we generated back in 2020, we can\nalways point to this paragraph we're adding here.\n\nBy \"include::\"-ing a small blurb, we should be able to be consistent\nthroughout the documentation and can eventually gradually tone down the\nseverity of this text. One day, we might even use it to start phasing\nout `--object-format=sha1`, but let's not get ahead of ourselves...\n\nThere's also `extensions.objectFormat`, but it's only mentioned three\ntimes. Twice where we're adding this new disclaimer and in the third\nspot we already have a \"do not edit\" warning. From there, interested\nreaders should eventually find this new one that we're adding here.\n\nBecause `GIT_DEFAULT_HASH` provides another entry point to this\nfunctionality, document the experimental nature of it too.\n\nSigned-off-by: Martin Ågren \n---\n Given the in-flight topics to document various SHA-256 details, I think\n it makes sense to simplify the commit message a bit. The diff is the\n same as in the original submission of this patch [1].\n\n [1] https://lore.kernel.org/git/20200806202358.2265705-1-martin.agren@gmail.com/\n\nRange-diff against v1:\n1: 19d5bb5028 ! 1: 0c2b124cfd Documentation: mark `--object-format=sha256` as experimental\n @@ Commit message\n your use of `git init --object-format=sha256` and have dug yourself into\n a fairly deep hole.\n \n - Workflows aside, let's consider a more technical aspect. Pack index\n - files (pack-*.idx) exist in two flavours: v1 and v2. The hash transition\n - document foresees a v3, which we do not yet support (and for all we\n - know, the final v3 might end up different from the one sketched in the\n - hash transition document).\n -\n - When the test suite is run with SHA-1 as the default hash algo, it\n - creates and consumes v2 pack files. But with SHA-256, we use an\n - undocumented, hybrid format where the header looks like v2, but where\n - the payload is not only \"not correct SHA1\", but where even the data\n - sizes are different. The trailing checksum is different, meaning no-one\n - (except us!) should/would try to interpret this file as a proper v2 pack\n - index.\n -\n - We could certainly (re)define v2 to match our SHA-256 behavior, but we\n - do foresee v3 for a reason. And that would still just fix this specific\n - issue. And even when everything around SHA-256 is well-defined and we\n - have SHA-1--SHA-256 interoperability, there's a risk, at least\n - initially, that somewhere we'd be permeating buggy data that we'd then\n - feel responsible for and need to be able to handle for a long time to\n - come.\n -\n - In short: we need some time and leeway.\n + There are currently topics in flight to document our data formats and\n + protocols regarding SHA-256 and in some cases (midx and commit-graph),\n + we're considering adjusting how the file formats indicate which object\n + format to use.\n \n Wherever `--object-format` is mentioned in our documentation, let's make\n it clear that using it with \"sha256\" is experimental. If we later need\n @@ Commit message\n functionality, document the experimental nature of it too.\n \n Signed-off-by: Martin Ågren \n - Signed-off-by: Junio C Hamano \n \n ## Documentation/git-index-pack.txt ##\n @@ Documentation/git-index-pack.txt: OPTIONS\n\n Documentation/git-index-pack.txt | 2 ++\n Documentation/git-init.txt | 2 ++\n Documentation/git-show-index.txt | 2 ++\n Documentation/git.txt | 3 ++-\n Documentation/object-format-disclaimer.txt | 6 ++++++\n 5 files changed, 14 insertions(+), 1 deletion(-)\n create mode 100644 Documentation/object-format-disclaimer.txt\n\ndiff --git a/Documentation/git-index-pack.txt b/Documentation/git-index-pack.txt\nindex ac74d058e0..af0c26232c 100644\n--- a/Documentation/git-index-pack.txt\n+++ b/Documentation/git-index-pack.txt\n@@ -100,6 +100,8 @@ OPTIONS\n \tvalue is set or outside a repository.\n +\n This option cannot be used with --stdin.\n++\n+include::object-format-disclaimer.txt[]\n \n NOTES\n -----\ndiff --git a/Documentation/git-init.txt b/Documentation/git-init.txt\nindex ddfe265da5..f35f70f13d 100644\n--- a/Documentation/git-init.txt\n+++ b/Documentation/git-init.txt\n@@ -53,6 +53,8 @@ current working directory.\n \n Specify the given object format (hash algorithm) for the repository. The valid\n values are 'sha1' and (if enabled) 'sha256'. 'sha1' is the default.\n++\n+include::object-format-disclaimer.txt[]\n \n --template=::\n \ndiff --git a/Documentation/git-show-index.txt b/Documentation/git-show-index.txt\nindex 39b1d8eaa1..e49318a5a0 100644\n--- a/Documentation/git-show-index.txt\n+++ b/Documentation/git-show-index.txt\n@@ -44,6 +44,8 @@ OPTIONS\n \tvalid values are 'sha1' and (if enabled) 'sha256'. The default is the\n \talgorithm for the current repository (set by `extensions.objectFormat`), or\n \t'sha1' if no value is set or outside a repository..\n++\n+include::object-format-disclaimer.txt[]\n \n GIT\n ---\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 3e50065198..e0a2690ba2 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -497,7 +497,8 @@ double-quotes and respecting backslash escapes. E.g., the value\n \tIf this variable is set, the default hash algorithm for new\n \trepositories will be set to this value. This value is currently\n \tignored when cloning; the setting of the remote repository\n-\tis used instead. The default is \"sha1\".\n+\tis used instead. The default is \"sha1\". THIS VARIABLE IS\n+\tEXPERIMENTAL! See `--object-format` in linkgit:git-init[1].\n \n Git Commits\n ~~~~~~~~~~~\ndiff --git a/Documentation/object-format-disclaimer.txt b/Documentation/object-format-disclaimer.txt\nnew file mode 100644\nindex 0000000000..4cb106f0d1\n--- /dev/null\n+++ b/Documentation/object-format-disclaimer.txt\n@@ -0,0 +1,6 @@\n+THIS OPTION IS EXPERIMENTAL! SHA-256 support is experimental and still\n+in an early stage. A SHA-256 repository will in general not be able to\n+share work with \"regular\" SHA-1 repositories. It should be assumed\n+that, e.g., Git internal file formats in relation to SHA-256\n+repositories may change in backwards-incompatible ways. Only use\n+`--object-format=sha256` for testing purposes.\n-- \n2.28.0.297.g1956fa8f8d\n\n"}]}