{"thread":{"id":"49758","subject":"Design of multiple hash support","startedAt":"2018-11-05T01:00:41Z","lastAt":"2018-11-06T00:13:48Z","messageCount":7,"participants":["brian m. carlson","Junio C Hamano","Stefan Beller","Duy Nguyen","Jonathan Nieder"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"362435","messageId":"20181105010032.GN731755@genre.crustytoothpaste.net","threadId":"49758","inReplyTo":null,"subject":"Design of multiple hash support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-05T01:00:33Z","receivedAt":"2018-11-05T01:00:41Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"I'm currently working on getting Git to support multiple hash algorithms\nin the same binary (SHA-1 and SHA-256). In order to have a fully\nfunctional binary, we'll need to have some way of indicating to certain\ncommands (such as init and show-index) that they should assume a certain\nhash algorithm.\n\nThere are basically two approaches I can take. The first is to provide\neach command that needs to learn about this with its own --hash\nargument. So we'd have:\n\n git init --hash=sha256\n git show-index --hash=sha256 writes:\n\n> I'm currently working on getting Git to support multiple hash algorithms\n> in the same binary (SHA-1 and SHA-256). In order to have a fully\n> functional binary, we'll need to have some way of indicating to certain\n> commands (such as init and show-index) that they should assume a certain\n> hash algorithm.\n>\n> There are basically two approaches I can take. The first is to provide\n> each command that needs to learn about this with its own --hash\n> argument. So we'd have:\n>\n> git init --hash=sha256\n> git show-index --hash=sha256 \n> The other alternative is that we provide a global option to git, which\n> is parsed by all programs, like so:\n>\n> git --hash=sha256 init\n> git --hash=sha256 show-index wrote:\n>\n> \"brian m. carlson\" writes:\n>\n> > I'm currently working on getting Git to support multiple hash algorithms\n> > in the same binary (SHA-1 and SHA-256). In order to have a fully\n> > functional binary, we'll need to have some way of indicating to certain\n> > commands (such as init and show-index) that they should assume a certain\n> > hash algorithm.\n> >\n> > There are basically two approaches I can take. The first is to provide\n> > each command that needs to learn about this with its own --hash\n> > argument. So we'd have:\n> >\n> > git init --hash=sha256\n> > git show-index --hash=sha256 >\n> > The other alternative is that we provide a global option to git, which\n> > is parsed by all programs, like so:\n> >\n> > git --hash=sha256 init\n> > git --hash=sha256 show-index \n> I am assuming that \"show-index\" above is a typo for something like\n> \"hash-object\"?\n\nActually both seem plausible, as both do not require\nRUN_SETUP, which means they cannot rely on the\nextensions.objectFormat setting.\n\nWhen having a global setting, would that override the configured\nobject format extension in a repository, or do we error out?\n\nSo maybe\n\n git -c extensions.objectFormat=sha256 init\n\nis the way to go, for now? (Are repository format extensions parsed\njust like normal config, such that non-RUN_SETUP commands\ncan rely on the (non-)existence to determine whether to use\nthe default or the given hash function?)\n\n> It is hard to answer the question without knowing what exactly does\n> \"(to) support multiple hash algorithms\" mean. For example, inside\n> today's repository, what should this command do?\n>\n> git --hash=sha256 cat-file commit HEAD\n\nThere is a section \"Object names on the command line\"\nin Documentation/technical/hash-function-transition.txt\nand I assume that this before the \"dark launch\"\nphase, so I would expect the latter to work (no error\nbut conversion/translation on the fly) eventually as a goal.\nBut the former might be in scope of one series.\n\n> It can work this way:\n>\n> - read HEAD, discover that I am on 'master' branch, read refs/heads/master\n> to learn the object name in 40-hex, realize that it cannot be\n> sha256 and report \"corrupt ref\".\n>\n> Or it can work this way:\n>\n> - read repository format, realize it is a good old sha1 repository.\n>\n> - do the usual thing to get to read_object() to read the commit\n> object data for the commit at HEAD, doing all of it in sha1.\n>\n> - in the commit object data, locate references to other objects\n> that use sha1 name.\n>\n> - replace these sha1 references with their sha256 counterparts and\n> show the result.\n>\n> I am guessing that you are doing the former as a good first step, in\n> which case, as an option that changes/affects the behaviour of git\n> globally, I think \"git --hash=sha256\" would make sense, like other\n> global options like --literal-pathspecs and --no-replace-objects.\n>\n> Thanks.\n"},{"id":"362503","messageId":"CACsJy8B+WAyjrthKs9nr=kLpx7f8k_Dug4rRdYDoBR+mmLHCuQ@mail.gmail.com","threadId":"49758","inReplyTo":"20181105010032.GN731755@genre.crustytoothpaste.net","subject":"Re: Design of multiple hash support","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-11-05T19:03:39Z","receivedAt":"2018-11-05T19:04:07Z","isPatch":false,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Mon, Nov 5, 2018 at 2:02 AM brian m. carlson\n wrote:\n>\n> I'm currently working on getting Git to support multiple hash algorithms\n> in the same binary (SHA-1 and SHA-256). In order to have a fully\n> functional binary, we'll need to have some way of indicating to certain\n> commands (such as init and show-index) that they should assume a certain\n> hash algorithm.\n>\n> There are basically two approaches I can take. The first is to provide\n> each command that needs to learn about this with its own --hash\n> argument. So we'd have:\n>\n> git init --hash=sha256\n> git show-index --hash=sha256 \n> The other alternative is that we provide a global option to git, which\n> is parsed by all programs, like so:\n>\n> git --hash=sha256 init\n> git --hash=sha256 show-index \n\nI suppose this is about the \"no repository/standalone\" mode, because\n\n - it's hard to pass global arguments down to builtin commands (we\noften have to rely on global variables which are on the way out)\n\n - global options confuse new people and also harder to reorder (if\nyou forget it, you have to alt-b all the way back to near the\nbeginning of the command line and add it there, instead of near the\nend)\n\n - there aren't that many standalone commands\n\nI'm leaning towards \"git foo --hash=\".\n\n> There's also the question of what we want to call the option. The\n> obvious name is --hash, which is intuitive and straightforward.\n> However, the transition plan names the config option\n> extensions.objectFormat, so --object-format is also a possibility. If\n> we ever decide to support, say, zstd compression instead of zlib, we\n> could leverage the same option (say, --object-format=sha256:zstd) and\n> avoid the need for an additional option. This might be planning for a\n> future that never occurs, though.\n\n--object-format is less vague than --hash. The downside is it's longer\n(more to type) but I'm counting on git-completion.bash and the guess\nthat people rarely need to use this option.\n-- \nDuy\n"},{"id":"362535","messageId":"20181105220042.GA162375@google.com","threadId":"49758","inReplyTo":"CACsJy8B+WAyjrthKs9nr=kLpx7f8k_Dug4rRdYDoBR+mmLHCuQ@mail.gmail.com","subject":"Re: Design of multiple hash support","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2018-11-05T22:00:42Z","receivedAt":"2018-11-05T22:00:47Z","isPatch":false,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nDuy Nguyen wrote:\n> On Mon, Nov 5, 2018 at 2:02 AM brian m. carlson\n> wrote:\n\n>> There are basically two approaches I can take. The first is to provide\n>> each command that needs to learn about this with its own --hash\n>> argument. So we'd have:\n>>\n>> git init --hash=sha256\n>> git show-index --hash=sha256 >\n>> The other alternative is that we provide a global option to git, which\n>> is parsed by all programs, like so:\n>>\n>> git --hash=sha256 init\n>> git --hash=sha256 show-index I'm leaning towards \"git foo --hash=\".\n\nCan you say a little more about the semantics of the option? For\ncommands like \"git init\", I tend to agree with Duy here, since it\nallows each command's manual to describe what the option means in the\ncontext of that command.\n\nFor \"git show-index\", ideally Git should use the object format named\nin the idx file.\n\n>> There's also the question of what we want to call the option. The\n>> obvious name is --hash, which is intuitive and straightforward.\n>> However, the transition plan names the config option\n>> extensions.objectFormat,\n[...]\n> --object-format is less vague than --hash. The downside is it's longer\n> (more to type) but I'm counting on git-completion.bash and the guess\n> that people rarely need to use this option.\n\nAgreed. --object-format makes more sense to me than --hash, since\nit's more precise about what the option affects.\n\nThanks for looking into this.\n\nSincerely,\nJonathan\n"},{"id":"362539","messageId":"20181105235424.GB890086@genre.crustytoothpaste.net","threadId":"49758","inReplyTo":"CAGZ79kZQQ2BqaANTQU_eHgOS0nBp9YzyTx1WAfpgADOyeLs4XQ@mail.gmail.com","subject":"Re: Design of multiple hash support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-05T23:54:24Z","receivedAt":"2018-11-05T23:54:35Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Nov 05, 2018 at 10:03:21AM -0800, Stefan Beller wrote:\n> On Sun, Nov 4, 2018 at 6:36 PM Junio C Hamano wrote:\n> >\n> > \"brian m. carlson\" writes:\n> >\n> > > I'm currently working on getting Git to support multiple hash algorithms\n> > > in the same binary (SHA-1 and SHA-256). In order to have a fully\n> > > functional binary, we'll need to have some way of indicating to certain\n> > > commands (such as init and show-index) that they should assume a certain\n> > > hash algorithm.\n> > >\n> > > There are basically two approaches I can take. The first is to provide\n> > > each command that needs to learn about this with its own --hash\n> > > argument. So we'd have:\n> > >\n> > > git init --hash=sha256\n> > > git show-index --hash=sha256 > >\n> > > The other alternative is that we provide a global option to git, which\n> > > is parsed by all programs, like so:\n> > >\n> > > git --hash=sha256 init\n> > > git --hash=sha256 show-index >\n> > I am assuming that \"show-index\" above is a typo for something like\n> > \"hash-object\"?\n\n> Actually both seem plausible, as both do not require\n> RUN_SETUP, which means they cannot rely on the\n> extensions.objectFormat setting.\n\nCorrect. In general, I assume that options that want a repository will\nuse the repository for that information. There are a small number of\nprograms, such as init, that need to either set up a repository (without\nreference to another repository) or need to inspect files without\nnecessarily being in a repository.\n\nFor example, we will want to have a way of indicating which hash we\nwould like to use in a fresh repository. I am for the moment assuming\nthat we're in a stage 4 configuration: that is, that we're all SHA-1 or\nall SHA-256. A clone will provide this for us, but a git init will not.\n\nAlso, our pack index v3 format knows about which hash algorithm is in\nuse, but packs are not labeled with the algorithm they use. This isn't\nreally a problem in normal use, since we always know from context which\nalgorithm is in use, but we'll need to indicate to index-pack (which\ntechnically need not run in a repository) which algorithm it should use.\n\nshow-index will eventually learn to parse the index itself to learn\nwhich algorithms are in use, so it is technically not required here.\n\n> When having a global setting, would that override the configured\n> object format extension in a repository, or do we error out?\n> \n> So maybe\n> \n> git -c extensions.objectFormat=sha256 init\n> \n> is the way to go, for now? (Are repository format extensions parsed\n> just like normal config, such that non-RUN_SETUP commands\n> can rely on the (non-)existence to determine whether to use\n> the default or the given hash function?)\n\nThe extensions callbacks are only handled in check_repo_format, so they\nnecessarily require a repository. This is not new with my code.\n\nFurthermore, one would have to specify \"-c\ncore.repositoryformatversion=1\" as well, as extensions require that\nversion in order to have any effect.\n\nMy current approach for the testsuite is to have git init honor a new\nGIT_DEFAULT_HASH environment variable so we need not modify every place\nin the testsuite that calls git init (of which there are many). That\nmay or may not be greeted with joy by reviewers, but it seemed to be the\nminimum viable approach.\n\n> There is a section \"Object names on the command line\"\n> in Documentation/technical/hash-function-transition.txt\n> and I assume that this before the \"dark launch\"\n> phase, so I would expect the latter to work (no error\n> but conversion/translation on the fly) eventually as a goal.\n> But the former might be in scope of one series.\n\nCurrently, I'm not implementing the stage 1-3 implementations. I'm\nmerely going from the point where we have a binary that does only\nSHA-256 and cannot perform SHA-1 operations at all to a stage 4\nimplementation, where the binary can do either, but a repository is\nwholly one or the other.\n\n> > It can work this way:\n> >\n> > - read HEAD, discover that I am on 'master' branch, read refs/heads/master\n> > to learn the object name in 40-hex, realize that it cannot be\n> > sha256 and report \"corrupt ref\".\n> >\n> > Or it can work this way:\n> >\n> > - read repository format, realize it is a good old sha1 repository.\n> >\n> > - do the usual thing to get to read_object() to read the commit\n> > object data for the commit at HEAD, doing all of it in sha1.\n> >\n> > - in the commit object data, locate references to other objects\n> > that use sha1 name.\n> >\n> > - replace these sha1 references with their sha256 counterparts and\n> > show the result.\n> >\n> > I am guessing that you are doing the former as a good first step, in\n> > which case, as an option that changes/affects the behaviour of git\n> > globally, I think \"git --hash=sha256\" would make sense, like other\n> > global options like --literal-pathspecs and --no-replace-objects.\n\nRight now, we always read the repository configuration when possible,\nand honor that. I'm not planning, even when we have a full\nimplementation, to let the configuration of input and output format be\nmodified by command-line options. That's a configuration of the\nrepository in the current transition plan, and I have no intention of\nchanging that (apart from possibly honoring \"git -c\").\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"},{"id":"362540","messageId":"20181106001340.GC890086@genre.crustytoothpaste.net","threadId":"49758","inReplyTo":"20181105220042.GA162375@google.com","subject":"Re: Design of multiple hash support","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2018-11-06T00:13:40Z","receivedAt":"2018-11-06T00:13:48Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On Mon, Nov 05, 2018 at 02:00:42PM -0800, Jonathan Nieder wrote:\n> Hi,\n> \n> Duy Nguyen wrote:\n> > On Mon, Nov 5, 2018 at 2:02 AM brian m. carlson\n> > wrote:\n> \n> >> There are basically two approaches I can take. The first is to provide\n> >> each command that needs to learn about this with its own --hash\n> >> argument. So we'd have:\n> >>\n> >> git init --hash=sha256\n> >> git show-index --hash=sha256 >>\n> >> The other alternative is that we provide a global option to git, which\n> >> is parsed by all programs, like so:\n> >>\n> >> git --hash=sha256 init\n> >> git --hash=sha256 show-index [...]\n> > I'm leaning towards \"git foo --hash=\".\n> \n> Can you say a little more about the semantics of the option? For\n> commands like \"git init\", I tend to agree with Duy here, since it\n> allows each command's manual to describe what the option means in the\n> context of that command.\n\nSure. The semantics for git init are \"produce a repository with this\nhash algorithm\". The semantics for git index-pack are \"the pack I want\nyou to index uses this hash algorithm\". Essentially, more generically,\nthe semantics are \"the repository or data object uses this hash\nalgorithm\".\n\n> For \"git show-index\", ideally Git should use the object format named\n> in the idx file.\n\nI agree that will be the eventual goal. It will also be what I ship in\nthe final series, in all likelihood. I have most of pack v3\nimplemented, but it's not complete yet.\n-- \nbrian m. carlson: Houston, Texas, US\nOpenPGP: https://keybase.io/bk2204\n"}]}