{"thread":{"id":"55148","subject":"Submodules and SHA-256/SHA-1 interoperability","startedAt":"2021-02-14T21:52:13Z","lastAt":"2021-03-19T14:24:19Z","messageCount":4,"participants":["brian m. carlson","Johannes Schindelin"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"416886","messageId":"YCmbKrTsJhgPHYLc@camp.crustytoothpaste.net","threadId":"55148","inReplyTo":null,"subject":"Submodules and SHA-256/SHA-1 interoperability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-02-14T21:50:34Z","receivedAt":"2021-02-14T21:52:13Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"I'm currently working on the next step of the SHA-256 transition code,\nwhich is SHA-256/SHA-1 interoperability.  Essentially, when we write a\nloose object into the store, or when we index a pack, we take one form\nof the object, usually the SHA-256 form, and rewrite it so that it is in\nits SHA-1 form, and then hash it to determine its SHA-1 name.  We then\nwrite this correspondence either into the loose object index (for loose\nobjects) or a v3 index (for packs).\n\nBlobs are simply hashed with both algorithms, but trees, commits, and\ntags need to be rewritten to use the SHA-1 names of the objects they\nrefer to.  For most situations, we already have this data, since it will\nexist in the loose object index, in some pack index, or elsewhere in the\npack we're indexing.\n\nHowever, for submodules, we have a problem.  By definition, the object\nexists in a different repository.  If we have the submodule locally on\nthe system, then this works fine, but if we're performing a fetch or\nclone and the submodule is not present, then we cannot rewrite the tree\nor anything that refers to it, directly or indirectly.\n\nSo there are some possible courses of action:\n\n* Disallow compatibility algorithms when using submodules.  This is\n  simple, but inconvenient.\n* Force users to always clone submodules and fetch them before fetching\n  the main repository.  This is also relatively simple, but\n  inconvenient.\n* Have the remote server keep a list of correspondences and send them in\n  a protocol extension.\n* Just skip rewriting objects until the data is filled in later and\n  admit the data will be incomplete.  This means that pushing to or\n  pulling from a repository using a incompatible algorithm will be\n  impossible.\n* Something else I haven't thought of.\n\nThe third option is where I'm leaning, but it has some potential\ndownsides.  First, the server must support both hash algorithms and have\nthis data.  Second, it essentially requires all submodule updates to be\npushed from a compatible client.  Third, we need to trust that the\nserver hasn't tampered with the data, which should be possible by doing\nan fsck on both forms (I think).  Fourth, we need to store this\nsomewhere, and the only place we have right now is the loose object\nindex, which would potentially grow to inefficient sizes.\n\nWe could potentially change this to be slightly different by asking the\nsubmodule server for a list of correspondences instead via a new\nprotocol extension, but it has the same downsides except for the second\none, and additionally means that we'd need to make multiple connections.\n\nSo I'm seeking some ideas on which approach we want to use here before\nI start sinking a lot of work into this.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"418061","messageId":"nycvar.QRO.7.76.6.2103011621000.57@tvgsbejvaqbjf.bet","threadId":"55148","inReplyTo":"YCmbKrTsJhgPHYLc@camp.crustytoothpaste.net","subject":"Re: Submodules and SHA-256/SHA-1 interoperability","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-03-01T19:28:13Z","receivedAt":"2021-03-01T19:36:09Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi brian,\n\nOn Sun, 14 Feb 2021, brian m. carlson wrote:\n\n> I'm currently working on the next step of the SHA-256 transition code,\n> which is SHA-256/SHA-1 interoperability.  Essentially, when we write a\n> loose object into the store, or when we index a pack, we take one form\n> of the object, usually the SHA-256 form, and rewrite it so that it is in\n> its SHA-1 form, and then hash it to determine its SHA-1 name.  We then\n> write this correspondence either into the loose object index (for loose\n> objects) or a v3 index (for packs).\n>\n> Blobs are simply hashed with both algorithms, but trees, commits, and\n> tags need to be rewritten to use the SHA-1 names of the objects they\n> refer to.  For most situations, we already have this data, since it will\n> exist in the loose object index, in some pack index, or elsewhere in the\n> pack we're indexing.\n>\n> However, for submodules, we have a problem.  By definition, the object\n> exists in a different repository.  If we have the submodule locally on\n> the system, then this works fine, but if we're performing a fetch or\n> clone and the submodule is not present, then we cannot rewrite the tree\n> or anything that refers to it, directly or indirectly.\n>\n> So there are some possible courses of action:\n>\n> * Disallow compatibility algorithms when using submodules.  This is\n>   simple, but inconvenient.\n> * Force users to always clone submodules and fetch them before fetching\n>   the main repository.  This is also relatively simple, but\n>   inconvenient.\n> * Have the remote server keep a list of correspondences and send them in\n>   a protocol extension.\n> * Just skip rewriting objects until the data is filled in later and\n>   admit the data will be incomplete.  This means that pushing to or\n>   pulling from a repository using a incompatible algorithm will be\n>   impossible.\n> * Something else I haven't thought of.\n\nWhile my strong urge is to add \"Remove support for submodules\" (which BTW\nwould also plug so many attack vectors that have lead to many a\nvulnerability in the past), I understand that this would be impractical:\nthe figurative barn door has been open for way too long to do that.\n\nBut I'd like to put another idea into the fray: store the mapping in\n`.gitmodules`. That is, each time `git submodule add <...>` is called, it\nwould update `.gitmodules` to list SHA-1 *and* SHA-256 for the given path.\n\nThat would relieve us of the problem where we rely on a server's ability\nto give us that mapping.\n\nCiao,\nDscho\n\n> The third option is where I'm leaning, but it has some potential\n> downsides.  First, the server must support both hash algorithms and have\n> this data.  Second, it essentially requires all submodule updates to be\n> pushed from a compatible client.  Third, we need to trust that the\n> server hasn't tampered with the data, which should be possible by doing\n> an fsck on both forms (I think).  Fourth, we need to store this\n> somewhere, and the only place we have right now is the loose object\n> index, which would potentially grow to inefficient sizes.\n>\n> We could potentially change this to be slightly different by asking the\n> submodule server for a list of correspondences instead via a new\n> protocol extension, but it has the same downsides except for the second\n> one, and additionally means that we'd need to make multiple connections.\n>\n> So I'm seeking some ideas on which approach we want to use here before\n> I start sinking a lot of work into this.\n> --\n> brian m. carlson (he/him or they/them)\n> Houston, Texas, US\n>\n"},{"id":"419049","messageId":"YE0Vki0HNOYhYIYh@camp.crustytoothpaste.net","threadId":"55148","inReplyTo":"nycvar.QRO.7.76.6.2103011621000.57@tvgsbejvaqbjf.bet","subject":"Re: Submodules and SHA-256/SHA-1 interoperability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2021-03-13T19:42:10Z","receivedAt":"2021-03-13T19:43:10Z","isPatch":false,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2021-03-01 at 19:28:13, Johannes Schindelin wrote:\n> While my strong urge is to add \"Remove support for submodules\" (which BTW\n> would also plug so many attack vectors that have lead to many a\n> vulnerability in the past), I understand that this would be impractical:\n> the figurative barn door has been open for way too long to do that.\n> \n> But I'd like to put another idea into the fray: store the mapping in\n> `.gitmodules`. That is, each time `git submodule add <...>` is called, it\n> would update `.gitmodules` to list SHA-1 *and* SHA-256 for the given path.\n> \n> That would relieve us of the problem where we rely on a server's ability\n> to give us that mapping.\n\nThis is true, but it ends up causing problems because we don't know\nwhere the .gitmodules file is for a given revision.  If we're indexing a\npack file, we lack the ability to know which .gitmodules file is\nassociated with the blobs in a given revision, and we can't finish\nindexing that file until we have both hashes for every object.\n\nWhile we could change the way we do indexing, we'd end up having to\ncrawl the history and that would be very slow.\n-- \nbrian m. carlson (he/him or they/them)\nHouston, Texas, US\n"},{"id":"419708","messageId":"nycvar.QRO.7.76.6.2103191521290.57@tvgsbejvaqbjf.bet","threadId":"55148","inReplyTo":"YE0Vki0HNOYhYIYh@camp.crustytoothpaste.net","subject":"Re: Submodules and SHA-256/SHA-1 interoperability","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2021-03-19T14:23:09Z","receivedAt":"2021-03-19T14:24:19Z","isPatch":false,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi brian,\n\nOn Sat, 13 Mar 2021, brian m. carlson wrote:\n\n> On 2021-03-01 at 19:28:13, Johannes Schindelin wrote:\n> > While my strong urge is to add \"Remove support for submodules\" (which BTW\n> > would also plug so many attack vectors that have lead to many a\n> > vulnerability in the past), I understand that this would be impractical:\n> > the figurative barn door has been open for way too long to do that.\n> >\n> > But I'd like to put another idea into the fray: store the mapping in\n> > `.gitmodules`. That is, each time `git submodule add <...>` is called, it\n> > would update `.gitmodules` to list SHA-1 *and* SHA-256 for the given path.\n> >\n> > That would relieve us of the problem where we rely on a server's ability\n> > to give us that mapping.\n>\n> This is true, but it ends up causing problems because we don't know\n> where the .gitmodules file is for a given revision.  If we're indexing a\n> pack file, we lack the ability to know which .gitmodules file is\n> associated with the blobs in a given revision, and we can't finish\n> indexing that file until we have both hashes for every object.\n>\n> While we could change the way we do indexing, we'd end up having to\n> crawl the history and that would be very slow.\n\nHrm, that's a valid point.\n\nI just wish that we could make this more independent of servers. There\n_might_ be a way to work around the flaw you pointed out, e.g. adding the\nmappings from `.gitmodules` to the repository-local SHA-1 <-> SHA-256\nmapping. But maybe somebody else can think of a better way?\n\nCiao,\nDscho\n"}]}