{"thread":{"id":"65818","subject":"SHA-1/SHA-256 interoperability work is functional","startedAt":"2026-06-16T00:17:16Z","lastAt":"2026-06-16T21:31:29Z","messageCount":3,"participants":["brian m. carlson","Junio C Hamano"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"545617","messageId":"ajCWBG9RHBrm8jMZ@fruit.crustytoothpaste.net","threadId":"65818","inReplyTo":null,"subject":"SHA-1/SHA-256 interoperability work is functional","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2026-06-16T00:17:08Z","receivedAt":"2026-06-16T00:17:16Z","isPatch":false,"body":"I'm pleased to announce that I have Git fully passing the testsuite and\nCI in interoperability mode, both with SHA-256 and SHA-1 as the main\nalgorithm.  While this is very exciting, the work is not ready to send\nto the list and is effectively a draft, since there is still cleanup\nand efficiency work to be done.\n\nWhat is fully functional:\n\n* The testsuite.\n* The protocol, including extensions for mapping objects to support\n  shallow clones, submodules, and partial clones and interoperability\n  with remotes of different algorithms.\n* A full complement of functionality for everyday use cases.\n\nFeatures which are currently unsupported (and which may or may not be\nsupported in the future):\n\n* Filtered bundles are unsupported because there is currently no way to provide\n\ta mapping.\n* Multi-pack index cannot be used as the sole pack index format because it does\n\tnot yet provide mappings.\n* Pack index v1 and v2 cannot be used because they do not provide object\n\tmappings.  Git automatically uses pack index v3 instead when necessary, which\n\tdoes handle mappings.\n* Packfile URIs are not supported because the protocol-provided packfile is not\n\tcomplete and its objects cannot be mapped.\n* Large object promisors cannot be used if the server does not actually have\n\tthe entire history, since the server must have a complete history in order to\n\tprovide object mappings.\n* `git fast-import` does not accept submodules in compatibility mode because\n\tthere is no provision for mappings.\n* Remote helpers do not emit signatures in the compatibility algorithm for\n\tsigned tags.\n* The WebDAV-based HTTP protocol doesn't support interoperability due to\n  the lack of a way to distribute mappings.\n\nSome additional things that may need to be improved:\n\n* We have some recursive delta resolution code in `git index-pack` that\n  will need to be made iterative to avoid stack overflows.\n* We need to batch object maps whenever we write them, since having too\n  many causes `git gc` to kick off frequently (which can be seen in some\n  tests).  This will require substantial refactoring of code like `git\n  add`, since any time we write an object we must be sure to always\n  write the mapping (even if we `die`).\n* We will probably want to move the object map repacking code out of\n  `git gc` into a separate command that we can call manually.\n* We may want some debugging tools for pack index v3 and other data\n  formats so that we can show the mapping of objects.\n* We will probably want to be able to sign in only one algorithm instead\n  of always in both.  Users may not want to sign the SHA-1 format for\n  security reasons.\n* There is some extremely basic code for `^{sha1}` and `^{sha256}` to\n  help `git rev-list` perform connectivity checks for remotes using the\n  compatibility algorithm, but it is very much incomplete and will need\n  to be completed or fenced off.\n* Operations can be rather slow in some cases and we'll want to see what\n  speedups we can perform.\n* Probably other things I have forgotten.\n\nThere is new documentation in `Documentation/gitformat-hash.adoc` that\noutlines the requirements for using the protocol.  The protocol\nrestrictions described there are hard technical limitations that cannot\nbe avoided; I've intentionally made things as featureful as they can be.\nThis imposes real restrictions on using protocol interoperability with many\nprojects, including Git and Linux[0].  Interested parties may wish to look\nat t1017 to see what's tested vis-à-vis the protocol and\ninteroperability.\n\nOn the end of the series is a small amount of new Rust code that moves\nus in the direction of object file conversion in Rust.  All the pointer\narithmetic makes me very nervous from a security perspective, especially\nin network-facing code, so my hope is to eventually port that code over.\nNote that Rust is already required for the interoperability work, so\nthis doesn't add any new dependencies.\n\nI also have some unpublished code that could start on in-place migration\nfunctionality much like is done with `git refs migrate` once the Rust\nprerequisites above are ready.  This could also make it possible at some\npoint to migrate any relevant submodules as part of the migration of the\nmain repository.  I may or may not complete this work, but perhaps\nsomeone else will want to pick it up if I do not.  I'm certain that such\ncode would be valuable to many people, including forges.\n\nI intend to rebase and tidy this work for at least a little while, but\ndon't actually intend to send it upstream, since there are some\ntechnical limitations that prevent me from doing so.  I have, however,\nbeen in contact with someone (who may identify themselves if they\nchoose) who is interested in getting some of the polishing and\nupstreaming work done, which I deeply appreciate.\n\nIf you're interested in testing or perusing the work, you may get it\nfrom the `sha256-interop` branch of https://github.com/bk2204/git.git.\nPlease note that it may be rebased, rewound, or otherwise folded,\nspindled, or mutilated at any time.\n\nEven though the testsuite is passing earlier than I expected, I don't\nexpect it to make Git 3.0, nor do I think we should delay Git 3.0 for\nthis work.  There are approximately 200 patches currently (and more to\ncome if we add in-place migration tooling), so it seems very unlikely\nthat we could get the entire series upstream in any reasonable amount of\ntime.  We will also very much want to give this time in an experimental\nstate so people can try it out and report back on things that should be\nimproved, which is further evidence that it's not right for Git 3.0.\n\nI'm happy to answer any further questions if folks have any.\n\n[0] Git uses submodules, which would need to be rewritten first in order\nto migrate, and Linux uses mergetags for which the tagged object is not\nin the history (which makes mapping the commit containing it fail).\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"545681","messageId":"xmqqldce2r1a.fsf@gitster.g","threadId":"65818","inReplyTo":"ajCWBG9RHBrm8jMZ@fruit.crustytoothpaste.net","subject":"Re: SHA-1/SHA-256 interoperability work is functional","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2026-06-16T20:01:37Z","receivedAt":"2026-06-16T20:01:39Z","isPatch":false,"body":"\"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n\n> I'm pleased to announce that I have Git fully passing the testsuite and\n> CI in interoperability mode, both with SHA-256 and SHA-1 as the main\n> algorithm.  While this is very exciting, the work is not ready to send\n> to the list and is effectively a draft, since there is still cleanup\n> and efficiency work to be done.\n\nGreat to hear about a great milestone.\n\n> Features which are currently unsupported (and which may or may not be\n> supported in the future):\n>\n> * Filtered bundles are unsupported because there is currently no way to provide\n> \ta mapping.\n> * Multi-pack index cannot be used as the sole pack index format because it does\n> \tnot yet provide mappings.\n> * Pack index v1 and v2 cannot be used because they do not provide object\n> \tmappings.  Git automatically uses pack index v3 instead when necessary, which\n> \tdoes handle mappings.\n\n> * Packfile URIs are not supported because the protocol-provided packfile is not\n> \tcomplete and its objects cannot be mapped.\n\nNot that I specifically care about packfile URI, but this one is\ncurious.  How would regular \"fetch\" and \"push\" traffic work under\nthe new world order?  Presumably we will keep one characteristic of\nthe protocol, that the packdata stream is the only thing that is\ngiven to the other side and no object names are given, because the\nreceiving end would not want to blindly trust the object name the\nsending end _claims_ to have sent and instead recomputes the object\nname out of the packed objects in the data stream (\"if we rehash\nand recompute the object names from the datastream, the other side\ncannot lie to us\" IIRC was a security measure).\n\nFor a regular \"fetch\" and \"push\" to work, we would need to recompute\nthe native object names and also somehow compute the compatibility\nobject names if we are in interoperability mode, no?\n\nIf we download *.pack files from a packfile URI, wouldn't it be the\nsame story?\n\n> * Large object promisors cannot be used if the server does not actually have\n> \tthe entire history, since the server must have a complete history in order to\n> \tprovide object mappings.\n\nAgain, this one worries me a bit, but perhaps I am not reading it\ncorrectly.  Does this mean that the server side says \"this is the\ndata for object whose name is X in the SHA-1 world, which translates\nto X256 in the SHA-256 world\", the receiving end blindly trusts\nwithout having a way to verify?\n\n> There is new documentation in `Documentation/gitformat-hash.adoc` that\n> outlines the requirements for using the protocol.  The protocol\n> restrictions described there are hard technical limitations that cannot\n> be avoided; I've intentionally made things as featureful as they can be.\n> This imposes real restrictions on using protocol interoperability with many\n> projects, including Git and Linux[0].  Interested parties may wish to look\n> at t1017 to see what's tested vis-à-vis the protocol and\n> interoperability.\n> ...\n> If you're interested in testing or perusing the work, you may get it\n> from the `sha256-interop` branch of https://github.com/bk2204/git.git.\n> Please note that it may be rebased, rewound, or otherwise folded,\n> spindled, or mutilated at any time.\n\nSounds exciting.\n\nThanks.\n"},{"id":"545693","messageId":"ajHAr5XL3Tzery7J@fruit.crustytoothpaste.net","threadId":"65818","inReplyTo":"xmqqldce2r1a.fsf@gitster.g","subject":"Re: SHA-1/SHA-256 interoperability work is functional","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2026-06-16T21:31:27Z","receivedAt":"2026-06-16T21:31:29Z","isPatch":false,"body":"On 2026-06-16 at 20:01:37, Junio C Hamano wrote:\n> Not that I specifically care about packfile URI, but this one is\n> curious.  How would regular \"fetch\" and \"push\" traffic work under\n> the new world order?  Presumably we will keep one characteristic of\n> the protocol, that the packdata stream is the only thing that is\n> given to the other side and no object names are given, because the\n> receiving end would not want to blindly trust the object name the\n> sending end _claims_ to have sent and instead recomputes the object\n> name out of the packed objects in the data stream (\"if we rehash\n> and recompute the object names from the datastream, the other side\n> cannot lie to us\" IIRC was a security measure).\n> \n> For a regular \"fetch\" and \"push\" to work, we would need to recompute\n> the native object names and also somehow compute the compatibility\n> object names if we are in interoperability mode, no?\n> \n> If we download *.pack files from a packfile URI, wouldn't it be the\n> same story?\n\nLet me explain how conversion works.  Say you have an empty local\nrepository with SHA-256 as the main algorithm and SHA-1 as the\ncompatibility algorithm, plus a SHA-1 remote.  When you do `git fetch`,\nyou get a SHA-1 pack.  You cannot write this into the repository because\nyour repository doesn't use SHA-1 as the main algorithm, so `git\nindex-pack` takes the pack and maps any objects.  If the objects are in\nthe new pack, they get rewritten based on the dependencies; otherwise,\nGit uses the existing maps in the repository to rewrite the objects.\n`git index-pack` then writes a completely new SHA-256 pack with an index\ncontaining the SHA-1 mapping, using the corresponding deltas[0].\n\nHowever, `git index-pack` can only index and map objects for one pack at\na time.  We therefore need any pack that we get to be connected to our\nexisting history so that we can rely on our existing maps to remap\nobjects that are not in the pack.  For instance, if we get a commit\nwithout its parents, then we'll simply die because those objects cannot\nbe mapped and we can't write the mapping in the index.\n\nThe problem is that that packfile URIs result in multiple packs (one of\nwhich is the dynamically generated protocol pack) that, _in total_,\nprovide a complete history with what we have, but are not necessarily\nindividually connected to our existing history.  Moreover, the\ndynamically generated protocol pack is sent _first_, so if we have\npackfile URIs, that pack is almost certainly guaranteed _not_ to connect\nto our existing history.  We would therefore have to pause index-pack,\ndownload all the packfile URIs, index those packfiles (which would have\nto be connected to our history), and then unpause index-pack to rewrite\nthe history.  This is not impossible, but it's tricky, and it has yet to\nbe implemented.  Someone may decide that this is a valuable feature and\nimplement it, but it's not on my to-do list.\n\nThis doesn't pose a problem with single-algorithm repositories because\nif you have unreferenced and unconnected objects, no big deal.  They\njust don't get used and will eventually get GC'd.  But since we can't\nmap those objects in a multi-algorithm world, that's fatal and those\npacks can't be indexed.\n\n> > * Large object promisors cannot be used if the server does not actually have\n> > \tthe entire history, since the server must have a complete history in order to\n> > \tprovide object mappings.\n> \n> Again, this one worries me a bit, but perhaps I am not reading it\n> correctly.  Does this mean that the server side says \"this is the\n> data for object whose name is X in the SHA-1 world, which translates\n> to X256 in the SHA-256 world\", the receiving end blindly trusts\n> without having a way to verify?\n\nThe server provides algorithm mappings for for submodules, shallow\nclones, and partial clones.  For shallow clones and partial clones, you\nhave to trust the server anyway because you're already getting a\ntruncated history.  If you complete the history by fetching the missing\nobjects and run `git fsck`, then it will detect if the mapping is\ninvalid because the server was dishonest and complain.  You will have a\ncorrupt set of mappings to the compatibility algorithm, but those could\ntheoretically be repaired.\n\nHowever, in order for the server to produce those mappings, it has to\nknow the entire history.  If there are objects that are outside the\nrepository in a secondary location, the server will not have mappings\nfor those objects and so it will abort the protocol.\n\nThe server does not normally provide mappings for non-submodules if\nyou're doing a regular fetch or clone, since the client has a\nself-contained history and does not need those objects to compute the\nmapping.  That means that regular clones and fetches work just fine\nagainst existing servers as long as no submodules are involved[1].\n\nThe tricky part is submodules.  Because the data is in a separate\nrepository, we cannot be certain of the mapping.  The documentation says\nthis:\n\n  There is a potential security problem with providing mappings of\n  submodules over the protocol.  Namely, there is no way to guarantee\n  that the SHA-1 object ID and the SHA-256 object ID correspond to the\n  same commit.  This means that, for example, a malicious server could\n  provide a SHA-256 object ID for a submodule that was up to date with\n  all security fixes, but map that to a SHA-1 object ID for an older\n  commit with security problems.\n\nWe therefore reject submodule mappings if fsck verification for\ntransferred objects is enabled unless the user has explicitly enabled\nsubmodule mappings.\n\n[0] If A deltas against B in SHA-1, then when those are rewritten into\nSHA-256, we delta the SHA-256 A against the SHA-256 B.  This does not\nguarantee the best possible delta, but it is much cheaper than\nredeltifying and because we expect remapped objects to have the same\nshape, it should delta well enough in most cases.\n[1] For instance, if you build my branch, you can do\n`git clone --object-format=sha256:sha1 https://github.com/bk2204/lawn.git`\nand it just works since there are no submodules.  My dotfiles, on the\nother hand, have submodules and will not work without protocol support.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"}]}