{"thread":{"id":"54039","subject":"[PATCH 0/2] Documentation updates for SHA-256","startedAt":"2020-08-13T22:49:43Z","lastAt":"2020-08-15T22:01:50Z","messageCount":35,"participants":["brian m. carlson","Derrick Stolee","Junio C Hamano","Martin Ågren"],"isPatch":true,"patchVersion":1,"patchTotal":2},"messages":[{"id":"403597","messageId":"20200813224901.2652387-1-sandals@crustytoothpaste.net","threadId":"54039","inReplyTo":null,"subject":"[PATCH 0/2] Documentation updates for SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-08-13T22:48:59Z","receivedAt":"2020-08-13T22:49:43Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"As was pointed out recently, some of our documentation doesn't properly\nreflect the new support for SHA-256. This series updates the pack and\nindex documentation to reflect that these formats can handle SHA-256,\nand updates the transition plan to reflect what we've implemented and\nwhat the next steps are.\n\nbrian m. carlson (2):\n docs: document SHA-256 pack and indices\n docs: fix step in transition plan\n\n .../technical/hash-function-transition.txt | 2 +-\n Documentation/technical/pack-format.txt | 36 +++++++++++--------\n 2 files changed, 22 insertions(+), 16 deletions(-)\n\n"},{"id":"403598","messageId":"20200813224901.2652387-2-sandals@crustytoothpaste.net","threadId":"54039","inReplyTo":"20200813224901.2652387-1-sandals@crustytoothpaste.net","subject":"[PATCH 1/2] docs: document SHA-256 pack and indices","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-08-13T22:49:00Z","receivedAt":"2020-08-13T22:49:43Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Now that we have SHA-256 support for packs and indices, let's document\nthat in SHA-256 repositories, we use SHA-256 instead of SHA-1 for object\nnames and checksums. Instead of duplicating this information throughout\nthe document, let's just document that in SHA-1 repositories, we use\nSHA-1 for these purposes, and in SHA-256 repositories, we use SHA-256.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/technical/pack-format.txt | 36 ++++++++++++++-----------\n 1 file changed, 21 insertions(+), 15 deletions(-)\n\ndiff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\nindex d3a142c652..f4c8d94f73 100644\n--- a/Documentation/technical/pack-format.txt\n+++ b/Documentation/technical/pack-format.txt\n@@ -1,6 +1,12 @@\n Git pack format\n ===============\n \n+== Checksums and object IDs\n+\n+In a repository using the traditional SHA-1, pack checksums, index checksums,\n+and object IDs (object names) mentioned below are all computed using SHA-1.\n+Similarly, in SHA-256 repositories, these values are computed using SHA-256.\n+\n == pack-*.pack files have the following format:\n \n - A header appears at the beginning and consists of the following:\n@@ -26,7 +32,7 @@ Git pack format\n \n (deltified representation)\n n-byte type and length (3-bit type, (n-1)*7+4-bit length)\n- 20-byte base object name if OBJ_REF_DELTA or a negative relative\n+ base object name if OBJ_REF_DELTA or a negative relative\n \t offset from the delta object's position in the pack if this\n \t is an OBJ_OFS_DELTA object\n compressed delta data\n@@ -34,7 +40,7 @@ Git pack format\n Observation: length of each object is encoded in a variable\n length format and is not constrained to 32-bit or anything.\n \n- - The trailer records 20-byte SHA-1 checksum of all of the above.\n+ - The trailer records a pack checksum of all of the above.\n \n === Object types\n \n@@ -58,8 +64,8 @@ ofs-delta and ref-delta, which is only valid in a pack file.\n \n Both ofs-delta and ref-delta store the \"delta\" to be applied to\n another object (called 'base object') to reconstruct the object. The\n-difference between them is, ref-delta directly encodes 20-byte base\n-object name. If the base object is in the same pack, ofs-delta encodes\n+difference between them is, ref-delta directly encodes base object\n+name. If the base object is in the same pack, ofs-delta encodes\n the offset of the base object in the pack instead.\n \n The base object could also be deltified if it's in the same pack.\n@@ -143,14 +149,14 @@ This is the instruction reserved for future expansion.\n object is stored in the packfile as the offset from the\n beginning.\n \n- 20-byte object name.\n+ one object name of the appropriate size.\n \n - The file is concluded with a trailer:\n \n- A copy of the 20-byte SHA-1 checksum at the end of\n- corresponding packfile.\n+ A copy of the pack checksum at the end of the corresponding\n+ packfile.\n \n- 20-byte SHA-1-checksum of all of the above.\n+ Index checksum of all of the above.\n \n Pack Idx file:\n \n@@ -198,7 +204,7 @@ Pack file entry: <+\n If it is not DELTA, then deflated bytes (the size above\n \t\tis the size before compression).\n \tIf it is REF_DELTA, then\n-\t 20-byte base object name SHA-1 (the size above is the\n+\t base object name (the size above is the\n \t\tsize of the delta data that follows).\n delta data, deflated.\n \tIf it is OFS_DELTA, then\n@@ -227,9 +233,9 @@ Pack file entry: <+\n \n - A 256-entry fan-out table just like v1.\n \n- - A table of sorted 20-byte SHA-1 object names. These are\n- packed together without offset values to reduce the cache\n- footprint of the binary search for a specific object name.\n+ - A table of sorted object names. These are packed together\n+ without offset values to reduce the cache footprint of the\n+ binary search for a specific object name.\n \n - A table of 4-byte CRC32 values of the packed object data.\n This is new in v2 so compressed data can be copied directly\n@@ -248,10 +254,10 @@ Pack file entry: <+\n \n - The same trailer as a v1 pack file:\n \n- A copy of the 20-byte SHA-1 checksum at the end of\n+ A copy of the pack checksum at the end of\n corresponding packfile.\n \n- 20-byte SHA-1-checksum of all of the above.\n+ Index checksum of all of the above.\n \n == multi-pack-index (MIDX) files have the following format:\n \n@@ -329,4 +335,4 @@ CHUNK DATA:\n \n TRAILER:\n \n-\t20-byte SHA1-checksum of the above contents.\n+\tIndex checksum of the above contents.\n"},{"id":"403599","messageId":"20200813224901.2652387-3-sandals@crustytoothpaste.net","threadId":"54039","inReplyTo":"20200813224901.2652387-1-sandals@crustytoothpaste.net","subject":"[PATCH 2/2] docs: fix step in transition plan","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-08-13T22:49:01Z","receivedAt":"2020-08-13T22:49:43Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"One of the required steps for the objectFormat extension is to implement\nthe loose object index. However, without support for\ncompatObjectFormat, we don't even know if the loose object index is\nneeded, so it makes sense to move that step to the compatObjectFormat\nsection. Do so.\n\nSigned-off-by: brian m. carlson \n---\n Documentation/technical/hash-function-transition.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/hash-function-transition.txt b/Documentation/technical/hash-function-transition.txt\nindex 5b2db3be1e..6fd20ebbc2 100644\n--- a/Documentation/technical/hash-function-transition.txt\n+++ b/Documentation/technical/hash-function-transition.txt\n@@ -650,7 +650,6 @@ Some initial steps can be implemented independently of one another:\n \n The first user-visible change is the introduction of the objectFormat\n extension (without compatObjectFormat). This requires:\n-- implementing the loose-object-idx\n - teaching fsck about this mode of operation\n - using the hash function API (vtable) when computing object names\n - signing objects and verifying signatures\n@@ -658,6 +657,7 @@ extension (without compatObjectFormat). This requires:\n repository\n \n Next comes introduction of compatObjectFormat:\n+- implementing the loose-object-idx\n - translating object names between object formats\n - translating object content between object formats\n - generating and verifying signatures in the compat format\n"},{"id":"403615","messageId":"ffb5868b-a1af-6f77-9900-aac530201cdc@gmail.com","threadId":"54039","inReplyTo":"20200813224901.2652387-2-sandals@crustytoothpaste.net","subject":"Re: [PATCH 1/2] docs: document SHA-256 pack and indices","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-14T02:26:52Z","receivedAt":"2020-08-14T02:26:56Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/13/2020 6:49 PM, brian m. carlson wrote:\n> Now that we have SHA-256 support for packs and indices, let's document\n> that in SHA-256 repositories, we use SHA-256 instead of SHA-1 for object\n> names and checksums. Instead of duplicating this information throughout\n> the document, let's just document that in SHA-1 repositories, we use\n> SHA-1 for these purposes, and in SHA-256 repositories, we use SHA-256.\n> \n> Signed-off-by: brian m. carlson \n> ---\n> Documentation/technical/pack-format.txt | 36 ++++++++++++++-----------\n> 1 file changed, 21 insertions(+), 15 deletions(-)\n> \n> diff --git a/Documentation/technical/pack-format.txt b/Documentation/technical/pack-format.txt\n> index d3a142c652..f4c8d94f73 100644\n> --- a/Documentation/technical/pack-format.txt\n> +++ b/Documentation/technical/pack-format.txt\n> @@ -1,6 +1,12 @@\n> Git pack format\n> ===============\n> \n> +== Checksums and object IDs\n> +\n> +In a repository using the traditional SHA-1, pack checksums, index checksums,\n> +and object IDs (object names) mentioned below are all computed using SHA-1.\n> +Similarly, in SHA-256 repositories, these values are computed using SHA-256.\n> +\n> == pack-*.pack files have the following format:\n> \n> - A header appears at the beginning and consists of the following:\n> @@ -26,7 +32,7 @@ Git pack format\n> \n> (deltified representation)\n> n-byte type and length (3-bit type, (n-1)*7+4-bit length)\n> - 20-byte base object name if OBJ_REF_DELTA or a negative relative\n> + base object name if OBJ_REF_DELTA or a negative relative\n> \t offset from the delta object's position in the pack if this\n> \t is an OBJ_OFS_DELTA object\n> compressed delta data\n> @@ -34,7 +40,7 @@ Git pack format\n> Observation: length of each object is encoded in a variable\n> length format and is not constrained to 32-bit or anything.\n> \n> - - The trailer records 20-byte SHA-1 checksum of all of the above.\n> + - The trailer records a pack checksum of all of the above.\n> \n> === Object types\n> \n> @@ -58,8 +64,8 @@ ofs-delta and ref-delta, which is only valid in a pack file.\n> \n> Both ofs-delta and ref-delta store the \"delta\" to be applied to\n> another object (called 'base object') to reconstruct the object. The\n> -difference between them is, ref-delta directly encodes 20-byte base\n> -object name. If the base object is in the same pack, ofs-delta encodes\n> +difference between them is, ref-delta directly encodes base object\n> +name. If the base object is in the same pack, ofs-delta encodes\n> the offset of the base object in the pack instead.\n> \n> The base object could also be deltified if it's in the same pack.\n> @@ -143,14 +149,14 @@ This is the instruction reserved for future expansion.\n> object is stored in the packfile as the offset from the\n> beginning.\n> \n> - 20-byte object name.\n> + one object name of the appropriate size.\n> \n> - The file is concluded with a trailer:\n> \n> - A copy of the 20-byte SHA-1 checksum at the end of\n> - corresponding packfile.\n> + A copy of the pack checksum at the end of the corresponding\n> + packfile.\n> \n> - 20-byte SHA-1-checksum of all of the above.\n> + Index checksum of all of the above.\n> \n> Pack Idx file:\n> \n> @@ -198,7 +204,7 @@ Pack file entry: <+\n> If it is not DELTA, then deflated bytes (the size above\n> \t\tis the size before compression).\n> \tIf it is REF_DELTA, then\n> -\t 20-byte base object name SHA-1 (the size above is the\n> +\t base object name (the size above is the\n> \t\tsize of the delta data that follows).\n> delta data, deflated.\n> \tIf it is OFS_DELTA, then\n> @@ -227,9 +233,9 @@ Pack file entry: <+\n> \n> - A 256-entry fan-out table just like v1.\n> \n> - - A table of sorted 20-byte SHA-1 object names. These are\n> - packed together without offset values to reduce the cache\n> - footprint of the binary search for a specific object name.\n> + - A table of sorted object names. These are packed together\n> + without offset values to reduce the cache footprint of the\n> + binary search for a specific object name.\n> \n> - A table of 4-byte CRC32 values of the packed object data.\n> This is new in v2 so compressed data can be copied directly\n> @@ -248,10 +254,10 @@ Pack file entry: <+\n> \n> - The same trailer as a v1 pack file:\n> \n> - A copy of the 20-byte SHA-1 checksum at the end of\n> + A copy of the pack checksum at the end of\n> corresponding packfile.\n> \n> - 20-byte SHA-1-checksum of all of the above.\n> + Index checksum of all of the above.\n> \n> == multi-pack-index (MIDX) files have the following format:\n> \n> @@ -329,4 +335,4 @@ CHUNK DATA:\n> \n> TRAILER:\n> \n> -\t20-byte SHA1-checksum of the above contents.\n> +\tIndex checksum of the above contents.\n\nI immediately got concerned and looked at the existing MIDX\nformat to see how we deal with hash length and the contents\nmissing in this diff are suitably vague about the hash-length:\n\n \"The OIDs for all objects in the MIDX are stored...\"\n\nThese changes are good.\n\nThanks,\n-Stolee\n"},{"id":"403616","messageId":"da077fb0-14bb-b84f-c526-d759ebc9f5eb@gmail.com","threadId":"54039","inReplyTo":"20200813224901.2652387-1-sandals@crustytoothpaste.net","subject":"Re: [PATCH 0/2] Documentation updates for SHA-256","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-14T02:33:20Z","receivedAt":"2020-08-14T02:33:24Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/13/2020 6:48 PM, brian m. carlson wrote:\n> As was pointed out recently, some of our documentation doesn't properly\n> reflect the new support for SHA-256.\n\nJust to bring up some subtlety in Martin's message [1], there is some\nvalid concern that the existing file formats are not correctly versioned\n\"in a vacuum.\" When they are in a repository that has SHA-256 set as its\nhash algorithm, Git interprets these file formats correctly, but if a\n*.pack file (and its *.idx) happened to be copied into the pack directory\nof a Git repository still in SHA-1 mode, then Git would get confused and\nprobably fail miserably.\n\nIs that really a concern? Maybe, but also Git will never move data like\nthat. The main thing is to focus on compatibility of the .git directory\nas a whole (and the protocol, as we move into inter-operability mode).\n\n[1] https://lore.kernel.org/git/CAN0heSptiJL9d86ZeNPMUaZeTA68juwTyf3K-uWR=K-vt=1Hrg@mail.gmail.com/\n\n> This series updates the pack and\n> index documentation to reflect that these formats can handle SHA-256,\n> and updates the transition plan to reflect what we've implemented and\n> what the next steps are.\n\nThese patches are good to help clarify these formats in the new world.\n\nThanks,\n-Stolee\n"},{"id":"403625","messageId":"xmqqh7t5zve0.fsf@gitster.c.googlers.com","threadId":"54039","inReplyTo":"da077fb0-14bb-b84f-c526-d759ebc9f5eb@gmail.com","subject":"Re: [PATCH 0/2] Documentation updates for SHA-256","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-14T04:47:19Z","receivedAt":"2020-08-14T04:47:25Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Derrick Stolee writes:\n\n> Is that really a concern? Maybe, but also Git will never move data like\n> that.\n\nI would say that we can safely say that this year ;-) as dumb HTTP\nwould be mostly dead.\n\n> The main thing is to focus on compatibility of the .git directory\n> as a whole (and the protocol, as we move into inter-operability mode).\n>\n> [1] https://lore.kernel.org/git/CAN0heSptiJL9d86ZeNPMUaZeTA68juwTyf3K-uWR=K-vt=1Hrg@mail.gmail.com/\n>\n>> This series updates the pack and\n>> index documentation to reflect that these formats can handle SHA-256,\n>> and updates the transition plan to reflect what we've implemented and\n>> what the next steps are.\n>\n> These patches are good to help clarify these formats in the new world.\n\nYup.\n"},{"id":"403645","messageId":"CAN0heSq52VrEaUxxCb91Adw5YByz=H_r60px9UHbTcQHLb1PmA@mail.gmail.com","threadId":"54039","inReplyTo":"20200813224901.2652387-3-sandals@crustytoothpaste.net","subject":"Re: [PATCH 2/2] docs: fix step in transition plan","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T12:21:33Z","receivedAt":"2020-08-14T12:21:55Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 14 Aug 2020 at 00:49, brian m. carlson\n wrote:\n>\n> One of the required steps for the objectFormat extension is to implement\n> the loose object index. However, without support for\n> compatObjectFormat, we don't even know if the loose object index is\n> needed, so it makes sense to move that step to the compatObjectFormat\n> section. Do so.\n\nThis makes sense to me. I know I thought out loud before that maybe\nthere's some intention here and more specifically, maybe we want to\n*know* that this loose-object-idx is always there. But we'd still need\nto tiptoe around it in a SHA-1 repo, so even if we'd know that all\nproper SHA-256 repos have been generating such a file since day 1, that\nprobably wouldn't help us much in terms of implementation/fallback\nstrategies and whatnot.\n\nMartin\n"},{"id":"403646","messageId":"cover.1597406877.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"20200813224901.2652387-1-sandals@crustytoothpaste.net","subject":"[PATCH 0/5] more SHA-256 documentation","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T12:21:41Z","receivedAt":"2020-08-14T12:22:04Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Hi brian,\n\nOn Fri, 14 Aug 2020 at 00:49, brian m. carlson wrote:\n>\n> As was pointed out recently, some of our documentation doesn't properly\n> reflect the new support for SHA-256. This series updates the pack and\n> index documentation to reflect that these formats can handle SHA-256,\n> and updates the transition plan to reflect what we've implemented and\n> what the next steps are.\n\nThanks, this looks great. Now we're making clear what it is we intend to\nbe doing.\n\nWhat about these additional patches on top? These are based on my\nunderstanding, but hopefully they're not *too* wrong. I'm a bit hesitant\nabout the final patch and it would be interesting to know what you\nthink.\n\nMartin Ågren (5):\n http-protocol.txt: document SHA-256 \"want\"/\"have\" format\n index-format.txt: document SHA-256 index format\n protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256\n shallow.txt: document SHA-256 shallow format\n commit-graph-format.txt: fix \"Hash Version\" description\n\n .../technical/commit-graph-format.txt | 4 +--\n Documentation/technical/http-protocol.txt | 5 ++--\n Documentation/technical/index-format.txt | 27 ++++++++++---------\n .../technical/protocol-capabilities.txt | 11 +++++---\n Documentation/technical/shallow.txt | 2 +-\n 5 files changed, 28 insertions(+), 21 deletions(-)\n\n-- \n2.28.0.277.g9b3c35fffd\n\n"},{"id":"403647","messageId":"5590a68c5ba7081cd7e64c708b5c25db23f5e95b.1597406877.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597406877.git.martin.agren@gmail.com","subject":"[PATCH 1/5] http-protocol.txt: document SHA-256 \"want\"/\"have\" format","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T12:21:42Z","receivedAt":"2020-08-14T12:22:06Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Document that in SHA-1 repositories, we use SHA-1 for \"want\"s and\n\"have\"s, and in SHA-256 repositories, we use SHA-256.\n\nSigned-off-by: Martin Ågren \n---\n Documentation/technical/http-protocol.txt | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt\nindex 51a79e63de..507f28f9b3 100644\n--- a/Documentation/technical/http-protocol.txt\n+++ b/Documentation/technical/http-protocol.txt\n@@ -401,8 +401,9 @@ at all in the request stream:\n The stream is terminated by a pkt-line flush (`0000`).\n \n A single \"want\" or \"have\" command MUST have one hex formatted\n-SHA-1 as its value. Multiple SHA-1s MUST be sent by sending\n-multiple commands.\n+object name as its value. Multiple object names MUST be sent by sending\n+multiple commands. (An object name is a SHA-1 hash in a SHA-1 repo\n+and a SHA-256 hash in a SHA-256 repo.)\n \n The `have` list is created by popping the first 32 commits\n from `c_pending`. Less can be supplied if `c_pending` empties.\n-- \n2.28.0.277.g9b3c35fffd\n\n"},{"id":"403648","messageId":"8e498a3b400a2c8390e78de08b83d41e354e7564.1597406877.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597406877.git.martin.agren@gmail.com","subject":"[PATCH 4/5] shallow.txt: document SHA-256 shallow format","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T12:21:45Z","receivedAt":"2020-08-14T12:22:14Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Similar to recent commits, document that in SHA-1 repositories, we use\nSHA-1 for these purposes, and in SHA-256 repositories, we use SHA-256.\n\nSigned-off-by: Martin Ågren \n---\n Documentation/technical/shallow.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/shallow.txt b/Documentation/technical/shallow.txt\nindex 01dedfe9ff..f3738baa0f 100644\n--- a/Documentation/technical/shallow.txt\n+++ b/Documentation/technical/shallow.txt\n@@ -13,7 +13,7 @@ pretend as if they are root commits (e.g. \"git log\" traversal\n stops after showing them; \"git fsck\" does not complain saying\n the commits listed on their \"parent\" lines do not exist).\n \n-Each line contains exactly one SHA-1. When read, a commit_graft\n+Each line contains exactly one object name. When read, a commit_graft\n will be constructed, which has nr_parent < 0 to make it easier\n to discern from user provided grafts.\n \n-- \n2.28.0.277.g9b3c35fffd\n\n"},{"id":"403649","messageId":"13798dfb121c9631e9d811f5f74f9a78c1fcfdf9.1597406877.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597406877.git.martin.agren@gmail.com","subject":"[PATCH 3/5] protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T12:21:44Z","receivedAt":"2020-08-14T12:22:15Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Two of our extensions contain \"sha1\" in their names, but that's\nhistorical. The \"want\"s will take object names that are not necessarily\nSHA-1s. Make this clear, but also make it clear how there's still just\none correct hash algo: These extensions don't somehow make the \"want\"s\ntake object names derived using *any* hash algorithm.\n\nSigned-off-by: Martin Ågren \n---\n Documentation/technical/protocol-capabilities.txt | 11 +++++++----\n 1 file changed, 7 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 36ccd14f97..47f1b30090 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -324,15 +324,18 @@ allow-tip-sha1-in-want\n ----------------------\n \n If the upload-pack server advertises this capability, fetch-pack may\n-send \"want\" lines with SHA-1s that exist at the server but are not\n-advertised by upload-pack.\n+send \"want\" lines with object names that exist at the server but are not\n+advertised by upload-pack. (Note that the name of the capability\n+contains \"sha1\", but that it's more general than that: in SHA-1\n+repositories, the \"want\" lines provide SHA-1 values, but in SHA-256\n+repositories, they provide SHA-256 values.)\n \n allow-reachable-sha1-in-want\n ----------------------------\n \n If the upload-pack server advertises this capability, fetch-pack may\n-send \"want\" lines with SHA-1s that exist at the server but are not\n-advertised by upload-pack.\n+send \"want\" lines with object names that exist at the server but are not\n+advertised by upload-pack. (Same remark about \"sha1\" as above.)\n \n push-cert=\n -----------------\n-- \n2.28.0.277.g9b3c35fffd\n\n"},{"id":"403650","messageId":"bbcd97db8a2c94343950bb0cf8cbd5c9c01b4577.1597406877.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597406877.git.martin.agren@gmail.com","subject":"[PATCH 5/5] commit-graph-format.txt: fix \"Hash Version\" description","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T12:21:46Z","receivedAt":"2020-08-14T12:22:20Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"We say that value 1 means \"SHA-1\", but in fact, it means \"whatever\nthe_hash_algo is\", see commit c166599862 (\"commit-graph: convert to\nusing the_hash_algo\", 2018-11-14).\n\nSigned-off-by: Martin Ågren \n---\n If we want to be more fine-grained in the future, we'll need to say,\n e.g., \"2 means SHA-1, 3 means SHA-256\" or, perhaps preferrably, bump the\n version number.\n\n I wonder: Should we instead say \"1 means SHA-1, 2 means SHA-256\"? It\n could be implemented as \"easily\" as \"if (value_from_header !=\n value_from_the_hash_algo) die(...);\" for now. Might that pay off in the\n long run?\n\n This relates to Stolee's \"in a vacuum\" comment [1] ... so maybe we're\n fine.\n\n [1] https://lore.kernel.org/git/da077fb0-14bb-b84f-c526-d759ebc9f5eb@gmail.com/\n\n Documentation/technical/commit-graph-format.txt | 4 ++--\n 1 file changed, 2 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\nindex 440541045d..3535426d32 100644\n--- a/Documentation/technical/commit-graph-format.txt\n+++ b/Documentation/technical/commit-graph-format.txt\n@@ -42,8 +42,8 @@ HEADER:\n 1-byte version number:\n Currently, the only valid version is 1.\n \n- 1-byte Hash Version (1 = SHA-1)\n- We infer the hash length (H) from this value.\n+ 1-byte Hash Version (1 = SHA-1 in SHA-1 repo, SHA-256 in SHA-256 repo)\n+ We infer the hash length (H) from the hash algo derived from this value.\n \n 1-byte number (C) of \"chunks\"\n \n-- \n2.28.0.277.g9b3c35fffd\n\n"},{"id":"403651","messageId":"e811455d55cdb222a85d880f3cf3d5e28a8d4c91.1597406877.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597406877.git.martin.agren@gmail.com","subject":"[PATCH 2/5] index-format.txt: document SHA-256 index format","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T12:21:43Z","receivedAt":"2020-08-14T12:22:20Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Similar to a recent commit, document that in SHA-1 repositories, we use\nSHA-1 and in SHA-256 repositories, we use SHA-256, then replace all\nother uses of \"SHA-1\" with something more neutral.\n\nSigned-off-by: Martin Ågren \n---\n Documentation/technical/index-format.txt | 27 +++++++++++++-----------\n 1 file changed, 15 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex faa25c5c52..827ece2ed1 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -3,8 +3,11 @@ Git index format\n \n == The Git index file has the following format\n \n- All binary numbers are in network byte order. Version 2 is described\n- here unless stated otherwise.\n+ All binary numbers are in network byte order.\n+ In a repository using the traditional SHA-1, checksums and object IDs\n+ (object names) mentioned below are all computed using SHA-1. Similarly,\n+ in SHA-256 repositories, these values are computed using SHA-256.\n+ Version 2 is described here unless stated otherwise.\n \n - A 12-byte header consisting of\n \n@@ -32,7 +35,7 @@ Git index format\n \n Extension data\n \n- - 160-bit SHA-1 over the content of the index file before this\n+ - 160-bit hash checksum over the content of the index file before this\n checksum.\n \n == Index entry\n@@ -80,7 +83,7 @@ Git index format\n 32-bit file size\n This is the on-disk size from stat(2), truncated to 32-bit.\n \n- 160-bit SHA-1 for the represented object\n+ 160-bit object name for the represented object\n \n A 16-bit 'flags' field split into (high to low bits)\n \n@@ -211,8 +214,8 @@ Git index format\n \n The extension consists of:\n \n- - 160-bit SHA-1 of the shared index file. The shared index file path\n- is $GIT_DIR/sharedindex.. If all 160 bits are zero, the\n+ - Hash of the shared index file. The shared index file path\n+ is $GIT_DIR/sharedindex.. If all bits are zero, the\n index does not require a shared index file.\n \n - An ewah-encoded delete bitmap, each bit represents an entry in the\n@@ -253,10 +256,10 @@ Git index format\n \n - 32-bit dir_flags (see struct dir_struct)\n \n- - 160-bit SHA-1 of $GIT_DIR/info/exclude. Null SHA-1 means the file\n+ - Hash of $GIT_DIR/info/exclude. A null hash means the file\n does not exist.\n \n- - 160-bit SHA-1 of core.excludesfile. Null SHA-1 means the file does\n+ - Hash of core.excludesfile. A null hash means the file does\n not exist.\n \n - NUL-terminated string of per-dir exclude file name. This usually\n@@ -285,13 +288,13 @@ The remaining data of each directory block is grouped by type:\n - An ewah bitmap, the n-th bit records \"check-only\" bit of\n read_directory_recursive() for the n-th directory.\n \n- - An ewah bitmap, the n-th bit indicates whether SHA-1 and stat data\n+ - An ewah bitmap, the n-th bit indicates whether hash and stat data\n is valid for the n-th directory and exists in the next data.\n \n - An array of stat data. The n-th data corresponds with the n-th\n \"one\" bit in the previous ewah bitmap.\n \n- - An array of SHA-1. The n-th SHA-1 corresponds with the n-th \"one\" bit\n+ - An array of hashes. The n-th hash corresponds with the n-th \"one\" bit\n in the previous ewah bitmap.\n \n - One NUL.\n@@ -330,12 +333,12 @@ The remaining data of each directory block is grouped by type:\n \n - 32-bit offset to the end of the index entries\n \n- - 160-bit SHA-1 over the extension types and their sizes (but not\n+ - Hash over the extension types and their sizes (but not\n \ttheir contents). E.g. if we have \"TREE\" extension that is N-bytes\n \tlong, \"REUC\" extension that is M-bytes long, followed by \"EOIE\",\n \tthen the hash would be:\n \n-\tSHA-1(\"TREE\" + +\n+\tHash(\"TREE\" + +\n \t\t\"REUC\" + )\n \n == Index Entry Offset Table\n-- \n2.28.0.277.g9b3c35fffd\n\n"},{"id":"403654","messageId":"e9963ef1-cecd-cdf8-a5cf-2be884d8ced6@gmail.com","threadId":"54039","inReplyTo":"e811455d55cdb222a85d880f3cf3d5e28a8d4c91.1597406877.git.martin.agren@gmail.com","subject":"Re: [PATCH 2/5] index-format.txt: document SHA-256 index format","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-14T12:28:13Z","receivedAt":"2020-08-14T12:28:26Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/14/2020 8:21 AM, Martin Ågren wrote:\n> Similar to a recent commit, document that in SHA-1 repositories, we use\n> SHA-1 and in SHA-256 repositories, we use SHA-256, then replace all\n> other uses of \"SHA-1\" with something more neutral.\n> \n> Signed-off-by: Martin Ågren \n> ---\n> Documentation/technical/index-format.txt | 27 +++++++++++++-----------\n> 1 file changed, 15 insertions(+), 12 deletions(-)\n> \n> diff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\n> index faa25c5c52..827ece2ed1 100644\n> --- a/Documentation/technical/index-format.txt\n> +++ b/Documentation/technical/index-format.txt\n> @@ -3,8 +3,11 @@ Git index format\n> \n> == The Git index file has the following format\n> \n> - All binary numbers are in network byte order. Version 2 is described\n> - here unless stated otherwise.\n> + All binary numbers are in network byte order.\n> + In a repository using the traditional SHA-1, checksums and object IDs\n> + (object names) mentioned below are all computed using SHA-1. Similarly,\n> + in SHA-256 repositories, these values are computed using SHA-256.\n> + Version 2 is described here unless stated otherwise.\n> \n> - A 12-byte header consisting of\n> \n> @@ -32,7 +35,7 @@ Git index format\n> \n> Extension data\n> \n> - - 160-bit SHA-1 over the content of the index file before this\n> + - 160-bit hash checksum over the content of the index file before this\n> checksum.\n\nIf this hash is flexible, then \"160-bit\" is not correct anymore, right?\n\n> == Index entry\n> @@ -80,7 +83,7 @@ Git index format\n> 32-bit file size\n> This is the on-disk size from stat(2), truncated to 32-bit.\n> \n> - 160-bit SHA-1 for the represented object\n> + 160-bit object name for the represented object\n\nSame here. The later instances of \"160-bit\" were dropped.\n\n> A 16-bit 'flags' field split into (high to low bits)\n> \n> @@ -211,8 +214,8 @@ Git index format\n> \n> The extension consists of:\n> \n> - - 160-bit SHA-1 of the shared index file. The shared index file path\n> - is $GIT_DIR/sharedindex.. If all 160 bits are zero, the\n> + - Hash of the shared index file. The shared index file path\n> + is $GIT_DIR/sharedindex.. If all bits are zero, the\n> index does not require a shared index file.\n> \n> - An ewah-encoded delete bitmap, each bit represents an entry in the\n> @@ -253,10 +256,10 @@ Git index format\n> \n> - 32-bit dir_flags (see struct dir_struct)\n> \n> - - 160-bit SHA-1 of $GIT_DIR/info/exclude. Null SHA-1 means the file\n> + - Hash of $GIT_DIR/info/exclude. A null hash means the file\n> does not exist.\n> \n> - - 160-bit SHA-1 of core.excludesfile. Null SHA-1 means the file does\n> + - Hash of core.excludesfile. A null hash means the file does\n> not exist.\n> \n> - NUL-terminated string of per-dir exclude file name. This usually\n> @@ -285,13 +288,13 @@ The remaining data of each directory block is grouped by type:\n> - An ewah bitmap, the n-th bit records \"check-only\" bit of\n> read_directory_recursive() for the n-th directory.\n> \n> - - An ewah bitmap, the n-th bit indicates whether SHA-1 and stat data\n> + - An ewah bitmap, the n-th bit indicates whether hash and stat data\n> is valid for the n-th directory and exists in the next data.\n> \n> - An array of stat data. The n-th data corresponds with the n-th\n> \"one\" bit in the previous ewah bitmap.\n> \n> - - An array of SHA-1. The n-th SHA-1 corresponds with the n-th \"one\" bit\n> + - An array of hashes. The n-th hash corresponds with the n-th \"one\" bit\n> in the previous ewah bitmap.\n> \n> - One NUL.\n> @@ -330,12 +333,12 @@ The remaining data of each directory block is grouped by type:\n> \n> - 32-bit offset to the end of the index entries\n> \n> - - 160-bit SHA-1 over the extension types and their sizes (but not\n> + - Hash over the extension types and their sizes (but not\n> \ttheir contents). E.g. if we have \"TREE\" extension that is N-bytes\n> \tlong, \"REUC\" extension that is M-bytes long, followed by \"EOIE\",\n> \tthen the hash would be:\n> \n> -\tSHA-1(\"TREE\" + +\n> +\tHash(\"TREE\" + +\n> \t\t\"REUC\" + )\n> \n> == Index Entry Offset Table\n> \n\nThanks,\n-Stolee\n\n\n"},{"id":"403655","messageId":"73e1ce78-a461-1204-99d5-a588f8e65fc9@gmail.com","threadId":"54039","inReplyTo":"13798dfb121c9631e9d811f5f74f9a78c1fcfdf9.1597406877.git.martin.agren@gmail.com","subject":"Re: [PATCH 3/5] protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-14T12:31:05Z","receivedAt":"2020-08-14T12:31:09Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/14/2020 8:21 AM, Martin Ågren wrote:\n> Two of our extensions contain \"sha1\" in their names, but that's\n> historical. The \"want\"s will take object names that are not necessarily\n> SHA-1s. Make this clear, but also make it clear how there's still just\n> one correct hash algo: These extensions don't somehow make the \"want\"s\n> take object names derived using *any* hash algorithm.\n> \n> Signed-off-by: Martin Ågren \n> ---\n> Documentation/technical/protocol-capabilities.txt | 11 +++++++----\n> 1 file changed, 7 insertions(+), 4 deletions(-)\n> \n> diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\n> index 36ccd14f97..47f1b30090 100644\n> --- a/Documentation/technical/protocol-capabilities.txt\n> +++ b/Documentation/technical/protocol-capabilities.txt\n> @@ -324,15 +324,18 @@ allow-tip-sha1-in-want\n> ----------------------\n> \n> If the upload-pack server advertises this capability, fetch-pack may\n> -send \"want\" lines with SHA-1s that exist at the server but are not\n> -advertised by upload-pack.\n> +send \"want\" lines with object names that exist at the server but are not\n> +advertised by upload-pack. (Note that the name of the capability\n> +contains \"sha1\", but that it's more general than that: in SHA-1\n> +repositories, the \"want\" lines provide SHA-1 values, but in SHA-256\n> +repositories, they provide SHA-256 values.)\n> \n> allow-reachable-sha1-in-want\n> ----------------------------\n> \n> If the upload-pack server advertises this capability, fetch-pack may\n> -send \"want\" lines with SHA-1s that exist at the server but are not\n> -advertised by upload-pack.\n> +send \"want\" lines with object names that exist at the server but are not\n> +advertised by upload-pack. (Same remark about \"sha1\" as above.)\n\nThis \"as above\" is brittle to future changes. I think it\ncould be improved with\n\n\t(As in \"allow-tip-sha1-in-want\", the \"sha1\" in this capability\n\trefers to object names, not the hash algorithm chosen for the\n\trepository.)\n\nOr, just repeat the same note again.\n\nThanks,\n-Stolee\n"},{"id":"403656","messageId":"e216492c-d11a-6733-db64-720d027d3d42@gmail.com","threadId":"54039","inReplyTo":"bbcd97db8a2c94343950bb0cf8cbd5c9c01b4577.1597406877.git.martin.agren@gmail.com","subject":"Re: [PATCH 5/5] commit-graph-format.txt: fix \"Hash Version\" description","fromName":"Derrick Stolee","fromEmail":"stolee@gmail.com","sentAt":"2020-08-14T12:37:31Z","receivedAt":"2020-08-14T12:37:36Z","isPatch":true,"sender":{"key":"stolee@gmail.com","avatar":"https://avatars.githubusercontent.com/u/570044?v=4"},"body":"On 8/14/2020 8:21 AM, Martin Ågren wrote:\n> We say that value 1 means \"SHA-1\", but in fact, it means \"whatever\n> the_hash_algo is\", see commit c166599862 (\"commit-graph: convert to\n> using the_hash_algo\", 2018-11-14).\n> \n> Signed-off-by: Martin Ågren \n> ---\n> If we want to be more fine-grained in the future, we'll need to say,\n> e.g., \"2 means SHA-1, 3 means SHA-256\" or, perhaps preferrably, bump the\n> version number.\n> \n> I wonder: Should we instead say \"1 means SHA-1, 2 means SHA-256\"? It\n> could be implemented as \"easily\" as \"if (value_from_header !=\n> value_from_the_hash_algo) die(...);\" for now. Might that pay off in the\n> long run?\n> \n> This relates to Stolee's \"in a vacuum\" comment [1] ... so maybe we're\n> fine.\n\nI think that was the intention of the byte, but that is not what ended\nup happening. If we want that to be the case, then we should do that\nwork as part of the 2.29 cycle before we release with the ability to\ncreate SHA-256 repos (which will lock the commit-graph format for these\nrepos).\n\n(By \"we\" I mean that I would try to do this work in a way that minimizes\nconflicts with the current commit-graph work in flight [1] [2].)\n\n[1] https://lore.kernel.org/git/pull.676.v2.git.1596941624.gitgitgadget@gmail.com/\n\n[2] https://lore.kernel.org/git/cover.1597178914.git.me@ttaylorr.com/\n\n> [1] https://lore.kernel.org/git/da077fb0-14bb-b84f-c526-d759ebc9f5eb@gmail.com/\n> \n> Documentation/technical/commit-graph-format.txt | 4 ++--\n> 1 file changed, 2 insertions(+), 2 deletions(-)\n> \n> diff --git a/Documentation/technical/commit-graph-format.txt b/Documentation/technical/commit-graph-format.txt\n> index 440541045d..3535426d32 100644\n> --- a/Documentation/technical/commit-graph-format.txt\n> +++ b/Documentation/technical/commit-graph-format.txt\n> @@ -42,8 +42,8 @@ HEADER:\n> 1-byte version number:\n> Currently, the only valid version is 1.\n> \n> - 1-byte Hash Version (1 = SHA-1)\n> - We infer the hash length (H) from this value.\n> + 1-byte Hash Version (1 = SHA-1 in SHA-1 repo, SHA-256 in SHA-256 repo)\n> + We infer the hash length (H) from the hash algo derived from this value.\n\nIf we are _not_ changing the format to have a meaningful value in\nthis byte, then this documentation should be updated to state that\nthis byte must always have value 1, as it does not provide any\ninformation.\n\nThanks,\n-Stolee\n"},{"id":"403658","messageId":"CAN0heSrOyLJ-fpjJMY9tYXjnz5s8tAdcGMnef8WfJBUgAU1f_A@mail.gmail.com","threadId":"54039","inReplyTo":"e9963ef1-cecd-cdf8-a5cf-2be884d8ced6@gmail.com","subject":"Re: [PATCH 2/5] index-format.txt: document SHA-256 index format","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T14:05:11Z","receivedAt":"2020-08-14T14:05:43Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 14 Aug 2020 at 14:28, Derrick Stolee wrote:\n>\n> On 8/14/2020 8:21 AM, Martin Ågren wrote:\n> > - - 160-bit SHA-1 over the content of the index file before this\n> > + - 160-bit hash checksum over the content of the index file before this\n> > checksum.\n>\n> If this hash is flexible, then \"160-bit\" is not correct anymore, right?\n>\n> > - 160-bit SHA-1 for the represented object\n> > + 160-bit object name for the represented object\n>\n> Same here. The later instances of \"160-bit\" were dropped.\n\nThanks for pointing out these errors.\n\n\nMartin\n"},{"id":"403659","messageId":"CAN0heSraHteU1fG7Eu_S7aE6r60vuVUFC=DnxdZNgpvHk4G0MQ@mail.gmail.com","threadId":"54039","inReplyTo":"73e1ce78-a461-1204-99d5-a588f8e65fc9@gmail.com","subject":"Re: [PATCH 3/5] protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T14:05:44Z","receivedAt":"2020-08-14T14:06:25Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 14 Aug 2020 at 14:31, Derrick Stolee wrote:\n>\n> On 8/14/2020 8:21 AM, Martin Ågren wrote:\n> >\n> > If the upload-pack server advertises this capability, fetch-pack may\n> > -send \"want\" lines with SHA-1s that exist at the server but are not\n> > -advertised by upload-pack.\n> > +send \"want\" lines with object names that exist at the server but are not\n> > +advertised by upload-pack. (Note that the name of the capability\n> > +contains \"sha1\", but that it's more general than that: in SHA-1\n> > +repositories, the \"want\" lines provide SHA-1 values, but in SHA-256\n> > +repositories, they provide SHA-256 values.)\n> >\n> > allow-reachable-sha1-in-want\n> > ----------------------------\n> >\n> > If the upload-pack server advertises this capability, fetch-pack may\n> > -send \"want\" lines with SHA-1s that exist at the server but are not\n> > -advertised by upload-pack.\n> > +send \"want\" lines with object names that exist at the server but are not\n> > +advertised by upload-pack. (Same remark about \"sha1\" as above.)\n>\n> This \"as above\" is brittle to future changes.\n\nFair enough. :-) I actually thought this might be *less* brittle, since\nwe wouldn't need to do any additional changes twice.\n\n> I think it\n> could be improved with\n>\n> (As in \"allow-tip-sha1-in-want\", the \"sha1\" in this capability\n> refers to object names, not the hash algorithm chosen for the\n> repository.)\n>\n> Or, just repeat the same note again.\n\nThese two paragraphs are identical before this patch, so it might make\nsense not to change that property. Thanks.\n\n\nMartin\n"},{"id":"403660","messageId":"CAN0heSp024=Kyy7gdQ2VSetk_5iVhj_qdT8CMVPcry_AwWrhHQ@mail.gmail.com","threadId":"54039","inReplyTo":"e216492c-d11a-6733-db64-720d027d3d42@gmail.com","subject":"Re: [PATCH 5/5] commit-graph-format.txt: fix \"Hash Version\" description","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T14:10:35Z","receivedAt":"2020-08-14T14:10:51Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 14 Aug 2020 at 14:37, Derrick Stolee wrote:\n>\n> On 8/14/2020 8:21 AM, Martin Ågren wrote:\n> > We say that value 1 means \"SHA-1\", but in fact, it means \"whatever\n> > the_hash_algo is\", see commit c166599862 (\"commit-graph: convert to\n> > using the_hash_algo\", 2018-11-14).\n> >\n> > Signed-off-by: Martin Ågren \n> > ---\n> > If we want to be more fine-grained in the future, we'll need to say,\n> > e.g., \"2 means SHA-1, 3 means SHA-256\" or, perhaps preferrably, bump the\n> > version number.\n> >\n> > I wonder: Should we instead say \"1 means SHA-1, 2 means SHA-256\"? It\n> > could be implemented as \"easily\" as \"if (value_from_header !=\n> > value_from_the_hash_algo) die(...);\" for now. Might that pay off in the\n> > long run?\n> >\n> > This relates to Stolee's \"in a vacuum\" comment [1] ... so maybe we're\n> > fine.\n>\n> I think that was the intention of the byte, but that is not what ended\n> up happening.\n\nWhen I wrote this patch, I did go with \"fix \" rather than\n\"document SHA-256\". For the other patches, it's sort of obvious how\nthose formats are so old that it's no wonder they assumed SHA-1. But\nhere, we did go to some trouble to try and future-proof things and\nalready had NewHash in mind. So that calls for \"fix \". But I'm\nmore and more starting to think that it's the implementation that should\nbe fixed and that the docs should just be extended to add \"2 means\nSHA-256\".\n\n> If we want that to be the case, then we should do that\n> work as part of the 2.29 cycle before we release with the ability to\n> create SHA-256 repos (which will lock the commit-graph format for these\n> repos).\n\nPart of my reasoning behind [3] was that in exactly a situation like this,\nwe'd be able to say\n\n With extensions.objectFormat=sha256, Git 2.29-2.30 will barf at the\n commit-graph files that Git 2.31+ generate, and the other way around.\n Users will be able to remove \"old\" files and regenerate them, and\n shouldn't use a mixed environment.\n\nand know that those users knew this might happen.\n\nBut certainly, if we can avoid it altogether by changing behavior\nalready in 2.29, that would be better.\n\n[3] https://lore.kernel.org/git/20200806202358.2265705-1-martin.agren@gmail.com/\n\n> (By \"we\" I mean that I would try to do this work in a way that minimizes\n> conflicts with the current commit-graph work in flight [1] [2].)\n\nNone of those seems to touch `oid_version()`, so if we can just tweak\nthat function to return 1 or 2 instead of always 1, I guess that's one\nway.\n\n> [1] https://lore.kernel.org/git/pull.676.v2.git.1596941624.gitgitgadget@gmail.com/\n>\n> [2] https://lore.kernel.org/git/cover.1597178914.git.me@ttaylorr.com/\n>\n> >\n> > - 1-byte Hash Version (1 = SHA-1)\n> > - We infer the hash length (H) from this value.\n> > + 1-byte Hash Version (1 = SHA-1 in SHA-1 repo, SHA-256 in SHA-256 repo)\n> > + We infer the hash length (H) from the hash algo derived from this value.\n>\n> If we are _not_ changing the format to have a meaningful value in\n> this byte, then this documentation should be updated to state that\n> this byte must always have value 1, as it does not provide any\n> information.\n\nWe could still go\n\n 1 means whatever extensions.objectFormat says,\n 2 means SHA-1,\n 3 means SHA-256,\n ...\n\nBut maybe that would just be crazy.\n\nThanks for all your comments\nMartin\n"},{"id":"403690","messageId":"xmqqk0y1xhl0.fsf@gitster.c.googlers.com","threadId":"54039","inReplyTo":"5590a68c5ba7081cd7e64c708b5c25db23f5e95b.1597406877.git.martin.agren@gmail.com","subject":"Re: [PATCH 1/5] http-protocol.txt: document SHA-256 \"want\"/\"have\" format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-14T17:28:27Z","receivedAt":"2020-08-14T17:29:39Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Ågren writes:\n\n> Document that in SHA-1 repositories, we use SHA-1 for \"want\"s and\n> \"have\"s, and in SHA-256 repositories, we use SHA-256.\n\nEhh, doesn't this directly contradict the transition plan of \"on the\nwire everything will use SHA-1 version for now?\"\n\n\n\n> Signed-off-by: Martin Ågren \n> ---\n> Documentation/technical/http-protocol.txt | 5 +++--\n> 1 file changed, 3 insertions(+), 2 deletions(-)\n>\n> diff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt\n> index 51a79e63de..507f28f9b3 100644\n> --- a/Documentation/technical/http-protocol.txt\n> +++ b/Documentation/technical/http-protocol.txt\n> @@ -401,8 +401,9 @@ at all in the request stream:\n> The stream is terminated by a pkt-line flush (`0000`).\n> \n> A single \"want\" or \"have\" command MUST have one hex formatted\n> -SHA-1 as its value. Multiple SHA-1s MUST be sent by sending\n> -multiple commands.\n> +object name as its value. Multiple object names MUST be sent by sending\n> +multiple commands. (An object name is a SHA-1 hash in a SHA-1 repo\n> +and a SHA-256 hash in a SHA-256 repo.)\n> \n> The `have` list is created by popping the first 32 commits\n> from `c_pending`. Less can be supplied if `c_pending` empties.\n"},{"id":"403691","messageId":"xmqqft8pxhd9.fsf@gitster.c.googlers.com","threadId":"54039","inReplyTo":"13798dfb121c9631e9d811f5f74f9a78c1fcfdf9.1597406877.git.martin.agren@gmail.com","subject":"Re: [PATCH 3/5] protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-14T17:33:06Z","receivedAt":"2020-08-14T17:33:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Ågren writes:\n\n> Two of our extensions contain \"sha1\" in their names, but that's\n> historical. The \"want\"s will take object names that are not necessarily\n> SHA-1s. Make this clear, but also make it clear how there's still just\n> one correct hash algo: These extensions don't somehow make the \"want\"s\n> take object names derived using *any* hash algorithm.\n>\n> Signed-off-by: Martin Ågren \n> ---\n> Documentation/technical/protocol-capabilities.txt | 11 +++++++----\n> 1 file changed, 7 insertions(+), 4 deletions(-)\n>\n> diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\n> index 36ccd14f97..47f1b30090 100644\n> --- a/Documentation/technical/protocol-capabilities.txt\n> +++ b/Documentation/technical/protocol-capabilities.txt\n> @@ -324,15 +324,18 @@ allow-tip-sha1-in-want\n> ----------------------\n> \n> If the upload-pack server advertises this capability, fetch-pack may\n> -send \"want\" lines with SHA-1s that exist at the server but are not\n> -advertised by upload-pack.\n> +send \"want\" lines with object names that exist at the server but are not\n> +advertised by upload-pack. (Note that the name of the capability\n> +contains \"sha1\", but that it's more general than that: in SHA-1\n> +repositories, the \"want\" lines provide SHA-1 values, but in SHA-256\n> +repositories, they provide SHA-256 values.)\n\nI think we should have either a new sha256 capability or a more\ngeneric hash-algo capability whose value can be set to sha256.\nNeither the connection initiators or the acceptors should talk\nin sha256 until both ends agreed to do so. \n\nI do not think of any other way to make sure hosting sites to serve\nprojects that migrate at different pace. Per project, you might be\nable to have a flag day. You cannot have a flag day that spans the\nworld.\n\n"},{"id":"403710","messageId":"20200814202031.GM8085@camp.crustytoothpaste.net","threadId":"54039","inReplyTo":"xmqqh7t5zve0.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 0/2] Documentation updates for SHA-256","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-08-14T20:20:31Z","receivedAt":"2020-08-14T20:21:07Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-08-14 at 04:47:19, Junio C Hamano wrote:\n> Derrick Stolee writes:\n> \n> > Is that really a concern? Maybe, but also Git will never move data like\n> > that.\n> \n> I would say that we can safely say that this year ;-) as dumb HTTP\n> would be mostly dead.\n\nWe do fetch the refs first for dumb HTTP so last I checked, we correctly\ndetected this case and failed. I'd personally be happy to let the\nDAV-based protocol die, but there are folks who like it.\n-- \nbrian m. carlson: Houston, Texas, US\n"},{"id":"403714","messageId":"20200814202347.GN8085@camp.crustytoothpaste.net","threadId":"54039","inReplyTo":"xmqqk0y1xhl0.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 1/5] http-protocol.txt: document SHA-256 \"want\"/\"have\" format","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-08-14T20:23:47Z","receivedAt":"2020-08-14T20:24:24Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-08-14 at 17:28:27, Junio C Hamano wrote:\n> Martin Ågren writes:\n> \n> > Document that in SHA-1 repositories, we use SHA-1 for \"want\"s and\n> > \"have\"s, and in SHA-256 repositories, we use SHA-256.\n> \n> Ehh, doesn't this directly contradict the transition plan of \"on the\n> wire everything will use SHA-1 version for now?\"\n\nSHA-256 repositories interoperate currently using SHA-256 object IDs.\nIt was originally intended that we wouldn't update the protocol, but\nthat leads to much of the testsuite failing since it's impossible to\nmove objects from one place to another.\n\nIf we wanted to be more pedantically correct and optimize for the\nfuture, we could say that the values use the format negotiated by the\n\"object-format\" protocol extension and SHA-1 otherwise.\n-- \nbrian m. carlson: Houston, Texas, US\n"},{"id":"403715","messageId":"xmqqtux5ug9t.fsf@gitster.c.googlers.com","threadId":"54039","inReplyTo":"20200814202031.GM8085@camp.crustytoothpaste.net","subject":"Re: [PATCH 0/2] Documentation updates for SHA-256","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-14T20:25:02Z","receivedAt":"2020-08-14T20:25:10Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> On 2020-08-14 at 04:47:19, Junio C Hamano wrote:\n>> Derrick Stolee writes:\n>> \n>> > Is that really a concern? Maybe, but also Git will never move data like\n>> > that.\n>> \n>> I would say that we can safely say that this year ;-) as dumb HTTP\n>> would be mostly dead.\n>\n> We do fetch the refs first for dumb HTTP so last I checked, we correctly\n> detected this case and failed. I'd personally be happy to let the\n> DAV-based protocol die, but there are folks who like it.\n\nI didn't mean DAV. \n\nThe oldest dumb HTTP code grabs all packfiles listed in\nobjects/info/packs and there is nothing to prevent folks from\nrunning the current client to fetch from SHA-256 repository into a\nSHA-1 repository. The resulting packfiles that do not identify with\nthe version number what hash it uses would be very hard to use.\n\n\n"},{"id":"403716","messageId":"20200814202838.GO8085@camp.crustytoothpaste.net","threadId":"54039","inReplyTo":"cover.1597406877.git.martin.agren@gmail.com","subject":"Re: [PATCH 0/5] more SHA-256 documentation","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2020-08-14T20:28:38Z","receivedAt":"2020-08-14T20:28:46Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2020-08-14 at 12:21:41, Martin Ågren wrote:\n> Hi brian,\n> \n> On Fri, 14 Aug 2020 at 00:49, brian m. carlson wrote:\n> >\n> > As was pointed out recently, some of our documentation doesn't properly\n> > reflect the new support for SHA-256. This series updates the pack and\n> > index documentation to reflect that these formats can handle SHA-256,\n> > and updates the transition plan to reflect what we've implemented and\n> > what the next steps are.\n> \n> Thanks, this looks great. Now we're making clear what it is we intend to\n> be doing.\n> \n> What about these additional patches on top? These are based on my\n> understanding, but hopefully they're not *too* wrong. I'm a bit hesitant\n> about the final patch and it would be interesting to know what you\n> think.\n\nI think Stolee has a series so that the final patch isn't necessary, and\nother than the things he mentioned in this thread, I think these would\nbe fine on top.\n-- \nbrian m. carlson: Houston, Texas, US\n"},{"id":"403717","messageId":"CAN0heSqBfvVT18C5nHcAtO1YCCnVC7xUfjrYtS2vH6gw7uX7kQ@mail.gmail.com","threadId":"54039","inReplyTo":"20200814202347.GN8085@camp.crustytoothpaste.net","subject":"Re: [PATCH 1/5] http-protocol.txt: document SHA-256 \"want\"/\"have\" format","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T20:32:51Z","receivedAt":"2020-08-14T20:33:09Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 14 Aug 2020 at 22:23, brian m. carlson\n wrote:\n>\n> On 2020-08-14 at 17:28:27, Junio C Hamano wrote:\n> > Martin Ågren writes:\n> >\n> > > Document that in SHA-1 repositories, we use SHA-1 for \"want\"s and\n> > > \"have\"s, and in SHA-256 repositories, we use SHA-256.\n> >\n> > Ehh, doesn't this directly contradict the transition plan of \"on the\n> > wire everything will use SHA-1 version for now?\"\n\nYes, the transition plan would probably need updating there. I'm just\ntrying to document what we have.\n\n> SHA-256 repositories interoperate currently using SHA-256 object IDs.\n> It was originally intended that we wouldn't update the protocol, but\n> that leads to much of the testsuite failing since it's impossible to\n> move objects from one place to another.\n>\n> If we wanted to be more pedantically correct and optimize for the\n> future, we could say that the values use the format negotiated by the\n> \"object-format\" protocol extension and SHA-1 otherwise.\n\nHmm, I didn't think of that. Would we ever regret that we've painted\nsuch a \"big\" picture and wish to refine it somehow? Compared to\nadmittedly being fairly narrow as I am here, then loosen things later.\nI'll think about it, but I think I could go either way.\n\nMartin\n"},{"id":"403719","messageId":"CAN0heSpv5L67hYSX44LkwQ1pJ+_px_W=FQ4piXKS1anpMi0n2w@mail.gmail.com","threadId":"54039","inReplyTo":"xmqqft8pxhd9.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 3/5] protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-14T20:35:10Z","receivedAt":"2020-08-14T20:35:27Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On Fri, 14 Aug 2020 at 19:33, Junio C Hamano wrote:\n>\n> Martin Ågren writes:\n>\n> > Two of our extensions contain \"sha1\" in their names, but that's\n> > historical. The \"want\"s will take object names that are not necessarily\n> > SHA-1s. Make this clear, but also make it clear how there's still just\n> > one correct hash algo: These extensions don't somehow make the \"want\"s\n> > take object names derived using *any* hash algorithm.\n> >\n> > Signed-off-by: Martin Ågren \n> > ---\n> > Documentation/technical/protocol-capabilities.txt | 11 +++++++----\n> > 1 file changed, 7 insertions(+), 4 deletions(-)\n> >\n> > diff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\n> > index 36ccd14f97..47f1b30090 100644\n> > --- a/Documentation/technical/protocol-capabilities.txt\n> > +++ b/Documentation/technical/protocol-capabilities.txt\n> > @@ -324,15 +324,18 @@ allow-tip-sha1-in-want\n> > ----------------------\n> >\n> > If the upload-pack server advertises this capability, fetch-pack may\n> > -send \"want\" lines with SHA-1s that exist at the server but are not\n> > -advertised by upload-pack.\n> > +send \"want\" lines with object names that exist at the server but are not\n> > +advertised by upload-pack. (Note that the name of the capability\n> > +contains \"sha1\", but that it's more general than that: in SHA-1\n> > +repositories, the \"want\" lines provide SHA-1 values, but in SHA-256\n> > +repositories, they provide SHA-256 values.)\n>\n> I think we should have either a new sha256 capability or a more\n> generic hash-algo capability whose value can be set to sha256.\n> Neither the connection initiators or the acceptors should talk\n> in sha256 until both ends agreed to do so.\n\nI think we should, and I think we do. I haven't dug into the details,\nbut \"object-format\" looks like it's just that.\n\nMaybe instead of SHA-1 and SHA-256, this should talk about \"whatever has\nbeen negotiated through 'object-format', or SHA-1\", similar to brian's\nsuggestion elsewhere.\n\n> I do not think of any other way to make sure hosting sites to serve\n> projects that migrate at different pace. Per project, you might be\n> able to have a flag day. You cannot have a flag day that spans the\n> world.\n\nYeah, that makes sense.\n\n\nMartin\n"},{"id":"403721","messageId":"xmqqo8ndufli.fsf@gitster.c.googlers.com","threadId":"54039","inReplyTo":"20200814202347.GN8085@camp.crustytoothpaste.net","subject":"Re: [PATCH 1/5] http-protocol.txt: document SHA-256 \"want\"/\"have\" format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-14T20:39:37Z","receivedAt":"2020-08-14T20:39:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" writes:\n\n> On 2020-08-14 at 17:28:27, Junio C Hamano wrote:\n>> Martin Ågren writes:\n>> \n>> > Document that in SHA-1 repositories, we use SHA-1 for \"want\"s and\n>> > \"have\"s, and in SHA-256 repositories, we use SHA-256.\n>> \n>> Ehh, doesn't this directly contradict the transition plan of \"on the\n>> wire everything will use SHA-1 version for now?\"\n>\n> SHA-256 repositories interoperate currently using SHA-256 object IDs.\n> It was originally intended that we wouldn't update the protocol, but\n> that leads to much of the testsuite failing since it's impossible to\n> move objects from one place to another.\n>\n> If we wanted to be more pedantically correct and optimize for the\n> future, we could say that the values use the format negotiated by the\n> \"object-format\" protocol extension and SHA-1 otherwise.\n\nYup. I think a reasonable evolution path is\n\n 0) everything on the wire is SHA-1 and no local operation knows\n SHA-256 (i.e. a few releases ago)\n\n 1) local operations are either SHA-1 or SHA-256 but not both.\n On the wire, only protocol for SHA-1 repositories are\n defined, so SHA-256 repositories cannot talk with anybody\n using any official protocol, but a \"borked\" SHA-1 protocol\n that naturally extends the object names width exists and\n SHA-256 repositories can interoperate with each other. This\n will be a backward compatibility nightmare, as Git from\n SHA-256 repository that tries to talk to SHA-1 repository\n will fail but without grace (i.e. the current situation).\n\n 2) on-the-wire protocol gains just one new capability to safely\n unleash SHA-256 repositories to talk to the wider world. The\n \"borked\" SHA-1 protocol above will become official when the\n object-format=sha256 capability is negotiated by both ends.\n At this stage, SHA-256 repositories still cannot talk with\n SHA-1 repositories, but at least they can talk among\n themselves as long as they use new-enough version of Git that\n knows about the new capability.\n\n 3) on-the-fly SHA-1 vs SHA-256 migration gets implemented.\n SHA-256 reposotories trying to talk to somebody else, after\n discovering that the other end lacks object-format=sha256\n capability, on-the-fly converts its SHA-256 objecst to SHA-1\n and vice versa. Between SHA-256 repositories, the capability\n above in 2) will allow native conversation with SHA-256.\n\nReaching 3) may be a lot of work, but at least we should get to 2)\nto be able to safely let SHA-256 repositories to talk to the outside\nworld (yes, I consider it OK for SHA-256 repositories talking among\nthemselves in a private setting in the current state, and it would\nbe a good milestone and also test towards the eventual goal of\nreaching 3), and with much smaller effort.\n\nThanks.\n\n"},{"id":"403724","messageId":"xmqqk0y1uffn.fsf@gitster.c.googlers.com","threadId":"54039","inReplyTo":"CAN0heSpv5L67hYSX44LkwQ1pJ+_px_W=FQ4piXKS1anpMi0n2w@mail.gmail.com","subject":"Re: [PATCH 3/5] protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-14T20:43:08Z","receivedAt":"2020-08-14T20:43:12Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Ågren writes:\n\n>> I think we should have either a new sha256 capability or a more\n>> generic hash-algo capability whose value can be set to sha256.\n>> Neither the connection initiators or the acceptors should talk\n>> in sha256 until both ends agreed to do so.\n>\n> I think we should, and I think we do. I haven't dug into the details,\n> but \"object-format\" looks like it's just that.\n\nAh, Yes, my thinko.\n\n> Maybe instead of SHA-1 and SHA-256, this should talk about \"whatever has\n> been negotiated through 'object-format', or SHA-1\", similar to brian's\n> suggestion elsewhere.\n\nYup, that would be wonderful.\n\nThanks.\n"},{"id":"403725","messageId":"xmqqft8puf8d.fsf@gitster.c.googlers.com","threadId":"54039","inReplyTo":"xmqqo8ndufli.fsf@gitster.c.googlers.com","subject":"Re: [PATCH 1/5] http-protocol.txt: document SHA-256 \"want\"/\"have\" format","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2020-08-14T20:47:30Z","receivedAt":"2020-08-14T20:47:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> \"brian m. carlson\" writes:\n>\n>> On 2020-08-14 at 17:28:27, Junio C Hamano wrote:\n>>> Martin Ågren writes:\n>>> \n>>> > Document that in SHA-1 repositories, we use SHA-1 for \"want\"s and\n>>> > \"have\"s, and in SHA-256 repositories, we use SHA-256.\n>>> \n>>> Ehh, doesn't this directly contradict the transition plan of \"on the\n>>> wire everything will use SHA-1 version for now?\"\n>>\n>> SHA-256 repositories interoperate currently using SHA-256 object IDs.\n>> It was originally intended that we wouldn't update the protocol, but\n>> that leads to much of the testsuite failing since it's impossible to\n>> move objects from one place to another.\n>>\n>> If we wanted to be more pedantically correct and optimize for the\n>> future, we could say that the values use the format negotiated by the\n>> \"object-format\" protocol extension and SHA-1 otherwise.\n\nYes, that's wonderful. I was confused when I said about the\nevolution path. We still would want to eventually do the on-the-fly\nmigration over the wire to make SHA-1 and SHA-256 repositories\ninteroperate, but at least we already can allow SHA-256 repositories\nsafely attempt to talk to SHA-1 repositories and gracefully fail.\n\nThanks.\n\n"},{"id":"403737","messageId":"14bd0d93620a917f5373ccef2867184ab7bb0811.1597506837.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597506837.git.martin.agren@gmail.com","subject":"[PATCH v2 2/4] index-format.txt: document SHA-256 index format","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-15T16:06:00Z","receivedAt":"2020-08-15T21:51:46Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Document that in SHA-1 repositories, we use SHA-1 and in SHA-256\nrepositories, we use SHA-256, then replace all other uses of \"SHA-1\"\nwith something more neutral. Avoid referring to \"160-bit\" hash values.\n\nSigned-off-by: Martin Ågren \n---\n Documentation/technical/index-format.txt | 34 +++++++++++++-----------\n 1 file changed, 18 insertions(+), 16 deletions(-)\n\ndiff --git a/Documentation/technical/index-format.txt b/Documentation/technical/index-format.txt\nindex faa25c5c52..f9a3644711 100644\n--- a/Documentation/technical/index-format.txt\n+++ b/Documentation/technical/index-format.txt\n@@ -3,8 +3,11 @@ Git index format\n \n == The Git index file has the following format\n \n- All binary numbers are in network byte order. Version 2 is described\n- here unless stated otherwise.\n+ All binary numbers are in network byte order.\n+ In a repository using the traditional SHA-1, checksums and object IDs\n+ (object names) mentioned below are all computed using SHA-1. Similarly,\n+ in SHA-256 repositories, these values are computed using SHA-256.\n+ Version 2 is described here unless stated otherwise.\n \n - A 12-byte header consisting of\n \n@@ -32,8 +35,7 @@ Git index format\n \n Extension data\n \n- - 160-bit SHA-1 over the content of the index file before this\n- checksum.\n+ - Hash checksum over the content of the index file before this checksum.\n \n == Index entry\n \n@@ -80,7 +82,7 @@ Git index format\n 32-bit file size\n This is the on-disk size from stat(2), truncated to 32-bit.\n \n- 160-bit SHA-1 for the represented object\n+ Object name for the represented object\n \n A 16-bit 'flags' field split into (high to low bits)\n \n@@ -160,8 +162,8 @@ Git index format\n \n - A newline (ASCII 10); and\n \n- - 160-bit object name for the object that would result from writing\n- this span of index as a tree.\n+ - Object name for the object that would result from writing this span\n+ of index as a tree.\n \n An entry can be in an invalidated state and is represented by having\n a negative number in the entry_count field. In this case, there is no\n@@ -198,7 +200,7 @@ Git index format\n stage 1 to 3 (a missing stage is represented by \"0\" in this field);\n and\n \n- - At most three 160-bit object names of the entry in stages from 1 to 3\n+ - At most three object names of the entry in stages from 1 to 3\n (nothing is written for a missing stage).\n \n === Split index\n@@ -211,8 +213,8 @@ Git index format\n \n The extension consists of:\n \n- - 160-bit SHA-1 of the shared index file. The shared index file path\n- is $GIT_DIR/sharedindex.. If all 160 bits are zero, the\n+ - Hash of the shared index file. The shared index file path\n+ is $GIT_DIR/sharedindex.. If all bits are zero, the\n index does not require a shared index file.\n \n - An ewah-encoded delete bitmap, each bit represents an entry in the\n@@ -253,10 +255,10 @@ Git index format\n \n - 32-bit dir_flags (see struct dir_struct)\n \n- - 160-bit SHA-1 of $GIT_DIR/info/exclude. Null SHA-1 means the file\n+ - Hash of $GIT_DIR/info/exclude. A null hash means the file\n does not exist.\n \n- - 160-bit SHA-1 of core.excludesfile. Null SHA-1 means the file does\n+ - Hash of core.excludesfile. A null hash means the file does\n not exist.\n \n - NUL-terminated string of per-dir exclude file name. This usually\n@@ -285,13 +287,13 @@ The remaining data of each directory block is grouped by type:\n - An ewah bitmap, the n-th bit records \"check-only\" bit of\n read_directory_recursive() for the n-th directory.\n \n- - An ewah bitmap, the n-th bit indicates whether SHA-1 and stat data\n+ - An ewah bitmap, the n-th bit indicates whether hash and stat data\n is valid for the n-th directory and exists in the next data.\n \n - An array of stat data. The n-th data corresponds with the n-th\n \"one\" bit in the previous ewah bitmap.\n \n- - An array of SHA-1. The n-th SHA-1 corresponds with the n-th \"one\" bit\n+ - An array of hashes. The n-th hash corresponds with the n-th \"one\" bit\n in the previous ewah bitmap.\n \n - One NUL.\n@@ -330,12 +332,12 @@ The remaining data of each directory block is grouped by type:\n \n - 32-bit offset to the end of the index entries\n \n- - 160-bit SHA-1 over the extension types and their sizes (but not\n+ - Hash over the extension types and their sizes (but not\n \ttheir contents). E.g. if we have \"TREE\" extension that is N-bytes\n \tlong, \"REUC\" extension that is M-bytes long, followed by \"EOIE\",\n \tthen the hash would be:\n \n-\tSHA-1(\"TREE\" + +\n+\tHash(\"TREE\" + +\n \t\t\"REUC\" + )\n \n == Index Entry Offset Table\n-- \n2.28.0.297.g1956fa8f8d\n\n"},{"id":"403739","messageId":"cover.1597506837.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597406877.git.martin.agren@gmail.com","subject":"[PATCH v2 0/4] more SHA-256 documentation","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-15T16:05:58Z","receivedAt":"2020-08-15T21:52:42Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Thanks brian, Stolee and Junio for your comments on the initial\nsubmission. Changes since v1:\n\n * Dropped some \"160-bit\" I had missed.\n\n * Refer to the 'object-format' capability in a few spots rather than\n discussing SHA-1 vs SHA-256 repos.\n\n * Dropped the final patch, since Stolee has submitted a patch (series)\n for changing the implementation instead.\n\nThese could be part of bc/sha-256-doc-updates, since they are quite\nsimilar in spirit, or go in separately, so these series don't need to\nhold each other hostage. Whatever Junio and brian prefer will be fine\nby me.\n\nMartin\n\nMartin Ågren (4):\n http-protocol.txt: document SHA-256 \"want\"/\"have\" format\n index-format.txt: document SHA-256 index format\n protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256\n shallow.txt: document SHA-256 shallow format\n\n Documentation/technical/http-protocol.txt | 5 +--\n Documentation/technical/index-format.txt | 34 ++++++++++---------\n .../technical/protocol-capabilities.txt | 12 ++++---\n Documentation/technical/shallow.txt | 2 +-\n 4 files changed, 30 insertions(+), 23 deletions(-)\n\nRange-diff against v1:\n1: fcb26c81be ! 1: 2e9f6b9294 http-protocol.txt: document SHA-256 \"want\"/\"have\" format\n @@ Metadata\n ## Commit message ##\n http-protocol.txt: document SHA-256 \"want\"/\"have\" format\n \n - Document that in SHA-1 repositories, we use SHA-1 for \"want\"s and\n - \"have\"s, and in SHA-256 repositories, we use SHA-256.\n + Document that rather than always naming objects using SHA-1, we should\n + use whatever has been negotiated using the object-format capability.\n \n Signed-off-by: Martin Ågren \n \n @@ Documentation/technical/http-protocol.txt: at all in the request stream:\n -SHA-1 as its value. Multiple SHA-1s MUST be sent by sending\n -multiple commands.\n +object name as its value. Multiple object names MUST be sent by sending\n -+multiple commands. (An object name is a SHA-1 hash in a SHA-1 repo\n -+and a SHA-256 hash in a SHA-256 repo.)\n ++multiple commands. Object names MUST be given using the object format\n ++negotiated through the `object-format` capability (default SHA-1).\n \n The `have` list is created by popping the first 32 commits\n from `c_pending`. Less can be supplied if `c_pending` empties.\n2: 5c13a9478a ! 2: 14bd0d9362 index-format.txt: document SHA-256 index format\n @@ Metadata\n ## Commit message ##\n index-format.txt: document SHA-256 index format\n \n - Similar to a recent commit, document that in SHA-1 repositories, we use\n - SHA-1 and in SHA-256 repositories, we use SHA-256, then replace all\n - other uses of \"SHA-1\" with something more neutral.\n + Document that in SHA-1 repositories, we use SHA-1 and in SHA-256\n + repositories, we use SHA-256, then replace all other uses of \"SHA-1\"\n + with something more neutral. Avoid referring to \"160-bit\" hash values.\n \n Signed-off-by: Martin Ågren \n \n @@ Documentation/technical/index-format.txt: Git index format\n Extension data\n \n - - 160-bit SHA-1 over the content of the index file before this\n -+ - 160-bit hash checksum over the content of the index file before this\n - checksum.\n +- checksum.\n ++ - Hash checksum over the content of the index file before this checksum.\n \n == Index entry\n + \n @@ Documentation/technical/index-format.txt: Git index format\n 32-bit file size\n This is the on-disk size from stat(2), truncated to 32-bit.\n \n - 160-bit SHA-1 for the represented object\n -+ 160-bit object name for the represented object\n ++ Object name for the represented object\n \n A 16-bit 'flags' field split into (high to low bits)\n \n +@@ Documentation/technical/index-format.txt: Git index format\n + \n + - A newline (ASCII 10); and\n + \n +- - 160-bit object name for the object that would result from writing\n +- this span of index as a tree.\n ++ - Object name for the object that would result from writing this span\n ++ of index as a tree.\n + \n + An entry can be in an invalidated state and is represented by having\n + a negative number in the entry_count field. In this case, there is no\n +@@ Documentation/technical/index-format.txt: Git index format\n + stage 1 to 3 (a missing stage is represented by \"0\" in this field);\n + and\n + \n +- - At most three 160-bit object names of the entry in stages from 1 to 3\n ++ - At most three object names of the entry in stages from 1 to 3\n + (nothing is written for a missing stage).\n + \n + === Split index\n @@ Documentation/technical/index-format.txt: Git index format\n \n The extension consists of:\n3: 82e5c67b7c ! 3: 2e82be9e36 protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256\n @@ Metadata\n ## Commit message ##\n protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256\n \n - Two of our extensions contain \"sha1\" in their names, but that's\n - historical. The \"want\"s will take object names that are not necessarily\n - SHA-1s. Make this clear, but also make it clear how there's still just\n - one correct hash algo: These extensions don't somehow make the \"want\"s\n - take object names derived using *any* hash algorithm.\n + Two of our capabilities contain \"sha1\" in their names, but that's\n + historical. Clarify that object names are still to be given using\n + whatever object format has been negotiated using the \"object-format\"\n + capability.\n \n Signed-off-by: Martin Ågren \n \n @@ Documentation/technical/protocol-capabilities.txt: allow-tip-sha1-in-want\n -send \"want\" lines with SHA-1s that exist at the server but are not\n -advertised by upload-pack.\n +send \"want\" lines with object names that exist at the server but are not\n -+advertised by upload-pack. (Note that the name of the capability\n -+contains \"sha1\", but that it's more general than that: in SHA-1\n -+repositories, the \"want\" lines provide SHA-1 values, but in SHA-256\n -+repositories, they provide SHA-256 values.)\n ++advertised by upload-pack. For historical reasons, the name of this\n ++capability contains \"sha1\". Object names are always given using the\n ++object format negotiated through the 'object-format' capability.\n \n allow-reachable-sha1-in-want\n ----------------------------\n @@ Documentation/technical/protocol-capabilities.txt: allow-tip-sha1-in-want\n -send \"want\" lines with SHA-1s that exist at the server but are not\n -advertised by upload-pack.\n +send \"want\" lines with object names that exist at the server but are not\n -+advertised by upload-pack. (Same remark about \"sha1\" as above.)\n ++advertised by upload-pack. For historical reasons, the name of this\n ++capability contains \"sha1\". Object names are always given using the\n ++object format negotiated through the 'object-format' capability.\n \n push-cert=\n -----------------\n4: bcfbdd25e5 ! 4: 8680fc1af6 shallow.txt: document SHA-256 shallow format\n @@ Metadata\n ## Commit message ##\n shallow.txt: document SHA-256 shallow format\n \n - Similar to recent commits, document that in SHA-1 repositories, we use\n - SHA-1 for these purposes, and in SHA-256 repositories, we use SHA-256.\n + Similar to recent commits, document that we list object names rather\n + than SHA-1s.\n \n Signed-off-by: Martin Ågren \n \n5: f95e3f65e7 < -: ---------- commit-graph-format.txt: fix \"Hash Version\" description\n-- \n2.28.0.297.g1956fa8f8d\n\n"},{"id":"403741","messageId":"2e82be9e365d5157a604a599d764fa39664ab684.1597506837.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597506837.git.martin.agren@gmail.com","subject":"[PATCH v2 3/4] protocol-capabilities.txt: clarify \"allow-x-sha1-in-want\" re SHA-256","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-15T16:06:01Z","receivedAt":"2020-08-15T21:53:17Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Two of our capabilities contain \"sha1\" in their names, but that's\nhistorical. Clarify that object names are still to be given using\nwhatever object format has been negotiated using the \"object-format\"\ncapability.\n\nSigned-off-by: Martin Ågren \n---\n Documentation/technical/protocol-capabilities.txt | 12 ++++++++----\n 1 file changed, 8 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 36ccd14f97..124d716807 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -324,15 +324,19 @@ allow-tip-sha1-in-want\n ----------------------\n \n If the upload-pack server advertises this capability, fetch-pack may\n-send \"want\" lines with SHA-1s that exist at the server but are not\n-advertised by upload-pack.\n+send \"want\" lines with object names that exist at the server but are not\n+advertised by upload-pack. For historical reasons, the name of this\n+capability contains \"sha1\". Object names are always given using the\n+object format negotiated through the 'object-format' capability.\n \n allow-reachable-sha1-in-want\n ----------------------------\n \n If the upload-pack server advertises this capability, fetch-pack may\n-send \"want\" lines with SHA-1s that exist at the server but are not\n-advertised by upload-pack.\n+send \"want\" lines with object names that exist at the server but are not\n+advertised by upload-pack. For historical reasons, the name of this\n+capability contains \"sha1\". Object names are always given using the\n+object format negotiated through the 'object-format' capability.\n \n push-cert=\n -----------------\n-- \n2.28.0.297.g1956fa8f8d\n\n"},{"id":"403757","messageId":"8680fc1af6714b1d97b81b824184679593213d91.1597506837.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597506837.git.martin.agren@gmail.com","subject":"[PATCH v2 4/4] shallow.txt: document SHA-256 shallow format","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-15T16:06:02Z","receivedAt":"2020-08-15T21:58:35Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Similar to recent commits, document that we list object names rather\nthan SHA-1s.\n\nSigned-off-by: Martin Ågren \n---\n Documentation/technical/shallow.txt | 2 +-\n 1 file changed, 1 insertion(+), 1 deletion(-)\n\ndiff --git a/Documentation/technical/shallow.txt b/Documentation/technical/shallow.txt\nindex 01dedfe9ff..f3738baa0f 100644\n--- a/Documentation/technical/shallow.txt\n+++ b/Documentation/technical/shallow.txt\n@@ -13,7 +13,7 @@ pretend as if they are root commits (e.g. \"git log\" traversal\n stops after showing them; \"git fsck\" does not complain saying\n the commits listed on their \"parent\" lines do not exist).\n \n-Each line contains exactly one SHA-1. When read, a commit_graft\n+Each line contains exactly one object name. When read, a commit_graft\n will be constructed, which has nr_parent < 0 to make it easier\n to discern from user provided grafts.\n \n-- \n2.28.0.297.g1956fa8f8d\n\n"},{"id":"403763","messageId":"2e9f6b92946dc2fa645adc72f32d5d05a1720185.1597506837.git.martin.agren@gmail.com","threadId":"54039","inReplyTo":"cover.1597506837.git.martin.agren@gmail.com","subject":"[PATCH v2 1/4] http-protocol.txt: document SHA-256 \"want\"/\"have\" format","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2020-08-15T16:05:59Z","receivedAt":"2020-08-15T22:01:50Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"Document that rather than always naming objects using SHA-1, we should\nuse whatever has been negotiated using the object-format capability.\n\nSigned-off-by: Martin Ågren \n---\n Documentation/technical/http-protocol.txt | 5 +++--\n 1 file changed, 3 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt\nindex 51a79e63de..96d89ea9b2 100644\n--- a/Documentation/technical/http-protocol.txt\n+++ b/Documentation/technical/http-protocol.txt\n@@ -401,8 +401,9 @@ at all in the request stream:\n The stream is terminated by a pkt-line flush (`0000`).\n \n A single \"want\" or \"have\" command MUST have one hex formatted\n-SHA-1 as its value. Multiple SHA-1s MUST be sent by sending\n-multiple commands.\n+object name as its value. Multiple object names MUST be sent by sending\n+multiple commands. Object names MUST be given using the object format\n+negotiated through the `object-format` capability (default SHA-1).\n \n The `have` list is created by popping the first 32 commits\n from `c_pending`. Less can be supplied if `c_pending` empties.\n-- \n2.28.0.297.g1956fa8f8d\n\n"}]}