{"thread":{"id":"59204","subject":"[RFC PATCH 0/1] Canonical tar format for Git","startedAt":"2023-02-05T22:17:36Z","lastAt":"2023-02-09T01:39:18Z","messageCount":9,"participants":["brian m. carlson","Junio C Hamano","Ævar Arnfjörð Bjarmason"],"isPatch":true,"patchVersion":1,"patchTotal":1},"messages":[{"id":"471548","messageId":"20230205221728.4179674-1-sandals@crustytoothpaste.net","threadId":"59204","inReplyTo":null,"subject":"[RFC PATCH 0/1] Canonical tar format for Git","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-02-05T22:17:27Z","receivedAt":"2023-02-05T22:17:36Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"With recent discussions about guaranteeing some stability for our\narchives, here's a document which specifies a canonical tar format we\ncan use, based on the pax format.\n\nIt contains a sufficient amount of explanation about how the format\nworks to (hopefully) be easily understandable, and it explains the\nbenefits of the extremely rigid format it defines.\n\nIf we decide that we'd like to implement extensions to it in the future,\nit can be easily extended.  My original approach allowed specifying\nversion extensions to specify that things like permissions or timestamps\nwere persisted, but I simplified for ease of use, since we don't\ntechnically need those changes.\n\nFeedback about this approach is welcome, and after some discussion, I\nplan to send a future version with an actual implementation and tests.\n\nbrian m. carlson (1):\n  Document a fixed tar format for interoperability\n\n Documentation/technical/tarball.txt | 234 ++++++++++++++++++++++++++++\n 1 file changed, 234 insertions(+)\n create mode 100644 Documentation/technical/tarball.txt\n\n"},{"id":"471549","messageId":"20230205221728.4179674-2-sandals@crustytoothpaste.net","threadId":"59204","inReplyTo":"20230205221728.4179674-1-sandals@crustytoothpaste.net","subject":"[RFC PATCH 1/1] Document a fixed tar format for interoperability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-02-05T22:17:28Z","receivedAt":"2023-02-05T22:17:38Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"Right now, many people wish to have archives which are consistent among\nGit versions.  That is not something we currently offer, but users often\nrely on the fact that our tar format changes rarely and assume it will\nnever change, which has caused lots of problems with various sites in\nthe past.\n\nInstead of letting this go on indefinitely, let's explicitly document a\nversioned canonical tar format which is completely reproducible and\nwhich we guarantee will be permanently stable.  This format is more\nrigid than the current tar format, but it produces identical results for\nidentical trees, regardless of hash algorithm, and is easy to implement\nin other tools.  This is beneficial because lots of people want fixed,\nreproducible archives, and there's little reason to duplicate work.\n\nThis format, like the existing format, is actually a pax format archive,\nwhich is an extension to the ustar (Unix standard tar) format.  This\nformat was documented by POSIX in 2001 and is well understood by most\nmodern tar implementations, including GNU tar and libarchive, which\ncovers the versions of tar used on most major operating systems,\nincluding Windows, Linux, macOS, and most BSDs.\n\nThe format in this document does mandate a pax header for each file,\nwhich slightly increases the size of the archive.  However, to properly\nembed timestamps, GNU tar and libarchive's tar also do this when\ngenerating pax archives, and because the data is highly redundant, it\nwill compress extremely well.  A comparison between the two approaches\nusing GNU tar and libarchive's tar on Git's working tree shows that with\ndefault gzip compression, the increase in size is about 1.2%, which is\nfine for almost every use case.  In return, we get a substantially\nsimpler (and thus, likely, more correct) implementation which is much\neasier to test.\n---\n Documentation/technical/tarball.txt | 234 ++++++++++++++++++++++++++++\n 1 file changed, 234 insertions(+)\n create mode 100644 Documentation/technical/tarball.txt\n\ndiff --git a/Documentation/technical/tarball.txt b/Documentation/technical/tarball.txt\nnew file mode 100644\nindex 0000000000..fd23df2f33\n--- /dev/null\n+++ b/Documentation/technical/tarball.txt\n@@ -0,0 +1,234 @@\n+Git Canonical Tar Format\n+========================\n+\n+Overview\n+--------\n+\n+Many people find it convenient to have tar archives that are bit-for-bit\n+identical between versions.  This can be valuable to validate that an archive\n+has not changed using a cryptographic hash without needing to store the archive\n+itself.\n+\n+However, up to now, Git has not guaranteed a consistent format, although people\n+often make the assumption that Git's archives will always be bit-for-bit\n+identical.  This has led to several notable problems with various forges.\n+\n+This document proposes a canonical tar format based on the POSIX pax format that\n+is bit-for-bit identical.  It is referred to as ctar-v1 (canonical tar version 1).\n+\n+Goals and Rationale\n+-------------------\n+\n+The goals for this format are that it is first and foremost reproducible, that\n+identical trees produce identical results, that it is simple and easy to\n+implement correctly, and that it is useful in general.  While we don't consider\n+functionality needs beyond Git's at the moment (such as hardlinks, xattrs, or\n+sparse files), there is intense interest in reproducible builds, and so it makes\n+sense to design something that can see general use for software interchange.\n+\n+Because the goal is strict reproducibility, this format doesn't honor\n+`tar.umask` or other options that can produce different output.  It serializes\n+all timestamps as the Epoch, which produces identical results whether the tree\n+is serialized as a tree, commit, or tag.  This is consistent with the behaviour\n+of some other tar serializers, including the default for modern Rust crates, and\n+is not believed to pose any interoperability problems.\n+\n+Object IDs are not included in this version of the format because this produces\n+non-identical data when identical data is serialized with different hash\n+algorithms.\n+\n+Introduction to the Underlying Format\n+-------------------------------------\n+\n+A pax archive is an extended form of the ustar (Unix standard tar) archive, both\n+defined in POSIX.1-2001.  Each file in a ustar archive is preceded by a single\n+512-byte header block, followed by as many 512-byte data blocks as needed to\n+store the data, padded with zeros.  At the end of the archive are two 512-byte\n+blocks filled completely with zeros.\n+\n+A pax archive may additionally contain extended headers.  There is optionally\n+one for the entire archive, which is called the global header, and one for each\n+file.  If present, the global extended header is the first entry in the archive,\n+and the per-file header precedes the file to which it corresponds.  Every\n+extended header contains a normal ustar header block with either the `g`\n+(global) or `x` (per-file) type, followed by metadata in a textual\n+length-key-value form (`%d %s=%s\\n`) which is stored as the data of this\n+pseudo-file.\n+\n+A global extended header sets metadata for the entire file, and a per-file\n+extended header applies to only the to which it corresponds.  A per-file\n+extended header overrides any data specified in the global extended header, and\n+all extended headers override any data stored in a normal ustar per-file header\n+block.\n+\n+While pax extensions are widely supported by most modern versions of tar\n+(including versions on Windows and all major open-source OSes), some older\n+archivers and non-tar implementations which do not understand them typically\n+extract the extended headers as regular files.  Thus, it's helpful to have these\n+entries have reasonable permissions and unique names.\n+\n+General Architecture\n+--------------------\n+\n+All canonical tar archives are valid POSIX pax archives as that format is\n+defined in POSIX.1-2017.  Every archive will have a global header indicating the\n+version and format and what types of data are valid in the archive.\n+\n+Every file serialized in the archive is serialized in lexicographical order by\n+its bytes.  A directory is always serialized before its contents, and a\n+directory is never serialized with a trailing slash.  If a system uses a Unicode\n+encoding other than UTF-8, it encodes filenames as UTF-8.  Each file shall\n+contain a pax extended header record.\n+\n+It is possible to encode some extended headers in multiple ways because the\n+length in the header encodes its own length.  For example, in cases where the\n+length value can be encoded as either 99 or 100, both can lead to identical\n+header data.  The shortest possible encoding must always be used.\n+\n+In any event where multiple encodings are possible, the shortest and, if there\n+is still confusion, lexicographically first (by byte value) must always be used.\n+All unspecified padding is filled with NUL bytes.\n+\n+Version Number\n+--------------\n+\n+The version number for this version is `ctar-v1`.\n+\n+Extended Headers\n+----------------\n+\n+Global Extended Header\n+~~~~~~~~~~~~~~~~~~~~~~\n+\n+The global extended header (record `g`) shall contain one header:\n+`CTAR.version`, which contains the version number specified above.\n+\n+The contents of the ustar header for the global extended header are as below,\n+except that the `name` field contains `pax_global_header`.\n+\n+Per-File Extended Header\n+~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+Each file has a per-file extended header.\n+\n+The following per-file extended header fields are included:\n+\n+|===\n+| Field Name   | When Present  | Value\n+\n+| `atime`      | always        | `0`\n+| `mtime`      | always        | `0`\n+| `size`       | always        | size of the data in bytes\n+| `path`       | always        | full path name of the file\n+| `uid`        | always        | `0`\n+| `gid`        | always        | `0`\n+| `uname`      | always        | `root`\n+| `gname`      | always        | `root`\n+| `linkpath`   | symbolic link | full path name of the link destination\n+| `hdrcharset` | binary path   | `BINARY`\n+\n+Note that the `hdrcharset` entry appears if and only if the `path` or, if\n+present, the `linkpath`, header contains a non-UTF-8 encoded string.  Because\n+Git does not store the encoding of file names, it has no way of knowing whether\n+a file name which could be valid UTF-8 actually is, but for the purposes of\n+compatibility, such file names are assumed to be UTF-8 and are not declared as\n+binary.  This improves portability to systems which always use Unicode.\n+\n+However, we because we do not know for certain whether these values are UTF-8,\n+we avoid explicitly declaring them as such and rely on the default archiver\n+behavior, which may be more sensible.\n+\n+The `path` field contains the full path name without a leading slash or leading\n+`.` or `..` component.  The path never contains a directory component which is\n+`.` or `..`.\n+\n+The `linkpath` field contains the full symbolic link destination.  `.` and `..`\n+components are permitted if the destination contains those values.\n+\n+In all cases, path names use `/` as the directory separator.\n+\n+The reason for always including most of the entries in the archive is to aid in\n+implementing and testing correct serialization.  If these entries are always\n+present, then this process becomes much simpler, whereas if they are only\n+included as needed, then errors are more likely.\n+\n+The `name` field of the ustar header of this extended header is `paxheader.%d`,\n+where `%d` represents the shortest-form decimal integer encoding the index of\n+this file in the archive, starting with 0.  All files, directories, and links of\n+whatever kind are counted, but extended headers are not.\n+\n+Serialization of Extended Headers\n+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n+\n+When serializing the header block for an extended header, the following values\n+should be used.  Note that all text fields are be NUL-padded on the right when\n+they do not fill the field, and all octal fields are left-padded with zeros such\n+that they fill the field with a single trailing NUL.  An empty field contains\n+only NULs.\n+\n+|===\n+| Field Name | Value\n+\n+| `name`     | `pax_global_header` (global) or `paxheader.%d` (per-file) (see above)\n+| `mode`     | `0640`\n+| `uid`      | `0`\n+| `gid`      | `0`\n+| `size`     | the size of the extended header in bytes\n+| `mtime`    | `0` (the Epoch)\n+| `chksum`   | as specified in the standard\n+| `typeflag` | `g` (global) or `x` (per-file)\n+| `linkname` | empty\n+| `magic`    | as specified in the standard\n+| `version`  | as specified in the standard\n+| `uname`    | `root`\n+| `gname`    | `root`\n+| `devmajor` | `0`\n+| `devminor` | `0`\n+| `prefix`   | empty\n+|===\n+\n+When encoding the data for an extended header, all entries are sorted in order\n+by the byte values of their keys as encoded in UTF-8.  Duplicate keys are not\n+permitted.\n+\n+Because the format allows multiple length encodings of some values, the shortest\n+possible encoding must always be used.\n+\n+ustar headers\n+-------------\n+\n+The ustar header for each file is serialized as below.  Note that all text\n+fields are be NUL-padded on the right when they do not fill the field, and all\n+octal fields are left-padded with zeros such that they fill the field with a\n+single trailing NUL.  An empty field contains only NULs.\n+\n+|===\n+| Field Name | Value\n+\n+| `name`     | the last path component if it fits; otherwise, `path.%d`\n+| `mode`     | `0640` (regular file), `0777` (symbolic link), `0750` (directory)\n+| `uid`      | `0`\n+| `gid`      | `0`\n+| `size`     | the size of the data in bytes for regular files if it fits; otherwise, `0`\n+| `mtime`    | `0` (the Epoch)\n+| `chksum`   | as specified in the standard\n+| `typeflag` | `0` (regular file), `2` (symbolic link), `5` (directory)\n+| `linkname` | empty\n+| `magic`    | as specified in the standard\n+| `version`  | as specified in the standard\n+| `uname`    | `root`\n+| `gname`    | `root`\n+| `devmajor` | `0`\n+| `devminor` | `0`\n+| `prefix`   | all non-trailing path components if they fit; otherwise, empty\n+|===\n+\n+Note that the `size` field is always 0 for non-regular files.  The `typeflag`\n+value for regular files is always `0`, not NUL.\n+\n+`prefix` does not contain a trailing slash.\n+\n+If the `name` field cannot contain the last path component, then it is\n+serialized as `path.%d`, where `%d` represents the shortest-form decimal integer\n+encoding the index of this file in the archive, starting with 0.  The `%d` value\n+in this case is completely identical to the `%d` in the per-file pax header.\n"},{"id":"471600","messageId":"xmqq8rha5wno.fsf@gitster.g","threadId":"59204","inReplyTo":"20230205221728.4179674-2-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 1/1] Document a fixed tar format for interoperability","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2023-02-06T21:08:59Z","receivedAt":"2023-02-06T21:09:05Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"\"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n\n> +Overview\n> +--------\n> +\n> +Many people find it convenient to have tar archives that are bit-for-bit\n> +identical between versions.  This can be valuable to validate that an archive\n> +has not changed using a cryptographic hash without needing to store the archive\n> +itself.\n> +\n> +However, up to now, Git has not guaranteed a consistent format, although people\n> +often make the assumption that Git's archives will always be bit-for-bit\n> +identical.  This has led to several notable problems with various forges.\n> +\n> +This document proposes a canonical tar format based on the POSIX pax format that\n> +is bit-for-bit identical.  It is referred to as ctar-v1 (canonical tar version 1).\n\n\"is identical to what\"?  Ditto for the one in the previous\nparagraph.  The first paragraph is better in that there is \"between\nversions\", even though it would be easier to grok if we made it more\nclear that we are talking about versions of the software that is\nused to create the archive, not the version of contents being\narchived.\n\nOur goal is that serializing the same tree object or the same commit\nobject result in bit-for-bit identical result, no matter which\nversion of Git is used, and no matter what platform the Git used to\ncreate the archive was built on.  Mentioning both what we take an\narchive out of (i.e. tree or commit) and we can use different\nversions of Git to create archives, in the description would make it\neasier to grok.\n\n> +Goals and Rationale\n> +-------------------\n> +\n> +The goals for this format are that it is first and foremost reproducible, that\n> +identical trees produce identical results, that it is simple and easy to\n> +implement correctly, and that it is useful in general.  While we don't consider\n> +functionality needs beyond Git's at the moment (such as hardlinks, xattrs, or\n> +sparse files), there is intense interest in reproducible builds, and so it makes\n> +sense to design something that can see general use for software interchange.\n\nPerfect.\n\n> +Because the goal is strict reproducibility, this format doesn't honor\n> +`tar.umask` or other options that can produce different output.  It serializes\n> +all timestamps as the Epoch, which produces identical results whether the tree\n> +is serialized as a tree, commit, or tag.  This is consistent with the behaviour\n> +of some other tar serializers, including the default for modern Rust crates, and\n> +is not believed to pose any interoperability problems.\n\n> +Object IDs are not included in this version of the format because this produces\n> +non-identical data when identical data is serialized with different hash\n> +algorithms.\n\nDeclaring that we'll always peel a tag or a commit down to a tree is\none sure way to avoid having to worry about object name hashes, but\naren't we discarding too much utility by doing so?\n\nThis is probably debatable.  The commit object name embedded in the\nextended header of an archive makes it trivial to identify what\nversion the archive _claims_ to have been taken from (you could also\nembed it in the filename that stores archive, but the use of the\nembedded metainfo makes it more robust against file names).  And\nrunning \"git archive\" twice, with different versions of Git on\ndifferent architectures, should be reproducible as long as both\ninvokers expressed their desire to see the commit object name in the\narchive by passing the commit, not its tree, to the command, and\nthey are using the same hash algorithm.\n\nIn the world where multiple hash functions are in use, a commit that\nis being archived may have one or two \"object names\", but it should\nnot be hard to use one extended header item per each to store one or\nboth, I would imagine.\n\nHaving said all that, I think stripping the commit object name (or\ntags) is a better design.  Imagine that I see I created a tarball\nearlier and published its hash, but later lost the tarball.  By not\nallowing any commit object name in the archive, it would force me to\nsomehow name the tarball in such a way that I can tell which commit\nI used to create it, e.g. \"git-e83c516331.tar\".  Other people can\nnotice the filename and without having seen the bytes in it, they\ncan try running \"git archive e83c516331\" in their repository and see\nthe output matches the hash I published earlier.  Having commit or\ntag embedded in the archive would make it harder to do this kind of\nthings.\n\nBy the way, other potentially interesting points are:\n\n - Do we want to ignore \"export-subst\" for stability?\n\n - \"git archive\" can be invoked with pathspec to archive only a\n   subset of paths.\n\n - \"git archive\" could be extended to include submoudule trees\n   recursively in the same output.\n\nThe latter two are trivial to support, but we need to make sure that\nwe do not screw up the ordering of paths in the output, especially\nfor the last one, when we add it.\n\n> +Introduction to the Underlying Format\n> +-------------------------------------\n> ...\n> +A global extended header sets metadata for the entire file, and a per-file\n> +extended header applies to only the to which it corresponds.  A per-file\n\n\"only the to which\" -> \"only the file to which\"\n\n> +extended header overrides any data specified in the global extended header, and\n> +all extended headers override any data stored in a normal ustar per-file header\n> +block.\n\n> +While pax extensions are widely supported by most modern versions of tar\n> +(including versions on Windows and all major open-source OSes), some older\n> +archivers and non-tar implementations which do not understand them typically\n> +extract the extended headers as regular files.  Thus, it's helpful to have these\n> +entries have reasonable permissions and unique names.\n\nSurely, and to make things reproducible, they shouldn't just be\nreasonable and unique.  They should be exactly as we define in the\nspecification.\n\n> +General Architecture\n> +--------------------\n> +\n> +All canonical tar archives are valid POSIX pax archives as that format is\n> +defined in POSIX.1-2017.  Every archive will have a global header indicating the\n> +version and format and what types of data are valid in the archive.\n> +\n> +Every file serialized in the archive is serialized in lexicographical order by\n> +its bytes.  A directory is always serialized before its contents, and a\n\n\"by its bytes\" -> \"by the bytes in its filename\" or something?\nSurely we do not sort by contents ;-)\n\n> +directory is never serialized with a trailing slash.  If a system uses a Unicode\n> +encoding other than UTF-8, it encodes filenames as UTF-8.\n\nThis is a bit hard to grok.  Do you mean there may be UTF-16 system\nwhere the data in our tree objects, whose paths are recorded in UTF-8,\nbut \"git checkout\" of the tree may result in files in the native\nfilename on that system, i.e. UTF-16 not UTF-8?  And even on such a\nsystem, running \"git archive\" would record paths in the archive in\nUTF-8 (i.e. the same as what was in the tree object)?  Or do you\nmean something stronger, like on a Latin-1 system with Latin-1\nproject that used Latin-1 as pathnames even in the tree objects,\nwhen \"git archive\" produces an archive, the paths in it shall be\ntranscoded from the original Latin-1 pathnames to UTF-8?\n\n> +Each file shall contain a pax extended header record.\n> +\n> +It is possible to encode some extended headers in multiple ways because the\n> +length in the header encodes its own length.  For example, in cases where the\n> +length value can be encoded as either 99 or 100, both can lead to identical\n> +header data.  The shortest possible encoding must always be used.\n\n;-)\n\n> +In any event where multiple encodings are possible, the shortest and, if there\n> +is still confusion, lexicographically first (by byte value) must always be used.\n\n;-)\n\n> +All unspecified padding is filled with NUL bytes.\n\nPerhaps we should change the casual mention \"zero\"s we saw earlier\nabout with \"NUL bytes\", too.\n\n> +Version Number\n> +--------------\n> +\n> +The version number for this version is `ctar-v1`.\n> +\n> +Extended Headers\n> +----------------\n> +\n> +Global Extended Header\n> +~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +The global extended header (record `g`) shall contain one header:\n> +`CTAR.version`, which contains the version number specified above.\n> +\n> +The contents of the ustar header for the global extended header are as below,\n> +except that the `name` field contains `pax_global_header`.\n\n\"as below\" meaning...?  The same as what is listed in \"Per-File\nExtended Header\"?  There is no `name` field listed there, though.\n\n> +Per-File Extended Header\n> +~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +Each file has a per-file extended header.\n> +\n> +The following per-file extended header fields are included:\n> +\n> +|===\n> +| Field Name   | When Present  | Value\n> +\n> +| `atime`      | always        | `0`\n> +| `mtime`      | always        | `0`\n> +| `size`       | always        | size of the data in bytes\n> +| `path`       | always        | full path name of the file\n\nThese are length-prefixed data, so we do not have to worry about\noverly long pathnames or symlinks?\n\n> +| `uid`        | always        | `0`\n> +| `gid`        | always        | `0`\n> +| `uname`      | always        | `root`\n> +| `gname`      | always        | `root`\n> +| `linkpath`   | symbolic link | full path name of the link destination\n> +| `hdrcharset` | binary path   | `BINARY`\n> +\n> +Note that the `hdrcharset` entry appears if and only if the `path` or, if\n> +present, the `linkpath`, header contains a non-UTF-8 encoded string.  Because\n> +Git does not store the encoding of file names, it has no way of knowing whether\n> +a file name which could be valid UTF-8 actually is, but for the purposes of\n> +compatibility, such file names are assumed to be UTF-8 and are not declared as\n> +binary.  This improves portability to systems which always use Unicode.\n> +However, we because we do not know for certain whether these values are UTF-8,\n\n\"we because\" -> \"because\"\n\n> +we avoid explicitly declaring them as such and rely on the default archiver\n> +behavior, which may be more sensible.\n\nSo, do we or do we not store hdrcharset?  Producing Git does not know\nif the pathnames stored in the tree it is asked to produce archive\nfor are not in UTF-8, so it assumes everything is in UTF-8 hence\ndoes not see the need to add hdrcharset?\n\n> +The `path` field contains the full path name without a leading slash or leading\n> +`.` or `..` component.  The path never contains a directory component which is\n> +`.` or `..`.\n> +\n> +The `linkpath` field contains the full symbolic link destination.  `.` and `..`\n> +components are permitted if the destination contains those values.\n\nIn other words, we just store the contents of the blob that\nrepresents the symbolic link there?  I wonder if we do anything\nspecial if a blob, that is pointed at in an entry in a tree whose\nmode bits are 120000, has NUL in it (should we teach fsck to flag\nit, for example)?\n\n> +In all cases, path names use `/` as the directory separator.\n> +\n> +The reason for always including most of the entries in the archive is to aid in\n> +implementing and testing correct serialization.  If these entries are always\n> +present, then this process becomes much simpler, whereas if they are only\n> +included as needed, then errors are more likely.\n\nThe order of entries need to be specified when we aim for\nbit-for-bit reproduceability, no?\n\n> +The `name` field of the ustar header of this extended header is `paxheader.%d`,\n> +where `%d` represents the shortest-form decimal integer encoding the index of\n> +this file in the archive, starting with 0.  All files, directories, and links of\n> +whatever kind are counted, but extended headers are not.\n\n> +Serialization of Extended Headers\n> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\n> +\n> +When serializing the header block for an extended header, the following values\n\n\"the header block\" -> \"the ustar header block\" to match the next\nsection, probably.\n\n> +should be used.  Note that all text fields are be NUL-padded on the right when\n> +they do not fill the field, and all octal fields are left-padded with zeros such\n> +that they fill the field with a single trailing NUL.  An empty field contains\n> +only NULs.\n> +\n> +|===\n> +| Field Name | Value\n> +\n> +| `name`     | `pax_global_header` (global) or `paxheader.%d` (per-file) (see above)\n> +| `mode`     | `0640`\n> +| `uid`      | `0`\n> +| `gid`      | `0`\n> +| `size`     | the size of the extended header in bytes\n> +| `mtime`    | `0` (the Epoch)\n> +| `chksum`   | as specified in the standard\n> +| `typeflag` | `g` (global) or `x` (per-file)\n> +| `linkname` | empty\n> +| `magic`    | as specified in the standard\n> +| `version`  | as specified in the standard\n> +| `uname`    | `root`\n> +| `gname`    | `root`\n> +| `devmajor` | `0`\n> +| `devminor` | `0`\n> +| `prefix`   | empty\n> +|===\n\nThese are barebone header fields, not extended headers.  Do we want\nto refer to some canonical sources so that readers understand that\nunlike the extended headres we are talking about fixed-length fields? \nThe description above talks about \"padding\", but that of course\napplies to fixed width columns.\n\n> +When encoding the data for an extended header, all entries are sorted in order\n> +by the byte values of their keys as encoded in UTF-8.  Duplicate keys are not\n> +permitted.\n> +\n> +Because the format allows multiple length encodings of some values, the shortest\n> +possible encoding must always be used.\n"},{"id":"471620","messageId":"230206.86lela4ebq.gmgdl@evledraar.gmail.com","threadId":"59204","inReplyTo":"20230205221728.4179674-2-sandals@crustytoothpaste.net","subject":"Re: [RFC PATCH 1/1] Document a fixed tar format for interoperability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-06T22:18:47Z","receivedAt":"2023-02-06T22:31:07Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Sun, Feb 05 2023, brian m. carlson wrote:\n\n> +The goals for this format are that it is first and foremost reproducible, that\n> +identical trees produce identical results, that it is simple and easy to\n> +implement correctly, and that it is useful in general.  While we don't consider\n> +functionality needs beyond Git's at the moment (such as hardlinks, xattrs, or\n> +sparse files), there is intense interest in reproducible builds, and so it makes\n> +sense to design something that can see general use for software interchange.\n\nI think a goal should be to be bit-for-bit compatible with what we've\nhad historically, which...\n\n> +Object IDs are not included in this version of the format because this produces\n> +non-identical data when identical data is serialized with different hash\n> +algorithms.\n\n...this is inherntly at odds with. I had a longer comment about why I\nthink we can have our cake & eat it too at\nhttps://lore.kernel.org/git/230131.86tu06rkbp.gmgdl@evledraar.gmail.com/\n\nMaybe there are other changes in the proposed spec that put it at odds\nwith such a goal, it's unclear to me if this is the only difference.\n\nBut I don't see why we need bit-for-bit compatible output between SHA-1\nand SHA-256 git repos for the reasons noted in the linked-to reply, and\nremoving this will remove a *really useful* aspect of our tar format,\nwhich is that you can grab an arbitrary tarball, and see what commit\nit's produced from.\n\nEven if you want to retain SHA-1 and SHA-256 interop as far as tar is\nconcerned, an un-discussed alternative is to just stick the SHA-1 OID\ninto the SHA-256 archive.\n\nFor repos that are migrated we envision having such a bi-directional\nmapping anyway.\n\nAnd for those that started out as SHA-256, or where we no longer care\nabout compatibility with old SHA-1, we can just start including the\nSHA-256 OID, as all compatibility concerns have gone away when we\nstopped bothering to maintain the mapping, no?\n\n> +|===\n> +| Field Name | Value\n> +\n> +| `name`     | the last path component if it fits; otherwise, `path.%d`\n> +| `mode`     | `0640` (regular file), `0777` (symbolic link), `0750` (directory)\n> +| `uid`      | `0`\n> +| `gid`      | `0`\n> +| `size`     | the size of the data in bytes for regular files if it fits; otherwise, `0`\n> +| `mtime`    | `0` (the Epoch)\n> +| `chksum`   | as specified in the standard\n\nThis is the nth reference to \"the standard\". I think this would be\nimproved by linking to it, isn't it\nhttps://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html ?\n"},{"id":"471732","messageId":"Y+LR5rlFTqyLfoeF@tapette.crustytoothpaste.net","threadId":"59204","inReplyTo":"xmqq8rha5wno.fsf@gitster.g","subject":"Re: [RFC PATCH 1/1] Document a fixed tar format for interoperability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-02-07T22:34:14Z","receivedAt":"2023-02-07T22:34:21Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-02-06 at 21:08:59, Junio C Hamano wrote:\n> \"brian m. carlson\" <sandals@crustytoothpaste.net> writes:\n> \"is identical to what\"?  Ditto for the one in the previous\n> paragraph.  The first paragraph is better in that there is \"between\n> versions\", even though it would be easier to grok if we made it more\n> clear that we are talking about versions of the software that is\n> used to create the archive, not the version of contents being\n> archived.\n> \n> Our goal is that serializing the same tree object or the same commit\n> object result in bit-for-bit identical result, no matter which\n> version of Git is used, and no matter what platform the Git used to\n> create the archive was built on.  Mentioning both what we take an\n> archive out of (i.e. tree or commit) and we can use different\n> versions of Git to create archives, in the description would make it\n> easier to grok.\n\nI can update that to reflect things more accurately.\n\n> > +Goals and Rationale\n> > +-------------------\n> > +\n> > +The goals for this format are that it is first and foremost reproducible, that\n> > +identical trees produce identical results, that it is simple and easy to\n> > +implement correctly, and that it is useful in general.  While we don't consider\n> > +functionality needs beyond Git's at the moment (such as hardlinks, xattrs, or\n> > +sparse files), there is intense interest in reproducible builds, and so it makes\n> > +sense to design something that can see general use for software interchange.\n> \n> Perfect.\n> \n> > +Because the goal is strict reproducibility, this format doesn't honor\n> > +`tar.umask` or other options that can produce different output.  It serializes\n> > +all timestamps as the Epoch, which produces identical results whether the tree\n> > +is serialized as a tree, commit, or tag.  This is consistent with the behaviour\n> > +of some other tar serializers, including the default for modern Rust crates, and\n> > +is not believed to pose any interoperability problems.\n> \n> > +Object IDs are not included in this version of the format because this produces\n> > +non-identical data when identical data is serialized with different hash\n> > +algorithms.\n> \n> Declaring that we'll always peel a tag or a commit down to a tree is\n> one sure way to avoid having to worry about object name hashes, but\n> aren't we discarding too much utility by doing so?\n> \n> This is probably debatable.  The commit object name embedded in the\n> extended header of an archive makes it trivial to identify what\n> version the archive _claims_ to have been taken from (you could also\n> embed it in the filename that stores archive, but the use of the\n> embedded metainfo makes it more robust against file names).  And\n> running \"git archive\" twice, with different versions of Git on\n> different architectures, should be reproducible as long as both\n> invokers expressed their desire to see the commit object name in the\n> archive by passing the commit, not its tree, to the command, and\n> they are using the same hash algorithm.\n\nIt's true that it makes it easy to look up, but I can say I've never\nused that functionality.  I think very few people actually know it\nexists.\n\n> Having said all that, I think stripping the commit object name (or\n> tags) is a better design.  Imagine that I see I created a tarball\n> earlier and published its hash, but later lost the tarball.  By not\n> allowing any commit object name in the archive, it would force me to\n> somehow name the tarball in such a way that I can tell which commit\n> I used to create it, e.g. \"git-e83c516331.tar\".  Other people can\n> notice the filename and without having seen the bytes in it, they\n> can try running \"git archive e83c516331\" in their repository and see\n> the output matches the hash I published earlier.  Having commit or\n> tag embedded in the archive would make it harder to do this kind of\n> things.\n\nMost people do this anyway (except with a tag name), so I don't think\nit's a big deal to have this as the primary mechanism.\n\n> By the way, other potentially interesting points are:\n> \n>  - Do we want to ignore \"export-subst\" for stability?\n\nI think that would be a good idea.  I'll add it in v2.\n\n>  - \"git archive\" can be invoked with pathspec to archive only a\n>    subset of paths.\n\nTrue.  I don't think that's a problem as long as we generate paths\ncorrectly.  I'll be sure to add tests for it, though.\n\n> > +Introduction to the Underlying Format\n> > +-------------------------------------\n> > ...\n> > +A global extended header sets metadata for the entire file, and a per-file\n> > +extended header applies to only the to which it corresponds.  A per-file\n> \n> \"only the to which\" -> \"only the file to which\"\n\nWill fix.\n\n> > +While pax extensions are widely supported by most modern versions of tar\n> > +(including versions on Windows and all major open-source OSes), some older\n> > +archivers and non-tar implementations which do not understand them typically\n> > +extract the extended headers as regular files.  Thus, it's helpful to have these\n> > +entries have reasonable permissions and unique names.\n> \n> Surely, and to make things reproducible, they shouldn't just be\n> reasonable and unique.  They should be exactly as we define in the\n> specification.\n\nYes, of course.  This is more to indicate why we've made the decisions\nto name them as they are and give them the permissions we did.\n\n> > +Every file serialized in the archive is serialized in lexicographical order by\n> > +its bytes.  A directory is always serialized before its contents, and a\n> \n> \"by its bytes\" -> \"by the bytes in its filename\" or something?\n> Surely we do not sort by contents ;-)\n\nGood point.  We should avoid ambiguity.\n\n> > +directory is never serialized with a trailing slash.  If a system uses a Unicode\n> > +encoding other than UTF-8, it encodes filenames as UTF-8.\n> \n> This is a bit hard to grok.  Do you mean there may be UTF-16 system\n> where the data in our tree objects, whose paths are recorded in UTF-8,\n> but \"git checkout\" of the tree may result in files in the native\n> filename on that system, i.e. UTF-16 not UTF-8?  And even on such a\n> system, running \"git archive\" would record paths in the archive in\n> UTF-8 (i.e. the same as what was in the tree object)?  Or do you\n> mean something stronger, like on a Latin-1 system with Latin-1\n> project that used Latin-1 as pathnames even in the tree objects,\n> when \"git archive\" produces an archive, the paths in it shall be\n> transcoded from the original Latin-1 pathnames to UTF-8?\n\nThis means if, on Windows, someone uses --add-file or\n--add-virtual-file, those paths will be encoded in UTF-8, not UTF-16.\n\n> > +Version Number\n> > +--------------\n> > +\n> > +The version number for this version is `ctar-v1`.\n> > +\n> > +Extended Headers\n> > +----------------\n> > +\n> > +Global Extended Header\n> > +~~~~~~~~~~~~~~~~~~~~~~\n> > +\n> > +The global extended header (record `g`) shall contain one header:\n> > +`CTAR.version`, which contains the version number specified above.\n> > +\n> > +The contents of the ustar header for the global extended header are as below,\n> > +except that the `name` field contains `pax_global_header`.\n> \n> \"as below\" meaning...?  The same as what is listed in \"Per-File\n> Extended Header\"?  There is no `name` field listed there, though.\n\nI'll make a clearer reference.\n\n> > +Per-File Extended Header\n> > +~~~~~~~~~~~~~~~~~~~~~~~~\n> > +\n> > +Each file has a per-file extended header.\n> > +\n> > +The following per-file extended header fields are included:\n> > +\n> > +|===\n> > +| Field Name   | When Present  | Value\n> > +\n> > +| `atime`      | always        | `0`\n> > +| `mtime`      | always        | `0`\n> > +| `size`       | always        | size of the data in bytes\n> > +| `path`       | always        | full path name of the file\n> \n> These are length-prefixed data, so we do not have to worry about\n> overly long pathnames or symlinks?\n\nCorrect.  This data can be arbitrarily long as long as all the metadata\ncan be encoded in a ustar header, so we're limited to at least several\ngigabytes or so.  I don't think anybody thinks of that as a practical\nlimitation on filenames or other metadata.\n\n> \"we because\" -> \"because\"\n\nWill fix.\n\n> > +we avoid explicitly declaring them as such and rely on the default archiver\n> > +behavior, which may be more sensible.\n> \n> So, do we or do we not store hdrcharset?  Producing Git does not know\n> if the pathnames stored in the tree it is asked to produce archive\n> for are not in UTF-8, so it assumes everything is in UTF-8 hence\n> does not see the need to add hdrcharset?\n\npax says that these values are UTF-8 if not specified.  If they're\nclearly not UTF-8, we use `hdcharset` and say they're binary.  If they\nlook like valid UTF-8, we don't use `hdrcharset` and pretend they are in\nfact UTF-8, in case somebody just likes causing discord by using\nWindows-1252 that looks like UTF-8.\n\n> In other words, we just store the contents of the blob that\n> represents the symbolic link there?  I wonder if we do anything\n> special if a blob, that is pointed at in an entry in a tree whose\n> mode bits are 120000, has NUL in it (should we teach fsck to flag\n> it, for example)?\n\nThis is the destination of the symlink, yes.  We can simply check for\nNUL and abort; I don't think that's an unreasonable behaviour in any\ncase.\n\n> The order of entries need to be specified when we aim for\n> bit-for-bit reproduceability, no?\n\nYes.  That's specified in the next section, where we say this:\n\n  When encoding the data for an extended header, all entries are sorted in order\n  by the byte values of their keys as encoded in UTF-8.  Duplicate keys are not\n  permitted.\n\nI'll make a reference to that section and describe it more clearly.\n\n> \"the header block\" -> \"the ustar header block\" to match the next\n> section, probably.\n\nI'll update that.\n\n> These are barebone header fields, not extended headers.  Do we want\n> to refer to some canonical sources so that readers understand that\n> unlike the extended headres we are talking about fixed-length fields? \n> The description above talks about \"padding\", but that of course\n> applies to fixed width columns.\n\nCorrect.  I'll mention that these are the values in the ustar header for\nthe extended header.  I'll also put some references in to the\ndocumentation.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"471735","messageId":"Y+LYQYMS7ruvRbNW@tapette.crustytoothpaste.net","threadId":"59204","inReplyTo":"230206.86lela4ebq.gmgdl@evledraar.gmail.com","subject":"Re: [RFC PATCH 1/1] Document a fixed tar format for interoperability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-02-07T23:01:21Z","receivedAt":"2023-02-07T23:01:32Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-02-06 at 22:18:47, Ævar Arnfjörð Bjarmason wrote:\n> Maybe there are other changes in the proposed spec that put it at odds\n> with such a goal, it's unclear to me if this is the only difference.\n\nAs mentioned in the description, that doesn't address trees, which have\nnever been consistent traditionally.  We also have bad permissions for\npax headers (always 666), which is something we've tried to fix before\nand is not something we want to carry on with.\n\nYou specifically sent a patch stating that we're not guaranteeing that\nformat, and I agree with that assessment.  I'm proposing a format that\nwe would guarantee and which does not have any of the historical baggage\nor warts that we don't want to keep.\n\nThis format also doesn't serialize timestamps; everything is at the\nEpoch.  Again, that's because serializing a commit and its tree or even\na tag and its commit would produce different results.\n\n> But I don't see why we need bit-for-bit compatible output between SHA-1\n> and SHA-256 git repos for the reasons noted in the linked-to reply, and\n> removing this will remove a *really useful* aspect of our tar format,\n> which is that you can grab an arbitrary tarball, and see what commit\n> it's produced from.\n\nTrue, but this is a highly obscure feature and I've never used it\noutside of testing.  If you want it, you can have it: you just want the\ndefault format, which serializes it in the header, and not the extremely\nrestricted format I'm proposing here which is designed to never ever\nchange.  We might well decide to add cool new features and useful\ninformation to the default format, but this one will be fixed forever.\n\n> Even if you want to retain SHA-1 and SHA-256 interop as far as tar is\n> concerned, an un-discussed alternative is to just stick the SHA-1 OID\n> into the SHA-256 archive.\n> \n> For repos that are migrated we envision having such a bi-directional\n> mapping anyway.\n> \n> And for those that started out as SHA-256, or where we no longer care\n> about compatibility with old SHA-1, we can just start including the\n> SHA-256 OID, as all compatibility concerns have gone away when we\n> stopped bothering to maintain the mapping, no?\n\nWhether SHA-1 or SHA-256 or both are present in the repo is a local\ndecision.  The transition plan specifically anticipates people either\npreferring one hash or the other in output.  The behaviour is not \"use\nSHA-1 if there's SHA-1 and use SHA-256 otherwise\", because even if\neveryone has SHA-256 and prefers it on their system, some people may\nstill have SHA-1 for historical reasons and that would lead to different\noutput.\n\nPart of this is because I anticipate that once the interop work is done,\nGitHub may transition repositories on the server to SHA-256 with SHA-1\ninterop for existing SHA-1 repositories.  People are still going to have\na fit if tarball data breaks at some point because the repository owner\ndecided to flip the default hash algorithm, and I'm specifically\nproposing a format that is not going to direct hordes of angry users in\nmy direction or the repository owner's in that case.  Lots of people are\ngoing to avoid switching the default hash algorithm if it breaks\ntarballs, and I specifically don't want to encourage people sticking\nwith SHA-1 for that reason.\n\n> This is the nth reference to \"the standard\". I think this would be\n> improved by linking to it, isn't it\n> https://pubs.opengroup.org/onlinepubs/9699919799/utilities/pax.html ?\n\nYeah, I'll do that.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"471758","messageId":"230208.86wn4sz8a3.gmgdl@evledraar.gmail.com","threadId":"59204","inReplyTo":"Y+LYQYMS7ruvRbNW@tapette.crustytoothpaste.net","subject":"Re: [RFC PATCH 1/1] Document a fixed tar format for interoperability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-08T11:07:44Z","receivedAt":"2023-02-08T11:50:10Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Tue, Feb 07 2023, brian m. carlson wrote:\n\n> [[PGP Signed Part:Undecided]]\n> On 2023-02-06 at 22:18:47, Ævar Arnfjörð Bjarmason wrote:\n>> Maybe there are other changes in the proposed spec that put it at odds\n>> with such a goal, it's unclear to me if this is the only difference.\n>\n> As mentioned in the description, that doesn't address trees, which have\n> never been consistent traditionally.\n\nYou mention \"[...]it produces identical results for identical trees,\nregardless of hash algorithm\". I'm not familiar with how we encode trees\ndifferently based on the hash algorithm. Do we stick the tree OID in\nthere somewhere, or is it something else?\n\nIOW do these trees vary within the same hash algorithm, or is it another\nspecial-case where we now produce a different tarball with SHA-1 and\nSHA-256 with commits, but also with trees?\n\nB.t.w. are there some options to tar(1) to make it dump these headers\nyou're describing? I coludn't find anything when looking, it looks like\nlibtar might support it, but I was hoping for something more compatible\nwith my lazyness :)\n\n> We also have bad permissions for pax headers (always 666), which is\n> something we've tried to fix before and is not something we want to\n> carry on with.\n\nI'm concerned that you're expanding the scope of a \"stable\" tar format\nto necessarily include one-off fixing various things we've regretted\nover the years.\n\nMaybe that *needs* to happen, but so far I don't see why, you've\ndescribed:\n\n* We include the OID in the metadata\n* Something like that, but for trees?\n* The sucky 0666 permissions we'd like to fix.\n* We don't serialize timestamps (which is now optional, depending on how\n  you invoke it)\n\nI really applaud your efforts here, but I don't see if that's the extent\nof the changes why the v1 and default format shouldn't be something that\nproduces identical results to \"git archive\" as it stands today.\n\nThen a v1/v2 is just this pseudocode, isn't it?\n\n \tswitch (version) {\n\tcase 1:\n\t\tbreak; /* warts and all */\n\tcase 2:\n\t\tinclude_oid = 0;\n\t\tsatanic_permissions = 0;\n\t\tno_timestamps = 1;\n\t\tbreak;\n\t}\n\nThe reason we haven't promised to support an archive format isn't\nbecause we didn't find every aspect of it aesthetically pleasing, but\nbecause we didn't want to commit to some bug-for-bug compatibility with\nwhatever the code is doing right now.\n\nNow that you've done the work to specify it, it turns out that a\nproposed format you'd like going forward is almost identical to what we\ncurrently emit, to the point that supporting that as a v1 seems rather\ntrivial (but again, I may still be missing something).\n\nWe have a huge long-tail of users in the wild, forcing those users to go\nthrough a one-time breakage of their existing archives if we could avoid\nthat by making v1 the current format seems entirely unnecessary.\n\nI totally see your point about wanting byte-for-byte the same archives\nout of the SHA-1 and SHA-256 version of the same repo, I think that's a\ngood goal, and it's also a good goal to get rid of these other warts.\n\nBut I don't see why it needs to be required, or even the default.\n\n> You specifically sent a patch stating that we're not guaranteeing that\n> format, and I agree with that assessment.  I'm proposing a format that\n> we would guarantee and which does not have any of the historical baggage\n> or warts that we don't want to keep.\n\nPer the above I just don't see why that's a criteria. I think we should\nbe weighing the benefits of changing the existing default \"git archive\"\noutput v.s. the cost of maintaining the delta to some v2 wart-less\nformat.\n\n> This format also doesn't serialize timestamps; everything is at the\n> Epoch.  Again, that's because serializing a commit and its tree or even\n> a tag and its commit would produce different results.\n\nThis seems like further scope creep, and in this particular case I don't\nsee how *always* doing that helps you with reputability.\n\nI.e. for the cases where we're now given a top-level tree it's obvious\nhow this helps, we encode the time(), so every time it's different.\n\nBut in the case where we get a commit (or tag) ID we use the timestamp\nin the commit (or tag?) envelope.\n\nWhen producing a release archive, or packing up a given commit that's\ntherefore going to be stable, even between SHA-1 and SHA-256, although\nthose two would differ if the OID is put in the header, but that's\nanother matter.\n\nIf I understand you correctly here you seem to be in pursuit of another\ngoal entirely, which is that you'd like the same output for different\ncommits if they're TREESAME.\n\nOr, if you have a bunch of release archives a very nice attribute of\nthis is that with a bunch of similar archives on the same FS you could\ne.g. benefit more from block-level deduplication.\n\nAll of which is cool, but I don't see why it needs to be a hard\nrequirement in the design.\n\n>> But I don't see why we need bit-for-bit compatible output between SHA-1\n>> and SHA-256 git repos for the reasons noted in the linked-to reply, and\n>> removing this will remove a *really useful* aspect of our tar format,\n>> which is that you can grab an arbitrary tarball, and see what commit\n>> it's produced from.\n>\n> True, but this is a highly obscure feature and I've never used it\n> outside of testing.\n\nI admit that's a bit obscure, but one of those things that really comes\nin handy when you need it, I vaguely remember using it once or twice and\nbeing very happy it was there.\n\nBut related to that is setting everything to epoch:0, doesn't that mean\nthat when you unpack say a release archive that in common filesystem\nbrowsers all of the files will be dated in the 70s, as opposed to the\ntime of release as it is now?\n\n> If you want it, you can have it: you just want the\n> default format, which serializes it in the header, and not the extremely\n> restricted format I'm proposing here which is designed to never ever\n> change.\n\nOkey, so I might have to take back much of what I said about, so you're\nnot opposed to supporting the current format as a \"v1\" or whatever,\nyou'd just like this propsoed \"v2\" (or \"vstable\", or whatever) to have\nsome \"blessed\" status.\n\nI just don't get why we wouldn't support both, if the delta is as small\nas seems to be the case. If that's right this \"v2\" is less \"extremely\nrestricted\" to our current \"v1\", and more \"almost identical\", just \"a\nbit less wart-y\".\n\n> We might well decide to add cool new features and useful\n> information to the default format, but this one will be fixed forever.\n\nI just don't see the target audience for that. As the issues that\nprompted these on-list discussions show we have people in the wild who\ndeeply care about the current format.\n\nThey probably care enough about that that we're likely to try to support\nthat forever, at least I don't see any currently proposed change to the\nformat that seems worth breaking things for those users.\n\nSo, if that's the case we'd have a v1 (current), this \"vstable\" (never\nchanges), and a v2 (v1 + extra neat things), etc.\n\nThen we'd be maintaining 3 formats instead of 2 formats (a \"v1\" and\n\"vunstable\").\n\n>> Even if you want to retain SHA-1 and SHA-256 interop as far as tar is\n>> concerned, an un-discussed alternative is to just stick the SHA-1 OID\n>> into the SHA-256 archive.\n>> \n>> For repos that are migrated we envision having such a bi-directional\n>> mapping anyway.\n>> \n>> And for those that started out as SHA-256, or where we no longer care\n>> about compatibility with old SHA-1, we can just start including the\n>> SHA-256 OID, as all compatibility concerns have gone away when we\n>> stopped bothering to maintain the mapping, no?\n>\n> Whether SHA-1 or SHA-256 or both are present in the repo is a local\n> decision.  The transition plan specifically anticipates people either\n> preferring one hash or the other in output.  The behaviour is not \"use\n> SHA-1 if there's SHA-1 and use SHA-256 otherwise\", because even if\n> everyone has SHA-256 and prefers it on their system, some people may\n> still have SHA-1 for historical reasons and that would lead to different\n> output.\n\nYes, but who has this issue in practice? In practice people are\nproducing archives as part of some release process.\n\nAs long as they keep using SHA-1 to make release they're fine, at some\npoint they'll switch over to SHA-256 by default, and then their new\nreleases will use SHA-256.\n\nIf they then have to for some reason go back to an old commit when SHA-1\nwas the default it might be a tiny hassle, but no more than doing the\nsame if the format had changed entirely.\n\n> Part of this is because I anticipate that once the interop work is done,\n> GitHub may transition repositories on the server to SHA-256 with SHA-1\n> interop for existing SHA-1 repositories.  People are still going to have\n> a fit if tarball data breaks at some point because the repository owner\n> decided to flip the default hash algorithm, and I'm specifically\n> proposing a format that is not going to direct hordes of angry users in\n> my direction or the repository owner's in that case.  Lots of people are\n> going to avoid switching the default hash algorithm if it breaks\n> tarballs, and I specifically don't want to encourage people sticking\n> with SHA-1 for that reason.\n\nI see that, I don't see how your plan isn't a perfect recipe for\ncreating the problem you're trying to avoid.\n\nYou have tarballs generated with the current format today, 3rd party\nsystems are dynamically downloading e.g. v1.0.0.tar.gz or whatever, and\nexpecting it to byte-for-byte match previous downloads.\n\nIf you're going to switch to some stable format surely that would either\nneed to involve massive one-off breakage, or you'd have some \"flag day\",\nfrom today all new archives are produced with the new \"stable\" method.\n\nIf that \"stable\" format is different (among other things, but the others\nseem equally trival) because you wanted to extract the OID from the\nformat for SHA-1 and SHA-256 interop, why can't the day the repository\nswitched to SHA-256 be that flag day?\n"},{"id":"471815","messageId":"Y+Q1v/PM6n93xvqu@tapette.crustytoothpaste.net","threadId":"59204","inReplyTo":"230208.86wn4sz8a3.gmgdl@evledraar.gmail.com","subject":"Re: [RFC PATCH 1/1] Document a fixed tar format for interoperability","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2023-02-08T23:52:31Z","receivedAt":"2023-02-08T23:52:37Z","isPatch":true,"sender":{"key":"sandals@crustytoothpaste.net","avatar":"https://avatars.githubusercontent.com/u/497054?v=4"},"body":"On 2023-02-08 at 11:07:44, Ævar Arnfjörð Bjarmason wrote:\n> \n> On Tue, Feb 07 2023, brian m. carlson wrote:\n> \n> > [[PGP Signed Part:Undecided]]\n> > On 2023-02-06 at 22:18:47, Ævar Arnfjörð Bjarmason wrote:\n> >> Maybe there are other changes in the proposed spec that put it at odds\n> >> with such a goal, it's unclear to me if this is the only difference.\n> >\n> > As mentioned in the description, that doesn't address trees, which have\n> > never been consistent traditionally.\n> \n> You mention \"[...]it produces identical results for identical trees,\n> regardless of hash algorithm\". I'm not familiar with how we encode trees\n> differently based on the hash algorithm. Do we stick the tree OID in\n> there somewhere, or is it something else?\n\nIf you pass a commit or tag on the command line, you get the timestamp\nof the commit or tag.  If you pass a tree, you get the current\ntimestamp.  Thus, whether the output is reproducible depends on the type\nof object you specify.\n\n> IOW do these trees vary within the same hash algorithm, or is it another\n> special-case where we now produce a different tarball with SHA-1 and\n> SHA-256 with commits, but also with trees?\n\nWhen we write an archive, we embed a comment with the commit object ID\n(see next response).  That's using the hash algorithm in the repository.\nIf we write an archive for a tree, no object ID is embedded.\n\n> B.t.w. are there some options to tar(1) to make it dump these headers\n> you're describing? I coludn't find anything when looking, it looks like\n> libtar might support it, but I was hoping for something more compatible\n> with my lazyness :)\n\nI don't think so.  However, you can see them with `git archive --format=tar HEAD | env -u LESSOPEN less -R`.\n\nThe body of the global header looks like this (my indentation):\n\n  52 comment=7ff60001dae72ac39783ca536a4b673862b28587\n\nIf you want to see what GNU tar produces, you can run `tar -cf - --posix --exclude .git . | env -u LESSOPEN less -R`:\n\n  30 mtime=1675633909.844009705\n  30 atime=1675895555.716075364\n  30 ctime=1675633909.844009705\n\n> I'm concerned that you're expanding the scope of a \"stable\" tar format\n> to necessarily include one-off fixing various things we've regretted\n> over the years.\n\nWell, yes, because if we're specifying a stable format, we should make it\nsomething we want to support long term.  Right now, we don't guarantee\nanything; if we find something unsatisfactory, we just fix it.\n\n> Then a v1/v2 is just this pseudocode, isn't it?\n> \n>  \tswitch (version) {\n> \tcase 1:\n> \t\tbreak; /* warts and all */\n> \tcase 2:\n> \t\tinclude_oid = 0;\n> \t\tsatanic_permissions = 0;\n> \t\tno_timestamps = 1;\n> \t\tbreak;\n> \t}\n\nAs I mentioned in the doc, there are multiple ways to encode various\nthings like lengths and the order of headers.  It's not immediately\nobvious from the code how our length encoding works, and that's the kind\nof code that could easily have a small refactor or bug fix break things\nreally badly.\n\nAdditionally, we, like most other pax implementations, just encode\nheaders in whatever order we thought was most expedient when\nimplementing, and sometimes they're emitted and sometimes they're not.\nThat's a really great recipe for behaviour that is extremely hard to\ntest and extremely hard to reproduce.  For example, we'd have to test\nthe interaction with long paths and symlinks, long paths and large\nfiles, and several other sets of variants to make sure that a minor\nrefactor doesn't change output.  The current logic of the code is very\nsubtle.\n\n> Now that you've done the work to specify it, it turns out that a\n> proposed format you'd like going forward is almost identical to what we\n> currently emit, to the point that supporting that as a v1 seems rather\n> trivial (but again, I may still be missing something).\n\nIt's relatively similar.  The format I'm proposing is much stricter and\nmore regular than what we do now.\n\nI'm thinking that the changes will be limited to writing three or four\nfunctions.  It's not terribly invasive, but there will be some departure\nfrom the existing code.\n\n> We have a huge long-tail of users in the wild, forcing those users to go\n> through a one-time breakage of their existing archives if we could avoid\n> that by making v1 the current format seems entirely unnecessary.\n\nBecause right now, the current code is not amenable to producing or\ntesting reproducible output.  Any significant refactor of the existing\ncode will result in an output change unless the author is extremely\ncareful, and I'm not comfortable guaranteeing the current format with\nthat caveat.  The reason the data hasn't changed is because such a\nrefactor hasn't happened yet.\n\nI'm specifically thinking about the length calculation in\n`strbuf_append_ext_header`, which is extremely magical, and the path\nsplitting in `get_path_prefix`.  Those are both extremely subtle and\nlogical places to perform a refactor or adjustment that might change\noutput in a very minor way for a tiny subset of files.\n\n> When producing a release archive, or packing up a given commit that's\n> therefore going to be stable, even between SHA-1 and SHA-256, although\n> those two would differ if the OID is put in the header, but that's\n> another matter.\n> \n> If I understand you correctly here you seem to be in pursuit of another\n> goal entirely, which is that you'd like the same output for different\n> commits if they're TREESAME.\n> \n> Or, if you have a bunch of release archives a very nice attribute of\n> this is that with a bunch of similar archives on the same FS you could\n> e.g. benefit more from block-level deduplication.\n> \n> All of which is cool, but I don't see why it needs to be a hard\n> requirement in the design.\n\nI think it's valuable to have the same input data produce the same\noutput.  That means that I can use Git to produce the archive, or some\nother tool implementing the same format, and it just works.  If GNU tar,\nlibgit2, or libarchive implemented the same format with an option,\npeople would also be able to produce an identical archive as long as\nthey excluded the files in `.gitignore` and `.git`.  That approach is\nvery valuable if you need to slightly modify the contents of the archive\nthat Git produced in a way not supported by --add-file (and Junio used\nto do that himself for Git releases before --add-file).\n\n> But related to that is setting everything to epoch:0, doesn't that mean\n> that when you unpack say a release archive that in common filesystem\n> browsers all of the files will be dated in the 70s, as opposed to the\n> time of release as it is now?\n\nYes.  That's also the case for current Rust crates and lots of other\nreproducible archives.  I've heard exactly zero complaints about that\nbehaviour since I implemented it in Cargo.  Looking back at the history,\napparently there's some broken behaviour with the actual Epoch and lldb\n(because 0 is a sentinel), but the change is just to switch to a\ntimestamp of 1 instead of 0, which I can do in the next version of my\npatch.  No other problems seem to have come up with using a fixed\ntimestamp.\n\nThe only place where I could imagine this being a problem is if you used\nMake in a directory after unpacking a new archive over the old one, but\nthat is a terrible idea in the first place since that leaves now-removed\nfiles from the old version behind which will probably cause your build\nto fail at some point.  In any event, because almost everyone uses\n`--prefix` with the version number for their archives, it's difficult to\neven perform that extraction over top anyway, and so it's unlikely that\nanyone actually does such a thing.\n\nOtherwise, there's typically no functional difference.\n\n> Okey, so I might have to take back much of what I said about, so you're\n> not opposed to supporting the current format as a \"v1\" or whatever,\n> you'd just like this propsoed \"v2\" (or \"vstable\", or whatever) to have\n> some \"blessed\" status.\n\nNo, I'm not opposed to supporting both.  There's \"default\" (v0 if you\nlike) and \"v1\".  If you say, \"I'd like a tarball\", you get what we\nproduce now (or what it changes to in the future).  If you say, \"I want\nbit-for-bit compatibility\", then you get v1.\n\n> I just don't get why we wouldn't support both, if the delta is as small\n> as seems to be the case. If that's right this \"v2\" is less \"extremely\n> restricted\" to our current \"v1\", and more \"almost identical\", just \"a\n> bit less wart-y\".\n\nRight, I think it's very easy to do.\n\n> I just don't see the target audience for that. As the issues that\n> prompted these on-list discussions show we have people in the wild who\n> deeply care about the current format.\n> \n> They probably care enough about that that we're likely to try to support\n> that forever, at least I don't see any currently proposed change to the\n> format that seems worth breaking things for those users.\n\nI don't think there's any purpose in guaranteeing the current format,\ngiven what I've said above about testability and the risk of breakage\nduring a refactor with the current code, and I don't think the project\nshould do that.  However, downstream users, including various forges, may\nwish to do so, and if so I wish them all the best.\n\n> If you're going to switch to some stable format surely that would either\n> need to involve massive one-off breakage, or you'd have some \"flag day\",\n> from today all new archives are produced with the new \"stable\" method.\n\nNope.  There's simply a new option to produce v1 archives and people\nswitch over as part of their normal build system maintenance, and\neventually nobody cares about the ancient versions depending on the old\nformat.\n-- \nbrian m. carlson (he/him or they/them)\nToronto, Ontario, CA\n"},{"id":"471827","messageId":"230209.86bkm3zkg1.gmgdl@evledraar.gmail.com","threadId":"59204","inReplyTo":"Y+Q1v/PM6n93xvqu@tapette.crustytoothpaste.net","subject":"Re: [RFC PATCH 1/1] Document a fixed tar format for interoperability","fromName":"Ævar Arnfjörð Bjarmason","fromEmail":"avarab@gmail.com","sentAt":"2023-02-09T00:35:15Z","receivedAt":"2023-02-09T01:39:18Z","isPatch":true,"sender":{"key":"avarab@gmail.com","avatar":"https://avatars.githubusercontent.com/u/45301?v=4"},"body":"\nOn Wed, Feb 08 2023, brian m. carlson wrote:\n\n> [[PGP Signed Part:Undecided]]\n> On 2023-02-08 at 11:07:44, Ævar Arnfjörð Bjarmason wrote:\n>> \n>> On Tue, Feb 07 2023, brian m. carlson wrote:\n>> \n>> > [[PGP Signed Part:Undecided]]\n>> > On 2023-02-06 at 22:18:47, Ævar Arnfjörð Bjarmason wrote:\n>> >> Maybe there are other changes in the proposed spec that put it at odds\n>> >> with such a goal, it's unclear to me if this is the only difference.\n>> >\n>> > As mentioned in the description, that doesn't address trees, which have\n>> > never been consistent traditionally.\n>> \n>> You mention \"[...]it produces identical results for identical trees,\n>> regardless of hash algorithm\". I'm not familiar with how we encode trees\n>> differently based on the hash algorithm. Do we stick the tree OID in\n>> there somewhere, or is it something else?\n>\n> If you pass a commit or tag on the command line, you get the timestamp\n> of the commit or tag.  If you pass a tree, you get the current\n> timestamp.  Thus, whether the output is reproducible depends on the type\n> of object you specify.\n\nAh, that's all. So just the case documented in the openign pargraphs of\nthe DESCRIPTION of git-archive.\n\nI think it's probably a good thing to enforce the epoch:0 for the cases\nwhere the time varies now, or maybe we'd say \"if you want stable\narchives, point to a commit or tag\", but that's ultimately a UX issue.\n\nBut regardless of any other points we may differ on I think if that's\nwhat you meant this section of the doc is really confusing:\n\n\tIt serializes all timestamps as the Epoch, which produces\n\tidentical results whether the tree is serialized as a tree,\n\tcommit, or tag.\n\nBecause we do produce identical results now for commit or tag, so what's\nepoch:0 for those solving?\n\n>> IOW do these trees vary within the same hash algorithm, or is it another\n>> special-case where we now produce a different tarball with SHA-1 and\n>> SHA-256 with commits, but also with trees?\n>\n> When we write an archive, we embed a comment with the commit object ID\n> (see next response).  That's using the hash algorithm in the repository.\n> If we write an archive for a tree, no object ID is embedded.\n\nYeah, it was answered in the preceding. I.e. there's no special \"tree\"\nmagic here beyond the long-standing dynamic epoch behavior for the\n\"tree\" case, thanks.\n\n>> B.t.w. are there some options to tar(1) to make it dump these headers\n>> you're describing? I coludn't find anything when looking, it looks like\n>> libtar might support it, but I was hoping for something more compatible\n>> with my lazyness :)\n>\n> I don't think so.  However, you can see them with `git archive --format=tar HEAD | env -u LESSOPEN less -R`.\n>\n> The body of the global header looks like this (my indentation):\n>\n>   52 comment=7ff60001dae72ac39783ca536a4b673862b28587\n>\n> If you want to see what GNU tar produces, you can run `tar -cf - --posix --exclude .git . | env -u LESSOPEN less -R`:\n>\n>   30 mtime=1675633909.844009705\n>   30 atime=1675895555.716075364\n>   30 ctime=1675633909.844009705\n\n*nod*, some variant of that is how I've been looking at this,\nthanks.\n\n>> I'm concerned that you're expanding the scope of a \"stable\" tar format\n>> to necessarily include one-off fixing various things we've regretted\n>> over the years.\n>\n> Well, yes, because if we're specifying a stable format, we should make it\n> something we want to support long term.  Right now, we don't guarantee\n> anything; if we find something unsatisfactory, we just fix it.\n\nI won't repeat myself too much here, but we disagree on that.\n\nIf it's e.g. just the 666 permissions or whatever I don't see the value\nof changing that just for the sake of asthetics. Or to satisfy some more\nobscure use-cases (which must by definition be pretty obscure, as \"git\narchive\" is in wide production use now with those warts).\n\nBut having said that...\n\n>> Then a v1/v2 is just this pseudocode, isn't it?\n>> \n>>  \tswitch (version) {\n>> \tcase 1:\n>> \t\tbreak; /* warts and all */\n>> \tcase 2:\n>> \t\tinclude_oid = 0;\n>> \t\tsatanic_permissions = 0;\n>> \t\tno_timestamps = 1;\n>> \t\tbreak;\n>> \t}\n>\n> As I mentioned in the doc, there are multiple ways to encode various\n> things like lengths and the order of headers.  It's not immediately\n> obvious from the code how our length encoding works, and that's the kind\n> of code that could easily have a small refactor or bug fix break things\n> really badly.\n\n...I'm willing to be convinced otherwise.\n\nSo yes, if e.g. we find that to implement your proposed format we'd need\nto change some tricky existing archiving code from 500 current lines of\ntricky code, to either 100 neat lines in the new \"stable\" format, or 600\nif we keep both, I'd be much less included to argue for this.\n\nBut I also think your proposal isn't doing itself any favors by pointing\nout things that are clearly trivial to support without changes as the\nthings we should be \"fixing\" for a new stable format, e.g. modes and\nepochs.\n\n> Additionally, we, like most other pax implementations, just encode\n> headers in whatever order we thought was most expedient when\n> implementing, and sometimes they're emitted and sometimes they're not.\n> That's a really great recipe for behaviour that is extremely hard to\n> test and extremely hard to reproduce.  For example, we'd have to test\n> the interaction with long paths and symlinks, long paths and large\n> files, and several other sets of variants to make sure that a minor\n> refactor doesn't change output.  The current logic of the code is very\n> subtle.\n\nWhy do we need to exhaustively specify or test something we're not\ntesting or specifying now? Wouldn't something like this in some select\nplaces in archive-tar.c be sufficient (the write_tar_entry2() being some\nnew format):\n\n\tdiff --git a/archive-tar.c b/archive-tar.c\n\tindex f8fad2946ef..4f8ca02a82a 100644\n\t--- a/archive-tar.c\n\t+++ b/archive-tar.c\n\t@@ -257,8 +257,17 @@ static int write_tar_entry(struct archiver_args *args,\n\t \tunsigned long size_in_header;\n\t \tint err = 0;\n\t \n\t+\tswitch (args->version) {\n\t+\tcase 1: break;\n\t+\tcase 2: return write_tar_entry2(...);\n\t+\t}\n\t+\n\t \tmemset(&header, 0, sizeof(header));\n\t \n\t+\t/*\n\t+\t * This logic implements the tar v1 format, don't even change\n\t+\t * this, change write_tar_entry{2,3,...}() etc instead.\n\t+\t */\n\t \tif (S_ISDIR(mode) || S_ISGITLINK(mode)) {\n\t \t\t*header.typeflag = TYPEFLAG_DIR;\n\t \t\tmode = (mode | 0777) & ~tar_umask;\n\tdiff --git a/archive.h b/archive.h\n\tindex 08bed3ed3af..a2f72bbea7e 100644\n\t--- a/archive.h\n\t+++ b/archive.h\n\t@@ -24,6 +24,7 @@ struct archiver_args {\n\t \tint compression_level;\n\t \tstruct string_list extra_files;\n\t \tstruct pretty_print_context *pretty_ctx;\n\t+\tunsigned int tar_version;\n\t };\n\t \n\t /* main api */\n\nThis is analogous to what we did for \"git patch-id\n[--unstable|--stable]\", the latter is more specified, the former is\n\"we're keeping around whatever we did before\".\n\nBut this is all under my ongoing assumption that the main target\naudience for archive stability that cares about any of this someone with\nexisting archives who'd like to not go through some one-off migration if\nthey can help it.\n\n>> Now that you've done the work to specify it, it turns out that a\n>> proposed format you'd like going forward is almost identical to what we\n>> currently emit, to the point that supporting that as a v1 seems rather\n>> trivial (but again, I may still be missing something).\n>\n> It's relatively similar.  The format I'm proposing is much stricter and\n> more regular than what we do now.\n>\n> I'm thinking that the changes will be limited to writing three or four\n> functions.  It's not terribly invasive, but there will be some departure\n> from the existing code.\n\nRight, all of this sounds like mostly changes to write_tar_entry(),\nprepare_header() and write_extended_header() in archive-tar.c\n\n>> We have a huge long-tail of users in the wild, forcing those users to go\n>> through a one-time breakage of their existing archives if we could avoid\n>> that by making v1 the current format seems entirely unnecessary.\n>\n> Because right now, the current code is not amenable to producing or\n> testing reproducible output.  Any significant refactor of the existing\n> code will result in an output change unless the author is extremely\n> careful, and I'm not comfortable guaranteeing the current format with\n> that caveat.  The reason the data hasn't changed is because such a\n> refactor hasn't happened yet.\n>\n> I'm specifically thinking about the length calculation in\n> `strbuf_append_ext_header`, which is extremely magical, and the path\n> splitting in `get_path_prefix`.  Those are both extremely subtle and\n> logical places to perform a refactor or adjustment that might change\n> output in a very minor way for a tiny subset of files.\n\nYeah, this sort of thing is all stuff I'm very much willing to be sold\non, and I think if the proposal had led with some of these things...\n\n...but on the other hand the combined length of those two functions is\naround 30 lines, and if we just \"freeze\" it (along with the existing\nheader writing code) doesn't look like something that would be much of a\nhassle to just keep around, but maybe it all adds up.\n\n>> When producing a release archive, or packing up a given commit that's\n>> therefore going to be stable, even between SHA-1 and SHA-256, although\n>> those two would differ if the OID is put in the header, but that's\n>> another matter.\n>> \n>> If I understand you correctly here you seem to be in pursuit of another\n>> goal entirely, which is that you'd like the same output for different\n>> commits if they're TREESAME.\n>> \n>> Or, if you have a bunch of release archives a very nice attribute of\n>> this is that with a bunch of similar archives on the same FS you could\n>> e.g. benefit more from block-level deduplication.\n>> \n\n>> requirement in the design.\n>\n> I think it's valuable to have the same input data produce the same\n> output.  That means that I can use Git to produce the archive, or some\n> other tool implementing the same format, and it just works.  If GNU tar,\n> libgit2, or libarchive implemented the same format with an option,\n> people would also be able to produce an identical archive as long as\n> they excluded the files in `.gitignore` and `.git`.  That approach is\n> very valuable if you need to slightly modify the contents of the archive\n> that Git produced in a way not supported by --add-file (and Junio used\n> to do that himself for Git releases before --add-file).\n\nThey'd produce the same output if they just read the timestamp in the\ncommit envelope. GNU tar has a --mtime, so isn't this a matter of a\none-liner to \"git show\" that spews out the envelope time in a format it\ngroks?\n\n>> But related to that is setting everything to epoch:0, doesn't that mean\n>> that when you unpack say a release archive that in common filesystem\n>> browsers all of the files will be dated in the 70s, as opposed to the\n>> time of release as it is now?\n>\n> Yes.  That's also the case for current Rust crates and lots of other\n> reproducible archives.  I've heard exactly zero complaints about that\n> behaviour since I implemented it in Cargo.\n\nI don't think someone unpacking a bunch of files from different\narchives, doing \"sort by date\" in their file browser, and finding that\nit's all from the 70s is something people would open bug reports about.\n\nIt's just a small fringe UX benefit, but it also seems like a small\nmatter to keep the non-zero epochs.\n\n> Looking back at the history,\n> apparently there's some broken behaviour with the actual Epoch and lldb\n> (because 0 is a sentinel), but the change is just to switch to a\n> timestamp of 1 instead of 0, which I can do in the next version of my\n> patch.  No other problems seem to have come up with using a fixed\n> timestamp.\n\nHah, I wonder if in 10 years some other system will discover that epoch\n1 is used as a workaround for such issues, and hardcode that 1 is also a\nmeaningless sentinel value, so the next format will have epoch 2 etc.... :)\n\n> The only place where I could imagine this being a problem is if you used\n> Make in a directory after unpacking a new archive over the old one, but\n> that is a terrible idea in the first place since that leaves now-removed\n> files from the old version behind which will probably cause your build\n> to fail at some point.\n\nI hadn't thought of that, but that's another benefit to meaningful\ntimestamps.\n\nI wouldn't call it a \"terrible idea\", it's a really well supported\npattern in \"make\", and this project with its messy Makefile infra even\nsupports it at least 95% of the time :)\n\n(There's some messy picking up of old files logic with Documentation/*\nin particular).\n\n> in any event, because almost everyone uses\n> `--prefix` with the version number for their archives, it's difficult to\n> even perform that extraction over top anyway,[...]\n\nThere's going to be a lot of use cases on the edges, e.g. having\nmultiple prefix-unpacked archives, and wanting to use \"find\" with\n\"mtime\" to run a meaningful search over them.\n\n> and so it's unlikely that anyone actually does such a thing.\n\nI think it's unlikely that we'll ever have more than a double-digit\nnumber of people who'll implement any proposed archive format we come up\nwith, whereas the consumers will be a lot more digits :)\n\n> Otherwise, there's typically no functional difference.\n\nI think we're going to disagre on this point, but I think your propsal\nwould be improved if it clearly carved out this case, because it's\ncurrently claiming to be in service reproducible archves, which isn't\nreally the case...\n\n> [...]\n>> I just don't see the target audience for that. As the issues that\n>> prompted these on-list discussions show we have people in the wild who\n>> deeply care about the current format.\n>> \n>> They probably care enough about that that we're likely to try to support\n>> that forever, at least I don't see any currently proposed change to the\n>> format that seems worth breaking things for those users.\n>\n> I don't think there's any purpose in guaranteeing the current format,\n> given what I've said above about testability and the risk of breakage\n> during a refactor with the current code, and I don't think the project\n> should do that.  However, downstream users, including various forges, may\n> wish to do so, and if so I wish them all the best.\n>\n>> If you're going to switch to some stable format surely that would either\n>> need to involve massive one-off breakage, or you'd have some \"flag day\",\n>> from today all new archives are produced with the new \"stable\" method.\n>\n> Nope.  There's simply a new option to produce v1 archives and people\n> switch over as part of their normal build system maintenance, and\n> eventually nobody cares about the ancient versions depending on the old\n> format.\n\nOr more likely they'll never make that switch, or even read the docs\nabout this. If they did in the first place that we'd have had nobody\nrely on this implementation detail, but people do.\n\nAnd once we change the \"unstable\" format we've now given ourselves\nlicense to more freely change their systems break just as they did\nbefore.\n\nMaybe I'm just wrong here, and clearly in the long term your approach\nwill win out. If anyone's using these features in git in 10-20 years\nwe'll probably pat ourselves on the back in having gone through some\nsort of transition here (although I think if it's not the default it\nprobably won't happen).\n\nI just don't see why the problem at hand calls for a spec, a new format\nto migrate to etc., as opposed to just adding comments or something like\nthat to the existing code saying \"hey, people rely on this, consider\nformat changes carefully\".\n\nThen we kick the can down the road on any transition, which we can still\ndo if and when we find a really compelling reason for why we must change\nthe format.\n\nSo far the only \"we must change it\" that we have now AFAICT is the issue\nof the OID in the comment, which for the reasons upthread I think is\npretty much a non-issue, and in any case for those that are willing to\nmigrate to a new format much more easily solved by a\n--dont-add-the-oid-comment.\n\nBut if you're willing to work on all of this that's neat, although I do\nreserve the right to bikeshed about some of these changes :)\n\nIt just seems rather orthogonal to the in-the-wild problem, is all.\n\nCheers.\n"}]}