{"thread":{"id":"65561","subject":"Git generated tarballs and Debian","startedAt":"2026-04-28T08:50:09Z","lastAt":"2026-04-29T07:30:12Z","messageCount":6,"participants":["Simon Richter","brian m. carlson","Theodore Tso","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"542416","messageId":"9030b26d-02ed-4452-b212-a69a4ff21e2d@hogyros.de","threadId":"65561","inReplyTo":null,"subject":"Git generated tarballs and Debian","fromName":"Simon Richter","fromEmail":"simon.richter@hogyros.de","sentAt":"2026-04-28T08:40:05Z","receivedAt":"2026-04-28T08:50:09Z","isPatch":false,"body":"Hi,\n\nin Debian, we're shipping \"original\" tarballs for each software package, \nand the Debian specific changes in a separate file.\n\nHistorically, this users could do a bitwise comparison of the original \ntarball and the one in Debian to verify that these were unchanged.\n\nWith git, some authors have stopped releasing official tarballs, so \nwe're using git-archive a lot -- but this is reproducible only by \naccident. GitHub also prepares some release tarballs that may or not be \nbitwise identical to what git archive produces.\n\nI've written a small tool that generates the tree checksum for a given \ntarball (running inside a SECCOMP environment, not writing anything to \ndisk), that already goes a long way to make tarballs verifiable: one can \ncheck whether that ID is the same as the one mentioned in a commit (and \nthe comment inside a git-archive generated tarball is helpful in finding \nwhich commit).\n\nThe downsides of that are:\n\n1. that you still need a copy of the commit to verify it, as it's not \nincluded in the tarball.\n\nWe could add an ancillary file that contains the commit object (its \nchecksum being reproducible, and containing the tree checksum) and \npossibly a signed tag object as well, so that is solvable inside Debian.\n\nAnother option would be to extend the git-archive format to include them \nas a (longer) comment in the global pax header.\n\n2. that it doesn't work for submodules\n\nWhat we do currently is generate multiple archives with different \nprefixes, and concatenate them using tar. That loses all the pax global \nheaders though, so commit information is lost. In addition, putting the \nactual contents into a subdirectory instead of a commit reference means \nthat generating the tree object from the tarball contents means the \nchecksum does not match.\n\nWhat we could do is generate multiple archives, and keep them separate, \nbut the Debian toolchain can only unpack additional archives into a \ndirect subdirectory of the main archive (e.g. \"orig.tar.gz\" gets \nunpacked to \"foo-1.0\", then \"orig-addon.tar.gz\" gets unpacked into \n\"foo-1.0/addon\"). We can fix _that_ with symlinks, but it gets more and \nmore hacky.\n\nOne thing we could do inside git here is add a method to create archives \nthat include submodules (that gets rid of the concatenation), but in \norder for this to be easily verifiable, I still need to know where \nsubmodules are and what their commit objects are (so I know the commit \nchecksum and can verify the tree checksum).\n\nThe goal is to extend what I can already do inside the Linux kernel:\n\n$ git rev-parse HEAD\n94dfcc4a99b0cece77e73dc3011284050f95da89\n$ git rev-parse HEAD^{tree}\n2d14d43ce9f062160262f4e4f162f5ff0ed91a5e\n$ git archive --format=tar HEAD | git-treeof\nCommit-Hint: 94dfcc4a99b0cece77e73dc3011284050f95da89\nTree-SHA1: 2d14d43ce9f062160262f4e4f162f5ff0ed91a5e\n\nso the \"Commit-Hint\" can become a stronger statement \"I have seen a \ncommit object with this checksum that actually refers to the correct \ntree\", and to allow this to work for repositories with submodules.\n\nDoes it make sense to extend git here to allow this, or should I try to \nsolve this entirely within Debian?\n\n    Simon\n"},{"id":"542424","messageId":"afCLFJX86yEPKKfk@fruit.crustytoothpaste.net","threadId":"65561","inReplyTo":"9030b26d-02ed-4452-b212-a69a4ff21e2d@hogyros.de","subject":"Re: Git generated tarballs and Debian","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2026-04-28T10:25:24Z","receivedAt":"2026-04-28T10:25:27Z","isPatch":false,"body":"On 2026-04-28 at 08:40:05, Simon Richter wrote:\n> Hi,\n> \n> in Debian, we're shipping \"original\" tarballs for each software package, and\n> the Debian specific changes in a separate file.\n> \n> Historically, this users could do a bitwise comparison of the original\n> tarball and the one in Debian to verify that these were unchanged.\n> \n> With git, some authors have stopped releasing official tarballs, so we're\n> using git-archive a lot -- but this is reproducible only by accident. GitHub\n> also prepares some release tarballs that may or not be bitwise identical to\n> what git archive produces.\n\nI'll just note that we don't make any guarantees that `git archive`\nproduces identical output across versions.  Incorrectly making that\nassumption broke kernel.org when we changed the format in the past.\n\nAlso, if you use `export-subst`, then it's possible to emit short object\nIDs, which can differ in length depending on how many objects are in the\nrepository.  It's also possible to use zlib or pigz instead of gzip to\nproduce tarballs, in which case the compressed data will also differ.\n\nI had intended to create and emit a standard, reproducible format for\n`git archive`, but never got around to finishing that.  Perhaps I'll try\nto pick it up at some point; I expect it will be easier to implement now\nthat we have Rust support in the tree.\n\nWhen I was one of the maintainer of Git LFS, we intentionally produced\nsource tarballs specifically to emit bit-for-bit identical artifacts.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"542425","messageId":"a54d57b6-9270-406a-9056-ffaa939c6c21@hogyros.de","threadId":"65561","inReplyTo":"afCLFJX86yEPKKfk@fruit.crustytoothpaste.net","subject":"Re: Git generated tarballs and Debian","fromName":"Simon Richter","fromEmail":"simon.richter@hogyros.de","sentAt":"2026-04-28T11:32:37Z","receivedAt":"2026-04-28T11:32:43Z","isPatch":false,"body":"Hi,\n\nOn 4/28/26 7:25 PM, brian m. carlson wrote:\n\n> I'll just note that we don't make any guarantees that `git archive`\n> produces identical output across versions.  Incorrectly making that\n> assumption broke kernel.org when we changed the format in the past.\n\nExactly -- that's why I read the tarball and calculate the checksum of \nthe corresponding tree object, but we have a few cases where we need \nextra information that isn't in the archive, and I'm wondering where to \nput that extra information: inside the archive itself, or into an extra \nfile.\n\n> Also, if you use `export-subst`, then it's possible to emit short object\n> IDs, which can differ in length depending on how many objects are in the\n> repository.  It's also possible to use zlib or pigz instead of gzip to\n> produce tarballs, in which case the compressed data will also differ.\n\nexport-subst breaks verification completely as soon as a blob changes.\n\nCompression isn't an issue, because we're comparing tree checksums.\n\n    Simon\n"},{"id":"542426","messageId":"20260428115017.GA71700@macsyma-wired.lan","threadId":"65561","inReplyTo":"afCLFJX86yEPKKfk@fruit.crustytoothpaste.net","subject":"Re: Git generated tarballs and Debian","fromName":"Theodore Tso","fromEmail":"tytso@mit.edu","sentAt":"2026-04-28T11:50:17Z","receivedAt":"2026-04-28T11:51:37Z","isPatch":false,"body":"On Tue, Apr 28, 2026 at 10:25:24AM +0000, brian m. carlson wrote:\n> \n> I'll just note that we don't make any guarantees that `git archive`\n> produces identical output across versions.  Incorrectly making that\n> assumption broke kernel.org when we changed the format in the past.\n> \n> Also, if you use `export-subst`, then it's possible to emit short object\n> IDs, which can differ in length depending on how many objects are in the\n> repository.  It's also possible to use zlib or pigz instead of gzip to\n> produce tarballs, in which case the compressed data will also differ.\n\nThis is what I've been using to try get reproducible tarballs for\ne2fprogs:\n\ngit archive --prefix=e2fsprogs-${ver}/ ${commit} | gzip -9n > $fn\n\n,,, where $commit is a signed git tag.\n\nI know that in the past, using --format=tgz has broken based on\ndifferent compression parameters used by git (and whether it used an\nexternal or internal compressor).  I also know that if $commit is a\ntree-id, this can result in the timestamps being not reproduible.  I\nalso don't use export-subst.\n\nThere is also the difference in the prefix used by github and gitlab,\nbut that's arguably not git's fault.\n\nWhat other gotchas are there?  How is this likely to be inconsistent\nin the future?  How much work is there to provide that guarantee in\nthe future?\n\n   \t    \t    \t \t      \t  - Ted\n\nP.S.  Although I use pristine-tar in Debian because I didn't want to\ncount on git-archive being reproducible.  But it would be lovely if I\ncould make that guarantee starting on a particular git version.\n"},{"id":"542451","messageId":"afEko8bJnN3IOxcl@fruit.crustytoothpaste.net","threadId":"65561","inReplyTo":"20260428115017.GA71700@macsyma-wired.lan","subject":"Re: Git generated tarballs and Debian","fromName":"brian m. carlson","fromEmail":"sandals@crustytoothpaste.net","sentAt":"2026-04-28T21:20:35Z","receivedAt":"2026-04-28T21:20:37Z","isPatch":false,"body":"On 2026-04-28 at 11:50:17, Theodore Tso wrote:\n> On Tue, Apr 28, 2026 at 10:25:24AM +0000, brian m. carlson wrote:\n> > \n> > I'll just note that we don't make any guarantees that `git archive`\n> > produces identical output across versions.  Incorrectly making that\n> > assumption broke kernel.org when we changed the format in the past.\n> > \n> > Also, if you use `export-subst`, then it's possible to emit short object\n> > IDs, which can differ in length depending on how many objects are in the\n> > repository.  It's also possible to use zlib or pigz instead of gzip to\n> > produce tarballs, in which case the compressed data will also differ.\n> \n> This is what I've been using to try get reproducible tarballs for\n> e2fprogs:\n> \n> git archive --prefix=e2fsprogs-${ver}/ ${commit} | gzip -9n > $fn\n> \n> ,,, where $commit is a signed git tag.\n> \n> I know that in the past, using --format=tgz has broken based on\n> different compression parameters used by git (and whether it used an\n> external or internal compressor).  I also know that if $commit is a\n> tree-id, this can result in the timestamps being not reproduible.  I\n> also don't use export-subst.\n> \n> There is also the difference in the prefix used by github and gitlab,\n> but that's arguably not git's fault.\n> \n> What other gotchas are there?  How is this likely to be inconsistent\n> in the future?  How much work is there to provide that guarantee in\n> the future?\n\nWe could in theory provide reproducible tar output, but again, nobody\nhas committed to doing that yet.  If we did that, we would add a special\noption that produces, say, reproducible format v1, and if we needed to\nmake a change, then we would provide reproducible format v2, and so on.\nThat would also necessarily disable `export-subst`, since that\nintroduces non-reproducibility.\n\nOf course, if you're using filters, then those can also be a source of\nissues.  Git LFS doesn't have that problem because it identifies objects\nby SHA-256 hash, but there are many people who _do_ have unreproducible\nfilters (for instance, inserting the current date and time), so we might\nneed to disable those as well.\n\nMy approach would be to document a format and then implement it and\nthoroughly test it.  I was hoping, in fact, to define a format that\nother tarball-generating implementations could _also_ implement, since\nreproducible tarballs are also an issue for other tools like Cargo.  I\nhave some code somewhere in some branch to do part of that, but it ran\ninto complexities due to handling `--add-file` and `--add-virtual-file`,\nwhich are always appended, when we'd actually want them inserted in\nsorted order.  This will almost certainly be easier to write in Rust\nbecause of better data structures and easier unit testing, so I may pick\nit up at some point.\n\nWe cannot guarantee providing reproducible tar.gz output because we\ndon't control the compressor.  If gzip tomorrow decides to release a\nversion that produces a different bitstream for some output, we're not\ngoing to ship our own gzip.  Same goes for zlib, especially since\ndifferent distros use different libraries to implement that interface.\n\nOur zip files also have the undesirable attribute that they contain both\na local and a UTC timestamp, so the timezone is a problem.  This bit us\nat $DAYJOB when we started generating archives inside a container, since\nthe local timezone changed to UTC[0] and thus the archives were no\nlonger bit-for-bit identical.  Of course, zip files also have\ncompression, which adds additional potential for reproducibility\nproblems.\n\n[0] Yes, I know all servers should be using UTC and I agree, but that\ndecision got made well before my time.\n-- \nbrian m. carlson (they/them)\nToronto, Ontario, CA\n"},{"id":"542459","messageId":"20260429073002.GA717507@coredump.intra.peff.net","threadId":"65561","inReplyTo":"20260428115017.GA71700@macsyma-wired.lan","subject":"Re: Git generated tarballs and Debian","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2026-04-29T07:30:02Z","receivedAt":"2026-04-29T07:30:12Z","isPatch":false,"body":"On Tue, Apr 28, 2026 at 07:50:17AM -0400, Theodore Tso wrote:\n\n> I know that in the past, using --format=tgz has broken based on\n> different compression parameters used by git (and whether it used an\n> external or internal compressor).  I also know that if $commit is a\n> tree-id, this can result in the timestamps being not reproduible.  I\n> also don't use export-subst.\n> \n> There is also the difference in the prefix used by github and gitlab,\n> but that's arguably not git's fault.\n> \n> What other gotchas are there?  How is this likely to be inconsistent\n> in the future?  How much work is there to provide that guarantee in\n> the future?\n\nThe biggest unexpected change I recall was caused by a bug/compatibility\nfix. 22f0dcd963 (archive-tar: split long paths more carefully,\n2013-01-05) changed how some long paths were represented to be more\ncompatible between GNU tar and NetBSD. Lots of Homebrew recipes, etc,\nwere broken when GitHub deployed a version of Git with that commit.\n\nI think there was a more recent one in 2023-ish caused by some\ngzip-related changes (but it was after my time and I don't know the\ndetails).\n\nI feel like there was one in the middle, too, but I'm having trouble\ndigging it up (I think GitHub reverted 22f0dcd963 at the time and\nfinally reinstated it in 2017 after a warning period, so that might be\nwhat I'm thinking of).\n\nBut I'm not sure how often we'd do fixes like that. Not a lot, as the\ntar code is pretty stable. But is 82a46af13e (archive-tar: fix pax\nextended header length calculation, 2019-08-17), for example, likely to\nhave changed hashes for some repos? Probably.\n\nSo I think if you really want byte-for-byte compatibility of git-archive\nyou have to cement the behavior, bugs and all, behind some kind of\nversion flag, and every possible behavior change has to be analyzed for\na potential version bump.\n\nThough breaking some obscure cases once every 5-10 years is maybe not\n_so_ bad, and we can live with it. ;)\n\n-Peff\n"}]}