Stability of git-archive, breaking (?) the Github universe, and a possible solution

For those that haven't seen, github changed its checksums for all "source code" artifacts attached to any git repository with tags. This change is now reverted due to widespread breakage -- and the lack of advance warning. The technical details of the change appear simple: they upgraded git.

Probably the main discussion, complete with Github employees from this mailing list responding:

https://github.com/bazel-contrib/SIG-rules-authors/issues/11#issuecomment-1409438954

Consequences of that discussion, attempting to mitigate issues by warning people that it already happened:

https://github.blog/changelog/2023-01-30-git-archive-checksums-may-change/

And where I first saw it: https://github.com/mesonbuild/wrapdb/pull/884

Historically speaking, git-archive has been stable minus... a bug fix or two in rare cases, specifically relating to an inability to transcribe the contents of the git repo at all, I think? And the other factor is the compression algorithm used, which is generally GNU gzip, and historically whatever the system `gzip` command is.

And gzip is a stable format. It's a worn-out, battle-weary format, even -- it's not the best at compressing, and it's not the best at decompressing, and "all the cool kids" are working on cooler formats, such as zstd which does indeed regularly change its byte output between versions. But the advantage of gzip is that it's good *enough*, and it's probably *everywhere*, and it's *reliable*.

GNU gzip is reproducible. busybox gzip was fixed to agree with GNU gzip (this is relevant to the handful of people running software forges on, say, Alpine Linux):

https://reproducible-builds.org/reports/2019-08/#upstream-news

...

Nevertheless, I've seen the sentiment a few times that git doesn't like committing to output stability of git-archive, because it isn't officially documented (but it's not entirely clear what the benefits of changing are). And yet, git endeavors to do so, in order to prevent unnecessary breakage of people who embody Hyrum's Law and need that stability.

Even with the new change to the compressor, git-archive is still reproducible, it's the internal gzip compressor that isn't. (This may be fixable, possibly by embedding an implementation from busybox or from GNU gzip? I'm not going to discuss that right now, though I think it's an interesting avenue of exploration.)

I've thought about this now and then over the last couple of years, because I think I have a reasonable compromise that might make everyone (or at least most people) happy, and now seems like a good idea to mention it.

What does everyone think about offering versioned git-archive outputs? This could be user-selectable as an option to `git archive`, but the main goal would be to select a good versioned output format depending on what is being archived. So:

- first things first, un-default the internal compressor again
- implement a v2 archive format, where the internal compressor is the
  default -- no other changes
- teach git to select an archive format based on the date of the object
  being archived
  - when given a commit/tag ID to archive, check which support frame the
    committer date falls inside
  - for tree IDs, always use the latest format (it always uses the
    current date anyway)
- schedule a date, for the sake of argument, 6 months after the next
  scheduled release date of git version X.Y in which this change goes
  live; bake this into the git sources as a transition date, all commits
  or tags generated after this date fall into the next format support
  frame

The end result is that for all historic commits or tags, `git archive` will always produce the same output. This can be documented in the git-archive manpage: "the produced archive is guaranteed to be reproducible, unless you override the `tar.<format>.command` or your system compressor is not reproducible".

For *new* commits or tags, everyone gets the benefit of fascinating, cool new archive formats with useful improvements at the tar container level, which is apparently a very desirable feature. The git project no longer has to worry, at all, about whether users will come to complain about how their build pipelines suddenly fail with checksum issues. The git project can simply, fearlessly, go implement innovative new changes without giving any thought to backwards compatibility.

It is, simply, that those new changes only apply to projects which are still under active development, and which push new commits or tag new releases after the transition date.

Old states of existing projects (regardless of whether they are still actively updating) can go have their old and apparently inefficient archives and don't get cool new stuff. That's fine. They're also increasingly rarely used, because they are, after all, old -- and most likely only used for historic archival purposes. If the worst comes to worst, well, they managed to produce a somehow useful archive with an older version of git -- nothing will *break* if they don't get the cool new stuff.

And for the vast majority of new downloads for new stuff, the in-process compressor saves one fork+exec and is a bit more efficient, I guess?

A note on the transition date: I suggested 6 months after the scheduled release date, because this gives everyone running a software forge time to update git itself, and have everything ready, in time to handle the first wave of commits and tags that naturally occur after the transition date. And you don't want it to be immediate, because then people will take days or weeks to deploy and the most recent archives will change

For the purposes of this thought experiment, we assume that people don't routinely set the system time to a year in the future. This will only be done in situations such as, say, testing a git upgrade deployment for a software forge.

...

"And then no one ever complained about archive checksums changing again."

🤞🙏🥺