Re: [PATCH 0/9] git archive: use gzip again by default, document output stabilty

From: Theodore Ts'o <tytso@mit.edu>
Date: Feb 3, 2023, 15:47 UTC
Message-ID: <Y90soPW6KRB7PQCY@mit.edu>
In-Reply-To: <771a98ca-9540-ad4e-dfba-9d304e1dff09@dunelm.org.uk>

On Thu, Feb 02, 2023 at 04:17:09PM +0000, Phillip Wood wrote:

Show 15 quoted lines

> Playing devil's advocate for a moment as we're not going to promise that the
> compressed output of "git archive" will be stable in the future perhaps we
> should use this breakage as an opportunity to highlight that to users and to
> advertize the config setting that allows them to use gzip for compressing
> archives. Reverting the change gives the misleading impression that we're
> making a commitment to keeping the output stable. The focus of this thread
> seems to be the problems relating to github which they have already
> addressed.
> 
> I think there is general agreement that it is not practical to promise that
> the compressed output of "git archive" is stable so maybe it is better to
> make that clear now while users can work around it in the short term with a
> config setting rather than waiting until we're faced with some security or
> other issue that forces a change to the output which users cannot work
> around so easily.

I would be in favor of adding a config option that allows using the internal gzip option, although leave the default to be keep things compatible.

The reason for that it should be easy for a forge provider such as GitHub to break things, deliberately. Sound insane? Hear me out.

At $WORK, we have a highly reliable system, Paxos. It is a highly fault-tolerant system, so it rarely fails. But "rarely fails" is not the same as "never fails". And hopefully, things should degrade gracefully if there is a Paxos outage. But as the Google SRE's are fond of saying, "Hope is not a strategy".

So periodically, the people who run the Paxos service will deliberately force downtime for a short amount of time. The fact that they will do this is well advertised, and scheduled ahead of time --- and teams responsible for user-facing services are supposed to make sure that end-users don't notice when this happens. Maybe they won't be able to update configurations as easily while Paxos is down, but it shouldn't cause a user-visible outage.

So what I would recommend to the GitHub product manager, is that once a quarter, on a well-advertised date, that they flip the switch and break the git archive checksums for say, an hour. Then next quarter, they advertise that the switch will be thrown for 2 hours, doubling each time, until it is ramped up to 16 hours.

This will provide the necessary nudge so that all of these badly designed systems that depend on downloaded archives of arbitrary git hubs to be stable will rethink their position, while minimizing the end-user customer impact. Otherwise, I predict that Bazel, homebrew, etc will consider to rely on this ill-considered assumption, and at some point in the future, when we *do* have a much better reason to want to make a change to the tar or compression algorithm, all of these end users will once again scream bloody murder.

Of course, this is going to be up to each forge provider to decide whether they want to do this. But we can make it easy for them to do this thing, and I'd argue it is in our interest to make it easy for them to do this. Otherwise we'll get constrained in the future by the fear of massive user blowback, no metter what we say in our documentation regarding "no promises --- and next time, we really mean it!"

	      	       	       	    	  - Ted