Re: Stability of git-archive, breaking (?) the Github universe, and a possible solution

From: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
Date: Feb 3, 2023, 13:18 UTC
Message-ID: <230203.86sffmc1tz.gmgdl@evledraar.gmail.com>
In-Reply-To: <01a901d93760$c690d970$53b28c50$@nexbridge.com>

On Thu, Feb 02 2023, rsbecker@nexbridge.com wrote:

Show 40 quoted lines

> On February 2, 2023 6:02 PM, brian m. carlson wrote:
>>On 2023-02-01 at 23:37:19, Junio C Hamano wrote:
>>> "brian m. carlson" <sandals@crustytoothpaste.net> writes:
>>>
>>> > I don't think a blurb is necessary, but you're basically
>>> > underscoring the problem, which is that nobody is willing to promise
>>> > that compression is consistent, but yet people want to rely on that
>>> > fact.  I'm willing to write and implement a consistent tar spec and
>>> > to guarantee compatibility with that, but the tension here is that
>>> > people also want gzip to never change its byte format ever, which
>>> > frankly seems unrealistic without explicit guarantees.  Maybe the
>>> > authors will agree to promise that, but it seems unlikely.
>>>
>>> Just to step back a bit, where does the distinction between
>>> guaranteeing the tar format stability and gzip compressed bitstream
>>> stability come from?  At both levels, the same thing can be expressed
>>> in multiple different ways, I think, but spelling out how exactly the
>>> compressor compresses is more involved than spelling out how entries
>>> in a tar archive is ordered and each entry is expressed, or something?
>>
>>Yes, at least with my understanding about how gzip and compression in general
>>work.
>>
>>The tar format (and the pax format which builds on it) can mostly be restricted by
>>explaining what data is to be included in the pax and tar headers and how it is to be
>>formatted.  If we say, we will always write such and such information in the pax
>>header and sort the keys, and we write such and such information in the tar header,
>>then the format is completely deterministic, and we can make nice guarantees.
>>
>>My understanding about how Lempel-Ziv-based compression algorithms work is that
>>there's a lot more freedom to decide how best to compress things and that there
>>isn't always a logical obvious choice, but I will admit my understanding is relatively
>>limited.  If someone thinks we can effectively succeed in supporting compression
>>more than just relying on gzip, I would be delighted to be shown to be wrong.
>
> The nice part about gzip is that it is generally available on
> virtually all platforms (or can be easily obtained). Other compression
> forms, like bz2, which sometimes produces more dense compression, are
> not necessarily available. Availability is something I would be
> worried about...

I agree with all of that, gzip is in such wide use for a reason.

>... (clone and checkout failures).

But how would a hypothetical obscure format for "git archive" contribute to clone or checkout failures? Are you thinking of our use of zlib for e.g. loose objects? That's unrelated to this discussion (and I don't think anyone relies on their compressed checksum).

> Tar formats are also to be used carefully. Not all platform
> implementations of tar support all variants. "ustar" is fairly common
> but there are others that are not. Interoperability needs to be the
> biggest factor in this decision, IMHO, rather than compression rates.

For "git archive" whether you care about interoperability depends on the target audience of your archive, and in any case I don't see why we need to worry about it, except to perhaps note that some are more portable than others if we e.g. had a built-in "tar.bz2" helper method.

> The alternative is having git supply its own implementation, but that
> is a longer term migration problem, resembling the SHA-256 migration.

I've noted elsewhere in this thread that I don't see the point of shipping a fallback "gzip" beyond the "git archive gzip" we have already, but even if we did that the scope of that seems pretty simple, and *much* easier than the SHA-256 migration.