Re: [PATCH 1/9] docs: update pack index v3 format

From: Patrick Steinhardt <ps@pks.im>
Date: Sep 24, 2025, 07:55 UTC
Message-ID: <aNOj8fFTvkQ6jsaT@pks.im>
In-Reply-To: <20250919010911.649831-2-sandals@crustytoothpaste.net>

On Fri, Sep 19, 2025 at 01:09:03AM +0000, brian m. carlson wrote:

Show 12 quoted lines

> Our current pack index v3 format uses 4-byte integers to find the
> trailer of the file.  This effectively means that the file cannot be
> much larger than 2^32.  While this might at first seem to be okay, we
> expect that each object will have at least 64 bytes worth of data, which
> means that no more than about 67 million objects can be stored.
> 
> Again, this might seem fine, but unfortunately, we know of many users
> who attempt to create repos with extremely large numbers of commits to
> get a "high score," and we've already seen repositories with at least 55
> million commits.  In the interests of gracefully handling repositories
> even for these well-intentioned but ultimately misguided users, let's
> change these lengths to 8 bytes.

Yeah, this makes sense. We can only assume that repositories will continue to grow, so it makes sense to future proof.

We also have the 4-byte number of objects contained in the pack. But as you explain, it's nothing we should need to worry about given that this is a mere counter, and not an offset into the file. I doubt that there's repositories out there that'll have more than 4 billion objects anytime soon.

Show 14 quoted lines

> For the checksums at the end of the file, we're producing 32-byte
> SHA-256 checksums because that's what we already do with pack index v2
> and SHA-256.  Truncating SHA-256 doesn't pose any actual security
> problems other than those related to the reduced size, but our pack
> checksum must already be 32 bytes (since SHA-256 packs have 32-byte
> checksums) and it simplifies the code to use the existing hashfile logic
> for these cases for the index checksum as well.
> 
> In addition, even though we may not need cryptographic security for the
> index checksum, we'd like to avoid arguments from auditors and such for
> organizations that may have compliance or security requirements.  Using
> the simple, boring choice of the full SHA-256 hash avoids all possible
> discussion related to hash truncation and removes impediments for these
> organizations.

For now we only have SHA256 and SHA1. But thinking about the future, there will be a time when SHA256 will be considered broken. I wonder whether we should safeguard against that and also specify the trailer hash to be agile? That is, instead of hardcoding the hash function, we add something like a "primary" hash to the packfile and then use the full output of that hash as checksum.

In any case, please feel free to say "no" to the above thought. It's just something that popped into my mind upon reading this.

I guess one thing that should be explicitly pointed out in the commit message is that there are no implementations of the v3 format yet, so this is basically updating our envisioned design, only. Otherwise one might wonder why we can update the spec just so.

Patrick