Re: [PATCH 1/9] docs: update pack index v3 format

From: brian m. carlson <sandals@crustytoothpaste.net>
Date: Sep 25, 2025, 21:39 UTC
Message-ID: <aNW2riVWtLQbacSR@fruit.crustytoothpaste.net>
In-Reply-To: <aNOj8fFTvkQ6jsaT@pks.im>

On 2025-09-24 at 07:55:29, Patrick Steinhardt wrote:

Show 22 quoted lines

> On Fri, Sep 19, 2025 at 01:09:03AM +0000, brian m. carlson wrote:
> > Our current pack index v3 format uses 4-byte integers to find the
> > trailer of the file.  This effectively means that the file cannot be
> > much larger than 2^32.  While this might at first seem to be okay, we
> > expect that each object will have at least 64 bytes worth of data, which
> > means that no more than about 67 million objects can be stored.
> > 
> > Again, this might seem fine, but unfortunately, we know of many users
> > who attempt to create repos with extremely large numbers of commits to
> > get a "high score," and we've already seen repositories with at least 55
> > million commits.  In the interests of gracefully handling repositories
> > even for these well-intentioned but ultimately misguided users, let's
> > change these lengths to 8 bytes.
> 
> Yeah, this makes sense. We can only assume that repositories will
> continue to grow, so it makes sense to future proof.
> 
> We also have the 4-byte number of objects contained in the pack. But as
> you explain, it's nothing we should need to worry about given that this
> is a mere counter, and not an offset into the file. I doubt that there's
> repositories out there that'll have more than 4 billion objects anytime
> soon.

There are certainly some users who try to do that at $DAYJOB, but they come to our attention (because our maintenance job fails due to taking too long) before they get there. I am not, however, aware of any actually legitimate and productive uses of repositories that threaten to break that limit, which is what I think what we should really care about.

In the event we start seeing those kinds of problems, it should be easy to implement pack v5 with a corresponding index, just with a larger number of objects.

Show 9 quoted lines

> For now we only have SHA256 and SHA1. But thinking about the future,
> there will be a time when SHA256 will be considered broken. I wonder
> whether we should safeguard against that and also specify the trailer
> hash to be agile? That is, instead of hardcoding the hash function, we
> add something like a "primary" hash to the packfile and then use the
> full output of that hash as checksum.
> 
> In any case, please feel free to say "no" to the above thought. It's
> just something that popped into my mind upon reading this.

It is actually that it's the main hash algorithm in use. So if we add a third algorithm which is SHA-3-512, then the trailer checksum will be SHA-3-512 when that's the main algorithm.

Technically, it's also SHA-1 if we're in a SHA-1 repository with SHA-256 compatibility. That's not a use case I really encourage, but it is a use case I'm testing because it exposes bugs in our codebase and I expect people will want to do in-place conversion from SHA-1 only to SHA-1 with SHA-256 at some point.

I'll fix that for v2.

> I guess one thing that should be explicitly pointed out in the commit
> message is that there are no implementations of the v3 format yet, so
> this is basically updating our envisioned design, only. Otherwise one
> might wonder why we can update the spec just so.

That isn't completely true. There is an implementation, but it is not yet on the list, and it follows the spec written here. I will provide documentation with the rest of the pack index code when index v3 comes in, but I wanted to update this in case people are trying to add it in other implementations as well.

-- 
brian m. carlson (they/them)
Toronto, Ontario, CA