Re: [PATCH v4 0/6] pack-objects: freshen objects with multi-cruft packs

From: Junio C Hamano <gitster@pobox.com>
Date: Mar 12, 2025, 19:46 UTC
Message-ID: <xmqqa59qxc9b.fsf@gitster.g>
In-Reply-To: <Z9Ha2mFXpojI+aIR@nand.local>

Taylor Blau <me@ttaylorr.com> writes:

> Maybe we should call the option something else that makes the cruft pack
> use-case clearer. But in the other thread I came around to the idea that
> this case is too niche to address completely, so I think we can discard
> this round as a terrible idea.

Yeah, I do not think "an earlier attempt wouldn't fill the bucket fully, so a later logic may try to fill the bucket with the same material in vain, failing the same way" is a problem that is limited to cruft pack use case. If you are repacking real data into 200MB chunks every week, the 198MB pack created last week that is lying in the object store that houses the oldest part of the history is unlikely to be improved by repacking it (since the attempt failed to fill the remaining 2MB last week, and the oldest part of the history hasn't changed) but it is entirely possible that with the pieces of information left by the current set of tools, the machinery may not realize that fact.

Allowing the resulting pack to deliberately exceed 200MB ceiling is one possible way (I still do not think it is the best way among possible ways) to mark the pack "frozen-do not further touch".

Now, if the essense of the problem is when to mark a pack that we stopped appending to due to max-size limit as "frozen and do not touch" to avoid this problem, how such a marking is done, and who uses it to avoid repeated work that is known (to us) to be futile, there should be better ways to do such a marking. We have .keep mechanism which is ugly but proven to work and we know how to deal with, for example.

I wonder if our pack header or trailer is extensible enough so that we can record "this pack was closed due to max-size limit of 200MB and appending even one more object in the queue would have busted the limit".

No matter how such a marking is done, later repack that sees a pack marked as such, when it is limited to 200MB, should be able to tell that and act a bit more intelligently than what it currently does.