Re: [PATCH 0/3] Implement filtering repacks

From: Junio C Hamano <gitster@pobox.com>
Date: Oct 14, 2022, 16:46 UTC
Message-ID: <xmqqilkm9wv6.fsf@gitster.g>
In-Reply-To: <20221012135114.294680-1-christian.couder@gmail.com>

Christian Couder <christian.couder@gmail.com> writes:

Show 10 quoted lines

> For example one might want to clone with a filter to avoid too many
> space to be taken by some large blobs, and one might realize after
> some time that a number of the large blobs have still be downloaded
> because some old branches referencing them were checked out. In this
> case a filtering repack could remove some of those large blobs.
>
> Some of the comments on the patch series that John sent were related
> to the possible data loss and repo corruption that a filtering repack
> could cause. It's indeed true that it could be very dangerous, and we
> agree that improvements were needed in this area.

The wish is understandable, but I do not think this gives a good UI.

This feature is, from an end-user's point of view, very similar to "git prune-packed", in that we prune data that is not necessary due to redundancy. Nobody runs "prune-packed" directly; most people are even unaware of it being run on their behalf when they run "git gc".

Reusing pack-objects as an underlying mechanism is OK, but this needs to be plumbed through to "git gc" for a more consistent experience for the end users.

Is there a way to check if the "promisor remote" is still willing to keep the previous promise it made, so that the users do not have to see "we may corrupt the repository as the result of this operation, you have been warned", by the way? Possibly with a protocol extension?

In a sense, once you made a partial clone, your repository is at the mercy of the remote. They can disappear any time with majority of the data you depend on, leaving only what you created locally and haven't pruned away, in a repository that may technically pass "fsck", because the things that are supposed to exist locally exists, but may not be usable in practice. So from that point of view, a simple check that asks "I earlier fetched from you with this filter and these tips of histories; are you still willing to support me?" and gets yes/no answer might be sufficient. A remote that is not trustworthy can say "yes" and still change their mind later, so such a check may not even be needed.

The above two big paragraphs is a way to say that I am not all that worried about losing objects that we should be able to refetch again by adding this feature. The perceived need for "--force" or "must run from terminal" may be overblown. I do not think this negatively affects correctness or robustness at all (as long as the pruning is not buggy, of course).

HOWEVER

Unlike prune-packed, pruning objects that could be refetched has negative performance impact. So adding an option to enable/disable such pruning is needed. If the frontmost UI entry point were "gc", it needs an opt-in option to invoke the "filtering repack", in other words. "pack-objects" should not need any more work than what you have here (with the "terminal" and "force" discarded), as "--filter" is such an opt-in option already.