Re: [PATCH v2 0/5] Fsck for lazy objects, and (now) actual invocation of loader

From: Junio C Hamano <gitster@pobox.com>
Date: Aug 1, 2017, 17:11 UTC
Message-ID: <xmqq379bgqlx.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <20170731160533.0a446244@twelve2.svl.corp.google.com>

Jonathan Tan <jonathantanmy@google.com> writes:

> Well, the fsck can still detect issues like corrupt objects (as you
> mention above) and dangling heads, which might be real issues. But it is
> true that it does not give you the guarantee you describe.

Which makes it pretty much useless. The whole point of running "fsck" is to make sure that we won't waste work by not finding a corruption long after it was introduced and spent a lot of effort building on top of a state that nobody can reproduce.

> From a user standpoint, this might be able to be worked around by
> providing a network-requiring object connectivity checking tool or by
> just having the user running a build to ensure that all necessary files
> are present.

I actually was hoping that you do not have to go to the network for the checking. And I have to say that "only the tip matters" is a horrible cop-out that is not even a workaround. Your users would be served better if you honestly admit that your fsck will not be useful when this feature is used---at least they won't be harmed by a false expectation that "fsck" would give them some assurance, which is not the case.

Let's step back a bit and think what already happens in the pre- lazy-object world. We record cut-off commits when a depth limited clone is created in "shallow". These essentially are promises, saying something like:

    Rest assured that everything in the history behind these commits
    are on the other side and you can retrieve them by unshallowing.

    If you traverse from your local tips and find no missing objects
    before reaching one of these commits, then you do not have any
    local corruption you need to worry about.

the other end made to us, when the shallow clone was made. And we take this promise and build more commits on top, and then we adjust these cut-off commits incrementally as we deepen our clone or make it even shallower. For this assurance to work, we of course need to assume a bit more than what we assume for a complete clone, namely, the "other side" will hold onto the history behind these, i.e. does not remind the tips it already has shown to us, or even if it does, the objects that are reachable from these cut-off points will somehow always be available to us on demand.

Can we do something similar, i.e. maintain minimum set of cut-off points and adjust that set incrementally, just sufficient to ensure the integrity of objects locally created and not yet safely stored away by pushing them the "other side"?

I haven't thought things through (and I know you, Ben and others have thought much longer and harder), but I would imagine if we have a commit object [*1*], some of whose parent commits, trees and blobs are locally missing, and know that the commit exists on the "other side", we know that all of these "missing" objects that are referenced by the commit are also available from the "other side". IOW, I suspect that the same principle "shallow" uses to give us the integrity guarantee can be naturally extended to allow us to see if a broken connectivity is OK.

[Footnote]

*1* The same can be said for a tag or a tree object that we know
    exist on the "other side"; they may refer, directly or
    indirectly through objects we locally have, to objects that that
    are missing locally, and as long as the starting point object
    are known to be available on the "other side", it is OK for them
    to be missing locally.