Re: Resumable clone

From: Kevin Wern <kevin.m.wern@gmail.com>
Date: Mar 9, 2016, 02:04 UTC
Message-ID: <CANtyZjRw57kru=7RvF1Q+uTsoF+0YKynA2Y0FQYK5PBFxSoPNQ@mail.gmail.com>
In-Reply-To: <xmqq4mcgnbkb.fsf@gitster.mtv.corp.google.com>

Junio,

Show 18 quoted lines

> Yes, that is very close to what I said in the "what remains?"
> section, but with a crucial difference in a detail.  Perhaps reading
> the message you are respoinding to again more carefully will clear
> the confusion.  This is what we want to allow the server to say
> (from the message you are responding to, but rephrased slightly,
> hoping that it would help unconfuse you):
>
>     I prefer not to serve a full clone to you in the usual route if
>     I can avoid it.  You can help me by populate your history first
>     with something else (which would bring you to a state as if you
>     cloned potentially a bit older version of me) and then coming
>     back to me for an additional fetch to complete the history.
>
> That "something else" does not have to be, and is not expected to
> be, the "full" history of the current state.  As long as it can be
> used to bring the cloner to a reasonably recent state, sufficient to
> make a follow up incremental fetch inexpesive enough, it is
> appropriate.

Sorry, I was thrown off by:

> - the type of resource, if we want this to be extensible.  I
>   think we should initially limit it to "a single full history
>   .pack",

I misinterpreted what you meant by "a single full history .pack," and used that to limit what you said earlier. A lot of my reasoning from there misses the point, then, because it involved finding some way to determine if a history .pack contains the full history, which obviously is irrelevant.

Show 10 quoted lines

>> I'm not sure how the server should determine the returned resource. A
>> packfile alone does not guarantee the full repo history, and I'm not
>> positive checking the idx file for HEAD's commit hash ensures every
>> sub-object is in that file (though I feel it should, because it is
>> delta-compressed).
>
> The above reasoning does not make much technical sense.  delta
> compression does not ensure connectivity in the commit history and
> commit->tree->blob containment.  Again I am not sure where you are
> going with this.

Related to the above, I was trying to find a way to determine if the packfile contained the full history, which actually doesn't matter. My technical reasoning was affected by the way deltas represent changes in other VCS's, but I realize now that git's delta compression is based around the similarity of objects, not their historical relation to each other.

Show 14 quoted lines

>> Which leaves me with questions on how to test the above condition. Is
>> there an expected place, such as config, where the user will specify
>> the type of alternate resource, and should we assume some default if
>> it isn't specified? Can the user optionally specify the exact file to
>> use (I can't see why because it only invites more errors)? Should the
>> specification of this option change git's behavior on update, such as
>> making sure the full history is compressed? Does the existence of the
>> HEAD object in the packfile ensure the repo's entire history is
>> contained in that file?
>
> Those (except for your assumption that no follow-up fetch is
> allowed, which requires you to limit yourself to "full" history,
> which is an unnecessary requirement) are good points one should be
> making design decisions on when building this part of the system.

Awesome, all of this is enough info to get me started. Thanks!

Kevin