Re: Proposal: object negotiation for partial clones

From: Jonathan Nieder <jrnieder@gmail.com>
Date: May 6, 2019, 19:46 UTC
Message-ID: <20190506194625.GB57368@google.com>
In-Reply-To: <20190506192800.213716-1-jonathantanmy@google.com>

Hi,

Jonathan Tan wrote:

> Matthew DeVore wrote:

Show 11 quoted lines

>> I'm considering implementing a feature in the Git protocol which would
>> enable efficient and accurate object negotiation when the client is a
>> partial clone. I'd like to refine and get some validation of my
>> approach before I start to write any code, so I've written a proposal
>> for anyone interested to review. Your comments would be appreciated.
>
> Thanks. Let me try to summarize: The issue is that, during a fetch,
> normally the client can say "have" to inform the server that it has a
> commit and all its referenced objects (barring shallow lines), but we
> can't do the same if the client is a partial clone (because having a
> commit doesn't necessarily mean that we have all referenced objects).

Ah, interesting. When this was discussed before, the proposal has been that the client can say "have" anyway. They don't have the commit and all referenced objects, but they have the commit and a *promise* that they can obtain all referenced objects, which is almost as good. That's what "git fetch" currently implements.

But there's a hitch: when doing the fetch-on-demand for an object access, the client currently does not say "have". Sure, even there, they have a *promise* that they can obtain all referenced objects, but this could get out of hand: the first pack may contain a delta against an object the client doesn't have, triggering another fetch which contains a delta against another object they don't have, and so on. Too many round trips.

> And not doing this means that the server sends a lot of unnecessary
> objects in the sent packfile. The solution is to do the fetch in 2
> parts: one to get the list of objects that would be sent, and after the
> client filters that, one to get the objects themselves.

This helps with object selection but not with delta base selection.

For object selection, I think the current approach already works okay, at least where tree and blob filters are involved. For commit filters, in the current approach the fetch-on-demand sends way too much because there's no "filter=commit:none" option to pass. Is that what this proposal aims to address?

For blob filters, if I ignore the capability advertisements (there's an optimization that hasn't yet been implemented to allow single-round-trip fetches), the current behavior takes the same number of round trips as this proposal. Where the current approach has been lacking is in delta base selection during fetch-on-demand. Ideas for improving that?

Thanks, Jonathan