Re: [RFC PATCH] During a shallow fetch, prevent sending over unneeded objects

From: Duy Nguyen <pclouds@gmail.com>
Date: Aug 8, 2013, 07:21 UTC
Message-ID: <CACsJy8BahoGcDcLjSaHA-62_KQE2wD-p5oeJOOA4nk8ZRfXrEA@mail.gmail.com>
In-Reply-To: <7vfvukbrqh.fsf@alter.siamese.dyndns.org>

On Thu, Aug 8, 2013 at 1:51 PM, Junio C Hamano <gitster@pobox.com> wrote:

Show 10 quoted lines

> Duy Nguyen <pclouds@gmail.com> writes:
>
>> I think this applies to general case as well, not just shallow.
>> Imagine I have a disconnected commit that points to the latest tree
>> (i.e. it contains most of latest changes). Because it's disconnected,
>> it'll be ignored by the server side. But if the servide side does
>> mark_tree_interesting on this commit, a bunch of blobs might be
>> excluded from sending.
>
> I think you meant mark_tree_UNinteresting.

Yes, thanks for correcting.

Show 14 quoted lines

>> ... So perhaps we could go over have_obj list
>> again, if it's not processed and is
>>
>>  - a tree-ish, mark_tree_uninteresting
>>  - a blob, just mark unintesting
>>
>> and this does regardless of shallow state or edges.
>
> As a general idea, I agree it may be worth trying out to see if your
> concern that the "have" list may be so big that this approach may be
> more costly than it is worth.
>
> If the recipient is known to have something, we do not have to send
> it.

OK. Mathijs, do you want make a patch for it?

Show 13 quoted lines

> The things that we decide not to send are not necessarily what the
> recipient has, which introduces a twist you need to watch out for if
> we want to go that route.
>
> If the recipient is known to have something, a thin transfer can
> send a delta against it.  You do not want to send the commits before
> the shallow boundary (i.e. the parents of the commits listed in
> .git/shallow) because the recipient does not want them, and that
> means you may have to use a different mark to record that fact.  The
> recipient does not have them, we do not want to send them, and they
> cannot be used as a delta base for what we do send.  Which is quite
> different from the ordinary "uninteresting" objects, those we decide
> not to send because the recipient has them.

I fail to see the point here. There are two different things: what we want to send, and what we can make deltas against. Shallow boundary affects the former. What the recipient has affects latter. What is the twist about?

As for considering objects before shallow boundary uninteresting, I have a plan for it: kill upload-pack.c:do_rev_list(). The function is created to make a cut at shallow boundary, but we already have a tool for that: grafting. In my ongoing shallow series I will create a temporary shallow file that contains new roots and pass the file to pack-objects with --shallow-file. pack-objects will never see anything outside what the recipient may want (i.e. commits before shallow boundary) to receive and pack-objects' rev-list should do what upload-pack.c:do_rev_list() currently does.

-- 
Duy