Re: Proposal for "fetch-any-blob Git protocol" and server design

From: Junio C Hamano <gitster@pobox.com>
Date: Mar 15, 2017, 17:59 UTC
Message-ID: <xmqqinnafml4.fsf@gitster.mtv.corp.google.com>
In-Reply-To: <ffd92ad9-39fe-c76b-178d-6e3d6a425037@google.com>

Jonathan Tan <jonathantanmy@google.com> writes:

Show 14 quoted lines

> == Design
>
> A new endpoint "server" is created. The client will send a message in
> the following format:
>
> ----
> fbp-request = PKT-LINE("fetch-blob-pack")
>               1*want
>               flush-pkt
> want = PKT-LINE("want" SP obj-id)
> ----
>
> The client may send one or more SHA-1s for which it wants blobs, then
> a flush-pkt.

By "SHA-1s for which it wants blobs", you mean that "want" only allows one exact blob object name? I think it is necessary to support that mode of operation as a base case, and it is a good starting point.

When you know

 - you have a "partial" clone that initially asked to contain only
   blobs that are smaller than 10MB, and

 - you are now trying to do a "git checkout v1.0 -- this/directory"
   so that the directory is fully populated

instead of enumerating all the missing blobs from the output of "ls-tree -r v1.0 this/directory" on separate "want" requests, you may want to say "I want all the blobs that are not smaller than 10MB in this tree object $(git rev-parse v1.0:this/directory)".

I am not saying that you should add something like this right away, but I am wondering how you would extend the proposed system to do so. Would you add "fetch-size-limited-blob-in-tree-pack" that runs parallel to "fetch-blob-pack" request? Would you add a new type of request packet "want-blob-with-expression" for fbp-request, which is protected by some "protocol capability" exchange?

If the former, how does a client discover if a particular server already supports the new "fetch-size-limited-blob-in-tree-pack" request, so that it does not have to send a bunch of "want" request by enumerating the blobs itself? If the latter, how does a client discover if a particular server's "fetch-blob-pack" already supports the new "want-blob-with-expression" request packet?

Show 7 quoted lines

> === Endpoint support for forward compatibility
>
> This "server" endpoint requires that the first line be understood, but
> will ignore any other lines starting with words that it does not
> understand. This allows new "commands" to be added (distinguished by
> their first lines) and existing commands to be "upgraded" with
> backwards compatibility.