Re: [PATCH 00/10] RFC Partial Clone and Fetch

Overall, this fetch/clone approach seems reasonable to me, except 
perhaps some unanswered questions (some of which are also being 
discussed elsewhere):
  - does the server need to tell us of missing blobs?
  - if yes, does the server need to tell us their file sizes?
  - do we need to store the list of missing blobs somewhere (whether the
    server told it to us or whether we inferred it from the fetched
    trees)

The answers to this probably depend on the answers in "B. Issues Backfilling Omitted Blobs" (especially the additional concepts I listed below).

Also, do you have any plans to implement other functionality, e.g. "git checkout" (which will allow fetches and clones to repositories with a working directory)? (I don't know what the mailing list consensus would be for the "acceptance criteria" for this patch set, but I would at least include "checkout".)

On 03/08/2017 10:50 AM, git@jeffhostetler.com wrote:

Another concept is "how to determine if a blob is really omitted" - do we store a list somewhere or do we assume that all missing blobs are purposely omitted (like in this patch set)?

Yet another concept is "whether to fetch" - for example, a checkout should almost certainly fetch, but a rev-list used by a connectivity check (like in patch 6 of this set) should not.

For example, for historical-blob-searching commands like "git log -S", 
should we:
  a) fetch everything missing (so users should use date-limiting
     arguments)
  b) fetch nothing missing
  c) use the file size to automatically exclude big files, but fetch
     everything else

For a) and b), we wouldn't need file size information for missing blobs, but for c), we do. This might determine if we need file size information in the fetch-pack/upload-pack protocol.

Context (like the commit SHA-1) would help in reachability checks, but I'm not sure if we can rely on that. It is true that I can't think of a way that the client would dissociate a blob that is missing from its tree or commit (because it would first need to "fault-in" that blob to do its operation). But clients operating on non-contextual SHA-1s (e.g. "git cat-file") and servers manipulating commits (so that the commit SHA-1 that the client had in its context is no longer reachable) are not uncommon, I think.

Having said that, it might be useful to include the context in the protocol anyway as an optional "hint".

I'm not sure what you mean by "last known SHA for each blob".

(If we do store the file size of a blob somewhere, we could also store some context there. I'm not sure how useful this is, though.)

<snip>

These 2 points would be part of "whether to fetch" above.