[RFC PATCH v2 0/4] Partial clone: promised objects (not only blobs)

Thanks for all your comments on the earlier version. This is a
substantially different version. In particular:
 - Now supports all types (tag, commit, tree) of objects, not only blobs
 - fsck works
 - Incorporates Ben Peart's code that uses a long-living process
   (lifetime of the Git invocation) to obtain objects
 - Implemented as a repository extension

If anyone would like to comment on the overall direction of this approach, that would be great. In particular, I'm not too sure on the names of things. I know that I have some things to still work on (documentation and coding style, more improvements to and tests for fsck, maybe division of commits?) but I would like some early feedback on the bigger picture first.

If you want to patch this in, this is built off my recent cleanup patch [1].

About inability to scale if we have the list of promised blobs: In this design, we will have a promised blob entry only if we have a concrete tree, so in a repository in which many trees are omitted, there will not be many promised blob entry objects. In fact, the minimal partial clone will be one in which there will be one promised commit for each ref (assuming no duplicates), no promised trees, and no promised blobs. (I'm not planning to implement such a clone, but someone else could do so.)

About having multiple promise lists: I have retained the single list, but am still open to change. The support of all the object types might be sufficient mitigation for the issues that caused us to investigate multiple promise lists in the first place.

[1] https://public-inbox.org/git/20170718222848.1453-1-jonathantanmy@google.com/

Jonathan Tan (4):
  object: remove "used" field from struct object
  promised-object, fsck: introduce promised objects
  sha1-array: support appending unsigned char hash
  sha1_file: support promised object hook

 Documentation/config.txt                         |   8 +
 Documentation/gitrepository-layout.txt           |   8 +
 Documentation/technical/read-object-protocol.txt | 102 +++++++
 Documentation/technical/repository-version.txt   |   6 +
 Makefile                                         |   1 +
 builtin/cat-file.c                               |   9 +
 builtin/fsck.c                                   |  42 ++-
 cache.h                                          |   4 +
 environment.c                                    |   1 +
 fsck.c                                           |   6 +-
 object.c                                         |  21 +-
 object.h                                         |  21 +-
 promised-object.c                                | 324 +++++++++++++++++++++++
 promised-object.h                                |  34 +++
 setup.c                                          |   7 +-
 sha1-array.c                                     |   7 +
 sha1-array.h                                     |   1 +
 sha1_file.c                                      |  44 ++-
 t/t3907-promised-object.sh                       |  73 +++++
 t/t3907/read-object                              | 114 ++++++++
 t/test-lib-functions.sh                          |   6 +
 21 files changed, 808 insertions(+), 31 deletions(-)
 create mode 100644 Documentation/technical/read-object-protocol.txt
 create mode 100644 promised-object.c
 create mode 100644 promised-object.h
 create mode 100755 t/t3907-promised-object.sh
 create mode 100755 t/t3907/read-object