{"thread":{"id":"46359","subject":"[RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing blobs\")","startedAt":"2017-07-11T19:48:46Z","lastAt":"2017-07-29T13:26:59Z","messageCount":45,"participants":["Jonathan Tan","Stefan Beller","Jeff Hostetler","Ben Peart","Jonathan Nieder","Philip Oakley","Junio C Hamano"],"isPatch":true,"patchVersion":1,"patchTotal":3},"messages":[{"id":"324223","messageId":"cover.1499800530.git.jonathantanmy@google.com","threadId":"46359","inReplyTo":null,"subject":"[RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing blobs\")","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-11T19:48:29Z","receivedAt":"2017-07-11T19:48:46Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"These patches are part of a set of patches implementing partial clone,\nas you can see here:\n\nhttps://github.com/jonathantanmy/git/tree/partialclone\n\nIn that branch, clone with batch checkout works, as you can see in the\nREADME. The code and tests are generally done, but some patches are\nstill missing documentation and commit messages.\n\nThese 3 patches implement the foundational concept - formerly known as\n\"missing blobs\" in the \"missing blob manifest\", I decided to call them\n\"promised blobs\". The repo knows their object names and sizes. It also\ndoes not have the blobs themselves, but can be configured to know how to\nfetch them.\n\nAn older version of these patches was sent as a single demonstration\npatch in versions 1 to 3 of [1]. In there, Junio suggested that I have\nonly one file containing missing blob information. I have made that\nsuggested change in this version.\n\nOne thing remaining is to add a repository extension [2] so that older\nversions of Git fail immediately instead of trying to read missing\nblobs, but I thought I'd send these first in order to get some initial\nfeedback.\n\n[1] https://public-inbox.org/git/cover.1497035376.git.jonathantanmy@google.com/\n[2] Documentation/technical/repository-version.txt\n\nJonathan Tan (3):\n  promised-blob, fsck: introduce promised blobs\n  sha1-array: support appending unsigned char hash\n  sha1_file: add promised blob hook support\n\n Documentation/config.txt               |   8 ++\n Documentation/gitrepository-layout.txt |   8 ++\n Makefile                               |   1 +\n builtin/cat-file.c                     |   9 ++\n builtin/fsck.c                         |  13 +++\n promised-blob.c                        | 170 +++++++++++++++++++++++++++++++++\n promised-blob.h                        |  27 ++++++\n sha1-array.c                           |   7 ++\n sha1-array.h                           |   1 +\n sha1_file.c                            |  44 ++++++---\n t/t3907-promised-blob.sh               |  65 +++++++++++++\n t/test-lib-functions.sh                |   6 ++\n 12 files changed, 345 insertions(+), 14 deletions(-)\n create mode 100644 promised-blob.c\n create mode 100644 promised-blob.h\n create mode 100755 t/t3907-promised-blob.sh\n\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324224","messageId":"f9c7d4b3f800ea31e85e4897ee7048fec1e3c2f0.1499800530.git.jonathantanmy@google.com","threadId":"46359","inReplyTo":"cover.1499800530.git.jonathantanmy@google.com","subject":"[RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-11T19:48:30Z","receivedAt":"2017-07-11T19:48:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, Git does not support repos with very large numbers of blobs\nor repos that wish to minimize manipulation of certain blobs (for\nexample, because they are very large) very well, even if the user\noperates mostly on part of the repo, because Git is designed on the\nassumption that every blob referenced by a tree object is available\nsomewhere in the repo storage.\n\nAs a first step to reducing this problem, introduce the concept of\npromised blobs. Each Git repo can contain a list of promised blobs and\ntheir sizes at $GIT_DIR/objects/promisedblob. This patch contains\nfunctions to query them; functions for creating and modifying that file\nwill be introduced in later patches.\n\nA repository that is missing a blob but has that blob promised is not\nconsidered to be in error, so also teach fsck this.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Makefile                 |  1 +\n builtin/fsck.c           | 13 +++++++\n promised-blob.c          | 95 ++++++++++++++++++++++++++++++++++++++++++++++++\n promised-blob.h          | 14 +++++++\n t/t3907-promised-blob.sh | 29 +++++++++++++++\n t/test-lib-functions.sh  |  6 +++\n 6 files changed, 158 insertions(+)\n create mode 100644 promised-blob.c\n create mode 100644 promised-blob.h\n create mode 100755 t/t3907-promised-blob.sh\n\ndiff --git a/Makefile b/Makefile\nindex 9c9c42f8f..e96163269 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -828,6 +828,7 @@ LIB_OBJS += preload-index.o\n LIB_OBJS += pretty.o\n LIB_OBJS += prio-queue.o\n LIB_OBJS += progress.o\n+LIB_OBJS += promised-blob.o\n LIB_OBJS += prompt.o\n LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 99dea7adf..7454be7f1 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -15,6 +15,7 @@\n #include \"progress.h\"\n #include \"streaming.h\"\n #include \"decorate.h\"\n+#include \"promised-blob.h\"\n \n #define REACHABLE 0x0001\n #define SEEN      0x0002\n@@ -223,6 +224,9 @@ static void check_reachable_object(struct object *obj)\n \tif (!(obj->flags & HAS_OBJ)) {\n \t\tif (has_sha1_pack(obj->oid.hash))\n \t\t\treturn; /* it is in pack - forget about it */\n+\t\tif (obj->type == OBJ_BLOB &&\n+\t\t    is_promised_blob(&obj->oid, NULL))\n+\t\t\treturn;\n \t\tprintf(\"missing %s %s\\n\", printable_type(obj),\n \t\t\tdescribe_object(obj));\n \t\terrors_found |= ERROR_REACHABLE;\n@@ -642,6 +646,13 @@ static int mark_packed_for_connectivity(const struct object_id *oid,\n \treturn 0;\n }\n \n+static int mark_promised_blob_for_connectivity(const struct object_id *oid,\n+\t\t\t\t\t       void *data)\n+{\n+\tmark_object_for_connectivity(oid);\n+\treturn 0;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [<options>] [<object>...]\"),\n \tNULL\n@@ -701,6 +712,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \tif (connectivity_only) {\n \t\tfor_each_loose_object(mark_loose_for_connectivity, NULL, 0);\n \t\tfor_each_packed_object(mark_packed_for_connectivity, NULL, 0);\n+\t\tfor_each_promised_blob(mark_promised_blob_for_connectivity,\n+\t\t\t\t       NULL);\n \t} else {\n \t\tfsck_object_dir(get_object_directory());\n \ndiff --git a/promised-blob.c b/promised-blob.c\nnew file mode 100644\nindex 000000000..493808ed2\n--- /dev/null\n+++ b/promised-blob.c\n@@ -0,0 +1,95 @@\n+#include \"cache.h\"\n+#include \"promised-blob.h\"\n+#include \"sha1-lookup.h\"\n+#include \"strbuf.h\"\n+\n+#define ENTRY_SIZE (GIT_SHA1_RAWSZ + 8)\n+/*\n+ * A mmap-ed byte array of size (missing_blob_nr * ENTRY_SIZE). Each\n+ * ENTRY_SIZE-sized entry consists of the SHA-1 of the promised blob and its\n+ * 64-bit size in network byte order. The entries are sorted in ascending SHA-1\n+ * order.\n+ */\n+static char *promised_blobs;\n+static int64_t promised_blob_nr = -1;\n+\n+static void prepare_promised_blobs(void)\n+{\n+\tchar *filename;\n+\tint fd;\n+\tstruct stat st;\n+\n+\tif (promised_blob_nr >= 0)\n+\t\treturn;\n+\n+\tif (getenv(\"GIT_IGNORE_PROMISED_BLOBS\")) {\n+\t\tpromised_blob_nr = 0;\n+\t\treturn;\n+\t}\n+\t\n+\tfilename = xstrfmt(\"%s/promisedblob\", get_object_directory());\n+\tfd = git_open(filename);\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\tpromised_blob_nr = 0;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t\tperror(\"prepare_promised_blobs\");\n+\t\tdie(\"Could not open %s\", filename);\n+\t}\n+\tif (fstat(fd, &st)) {\n+\t\tperror(\"prepare_promised_blobs\");\n+\t\tdie(\"Could not stat %s\", filename);\n+\t}\n+\tif (st.st_size == 0) {\n+\t\tpromised_blob_nr = 0;\n+\t\tgoto cleanup;\n+\t}\n+\tif (st.st_size % ENTRY_SIZE) {\n+\t\tdie(\"Size of %s is not a multiple of %d\", filename, ENTRY_SIZE);\n+\t}\n+\n+\tpromised_blobs = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tpromised_blob_nr = st.st_size / ENTRY_SIZE;\n+\n+cleanup:\n+\tfree(filename);\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+}\n+\n+int is_promised_blob(const struct object_id *oid, unsigned long *size)\n+{\n+\tint result;\n+\n+\tprepare_promised_blobs();\n+\tresult = sha1_entry_pos(promised_blobs, ENTRY_SIZE, 0, 0,\n+\t\t\t\tpromised_blob_nr, promised_blob_nr, oid->hash);\n+\tif (result >= 0) {\n+\t\tif (size) {\n+\t\t\tuint64_t size_nbo;\n+\t\t\tchar *sizeptr = promised_blobs +\n+\t\t\t\t\tresult * ENTRY_SIZE + GIT_SHA1_RAWSZ;\n+\t\t\tmemcpy(&size_nbo, sizeptr, sizeof(size_nbo));\n+\t\t\t*size = ntohll(size_nbo);\n+\t\t}\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int for_each_promised_blob(each_promised_blob_fn cb, void *data)\n+{\n+\tstruct object_id oid;\n+\tint i, r;\n+\n+\tprepare_promised_blobs();\n+\tfor (i = 0; i < promised_blob_nr; i++) {\n+\t\tmemcpy(oid.hash, &promised_blobs[i * ENTRY_SIZE],\n+\t\t       GIT_SHA1_RAWSZ);\n+\t\tr = cb(&oid, data);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\treturn 0;\n+}\ndiff --git a/promised-blob.h b/promised-blob.h\nnew file mode 100644\nindex 000000000..a303ea1ff\n--- /dev/null\n+++ b/promised-blob.h\n@@ -0,0 +1,14 @@\n+#ifndef PROMISED_BLOB_H\n+#define PROMISED_BLOB_H\n+\n+/*\n+ * Returns 1 if oid is the name of a promised blob. If size is not NULL, also\n+ * returns its size.\n+ */\n+extern int is_promised_blob(const struct object_id *oid,\n+\t\t\t    unsigned long *size);\n+\n+typedef int each_promised_blob_fn(const struct object_id *oid, void *data);\n+int for_each_promised_blob(each_promised_blob_fn, void *);\n+\n+#endif\ndiff --git a/t/t3907-promised-blob.sh b/t/t3907-promised-blob.sh\nnew file mode 100755\nindex 000000000..827072004\n--- /dev/null\n+++ b/t/t3907-promised-blob.sh\n@@ -0,0 +1,29 @@\n+#!/bin/sh\n+\n+test_description='promised blobs'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'fsck fails on missing blobs' '\n+\trm -rf repo &&\n+\n+\tgit init repo &&\n+\ttest_commit -C repo 1 &&\n+\tHASH=$(git hash-object repo/1.t) &&\n+\n+\trm repo/.git/objects/$(echo $HASH | cut -c1-2)/$(echo $HASH | cut -c3-40) &&\n+\ttest_must_fail git -C repo fsck\n+'\n+\n+test_expect_success '...but succeeds if it is a promised blob' '\n+\tprintf \"%s%016x\" \"$HASH\" \"$(wc -c <repo/1.t)\" |\n+\t\thex_pack >repo/.git/objects/promisedblob &&\n+\tgit -C repo fsck\n+'\n+\n+test_expect_success '...but fails again with GIT_IGNORE_PROMISED_BLOBS' '\n+\tGIT_IGNORE_PROMISED_BLOBS=1 test_must_fail git -C repo fsck &&\n+\tunset GIT_IGNORE_PROMISED_BLOBS\n+'\n+\n+test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex db622c355..1ebdd2d04 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1018,3 +1018,9 @@ nongit () {\n \t\t\"$@\"\n \t)\n }\n+\n+# Converts big-endian pairs of hexadecimal digits into bytes. For example,\n+# \"printf 61620d0a | hex_pack\" results in \"ab\\r\\n\".\n+hex_pack () {\n+\tperl -e '$/ = undef; $input = <>; print pack(\"H*\", $input)'\n+}\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324225","messageId":"523f64e1395124a02316071329bedf9884757479.1499800530.git.jonathantanmy@google.com","threadId":"46359","inReplyTo":"cover.1499800530.git.jonathantanmy@google.com","subject":"[RFC PATCH 2/3] sha1-array: support appending unsigned char hash","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-11T19:48:31Z","receivedAt":"2017-07-11T19:48:54Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In a subsequent patch, sha1_file will need to append object names in the\nform of \"unsigned char *\" to oid arrays. Teach sha1-array support for\nthat.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1-array.c | 7 +++++++\n sha1-array.h | 1 +\n 2 files changed, 8 insertions(+)\n\ndiff --git a/sha1-array.c b/sha1-array.c\nindex 838b3bf84..6e0e35391 100644\n--- a/sha1-array.c\n+++ b/sha1-array.c\n@@ -9,6 +9,13 @@ void oid_array_append(struct oid_array *array, const struct object_id *oid)\n \tarray->sorted = 0;\n }\n \n+void oid_array_append_sha1(struct oid_array *array, const unsigned char *sha1)\n+{\n+\tALLOC_GROW(array->oid, array->nr + 1, array->alloc);\n+\thashcpy(array->oid[array->nr++].hash, sha1);\n+\tarray->sorted = 0;\n+}\n+\n static int void_hashcmp(const void *a, const void *b)\n {\n \treturn oidcmp(a, b);\ndiff --git a/sha1-array.h b/sha1-array.h\nindex 04b075633..3479959e4 100644\n--- a/sha1-array.h\n+++ b/sha1-array.h\n@@ -11,6 +11,7 @@ struct oid_array {\n #define OID_ARRAY_INIT { NULL, 0, 0, 0 }\n \n void oid_array_append(struct oid_array *array, const struct object_id *oid);\n+void oid_array_append_sha1(struct oid_array *array, const unsigned char *sha1);\n int oid_array_lookup(struct oid_array *array, const struct object_id *oid);\n void oid_array_clear(struct oid_array *array);\n \n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324226","messageId":"34efd9e9936fdab331655f5a33a098a72dc134f4.1499800530.git.jonathantanmy@google.com","threadId":"46359","inReplyTo":"cover.1499800530.git.jonathantanmy@google.com","subject":"[RFC PATCH 3/3] sha1_file: add promised blob hook support","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-11T19:48:32Z","receivedAt":"2017-07-11T19:48:59Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Teach sha1_file to invoke a hook whenever a blob is requested and\nunavailable but is promised. The hook is a shell command that can be\nconfigured through \"git config\"; this hook takes in a list of hashes and\nwrites (if successful) the corresponding objects to the repo's local\nstorage.\n\nThis is meant as a temporary measure to ensure that all Git commands\nwork in such a situation. Future patches will update some commands to\neither tolerate promised blobs (without invoking the hook) or be more\nefficient in invoking the promised blob hook.\n\nIn order to determine the code changes in sha1_file.c necessary, I\ninvestigated the following:\n (1) functions in sha1_file that take in a hash, without the user\n     regarding how the object is stored (loose or packed)\n (2) functions in sha1_file that operate on packed objects (because I\n     need to check callers that know about the loose/packed distinction\n     and operate on both differently, and ensure that they can handle\n     the concept of objects that are neither loose nor packed)\n\n(1) is handled by the modification to sha1_object_info_extended().\n\nFor (2), I looked at for_each_packed_object and at the packed-related\nfunctions that take in a hash. For for_each_packed_object, the callers\neither already work or are fixed in this patch:\n - reachable - only to find recent objects\n - builtin/fsck - already knows about promised blobs\n - builtin/cat-file - fixed in this commit\n\nCallers of the other functions do not need to be changed:\n - parse_pack_index\n   - http - indirectly from http_get_info_packs\n - find_pack_entry_one\n   - this searches a single pack that is provided as an argument; the\n     caller already knows (through other means) that the sought object\n     is in a specific pack\n - find_sha1_pack\n   - fast-import - appears to be an optimization to not store a\n     file if it is already in a pack\n   - http-walker - to search through a struct alt_base\n   - http-push - to search through remote packs\n - has_sha1_pack\n   - builtin/fsck - already knows about promised blobs\n   - builtin/count-objects - informational purposes only (check if loose\n     object is also packed)\n   - builtin/prune-packed - check if object to be pruned is packed (if\n     not, don't prune it)\n   - revision - used to exclude packed objects if requested by user\n   - diff - just for optimization\n\nAn alternative design that I considered but rejected:\n\n - Adding a hook whenever a packed blob is requested, not on any blob.\n   That is, whenever we attempt to search the packfiles for a blob, if\n   it is missing (from the packfiles and from the loose object storage),\n   to invoke the hook (which must then store it as a packfile), open the\n   packfile the hook generated, and report that the blob is found in\n   that new packfile. This reduces the amount of analysis needed (in\n   that we only need to look at how packed blobs are handled), but\n   requires that the hook generate packfiles (or for sha1_file to pack\n   whatever loose objects are generated), creating one packfile for each\n   missing blob and potentially very many packfiles that must be\n   linearly searched. This may be tolerable now for repos that only have\n   a few missing blobs (for example, repos that only want to exclude\n   large blobs), and might be tolerable in the future if we have\n   batching support for the most commonly used commands, but is not\n   tolerable now for repos that exclude a large amount of blobs.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/config.txt               |  8 ++++\n Documentation/gitrepository-layout.txt |  8 ++++\n builtin/cat-file.c                     |  9 ++++\n promised-blob.c                        | 75 ++++++++++++++++++++++++++++++++++\n promised-blob.h                        | 13 ++++++\n sha1_file.c                            | 44 +++++++++++++-------\n t/t3907-promised-blob.sh               | 36 ++++++++++++++++\n 7 files changed, 179 insertions(+), 14 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex d5c9c4cab..c293ac921 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -393,6 +393,14 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n will probe and set core.ignoreCase true if appropriate when the repository\n is created.\n \n+core.promisedBlobCommand::\n+\tIf set, whenever a blob in the local repo is attempted to be read, but\n+\tis both missing and a promised blob, invoke this shell command to\n+\tgenerate or obtain that blob before reporting an error. This shell\n+\tcommand should take one or more hashes, each terminated by a newline,\n+\tas standard input, and (if successful) should write the corresponding\n+\tobjects to the local repo (packed or loose).\n+\n core.precomposeUnicode::\n \tThis option is only used by Mac OS implementation of Git.\n \tWhen core.precomposeUnicode=true, Git reverts the unicode decomposition\ndiff --git a/Documentation/gitrepository-layout.txt b/Documentation/gitrepository-layout.txt\nindex f51ed4e37..0ef9312fa 100644\n--- a/Documentation/gitrepository-layout.txt\n+++ b/Documentation/gitrepository-layout.txt\n@@ -47,6 +47,10 @@ use with dumb transports but otherwise is OK as long as\n `objects/info/alternates` points at the object stores it\n borrows from.\n +\n+. You could have blobs that are merely promised by another source. When\n+Git requires those blobs, it will invoke the command in the\n+`core.promisedBlobCommand` configuration variable.\n++\n This directory is ignored if $GIT_COMMON_DIR is set and\n \"$GIT_COMMON_DIR/objects\" will be used instead.\n \n@@ -91,6 +95,10 @@ objects/info/http-alternates::\n \tthis object store borrows objects from, to be used when\n \tthe repository is fetched over HTTP.\n \n+objects/promisedblob::\n+\tThis file records the sha1 object names and sizes of promised\n+\tblobs.\n+\n refs::\n \tReferences are stored in subdirectories of this\n \tdirectory.  The 'git prune' command knows to preserve\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 96b786e48..3df00bf91 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -12,6 +12,7 @@\n #include \"streaming.h\"\n #include \"tree-walk.h\"\n #include \"sha1-array.h\"\n+#include \"promised-blob.h\"\n \n struct batch_options {\n \tint enabled;\n@@ -432,6 +433,13 @@ static int batch_packed_object(const struct object_id *oid,\n \treturn 0;\n }\n \n+static int batch_promised_blob(const struct object_id *oid,\n+\t\t\t       void *data)\n+{\n+\toid_array_append(data, oid);\n+\treturn 0;\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -473,6 +481,7 @@ static int batch_objects(struct batch_options *opt)\n \n \t\tfor_each_loose_object(batch_loose_object, &sa, 0);\n \t\tfor_each_packed_object(batch_packed_object, &sa, 0);\n+\t\tfor_each_promised_blob(batch_promised_blob, &sa);\n \n \t\tcb.opt = opt;\n \t\tcb.expand = &data;\ndiff --git a/promised-blob.c b/promised-blob.c\nindex 493808ed2..0fe04daf2 100644\n--- a/promised-blob.c\n+++ b/promised-blob.c\n@@ -2,6 +2,9 @@\n #include \"promised-blob.h\"\n #include \"sha1-lookup.h\"\n #include \"strbuf.h\"\n+#include \"run-command.h\"\n+#include \"sha1-array.h\"\n+#include \"config.h\"\n \n #define ENTRY_SIZE (GIT_SHA1_RAWSZ + 8)\n /*\n@@ -93,3 +96,75 @@ int for_each_promised_blob(each_promised_blob_fn cb, void *data)\n \t}\n \treturn 0;\n }\n+\n+static char *promised_blob_command;\n+\n+static int promised_blob_config(const char *conf_key, const char *value,\n+\t\t\t\tvoid *cb)\n+{\n+\tif (!strcmp(conf_key, \"core.promisedblobcommand\")) {\n+\t\tpromised_blob_command = xstrdup(value);\n+\t}\n+\treturn 0;\n+}\n+\n+static void ensure_promised_blob_configured(void)\n+{\n+\tstatic int configured;\n+\tif (configured)\n+\t\treturn;\n+\tgit_config(promised_blob_config, NULL);\n+\tconfigured = 1;\n+}\n+\n+int request_promised_blobs(const struct oid_array *blobs)\n+{\n+\tstruct child_process cp = CHILD_PROCESS_INIT;\n+\tconst char *argv[] = {NULL, NULL};\n+\tconst char *env[] = {\"GIT_IGNORE_PROMISED_BLOBS=1\", NULL};\n+\tint blobs_requested = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < blobs->nr; i++) {\n+\t\tif (is_promised_blob(&blobs->oid[i], NULL))\n+\t\t\tbreak;\n+\t}\n+\n+\tif (i == blobs->nr)\n+\t\t/* Nothing to fetch */\n+\t\treturn 0;\n+\n+\tensure_promised_blob_configured();\n+\tif (!promised_blob_command)\n+\t\tdie(\"some promised blobs need to be fetched, but\\n\"\n+\t\t    \"no core.promisedblobcommand configured\");\n+\n+\targv[0] = promised_blob_command;\n+\tcp.argv = argv;\n+\tcp.env = env;\n+\tcp.use_shell = 1;\n+\tcp.in = -1;\n+\n+\tif (start_command(&cp))\n+\t\tdie(\"failed to start <%s>\", promised_blob_command);\n+\n+\tfor (; i < blobs->nr; i++) {\n+\t\tif (is_promised_blob(&blobs->oid[i], NULL)) {\n+\t\t\twrite_in_full(cp.in, oid_to_hex(&blobs->oid[i]),\n+\t\t\t\t      GIT_SHA1_HEXSZ);\n+\t\t\twrite_in_full(cp.in, \"\\n\", 1);\n+\t\t\tblobs_requested++;\n+\t\t}\n+\t}\n+\tclose(cp.in);\n+\n+\tif (finish_command(&cp))\n+\t\tdie(\"failed to finish <%s>\", promised_blob_command);\n+\n+\t/*\n+\t * The command above may have updated packfiles, so update our record\n+\t * of them.\n+\t */\n+\treprepare_packed_git();\n+\treturn blobs_requested;\n+}\ndiff --git a/promised-blob.h b/promised-blob.h\nindex a303ea1ff..34b1d3f9e 100644\n--- a/promised-blob.h\n+++ b/promised-blob.h\n@@ -1,6 +1,8 @@\n #ifndef PROMISED_BLOB_H\n #define PROMISED_BLOB_H\n \n+#include \"sha1-array.h\"\n+\n /*\n  * Returns 1 if oid is the name of a promised blob. If size is not NULL, also\n  * returns its size.\n@@ -11,4 +13,15 @@ extern int is_promised_blob(const struct object_id *oid,\n typedef int each_promised_blob_fn(const struct object_id *oid, void *data);\n int for_each_promised_blob(each_promised_blob_fn, void *);\n \n+/*\n+ * If any of the given blobs are promised blobs, invokes\n+ * core.promisedblobcommand with those blobs and returns the number of blobs\n+ * requested. No check is made as to whether the invocation actually populated\n+ * the repository with the promised blobs.\n+ *\n+ * If none of the given blobs are promised blobs, this function does not invoke\n+ * anything and returns 0.\n+ */\n+int request_promised_blobs(const struct oid_array *blobs);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 5862386cd..5778a6dbd 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -28,6 +28,11 @@\n #include \"list.h\"\n #include \"mergesort.h\"\n #include \"quote.h\"\n+#include \"iterator.h\"\n+#include \"dir-iterator.h\"\n+#include \"sha1-lookup.h\"\n+#include \"promised-blob.h\"\n+#include \"sha1-array.h\"\n \n #define SZ_FMT PRIuMAX\n static inline uintmax_t sz_fmt(size_t s) { return s; }\n@@ -2983,6 +2988,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n \t\t\t\t    lookup_replace_object(sha1) :\n \t\t\t\t    sha1;\n+\tint already_retried = 0;\n \n \tif (!oi)\n \t\toi = &blank_oi;\n@@ -3007,30 +3013,40 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n-\tif (!find_pack_entry(real, &e)) {\n-\t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n-\t\t\toi->whence = OI_LOOSE;\n-\t\t\treturn 0;\n-\t\t}\n+retry:\n+\tif (find_pack_entry(real, &e))\n+\t\tgoto found_packed;\n \n-\t\t/* Not a loose object; someone else may have just packed it. */\n-\t\tif (flags & OBJECT_INFO_QUICK) {\n-\t\t\treturn -1;\n-\t\t} else {\n-\t\t\treprepare_packed_git();\n-\t\t\tif (!find_pack_entry(real, &e))\n-\t\t\t\treturn -1;\n+\t/* Most likely it's a loose object. */\n+\tif (!sha1_loose_object_info(real, oi, flags)) {\n+\t\toi->whence = OI_LOOSE;\n+\t\treturn 0;\n+\t}\n+\n+\t/* Not a loose object; someone else may have just packed it. */\n+\treprepare_packed_git();\n+\tif (find_pack_entry(real, &e))\n+\t\tgoto found_packed;\n+\n+\t/* Check if it is a promised blob */\n+\tif (!already_retried) {\n+\t\tstruct oid_array promised = OID_ARRAY_INIT;\n+\t\toid_array_append_sha1(&promised, real);\n+\t\tif (request_promised_blobs(&promised)) {\n+\t\t\talready_retried = 1;\n+\t\t\tgoto retry;\n \t\t}\n \t}\n \n+\treturn -1;\n+\n+found_packed:\n \tif (oi == &blank_oi)\n \t\t/*\n \t\t * We know that the caller doesn't actually need the\n \t\t * information below, so return early.\n \t\t */\n \t\treturn 0;\n-\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\ndiff --git a/t/t3907-promised-blob.sh b/t/t3907-promised-blob.sh\nindex 827072004..778afd1f6 100755\n--- a/t/t3907-promised-blob.sh\n+++ b/t/t3907-promised-blob.sh\n@@ -26,4 +26,40 @@ test_expect_success '...but fails again with GIT_IGNORE_PROMISED_BLOBS' '\n \tunset GIT_IGNORE_PROMISED_BLOBS\n '\n \n+test_expect_success 'sha1_object_info_extended (through git cat-file)' '\n+\trm -rf server client &&\n+\n+\tgit init server &&\n+\ttest_commit -C server 1 1.t abcdefgh &&\n+\tHASH=$(git hash-object server/1.t) &&\n+\n+\tgit init client &&\n+\tgit -C client config core.promisedblobcommand \\\n+\t\t\"git -C \\\"$(pwd)/server\\\" pack-objects --stdout |\n+\t\t git unpack-objects\" &&\n+\t\n+\ttest_must_fail git -C client cat-file -p \"$HASH\"\n+'\n+\n+test_expect_success '...succeeds if it is a promised blob' '\n+\tprintf \"%s%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\thex_pack >client/.git/objects/promisedblob &&\n+\tgit -C client cat-file -p \"$HASH\"\n+'\n+\n+test_expect_success 'cat-file --batch-all-objects with promised blobs' '\n+\trm -rf client &&\n+\n+\tgit init client &&\n+\tgit -C client config core.promisedblobcommand \\\n+\t\t\"git -C \\\"$(pwd)/server\\\" pack-objects --stdout |\n+\t\t git unpack-objects\" &&\n+\tprintf \"%s%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\thex_pack >client/.git/objects/promisedblob &&\n+\n+\t# Verify that the promised blob is printed\n+\tgit -C client cat-file --batch --batch-all-objects | tee out |\n+\t\tgrep abcdefgh\n+'\n+\n test_done\n-- \n2.13.2.932.g7449e964c-goog\n\n"},{"id":"324232","messageId":"CAGZ79kZ8i0Xqp2Yft19JKzPFMUdbzEDNJ01zER9RoM8ZzKPu8A@mail.gmail.com","threadId":"46359","inReplyTo":"f9c7d4b3f800ea31e85e4897ee7048fec1e3c2f0.1499800530.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-11T22:02:09Z","receivedAt":"2017-07-11T22:02:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jul 11, 2017 at 12:48 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> Currently, Git does not support repos with very large numbers of blobs\n> or repos that wish to minimize manipulation of certain blobs (for\n> example, because they are very large) very well, even if the user\n> operates mostly on part of the repo, because Git is designed on the\n> assumption that every blob referenced by a tree object is available\n> somewhere in the repo storage.\n>\n> As a first step to reducing this problem, introduce the concept of\n> promised blobs. Each Git repo can contain a list of promised blobs and\n> their sizes at $GIT_DIR/objects/promisedblob. This patch contains\n> functions to query them; functions for creating and modifying that file\n> will be introduced in later patches.\n>\n> A repository that is missing a blob but has that blob promised is not\n> considered to be in error, so also teach fsck this.\n\nHere I wondered what this file looks like, in a later patch you\nadd documentation:\n\n  +objects/promisedblob::\n  +       This file records the sha1 object names and sizes of promised\n  +       blobs.\n  +\n\nbut that leaves more fundamental questions:\n* Is that a list of sha1s, separated by LF? (CRLF? How would Windows\n  users interact with it, are they supposed to ever modify this file?)\n* Similar to .git/packed-refs, would we want to have a first line\n  that has some sort of comment?\n* In the first question I assumed a linear list, will that be a sorted list,\n  or do we want to have some fancy data structure here?\n  (critbit tree, bloom filter)\n* is the contents in ASCII or binary? (What are the separators?)\n* In the future I presume we'd want to quickly answer \"Is X in the\n  promised blobs list?\" so would it be possible (in the future) to\n  improve the searching e.g. binary search?\n* ... I'll read on to see my questions answered, but I'd guess\n  others would prefer to see it in the docs. :)\n\n\n> + * A mmap-ed byte array of size (missing_blob_nr * ENTRY_SIZE). Each\n> + * ENTRY_SIZE-sized entry consists of the SHA-1 of the promised blob and its\n> + * 64-bit size in network byte order. The entries are sorted in ascending SHA-1\n> + * order.\n\nThis seems to be the answer to the docs. :)\nSo binary representation, the size variable takes a fixed amount, such\nthat the n-th element in the file is at n * ENTRY_SIZE.\n\n\n> +       if (st.st_size % ENTRY_SIZE) {\n> +               die(\"Size of %s is not a multiple of %d\", filename, ENTRY_SIZE);\n> +       }\n\nSo it looks as if the file format is an array of ENTRY_SIZE.\n\nSimilar to other files, would we want to prefix the file with\na 4 letter magic number and a version number?\n\n> +       prepare_promised_blobs();\n> +       result = sha1_entry_pos(promised_blobs, ENTRY_SIZE, 0, 0,\n> +                               promised_blob_nr, promised_blob_nr, oid->hash);\n\nIIRC, this is a binary search.\n\n> +int for_each_promised_blob(each_promised_blob_fn cb, void *data)\n> +{\n> +       struct object_id oid;\n> +       int i, r;\n> +\n> +       prepare_promised_blobs();\n> +       for (i = 0; i < promised_blob_nr; i++) {\n> +               memcpy(oid.hash, &promised_blobs[i * ENTRY_SIZE],\n> +                      GIT_SHA1_RAWSZ);\n> +               r = cb(&oid, data);\n> +               if (r)\n> +                       return r;\n> +       }\n> +       return 0;\n> +}\n> diff --git a/promised-blob.h b/promised-blob.h\n> new file mode 100644\n> index 000000000..a303ea1ff\n> --- /dev/null\n> +++ b/promised-blob.h\n> @@ -0,0 +1,14 @@\n> +#ifndef PROMISED_BLOB_H\n> +#define PROMISED_BLOB_H\n> +\n> +/*\n> + * Returns 1 if oid is the name of a promised blob. If size is not NULL, also\n> + * returns its size.\n> + */\n> +extern int is_promised_blob(const struct object_id *oid,\n> +                           unsigned long *size);\n> +\n> +typedef int each_promised_blob_fn(const struct object_id *oid, void *data);\n> +int for_each_promised_blob(each_promised_blob_fn, void *);\n> +\n> +#endif\n> diff --git a/t/t3907-promised-blob.sh b/t/t3907-promised-blob.sh\n> new file mode 100755\n> index 000000000..827072004\n> --- /dev/null\n> +++ b/t/t3907-promised-blob.sh\n> @@ -0,0 +1,29 @@\n> +#!/bin/sh\n> +\n> +test_description='promised blobs'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'fsck fails on missing blobs' '\n> +       rm -rf repo &&\n\nWe should not need to delete the repo here, this is the first test?\n\n> +       rm repo/.git/objects/$(echo $HASH | cut -c1-2)/$(echo $HASH | cut -c3-40) &&\n\nLater down the road, do we want to have a\n(plumbing) command to move an object from\nstandard blob to promised blob (as of now I'd think\nit would perform this rm call as well as an insert into\nthe promised blobs file) ?\n(Well, we'd also have to think about how to get objects\nout of a pack)\n\nWith such a command you can easily write your own custom\nfilter to free up blobs again.\n\n\n> +       test_must_fail git -C repo fsck\n\n    test_i18n_grep missing out ?\n\nmaybe, too? (Maybe that is already tested elsewhere,\nso no need for it)\n\n\n> +'\n> +\n> +test_expect_success '...but succeeds if it is a promised blob' '\n> +       printf \"%s%016x\" \"$HASH\" \"$(wc -c <repo/1.t)\" |\n> +               hex_pack >repo/.git/objects/promisedblob &&\n> +       git -C repo fsck\n> +'\n> +\n> +test_expect_success '...but fails again with GIT_IGNORE_PROMISED_BLOBS' '\n> +       GIT_IGNORE_PROMISED_BLOBS=1 test_must_fail git -C repo fsck &&\n> +       unset GIT_IGNORE_PROMISED_BLOBS\n> +'\n\nThis patch makes sense, so far.\n\nThanks,\nStefan\n"},{"id":"324237","messageId":"CAGZ79kZ9Ewxv=8fnRZZaQjpyAdv8df81a9aLRkJzEt5ZU0VcTA@mail.gmail.com","threadId":"46359","inReplyTo":"523f64e1395124a02316071329bedf9884757479.1499800530.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 2/3] sha1-array: support appending unsigned char hash","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-11T22:06:11Z","receivedAt":"2017-07-11T22:06:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jul 11, 2017 at 12:48 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> In a subsequent patch, sha1_file will need to append object names in the\n> form of \"unsigned char *\" to oid arrays. Teach sha1-array support for\n> that.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n\nThis breaks the oid/sha1 barrier?\n\nI would have expected the caller to do a\n\n  oid_array_append_oid(&array, sha1_to_oid(sha1));\n\nwith sha1_to_oid working off some static memory, such that the\ncall to oid_array_append_oid (which we have as oid_array_append)\nis just as cheap?\n\nCould you say more in the commit message on why we collude\nsha1 and oids here?\n\nThanks,\nStefan\n"},{"id":"324240","messageId":"CAGZ79kbdyOyu8ni6tLzb+eMcOuT0ahzVWZxgnSfO+8XYn48=Wg@mail.gmail.com","threadId":"46359","inReplyTo":"34efd9e9936fdab331655f5a33a098a72dc134f4.1499800530.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 3/3] sha1_file: add promised blob hook support","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-11T22:38:40Z","receivedAt":"2017-07-11T22:38:48Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Jul 11, 2017 at 12:48 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> Teach sha1_file to invoke a hook whenever a blob is requested and\n> unavailable but is promised. The hook is a shell command that can be\n> configured through \"git config\"; this hook takes in a list of hashes and\n> writes (if successful) the corresponding objects to the repo's local\n> storage.\n>\n> This is meant as a temporary measure to ensure that all Git commands\n> work in such a situation. Future patches will update some commands to\n> either tolerate promised blobs (without invoking the hook) or be more\n> efficient in invoking the promised blob hook.\n>\n> In order to determine the code changes in sha1_file.c necessary, I\n> investigated the following:\n>  (1) functions in sha1_file that take in a hash, without the user\n>      regarding how the object is stored (loose or packed)\n>  (2) functions in sha1_file that operate on packed objects (because I\n>      need to check callers that know about the loose/packed distinction\n>      and operate on both differently, and ensure that they can handle\n>      the concept of objects that are neither loose nor packed)\n>\n> (1) is handled by the modification to sha1_object_info_extended().\n>\n> For (2), I looked at for_each_packed_object and at the packed-related\n> functions that take in a hash. For for_each_packed_object, the callers\n> either already work or are fixed in this patch:\n>  - reachable - only to find recent objects\n>  - builtin/fsck - already knows about promised blobs\n>  - builtin/cat-file - fixed in this commit\n>\n> Callers of the other functions do not need to be changed:\n>  - parse_pack_index\n>    - http - indirectly from http_get_info_packs\n>  - find_pack_entry_one\n>    - this searches a single pack that is provided as an argument; the\n>      caller already knows (through other means) that the sought object\n>      is in a specific pack\n>  - find_sha1_pack\n>    - fast-import - appears to be an optimization to not store a\n>      file if it is already in a pack\n>    - http-walker - to search through a struct alt_base\n>    - http-push - to search through remote packs\n>  - has_sha1_pack\n>    - builtin/fsck - already knows about promised blobs\n>    - builtin/count-objects - informational purposes only (check if loose\n>      object is also packed)\n>    - builtin/prune-packed - check if object to be pruned is packed (if\n>      not, don't prune it)\n>    - revision - used to exclude packed objects if requested by user\n>    - diff - just for optimization\n>\n> An alternative design that I considered but rejected:\n>\n>  - Adding a hook whenever a packed blob is requested, not on any blob.\n>    That is, whenever we attempt to search the packfiles for a blob, if\n>    it is missing (from the packfiles and from the loose object storage),\n>    to invoke the hook (which must then store it as a packfile), open the\n>    packfile the hook generated, and report that the blob is found in\n>    that new packfile. This reduces the amount of analysis needed (in\n>    that we only need to look at how packed blobs are handled), but\n>    requires that the hook generate packfiles (or for sha1_file to pack\n>    whatever loose objects are generated), creating one packfile for each\n>    missing blob and potentially very many packfiles that must be\n>    linearly searched. This may be tolerable now for repos that only have\n>    a few missing blobs (for example, repos that only want to exclude\n>    large blobs), and might be tolerable in the future if we have\n>    batching support for the most commonly used commands, but is not\n>    tolerable now for repos that exclude a large amount of blobs.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n\n>\n> +core.promisedBlobCommand::\n> +       If set, whenever a blob in the local repo is attempted to be read, but\n> +       is both missing and a promised blob, invoke this shell command to\n> +       generate or obtain that blob before reporting an error. This shell\n> +       command should take one or more hashes, each terminated by a newline,\n> +       as standard input, and (if successful) should write the corresponding\n> +       objects to the local repo (packed or loose).\n\nAny git command invoked here behaves differently as\nGIT_IGNORE_PROMISED_BLOBS is set.\nGIT_IGNORE_PROMISED_BLOBS needs documentation.\n\n\n> +int request_promised_blobs(const struct oid_array *blobs)\n\n> +       if (i == blobs->nr)\n> +               /* Nothing to fetch */\n\ns/to fetch/to get/ ? Might be to nitpicky.\n\nSo this function takes an array of blobs and obtains any that\nare promised.\n\n> +\n> +       argv[0] = promised_blob_command;\n> +       cp.argv = argv;\n> +       cp.env = env;\n\nI had the impression that we'd favor .args and .env_array\nnowdays, such that we'd not need to construct the NULL\nterminated array ourselves. (e.g. whenever we'd change argv,\nwe'd need to make sure it contains enough NULL at the end)\n\n    argv_array_push(&cp.args, promised_blob_command);\n    argv_array_push(&cp.env_array, \\\n        \"GIT_IGNORE_PROMISED_BLOBS=1\");\n\nwould have the same effect.\n\n\n> +/*\n> + * If any of the given blobs are promised blobs, invokes\n> + * core.promisedblobcommand with those blobs and returns the number of blobs\n> + * requested. No check is made as to whether the invocation actually populated\n> + * the repository with the promised blobs.\n> + *\n> + * If none of the given blobs are promised blobs, this function does not invoke\n> + * anything and returns 0.\n\nThanks for the docs here.\n  offtopic: As a general thing for our patches, can we configure\n  (or even convince Git in general), that headers ought to be sent *before*\n  its accompanying source? I think that would help reviewers like me, who\n  tend to start reading linearly and then giving random thoughts, because the\n  header prepares the reviewer for the source code with expectations. Also\n  by having it the other way around, the review first focuses on design (Is this\n  function signature sane; the docs said it would do X while not doing\nY, is that\n  sane?) instead of code.\n\n  IIRC Gerrit has an option (or default) that the header is before the source,\n  despite the order violating the git sorting of files.\n\n\n\n> + */\n> +int request_promised_blobs(const struct oid_array *blobs);\n> +\n>  #endif\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 5862386cd..5778a6dbd 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -28,6 +28,11 @@\n>  #include \"list.h\"\n>  #include \"mergesort.h\"\n>  #include \"quote.h\"\n> +#include \"iterator.h\"\n> +#include \"dir-iterator.h\"\n> +#include \"sha1-lookup.h\"\n> +#include \"promised-blob.h\"\n> +#include \"sha1-array.h\"\n>\n>  #define SZ_FMT PRIuMAX\n>  static inline uintmax_t sz_fmt(size_t s) { return s; }\n> @@ -2983,6 +2988,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>         const unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n>                                     lookup_replace_object(sha1) :\n>                                     sha1;\n> +       int already_retried = 0;\n>\n>         if (!oi)\n>                 oi = &blank_oi;\n> @@ -3007,30 +3013,40 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>                 }\n>         }\n>\n> -       if (!find_pack_entry(real, &e)) {\n> -               /* Most likely it's a loose object. */\n> -               if (!sha1_loose_object_info(real, oi, flags)) {\n> -                       oi->whence = OI_LOOSE;\n> -                       return 0;\n> -               }\n> +retry:\n> +       if (find_pack_entry(real, &e))\n> +               goto found_packed;\n>\n> -               /* Not a loose object; someone else may have just packed it. */\n> -               if (flags & OBJECT_INFO_QUICK) {\n> -                       return -1;\n> -               } else {\n> -                       reprepare_packed_git();\n> -                       if (!find_pack_entry(real, &e))\n> -                               return -1;\n> +       /* Most likely it's a loose object. */\n> +       if (!sha1_loose_object_info(real, oi, flags)) {\n> +               oi->whence = OI_LOOSE;\n> +               return 0;\n> +       }\n> +\n> +       /* Not a loose object; someone else may have just packed it. */\n> +       reprepare_packed_git();\n> +       if (find_pack_entry(real, &e))\n> +               goto found_packed;\n> +\n> +       /* Check if it is a promised blob */\n> +       if (!already_retried) {\n> +               struct oid_array promised = OID_ARRAY_INIT;\n> +               oid_array_append_sha1(&promised, real);\n> +               if (request_promised_blobs(&promised)) {\n> +                       already_retried = 1;\n> +                       goto retry;\n\nOnce we have the promised blob, do we want to\nremove it from the list of promised blobs?\n(Maybe check first if we have it actually?)\n\nOr is the opposite true:\nWe rather want to keep the promised blobs list,\nsuch that we can easily get rid of good candidates\nto make space on the local fs?\n\n\n> @@ -26,4 +26,40 @@ test_expect_success '...but fails again with GIT_IGNORE_PROMISED_BLOBS' '\n>         unset GIT_IGNORE_PROMISED_BLOBS\n>  '\n>\n> +test_expect_success 'sha1_object_info_extended (through git cat-file)' '\n> +       rm -rf server client &&\n\nDid you take these rm -rf as the first line of a test from any other tests?\n(If so I'd call them out as bad examples)\n\nI think it is better if we'd do\n\n    test_when_finished \"rm -rf ...\" &&\n\nin a test, as then each test is self contained and cleans up\nafter itself as opposed to cleaning up for the guy in front of you.\n\n> +       git init server &&\n\ntest_create_repo maybe?\n\n> +       test_commit -C server 1 1.t abcdefgh &&\n> +       HASH=$(git hash-object server/1.t) &&\n> +\n> +       git init client &&\n> +       git -C client config core.promisedblobcommand \\\n> +               \"git -C \\\"$(pwd)/server\\\" pack-objects --stdout |\n> +                git unpack-objects\" &&\n\nNothing is wrong with absolute path, except relative paths\nmay be easier to read here. Is the core.promisedblobcommand\npromised to run in a particular directory? Do we want to make that\npromise (if so, we'd want the test with relative path IMHO).\n\nI like the series! A pleasant read, though I am tossing back\na lot of questions.\n\nThanks,\nStefan\n"},{"id":"324272","messageId":"890a36fc-be16-83bc-fec6-94e21d0f7d0a@jeffhostetler.com","threadId":"46359","inReplyTo":"f9c7d4b3f800ea31e85e4897ee7048fec1e3c2f0.1499800530.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-12T17:29:11Z","receivedAt":"2017-07-12T17:29:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/11/2017 3:48 PM, Jonathan Tan wrote:\n> Currently, Git does not support repos with very large numbers of blobs\n> or repos that wish to minimize manipulation of certain blobs (for\n> example, because they are very large) very well, even if the user\n> operates mostly on part of the repo, because Git is designed on the\n> assumption that every blob referenced by a tree object is available\n> somewhere in the repo storage.\n> \n> As a first step to reducing this problem, introduce the concept of\n> promised blobs. Each Git repo can contain a list of promised blobs and\n> their sizes at $GIT_DIR/objects/promisedblob. This patch contains\n> functions to query them; functions for creating and modifying that file\n> will be introduced in later patches.\n\n\nAs part of my on-going effort on partial/narrow clone/fetch I've also\nlooked at how to represent the set of omitted objects and whether or\nnot we should even try.\n\nMy primary concern is scale and managing the list of objects over time.\n\nMy fear is that this list will be quite large.  If we only want to omit\nthe very large blobs, then maybe not.  But if we want to expand that\nscope to also omit other objects (such as a clone synchronized with a\nsparse checkout), then that list will get large on large repos.  For\nexample, on the Windows repo we have (conservatively) 100M+ blobs (and\ngrowing).  Assuming 28 bytes per, gives a 2.8GB list to be manipulated.\n\nIf I understand your proposal, newly-omitted blobs would need to be\nmerged into the promised-blob list after each fetch.  The fetch itself\nmay not have that many new entries, but inserting them into the existing\nlist will be slow.  Also, mmap'ing and bsearch'ing will likely have\nissues.  And there's likely to be a very expensive step to remove\nentries from the list as new blobs are received (or locally created).\n\nIn such a \"sparse clone\", it would be nice to omit unneeded tree objects\nin addition to just blobs.   I say that because we are finding with GVFS\non the Windows repo, that even with commits-and-trees-only filtering,\nthe number of tree objects is overwhelming.  So I'm also concerned about\nlimiting the list to just blobs.  If we need to have this list, it\nshould be able to contain any object.  (Suggesting having an object type\nin the entry.)\n\nI assume that we'll also need a promised-blob.lock file to control\naccess during list manipulation.  This is already a sore spot with the\nindex; I'd hate to create another one.\n\nI also have to wonder about the need to have a complete list of omitted\nblobs up front.  It may be better to just relax the consistency checks\nand assume a missing blob is \"intentionally missing\" rather than\nindicating a corruption somewhere.  And then let the client do a later\nround-trip to either demand-load the object -or- demand-load the\nexistence/size info if/when it really matters.\n\nMaybe we should add a verb to your new fetch-blob endpoint to just get\nthe size of one or more objects to help with this.\n\nThanks,\nJeff\n\n"},{"id":"324274","messageId":"8366d1de-3552-50fc-7a6c-7cfc3219181b@gmail.com","threadId":"46359","inReplyTo":"34efd9e9936fdab331655f5a33a098a72dc134f4.1499800530.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 3/3] sha1_file: add promised blob hook support","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-12T17:40:49Z","receivedAt":"2017-07-12T17:41:05Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/11/2017 3:48 PM, Jonathan Tan wrote:\n> Teach sha1_file to invoke a hook whenever a blob is requested and\n> unavailable but is promised. The hook is a shell command that can be\n> configured through \"git config\"; this hook takes in a list of hashes and\n> writes (if successful) the corresponding objects to the repo's local\n> storage.\n> \n\nThis would seem to work well for promised blobs but does it currently \nsupport other object types?  In our large mono repos, we have to support \nmissing commits and trees as well.  I'd rather not have \nseparate/different ways to handle the various missing object types.\n\nThere are a couple of related patch series in flight that are playing in \nthis area. We should definitely coordinate these various efforts.\n\n[RFC] Add support for downloading blobs on demand\nhttps://public-inbox.org/git/009d01d28deb$158563a0$40902ae0$@gmail.com/\n\nThis series adds read-object-process support to download on demand any \nmissing object type.  This is based on the code we are using today. I'll \nupdate it to 1) take advantage of the refactored sub-process support \n(https://public-inbox.org/git/20170505152802.6724-1-benpeart@microsoft.com/) \nand 2) fix a bug we discovered with alternates support and resubmit it \nas an updated RFC.\n\n[RFC/PATCH v4 00/49] Add initial experimental external ODB support\nhttps://public-inbox.org/git/20170620075523.26961-1-chriscool@tuxfamily.org/\n\nThis is a broader patch series that enables you to register multiple \nhandlers for \"missing\" objects. It also already incorporates the \nread-object-process patch series above.\n\nThe advantage of these other patch series is that they already support \n_any_ object type, not just blobs or promised blobs.\n\n> This is meant as a temporary measure to ensure that all Git commands\n> work in such a situation. Future patches will update some commands to\n> either tolerate promised blobs (without invoking the hook) or be more\n> efficient in invoking the promised blob hook.\n> \n> In order to determine the code changes in sha1_file.c necessary, I\n> investigated the following:\n>   (1) functions in sha1_file that take in a hash, without the user\n>       regarding how the object is stored (loose or packed)\n>   (2) functions in sha1_file that operate on packed objects (because I\n>       need to check callers that know about the loose/packed distinction\n>       and operate on both differently, and ensure that they can handle\n>       the concept of objects that are neither loose nor packed)\n> \n> (1) is handled by the modification to sha1_object_info_extended().\n> \n> For (2), I looked at for_each_packed_object and at the packed-related\n> functions that take in a hash. For for_each_packed_object, the callers\n> either already work or are fixed in this patch:\n>   - reachable - only to find recent objects\n>   - builtin/fsck - already knows about promised blobs\n>   - builtin/cat-file - fixed in this commit\n> \n> Callers of the other functions do not need to be changed:\n>   - parse_pack_index\n>     - http - indirectly from http_get_info_packs\n>   - find_pack_entry_one\n>     - this searches a single pack that is provided as an argument; the\n>       caller already knows (through other means) that the sought object\n>       is in a specific pack\n>   - find_sha1_pack\n>     - fast-import - appears to be an optimization to not store a\n>       file if it is already in a pack\n>     - http-walker - to search through a struct alt_base\n>     - http-push - to search through remote packs\n>   - has_sha1_pack\n>     - builtin/fsck - already knows about promised blobs\n>     - builtin/count-objects - informational purposes only (check if loose\n>       object is also packed)\n>     - builtin/prune-packed - check if object to be pruned is packed (if\n>       not, don't prune it)\n>     - revision - used to exclude packed objects if requested by user\n>     - diff - just for optimization\n> \n> An alternative design that I considered but rejected:\n> \n>   - Adding a hook whenever a packed blob is requested, not on any blob.\n>     That is, whenever we attempt to search the packfiles for a blob, if\n>     it is missing (from the packfiles and from the loose object storage),\n>     to invoke the hook (which must then store it as a packfile), open the\n>     packfile the hook generated, and report that the blob is found in\n>     that new packfile. This reduces the amount of analysis needed (in\n>     that we only need to look at how packed blobs are handled), but\n>     requires that the hook generate packfiles (or for sha1_file to pack\n>     whatever loose objects are generated), creating one packfile for each\n>     missing blob and potentially very many packfiles that must be\n>     linearly searched. This may be tolerable now for repos that only have\n>     a few missing blobs (for example, repos that only want to exclude\n>     large blobs), and might be tolerable in the future if we have\n>     batching support for the most commonly used commands, but is not\n>     tolerable now for repos that exclude a large amount of blobs.\n> \n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>   Documentation/config.txt               |  8 ++++\n>   Documentation/gitrepository-layout.txt |  8 ++++\n>   builtin/cat-file.c                     |  9 ++++\n>   promised-blob.c                        | 75 ++++++++++++++++++++++++++++++++++\n>   promised-blob.h                        | 13 ++++++\n>   sha1_file.c                            | 44 +++++++++++++-------\n>   t/t3907-promised-blob.sh               | 36 ++++++++++++++++\n>   7 files changed, 179 insertions(+), 14 deletions(-)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index d5c9c4cab..c293ac921 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -393,6 +393,14 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n>   will probe and set core.ignoreCase true if appropriate when the repository\n>   is created.\n>   \n> +core.promisedBlobCommand::\n> +\tIf set, whenever a blob in the local repo is attempted to be read, but\n> +\tis both missing and a promised blob, invoke this shell command to\n> +\tgenerate or obtain that blob before reporting an error. This shell\n> +\tcommand should take one or more hashes, each terminated by a newline,\n> +\tas standard input, and (if successful) should write the corresponding\n> +\tobjects to the local repo (packed or loose).\n> +\n>   core.precomposeUnicode::\n>   \tThis option is only used by Mac OS implementation of Git.\n>   \tWhen core.precomposeUnicode=true, Git reverts the unicode decomposition\n> diff --git a/Documentation/gitrepository-layout.txt b/Documentation/gitrepository-layout.txt\n> index f51ed4e37..0ef9312fa 100644\n> --- a/Documentation/gitrepository-layout.txt\n> +++ b/Documentation/gitrepository-layout.txt\n> @@ -47,6 +47,10 @@ use with dumb transports but otherwise is OK as long as\n>   `objects/info/alternates` points at the object stores it\n>   borrows from.\n>   +\n> +. You could have blobs that are merely promised by another source. When\n> +Git requires those blobs, it will invoke the command in the\n> +`core.promisedBlobCommand` configuration variable.\n> ++\n>   This directory is ignored if $GIT_COMMON_DIR is set and\n>   \"$GIT_COMMON_DIR/objects\" will be used instead.\n>   \n> @@ -91,6 +95,10 @@ objects/info/http-alternates::\n>   \tthis object store borrows objects from, to be used when\n>   \tthe repository is fetched over HTTP.\n>   \n> +objects/promisedblob::\n> +\tThis file records the sha1 object names and sizes of promised\n> +\tblobs.\n> +\n>   refs::\n>   \tReferences are stored in subdirectories of this\n>   \tdirectory.  The 'git prune' command knows to preserve\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 96b786e48..3df00bf91 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -12,6 +12,7 @@\n>   #include \"streaming.h\"\n>   #include \"tree-walk.h\"\n>   #include \"sha1-array.h\"\n> +#include \"promised-blob.h\"\n>   \n>   struct batch_options {\n>   \tint enabled;\n> @@ -432,6 +433,13 @@ static int batch_packed_object(const struct object_id *oid,\n>   \treturn 0;\n>   }\n>   \n> +static int batch_promised_blob(const struct object_id *oid,\n> +\t\t\t       void *data)\n> +{\n> +\toid_array_append(data, oid);\n> +\treturn 0;\n> +}\n> +\n>   static int batch_objects(struct batch_options *opt)\n>   {\n>   \tstruct strbuf buf = STRBUF_INIT;\n> @@ -473,6 +481,7 @@ static int batch_objects(struct batch_options *opt)\n>   \n>   \t\tfor_each_loose_object(batch_loose_object, &sa, 0);\n>   \t\tfor_each_packed_object(batch_packed_object, &sa, 0);\n> +\t\tfor_each_promised_blob(batch_promised_blob, &sa);\n>   \n>   \t\tcb.opt = opt;\n>   \t\tcb.expand = &data;\n> diff --git a/promised-blob.c b/promised-blob.c\n> index 493808ed2..0fe04daf2 100644\n> --- a/promised-blob.c\n> +++ b/promised-blob.c\n> @@ -2,6 +2,9 @@\n>   #include \"promised-blob.h\"\n>   #include \"sha1-lookup.h\"\n>   #include \"strbuf.h\"\n> +#include \"run-command.h\"\n> +#include \"sha1-array.h\"\n> +#include \"config.h\"\n>   \n>   #define ENTRY_SIZE (GIT_SHA1_RAWSZ + 8)\n>   /*\n> @@ -93,3 +96,75 @@ int for_each_promised_blob(each_promised_blob_fn cb, void *data)\n>   \t}\n>   \treturn 0;\n>   }\n> +\n> +static char *promised_blob_command;\n> +\n> +static int promised_blob_config(const char *conf_key, const char *value,\n> +\t\t\t\tvoid *cb)\n> +{\n> +\tif (!strcmp(conf_key, \"core.promisedblobcommand\")) {\n> +\t\tpromised_blob_command = xstrdup(value);\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +static void ensure_promised_blob_configured(void)\n> +{\n> +\tstatic int configured;\n> +\tif (configured)\n> +\t\treturn;\n> +\tgit_config(promised_blob_config, NULL);\n> +\tconfigured = 1;\n> +}\n> +\n> +int request_promised_blobs(const struct oid_array *blobs)\n> +{\n> +\tstruct child_process cp = CHILD_PROCESS_INIT;\n> +\tconst char *argv[] = {NULL, NULL};\n> +\tconst char *env[] = {\"GIT_IGNORE_PROMISED_BLOBS=1\", NULL};\n> +\tint blobs_requested = 0;\n> +\tint i;\n> +\n> +\tfor (i = 0; i < blobs->nr; i++) {\n> +\t\tif (is_promised_blob(&blobs->oid[i], NULL))\n> +\t\t\tbreak;\n> +\t}\n> +\n> +\tif (i == blobs->nr)\n> +\t\t/* Nothing to fetch */\n> +\t\treturn 0;\n> +\n> +\tensure_promised_blob_configured();\n> +\tif (!promised_blob_command)\n> +\t\tdie(\"some promised blobs need to be fetched, but\\n\"\n> +\t\t    \"no core.promisedblobcommand configured\");\n> +\n> +\targv[0] = promised_blob_command;\n> +\tcp.argv = argv;\n> +\tcp.env = env;\n> +\tcp.use_shell = 1;\n> +\tcp.in = -1;\n> +\n> +\tif (start_command(&cp))\n> +\t\tdie(\"failed to start <%s>\", promised_blob_command);\n> +\n> +\tfor (; i < blobs->nr; i++) {\n> +\t\tif (is_promised_blob(&blobs->oid[i], NULL)) {\n> +\t\t\twrite_in_full(cp.in, oid_to_hex(&blobs->oid[i]),\n> +\t\t\t\t      GIT_SHA1_HEXSZ);\n> +\t\t\twrite_in_full(cp.in, \"\\n\", 1);\n> +\t\t\tblobs_requested++;\n> +\t\t}\n> +\t}\n> +\tclose(cp.in);\n> +\n> +\tif (finish_command(&cp))\n> +\t\tdie(\"failed to finish <%s>\", promised_blob_command);\n> +\n> +\t/*\n> +\t * The command above may have updated packfiles, so update our record\n> +\t * of them.\n> +\t */\n> +\treprepare_packed_git();\n> +\treturn blobs_requested;\n> +}\n> diff --git a/promised-blob.h b/promised-blob.h\n> index a303ea1ff..34b1d3f9e 100644\n> --- a/promised-blob.h\n> +++ b/promised-blob.h\n> @@ -1,6 +1,8 @@\n>   #ifndef PROMISED_BLOB_H\n>   #define PROMISED_BLOB_H\n>   \n> +#include \"sha1-array.h\"\n> +\n>   /*\n>    * Returns 1 if oid is the name of a promised blob. If size is not NULL, also\n>    * returns its size.\n> @@ -11,4 +13,15 @@ extern int is_promised_blob(const struct object_id *oid,\n>   typedef int each_promised_blob_fn(const struct object_id *oid, void *data);\n>   int for_each_promised_blob(each_promised_blob_fn, void *);\n>   \n> +/*\n> + * If any of the given blobs are promised blobs, invokes\n> + * core.promisedblobcommand with those blobs and returns the number of blobs\n> + * requested. No check is made as to whether the invocation actually populated\n> + * the repository with the promised blobs.\n> + *\n> + * If none of the given blobs are promised blobs, this function does not invoke\n> + * anything and returns 0.\n> + */\n> +int request_promised_blobs(const struct oid_array *blobs);\n> +\n>   #endif\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 5862386cd..5778a6dbd 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -28,6 +28,11 @@\n>   #include \"list.h\"\n>   #include \"mergesort.h\"\n>   #include \"quote.h\"\n> +#include \"iterator.h\"\n> +#include \"dir-iterator.h\"\n> +#include \"sha1-lookup.h\"\n> +#include \"promised-blob.h\"\n> +#include \"sha1-array.h\"\n>   \n>   #define SZ_FMT PRIuMAX\n>   static inline uintmax_t sz_fmt(size_t s) { return s; }\n> @@ -2983,6 +2988,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>   \tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n>   \t\t\t\t    lookup_replace_object(sha1) :\n>   \t\t\t\t    sha1;\n> +\tint already_retried = 0;\n>   \n>   \tif (!oi)\n>   \t\toi = &blank_oi;\n> @@ -3007,30 +3013,40 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>   \t\t}\n>   \t}\n>   \n> -\tif (!find_pack_entry(real, &e)) {\n> -\t\t/* Most likely it's a loose object. */\n> -\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n> -\t\t\toi->whence = OI_LOOSE;\n> -\t\t\treturn 0;\n> -\t\t}\n> +retry:\n> +\tif (find_pack_entry(real, &e))\n> +\t\tgoto found_packed;\n>   \n> -\t\t/* Not a loose object; someone else may have just packed it. */\n> -\t\tif (flags & OBJECT_INFO_QUICK) {\n> -\t\t\treturn -1;\n> -\t\t} else {\n> -\t\t\treprepare_packed_git();\n> -\t\t\tif (!find_pack_entry(real, &e))\n> -\t\t\t\treturn -1;\n> +\t/* Most likely it's a loose object. */\n> +\tif (!sha1_loose_object_info(real, oi, flags)) {\n> +\t\toi->whence = OI_LOOSE;\n> +\t\treturn 0;\n> +\t}\n> +\n> +\t/* Not a loose object; someone else may have just packed it. */\n> +\treprepare_packed_git();\n> +\tif (find_pack_entry(real, &e))\n> +\t\tgoto found_packed;\n> +\n> +\t/* Check if it is a promised blob */\n> +\tif (!already_retried) {\n> +\t\tstruct oid_array promised = OID_ARRAY_INIT;\n> +\t\toid_array_append_sha1(&promised, real);\n> +\t\tif (request_promised_blobs(&promised)) {\n> +\t\t\talready_retried = 1;\n> +\t\t\tgoto retry;\n>   \t\t}\n>   \t}\n>   \n> +\treturn -1;\n> +\n> +found_packed:\n>   \tif (oi == &blank_oi)\n>   \t\t/*\n>   \t\t * We know that the caller doesn't actually need the\n>   \t\t * information below, so return early.\n>   \t\t */\n>   \t\treturn 0;\n> -\n>   \trtype = packed_object_info(e.p, e.offset, oi);\n>   \tif (rtype < 0) {\n>   \t\tmark_bad_packed_object(e.p, real);\n> diff --git a/t/t3907-promised-blob.sh b/t/t3907-promised-blob.sh\n> index 827072004..778afd1f6 100755\n> --- a/t/t3907-promised-blob.sh\n> +++ b/t/t3907-promised-blob.sh\n> @@ -26,4 +26,40 @@ test_expect_success '...but fails again with GIT_IGNORE_PROMISED_BLOBS' '\n>   \tunset GIT_IGNORE_PROMISED_BLOBS\n>   '\n>   \n> +test_expect_success 'sha1_object_info_extended (through git cat-file)' '\n> +\trm -rf server client &&\n> +\n> +\tgit init server &&\n> +\ttest_commit -C server 1 1.t abcdefgh &&\n> +\tHASH=$(git hash-object server/1.t) &&\n> +\n> +\tgit init client &&\n> +\tgit -C client config core.promisedblobcommand \\\n> +\t\t\"git -C \\\"$(pwd)/server\\\" pack-objects --stdout |\n> +\t\t git unpack-objects\" &&\n> +\t\n> +\ttest_must_fail git -C client cat-file -p \"$HASH\"\n> +'\n> +\n> +test_expect_success '...succeeds if it is a promised blob' '\n> +\tprintf \"%s%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n> +\t\thex_pack >client/.git/objects/promisedblob &&\n> +\tgit -C client cat-file -p \"$HASH\"\n> +'\n> +\n> +test_expect_success 'cat-file --batch-all-objects with promised blobs' '\n> +\trm -rf client &&\n> +\n> +\tgit init client &&\n> +\tgit -C client config core.promisedblobcommand \\\n> +\t\t\"git -C \\\"$(pwd)/server\\\" pack-objects --stdout |\n> +\t\t git unpack-objects\" &&\n> +\tprintf \"%s%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n> +\t\thex_pack >client/.git/objects/promisedblob &&\n> +\n> +\t# Verify that the promised blob is printed\n> +\tgit -C client cat-file --batch --batch-all-objects | tee out |\n> +\t\tgrep abcdefgh\n> +'\n> +\n>   test_done\n> \n"},{"id":"324291","messageId":"20170712192831.GG93855@aiede.mtv.corp.google.com","threadId":"46359","inReplyTo":"890a36fc-be16-83bc-fec6-94e21d0f7d0a@jeffhostetler.com","subject":"Re: [RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-12T19:28:31Z","receivedAt":"2017-07-12T19:28:40Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJeff Hostetler wrote:\n\n> My primary concern is scale and managing the list of objects over time.\n[...]\n>                                                                  For\n> example, on the Windows repo we have (conservatively) 100M+ blobs (and\n> growing).  Assuming 28 bytes per, gives a 2.8GB list to be manipulated.\n>\n> If I understand your proposal, newly-omitted blobs would need to be\n> merged into the promised-blob list after each fetch.  The fetch itself\n> may not have that many new entries, but inserting them into the existing\n> list will be slow.\n\nThis is a good point.  An alternative would be to avoid storing the\nlist and instead use a repository extension that treats all missing\nblobs in the repository similarly to promised blobs (with a weaker\npromise, where the server is allowed to return 404).  The downsides:\n\n- blob sizes are not available without an additional request, e.g. for\n  directory listings\n\n- the semantics of has_object_file become more complicated.  If it\n  is allowed to return false for omitted blobs, then callers have to\n  be audited to tolerate that and try to look up the blob anyway.\n  If it has to contact the server to find out whether an omitted blob\n  is available, then callers have to be audited to skip this expensive\n  operation when possible.\n\n- similarly, the semantics of sha1_object_info{,_extended} become more\n  complicated.  If they are allowed to return -1 for omitted blobs,\n  then callers have to be audited to handle that. If they have to\n  contact the server to find the object type and size, it becomes\n  expensive in a way that affects callers.\n\n- causes futile repeated requests to the server for objects that don't\n  exist.  Caching negative lookups is fussy because a later push could\n  cause those objects to exist --- though it should be possible for\n  fetches to invalidate entries in such a cache using the list of\n  promised blobs sent by the server.\n\n[...]\n> In such a \"sparse clone\", it would be nice to omit unneeded tree objects\n> in addition to just blobs.   I say that because we are finding with GVFS\n> on the Windows repo, that even with commits-and-trees-only filtering,\n> the number of tree objects is overwhelming.  So I'm also concerned about\n> limiting the list to just blobs.  If we need to have this list, it\n> should be able to contain any object.  (Suggesting having an object type\n> in the entry.)\n\nWould it work to have a separate lists of blobs and trees (either in\nseparate files or in the same file)?\n\nOne option would be to add a version number / magic string to the start\nof the file.  That would allow making format changes later without a\nproliferation of distinct repository extensions.\n\n[...]\n> I assume that we'll also need a promised-blob.lock file to control\n> access during list manipulation.  This is already a sore spot with the\n> index; I'd hate to create another one.\n\nCan you say more about this concern?  My experience with concurrent\nfetches has already not been great (since one fetch process is not\naware of what the other has fetched) --- is your concern that the\npromised-blob facility would affect pushes as well some day?\n\n> I also have to wonder about the need to have a complete list of omitted\n> blobs up front.  It may be better to just relax the consistency checks\n> and assume a missing blob is \"intentionally missing\" rather than\n> indicating a corruption somewhere.\n\nWe've discussed this previously on list and gone back and forth. :)\n\n>                                     And then let the client do a later\n> round-trip to either demand-load the object -or- demand-load the\n> existence/size info if/when it really matters.\n\nThe cost of demand-loading this existence/size information is what\nultimately convinced me of this approach.\n\nBut I can see how the tradeoffs differ between the omit-large-blobs\ncase and the omit-all-blobs case.  We might end up having to support\nboth modes. :(\n\n> Maybe we should add a verb to your new fetch-blob endpoint to just get\n> the size of one or more objects to help with this.\n\nNo objections from me, though we don't need it yet.\n\nThanks,\nJonathan\n"},{"id":"324294","messageId":"20170712203822.GH93855@aiede.mtv.corp.google.com","threadId":"46359","inReplyTo":"8366d1de-3552-50fc-7a6c-7cfc3219181b@gmail.com","subject":"Re: [RFC PATCH 3/3] sha1_file: add promised blob hook support","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-12T20:38:22Z","receivedAt":"2017-07-12T20:38:32Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBen Peart wrote:\n> On 7/11/2017 3:48 PM, Jonathan Tan wrote:\n\n>> Teach sha1_file to invoke a hook whenever a blob is requested and\n>> unavailable but is promised. The hook is a shell command that can be\n>> configured through \"git config\"; this hook takes in a list of hashes and\n>> writes (if successful) the corresponding objects to the repo's local\n>> storage.\n>\n> This would seem to work well for promised blobs but does it\n> currently support other object types?  In our large mono repos, we\n> have to support missing commits and trees as well.\n\nCan you elaborate on this?  This is the first time I've heard about\nit.\n\nAre you omitting commits and trees today?  How does that work with\ncommits --- is it something different from shallow clone?\n\n[...]\n> There are a couple of related patch series in flight that are\n> playing in this area. We should definitely coordinate these various\n> efforts.\n\nIn that spirit, I've created\nhttps://github.com/jrn/git-large-repositories so we have a shared\nplace to keep track of pointers to the mailing list, etc.  I'll push\nthe links mentioned to there now.\n\nI'm also usually available on IRC (channel #git-devel on freenode,\nusername jrnieder) for real-time conversation.  Logs are available at\nhttp://colabti.org/irclogger/irclogger_logs/git-devel.  You can\nprepend a message with [off] if you want it not to be logged.\n\nThanks,\nJonathan\n"},{"id":"324350","messageId":"4b326d17-3896-a374-9d7a-d0c89954c943@jeffhostetler.com","threadId":"46359","inReplyTo":"20170712192831.GG93855@aiede.mtv.corp.google.com","subject":"Re: [RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T14:48:30Z","receivedAt":"2017-07-13T14:48:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/12/2017 3:28 PM, Jonathan Nieder wrote:\n> Hi,\n> \n> Jeff Hostetler wrote:\n> \n>> My primary concern is scale and managing the list of objects over time.\n> [...]\n>>                                                                   For\n>> example, on the Windows repo we have (conservatively) 100M+ blobs (and\n>> growing).  Assuming 28 bytes per, gives a 2.8GB list to be manipulated.\n>>\n>> If I understand your proposal, newly-omitted blobs would need to be\n>> merged into the promised-blob list after each fetch.  The fetch itself\n>> may not have that many new entries, but inserting them into the existing\n>> list will be slow.\n> \n> This is a good point.  An alternative would be to avoid storing the\n> list and instead use a repository extension that treats all missing\n> blobs in the repository similarly to promised blobs (with a weaker\n> promise, where the server is allowed to return 404).  The downsides:\n> \n> - blob sizes are not available without an additional request, e.g. for\n>    directory listings\n> \n> - the semantics of has_object_file become more complicated.  If it\n>    is allowed to return false for omitted blobs, then callers have to\n>    be audited to tolerate that and try to look up the blob anyway.\n>    If it has to contact the server to find out whether an omitted blob\n>    is available, then callers have to be audited to skip this expensive\n>    operation when possible.\n> \n> - similarly, the semantics of sha1_object_info{,_extended} become more\n>    complicated.  If they are allowed to return -1 for omitted blobs,\n>    then callers have to be audited to handle that. If they have to\n>    contact the server to find the object type and size, it becomes\n>    expensive in a way that affects callers.\n> \n> - causes futile repeated requests to the server for objects that don't\n>    exist.  Caching negative lookups is fussy because a later push could\n>    cause those objects to exist --- though it should be possible for\n>    fetches to invalidate entries in such a cache using the list of\n>    promised blobs sent by the server.\n\nAlso good points. :-)\n\nWould it be better to:\n[] let fetch-pack/index-pack on the client compute the set of missing\n    objects based upon the commit range in the response.\n[] build a \"pack-<id>.promised\" as a peer of the \"pack-<id>.[pack,idx]\"\n    files.\n[] Use Jonathan's binary format with (-1LL) for the sizes, but also add\n    a flags/object-type word.  Since the local commit/object traversal\n    usually knows the type of object (either by context or the mode field\n    in the parent entry), index-pack can set it immediately.\n[] Repack would \"do the right thing\" when building new packfiles from\n    existing packfiles (and merge/prune/etc. a new promise file).\n\nConceptually, these \"pack-<id>.promised\" files would be read-only once\ncreated, like the corresponding .pack/.idx files.  But since we have a\nfixed record size, we could let the dynamic object size requests update\nthe sizes in-place.   (And if an update failed for some reason, the\nclient could just go on and use the result (as we do when trying to\nupdate mtime info in the index).)\n\nThis gives us the full set of omitted objects without the network\noverhead and without the overhead of merging them into a single list.\nAnd we defer the cost of getting an object's size until we actually\nneed the info.\n\n\n> [...]\n>> In such a \"sparse clone\", it would be nice to omit unneeded tree objects\n>> in addition to just blobs.   I say that because we are finding with GVFS\n>> on the Windows repo, that even with commits-and-trees-only filtering,\n>> the number of tree objects is overwhelming.  So I'm also concerned about\n>> limiting the list to just blobs.  If we need to have this list, it\n>> should be able to contain any object.  (Suggesting having an object type\n>> in the entry.)\n> \n> Would it work to have a separate lists of blobs and trees (either in\n> separate files or in the same file)?\n> \n> One option would be to add a version number / magic string to the start\n> of the file.  That would allow making format changes later without a\n> proliferation of distinct repository extensions.\n\nAs I suggested above, we could just put them in the same file with a\nflags/object-type field.\n\nWithout getting too nerdy here, we could even say that the size doesn't\nneed to be a int:64 -- definitely bigger than int:32, but we don't need\nthe full int:64.  We could make the size a int:40 or int:48 and use the\nrest of the QWORD for flags/object-type.\n\n\n> [...]\n>> I assume that we'll also need a promised-blob.lock file to control\n>> access during list manipulation.  This is already a sore spot with the\n>> index; I'd hate to create another one.\n> \n> Can you say more about this concern?  My experience with concurrent\n> fetches has already not been great (since one fetch process is not\n> aware of what the other has fetched) --- is your concern that the\n> promised-blob facility would affect pushes as well some day?\n\n[I wrote this before I wrote the multi-peer-file suggestion above.]\n\nI'm assuming that anyone adding a new promised-blob to the single file\nwill use the standard \"read foo, write foo.lock, rename\" trick.  If we\nhave a large number of omitted blobs, this file will be large and the\ninsert will take a non-trivial amount of time.\n\nConcurrent fetches does sound a bit silly, so we only have to worry\nabout the time to copy the bits on disk.\n\nIf other operations (such as demand loading missing blobs in a log or\ndiff) need to remove promised-blobs from the list as they create\nloose blobs, then they will get stuck on the lock.  I suppose we could\njust say that the promised-blob list may have false-positives in it\n(because lookups in it won't happen until the packed and loose objects\nhave been searched and failed) and just wait for GC to clean it.\n\nAnd there's always that stale lock problem where something dies holding\nthe lock and doesn't delete it.\n\nSome of these could be addressed with a DB or something that allows\nincremental and atomic updates rather than a regular index-like\nflat file, but I don't want to dive into that here.\n\n\n>> I also have to wonder about the need to have a complete list of omitted\n>> blobs up front.  It may be better to just relax the consistency checks\n>> and assume a missing blob is \"intentionally missing\" rather than\n>> indicating a corruption somewhere.\n> \n> We've discussed this previously on list and gone back and forth. :)\n> \n>>                                      And then let the client do a later\n>> round-trip to either demand-load the object -or- demand-load the\n>> existence/size info if/when it really matters.\n> \n> The cost of demand-loading this existence/size information is what\n> ultimately convinced me of this approach.\n> \n> But I can see how the tradeoffs differ between the omit-large-blobs\n> case and the omit-all-blobs case.  We might end up having to support\n> both modes. :(\n\nI think I've been in both camps myself.\n\nI'm not sure it would be that hard to support both (other than being\nunfortunate), assuming pack-objects could build the same binary data\nas I suggested be computed by fetch-pack/index-pack above.  It could\nthen just be a run-time decision.\n\nI could even see it being useful -- do the initial clone with\nclient-side computation (because you're getting mostly ancient data\nthat you don't care about) and then let fetches use server-side\ncomputation (because you're more likely to care about the sizes of\nobjects near your branch heads).\n\n\n>> Maybe we should add a verb to your new fetch-blob endpoint to just get\n>> the size of one or more objects to help with this.\n> \n> No objections from me, though we don't need it yet.\n> \n> Thanks,\n> Jonathan\n> \n\nThanks\nJeff\n"},{"id":"324361","messageId":"e44c0778-326e-9f9c-ecb3-eba5da170bd0@jeffhostetler.com","threadId":"46359","inReplyTo":"4b326d17-3896-a374-9d7a-d0c89954c943@jeffhostetler.com","subject":"Re: [RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-13T15:05:09Z","receivedAt":"2017-07-13T15:05:15Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/13/2017 10:48 AM, Jeff Hostetler wrote:\n> \n> \n> On 7/12/2017 3:28 PM, Jonathan Nieder wrote:\n>> Hi,\n>>\n>> Jeff Hostetler wrote:\n>>\n>>> My primary concern is scale and managing the list of objects over time.\n>> [...]\n>>>                                                                   For\n>>> example, on the Windows repo we have (conservatively) 100M+ blobs (and\n>>> growing).  Assuming 28 bytes per, gives a 2.8GB list to be manipulated.\n>>>\n>>> If I understand your proposal, newly-omitted blobs would need to be\n>>> merged into the promised-blob list after each fetch.  The fetch itself\n>>> may not have that many new entries, but inserting them into the existing\n>>> list will be slow.\n>>\n>> This is a good point.  An alternative would be to avoid storing the\n>> list and instead use a repository extension that treats all missing\n>> blobs in the repository similarly to promised blobs (with a weaker\n>> promise, where the server is allowed to return 404).  The downsides:\n>>\n>> - blob sizes are not available without an additional request, e.g. for\n>>    directory listings\n>>\n>> - the semantics of has_object_file become more complicated.  If it\n>>    is allowed to return false for omitted blobs, then callers have to\n>>    be audited to tolerate that and try to look up the blob anyway.\n>>    If it has to contact the server to find out whether an omitted blob\n>>    is available, then callers have to be audited to skip this expensive\n>>    operation when possible.\n>>\n>> - similarly, the semantics of sha1_object_info{,_extended} become more\n>>    complicated.  If they are allowed to return -1 for omitted blobs,\n>>    then callers have to be audited to handle that. If they have to\n>>    contact the server to find the object type and size, it becomes\n>>    expensive in a way that affects callers.\n>>\n>> - causes futile repeated requests to the server for objects that don't\n>>    exist.  Caching negative lookups is fussy because a later push could\n>>    cause those objects to exist --- though it should be possible for\n>>    fetches to invalidate entries in such a cache using the list of\n>>    promised blobs sent by the server.\n> \n> Also good points. :-)\n> \n> Would it be better to:\n> [] let fetch-pack/index-pack on the client compute the set of missing\n>     objects based upon the commit range in the response.\n> [] build a \"pack-<id>.promised\" as a peer of the \"pack-<id>.[pack,idx]\"\n>     files.\n> [] Use Jonathan's binary format with (-1LL) for the sizes, but also add\n>     a flags/object-type word.  Since the local commit/object traversal\n>     usually knows the type of object (either by context or the mode field\n>     in the parent entry), index-pack can set it immediately.\n> [] Repack would \"do the right thing\" when building new packfiles from\n>     existing packfiles (and merge/prune/etc. a new promise file).\n> \n> Conceptually, these \"pack-<id>.promised\" files would be read-only once\n> created, like the corresponding .pack/.idx files.  But since we have a\n> fixed record size, we could let the dynamic object size requests update\n> the sizes in-place.   (And if an update failed for some reason, the\n> client could just go on and use the result (as we do when trying to\n> update mtime info in the index).)\n> \n> This gives us the full set of omitted objects without the network\n> overhead and without the overhead of merging them into a single list.\n> And we defer the cost of getting an object's size until we actually\n> need the info.\n> \n> \n>> [...]\n>>> In such a \"sparse clone\", it would be nice to omit unneeded tree objects\n>>> in addition to just blobs.   I say that because we are finding with GVFS\n>>> on the Windows repo, that even with commits-and-trees-only filtering,\n>>> the number of tree objects is overwhelming.  So I'm also concerned about\n>>> limiting the list to just blobs.  If we need to have this list, it\n>>> should be able to contain any object.  (Suggesting having an object type\n>>> in the entry.)\n>>\n>> Would it work to have a separate lists of blobs and trees (either in\n>> separate files or in the same file)?\n>>\n>> One option would be to add a version number / magic string to the start\n>> of the file.  That would allow making format changes later without a\n>> proliferation of distinct repository extensions.\n> \n> As I suggested above, we could just put them in the same file with a\n> flags/object-type field.\n> \n> Without getting too nerdy here, we could even say that the size doesn't\n> need to be a int:64 -- definitely bigger than int:32, but we don't need\n> the full int:64.  We could make the size a int:40 or int:48 and use the\n> rest of the QWORD for flags/object-type.\n> \n> \n>> [...]\n>>> I assume that we'll also need a promised-blob.lock file to control\n>>> access during list manipulation.  This is already a sore spot with the\n>>> index; I'd hate to create another one.\n>>\n>> Can you say more about this concern?  My experience with concurrent\n>> fetches has already not been great (since one fetch process is not\n>> aware of what the other has fetched) --- is your concern that the\n>> promised-blob facility would affect pushes as well some day?\n> \n> [I wrote this before I wrote the multi-peer-file suggestion above.]\n> \n> I'm assuming that anyone adding a new promised-blob to the single file\n> will use the standard \"read foo, write foo.lock, rename\" trick.  If we\n> have a large number of omitted blobs, this file will be large and the\n> insert will take a non-trivial amount of time.\n> \n> Concurrent fetches does sound a bit silly, so we only have to worry\n> about the time to copy the bits on disk.\n> \n> If other operations (such as demand loading missing blobs in a log or\n> diff) need to remove promised-blobs from the list as they create\n> loose blobs, then they will get stuck on the lock.  I suppose we could\n> just say that the promised-blob list may have false-positives in it\n> (because lookups in it won't happen until the packed and loose objects\n> have been searched and failed) and just wait for GC to clean it.\n> \n> And there's always that stale lock problem where something dies holding\n> the lock and doesn't delete it.\n> \n> Some of these could be addressed with a DB or something that allows\n> incremental and atomic updates rather than a regular index-like\n> flat file, but I don't want to dive into that here.\n> \n> \n>>> I also have to wonder about the need to have a complete list of omitted\n>>> blobs up front.  It may be better to just relax the consistency checks\n>>> and assume a missing blob is \"intentionally missing\" rather than\n>>> indicating a corruption somewhere.\n>>\n>> We've discussed this previously on list and gone back and forth. :)\n>>\n>>>                                      And then let the client do a later\n>>> round-trip to either demand-load the object -or- demand-load the\n>>> existence/size info if/when it really matters.\n>>\n>> The cost of demand-loading this existence/size information is what\n>> ultimately convinced me of this approach.\n>>\n>> But I can see how the tradeoffs differ between the omit-large-blobs\n>> case and the omit-all-blobs case.  We might end up having to support\n>> both modes. :(\n> \n> I think I've been in both camps myself.\n> \n> I'm not sure it would be that hard to support both (other than being\n> unfortunate), assuming pack-objects could build the same binary data\n> as I suggested be computed by fetch-pack/index-pack above.  It could\n> then just be a run-time decision.\n> \n> I could even see it being useful -- do the initial clone with\n> client-side computation (because you're getting mostly ancient data\n> that you don't care about) and then let fetches use server-side\n> computation (because you're more likely to care about the sizes of\n> objects near your branch heads).\n> \n> \n>>> Maybe we should add a verb to your new fetch-blob endpoint to just get\n>>> the size of one or more objects to help with this.\n>>\n>> No objections from me, though we don't need it yet.\n\nI think it would also be helpful to consider more than one type of size\nrequest.\n[1] get the size of a single objects\n[2] get the sizes of a set of objects\n[3] get the sizes for all objects directly referenced by a tree.\n     if both sides have a tree object, then we don't need to send a list\n     blobs -- just ask for all the blobs referenced by the tree object.\n     the server could just return an array of sizes ordered by blob\n     position in the tree.\n[4] recursive version of [3].\n[5] recursive for a commit (basically just a convenience on [4] and the\n     tree root).\n\nThis would let the clone do a client-side computation and then (in a\nrequest to the new end-point) get the sizes of everything in HEAD, for\nexample.\n\nThanks\nJeff\n"},{"id":"324427","messageId":"20170713123951.5cab1adc@twelve2.svl.corp.google.com","threadId":"46359","inReplyTo":"890a36fc-be16-83bc-fec6-94e21d0f7d0a@jeffhostetler.com","subject":"Re: [RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-13T19:39:51Z","receivedAt":"2017-07-13T19:40:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 12 Jul 2017 13:29:11 -0400\nJeff Hostetler <git@jeffhostetler.com> wrote:\n\n> My primary concern is scale and managing the list of objects over time.\n> \n> My fear is that this list will be quite large.  If we only want to omit\n> the very large blobs, then maybe not.  But if we want to expand that\n> scope to also omit other objects (such as a clone synchronized with a\n> sparse checkout), then that list will get large on large repos.  For\n> example, on the Windows repo we have (conservatively) 100M+ blobs (and\n> growing).  Assuming 28 bytes per, gives a 2.8GB list to be manipulated.\n> \n> If I understand your proposal, newly-omitted blobs would need to be\n> merged into the promised-blob list after each fetch.  The fetch itself\n> may not have that many new entries, but inserting them into the existing\n> list will be slow.  Also, mmap'ing and bsearch'ing will likely have\n> issues.  And there's likely to be a very expensive step to remove\n> entries from the list as new blobs are received (or locally created).\n> \n> In such a \"sparse clone\", it would be nice to omit unneeded tree objects\n> in addition to just blobs.   I say that because we are finding with GVFS\n> on the Windows repo, that even with commits-and-trees-only filtering,\n> the number of tree objects is overwhelming.\n\nI know that discussion has shifted to the possibility of not having this\nlist at all, and not sending size information together with the fetch,\nbut going back to this...maybe omitting trees *is* the solution to both\nthe large local list and the large amount of size information needing to\nbe transferred.\n\nSo the large-blob (e.g. Android) and many-blob (e.g. Windows) cases\nwould look like this:\n\n * Large-blob repositories have no trees omitted and a few blobs\n   omitted, and we have sizes for all of them.\n * Many-blob repositories have many trees omitted and either all\n   blobs omitted (and we have size information for them, useful for FUSE\n   or FUSE-like things, for example) or possibly no blobs omitted (for\n   example, if shallow clones are going to be the norm, there won't be\n   many blobs to begin with if trees are omitted).\n\nThis seems better than an intermediate solution for the many-blob\nrepository case in which we still keep all the trees but also try to\navoid sending and storing as much information about the blobs as\npossible, because that doesn't seem to provide us with much savings\n(because the trees as a whole are just as large, if not larger, than the\nblob information).\n\n> So I'm also concerned about\n> limiting the list to just blobs.  If we need to have this list, it\n> should be able to contain any object.  (Suggesting having an object type\n> in the entry.)\n\nThis makes sense - I'll add it in.\n\n> I also have to wonder about the need to have a complete list of omitted\n> blobs up front.  It may be better to just relax the consistency checks\n> and assume a missing blob is \"intentionally missing\" rather than\n> indicating a corruption somewhere.  And then let the client do a later\n> round-trip to either demand-load the object -or- demand-load the\n> existence/size info if/when it really matters.\n> \n> Maybe we should add a verb to your new fetch-blob endpoint to just get\n> the size of one or more objects to help with this.\n\nIf we allow the omission of trees, I don't think the added complexity of\ndemand-loading sizes is worth it.\n\nWhat do you think of doing this:\n * add a \"type\" field to the list of promised objects (formerly the list\n   of promised blobs)\n * retain mandatory size for blobs\n * retain single file containing list of promised objects (I don't feel\n   too strongly about this, but it has a slight simplicity and\n   in-between-GC performance advantage)\n"},{"id":"324549","messageId":"4d0849b0-1340-5b82-ba3c-03a1f5c42f33@jeffhostetler.com","threadId":"46359","inReplyTo":"20170713123951.5cab1adc@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-07-14T20:03:45Z","receivedAt":"2017-07-14T20:03:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 7/13/2017 3:39 PM, Jonathan Tan wrote:\n> On Wed, 12 Jul 2017 13:29:11 -0400\n> Jeff Hostetler <git@jeffhostetler.com> wrote:\n> \n>> My primary concern is scale and managing the list of objects over time.\n>>\n>> My fear is that this list will be quite large.  If we only want to omit\n>> the very large blobs, then maybe not.  But if we want to expand that\n>> scope to also omit other objects (such as a clone synchronized with a\n>> sparse checkout), then that list will get large on large repos.  For\n>> example, on the Windows repo we have (conservatively) 100M+ blobs (and\n>> growing).  Assuming 28 bytes per, gives a 2.8GB list to be manipulated.\n>>\n>> If I understand your proposal, newly-omitted blobs would need to be\n>> merged into the promised-blob list after each fetch.  The fetch itself\n>> may not have that many new entries, but inserting them into the existing\n>> list will be slow.  Also, mmap'ing and bsearch'ing will likely have\n>> issues.  And there's likely to be a very expensive step to remove\n>> entries from the list as new blobs are received (or locally created).\n>>\n>> In such a \"sparse clone\", it would be nice to omit unneeded tree objects\n>> in addition to just blobs.   I say that because we are finding with GVFS\n>> on the Windows repo, that even with commits-and-trees-only filtering,\n>> the number of tree objects is overwhelming.\n> \n> I know that discussion has shifted to the possibility of not having this\n> list at all, and not sending size information together with the fetch,\n> but going back to this...maybe omitting trees *is* the solution to both\n> the large local list and the large amount of size information needing to\n> be transferred.\n> \n> So the large-blob (e.g. Android) and many-blob (e.g. Windows) cases\n> would look like this:\n> \n>   * Large-blob repositories have no trees omitted and a few blobs\n>     omitted, and we have sizes for all of them.\n>   * Many-blob repositories have many trees omitted and either all\n>     blobs omitted (and we have size information for them, useful for FUSE\n>     or FUSE-like things, for example) or possibly no blobs omitted (for\n>     example, if shallow clones are going to be the norm, there won't be\n>     many blobs to begin with if trees are omitted).\n\nI'm not sure I understand what you're saying here.  Does omitting a tree\nobject change the set of blob sizes we receive?  Are you saying that if\nwe omit a tree, then we implicitly omit all the blobs it references and\ndon't send size info those blobs?  So that the local list only has\nreachable objects?  So faulting-in a tree would also have to send size\ninfo for the newly referenced blobs?\n\nWould this make it more similar to a shallow clone (in that none of the\nhave_object tests work for items beyond the cut point) ?\n\n\n> This seems better than an intermediate solution for the many-blob\n> repository case in which we still keep all the trees but also try to\n> avoid sending and storing as much information about the blobs as\n> possible, because that doesn't seem to provide us with much savings\n> (because the trees as a whole are just as large, if not larger, than the\n> blob information).\n\nOr are you saying that by omitting mostly-unused trees (and sending just\ntheir sizes) we can shrink the packfile payload and therefore afford to\nsend the full set of blob sizes in the promises payload ?\n\n\n>> So I'm also concerned about\n>> limiting the list to just blobs.  If we need to have this list, it\n>> should be able to contain any object.  (Suggesting having an object type\n>> in the entry.)\n> \n> This makes sense - I'll add it in.\n\nGood.  That will solve some cases where the client is just asking if\nwe have an object and is it of type x.\n\n\n>> I also have to wonder about the need to have a complete list of omitted\n>> blobs up front.  It may be better to just relax the consistency checks\n>> and assume a missing blob is \"intentionally missing\" rather than\n>> indicating a corruption somewhere.  And then let the client do a later\n>> round-trip to either demand-load the object -or- demand-load the\n>> existence/size info if/when it really matters.\n>>\n>> Maybe we should add a verb to your new fetch-blob endpoint to just get\n>> the size of one or more objects to help with this.\n> \n> If we allow the omission of trees, I don't think the added complexity of\n> demand-loading sizes is worth it.\n\nAs you suggested earlier, we're trying to solve 2 different problems.\nThe \"few omitted very large blobs\" and the \"way too many unneeded blobs\"\nproblems.\n\nFor the former case, if you just have a few omitted objects, then a\nsecond round-trip to mget their sizes isn't that much work.\n\nI think for the latter, forcing a full promise-list on clone is just\ntoo much data to send -- data that we likely won't ever need.  If the\nclient can compute the promise-set from the received packfile, we can\ndo the \"is an object and type\" tests completely locally.  It's only\nwhen the client asks for the size do we need to decide whether to fetch\njust the size or fetch the actual object.  Either type of request\ndoesn't seem like that big of a deal.  But we save a lot of IO\n(especially during the initial clone, when it hurts the most).\n\nIn both cases, we benefit by letting the client decide if/when it\nwants them.  And whether it wants the full set or a subset.\n\nI suppose we could make it an option for the upload-pack protocol\nto send or not send the promise-set.  That feels a little like a\nhack in that it is another knob we have to support, but would let\nus work either way.\n\nI think it would be better to have 2 round-trips.  We could also have\nan fsck option to verify and back-fill missing sizes if you want to\nbe able to force them.\n\n\n> What do you think of doing this:\n>   * add a \"type\" field to the list of promised objects (formerly the list\n>     of promised blobs)\n>   * retain mandatory size for blobs\n>   * retain single file containing list of promised objects (I don't feel\n>     too strongly about this, but it has a slight simplicity and\n>     in-between-GC performance advantage)\n> \n\nThe single promise-set is problematic.  I think it will grow too\nlarge (in our case) and will need all the usual lock juggling\nand merging.\n\nI still prefer my suggestion for a per-packfile promise-set for all\nof the reasons I stated the other day.  This can be computed quickly\nduring index-pack, is (nearly) read-only, and doesn't require the\nwhole file rewrite lock file.  It also has the benefit of being\nportable -- in that I can also copy the .promise file if I copy the\n.pack and .idx file to another repo.\n\n\nSo to summarize, I think we need:\n* an object-list concept\n      (that is, we believe that just assuming missing objects\n       are \"intentionally omitted\" has issues.)\n* an object-list-entry containing:\n      {object-id,\n       optional object-size,  (use -1 when unknown)\n       object-type}\n* server end-point for fetching 1 or more sizes\n      (by sending 1 or more object-ids)\n* server end-point for fetching 1 or more objects\n      (by sending 1 or more object-ids)\n      and receiving a packfile or loose object.\n* per-packfile object-list (rather than single combined one).\n* client-side (index-pack) compute of per-packfile object-list.\n* repack to \"do the right thing\".\n\nOptional:\n* pack-protocol option.\n* server-side (pack-objects) compute of per-packfile object-list.\n* fsck to verify and back-fill all sizes using fetch-sizes.\n\nDon't need:\n* fetch-object or fetch-sizes end-points do not need to support\n   requests for the set of objects referenced by a directory\n   (recursively) from my earlier email.\n       keep it simple and just explicitly list the object-ids wanted.\n* the client side of fetch-object DOES NOT need to remove\n   object-list-entries from the object-list.\n       we must assume that the list will have false-positives\n       in it as loose objects are created locally.\n\n\nThanks,\nJeff\n"},{"id":"324555","messageId":"20170714213018.GK93855@aiede.mtv.corp.google.com","threadId":"46359","inReplyTo":"4d0849b0-1340-5b82-ba3c-03a1f5c42f33@jeffhostetler.com","subject":"Re: [RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-14T21:30:18Z","receivedAt":"2017-07-14T21:30:27Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jeff Hostetler wrote:\n> On 7/13/2017 3:39 PM, Jonathan Tan wrote:\n\n>> I know that discussion has shifted to the possibility of not having this\n>> list at all, and not sending size information together with the fetch,\n>> but going back to this...maybe omitting trees *is* the solution to both\n>> the large local list and the large amount of size information needing to\n>> be transferred.\n>>\n>> So the large-blob (e.g. Android) and many-blob (e.g. Windows) cases\n>> would look like this:\n>>\n>>   * Large-blob repositories have no trees omitted and a few blobs\n>>     omitted, and we have sizes for all of them.\n>>   * Many-blob repositories have many trees omitted and either all\n>>     blobs omitted (and we have size information for them, useful for FUSE\n>>     or FUSE-like things, for example) or possibly no blobs omitted (for\n>>     example, if shallow clones are going to be the norm, there won't be\n>>     many blobs to begin with if trees are omitted).\n>\n> I'm not sure I understand what you're saying here.  Does omitting a tree\n> object change the set of blob sizes we receive?  Are you saying that if\n> we omit a tree, then we implicitly omit all the blobs it references and\n> don't send size info those blobs?  So that the local list only has\n> reachable objects?  So faulting-in a tree would also have to send size\n> info for the newly referenced blobs?\n>\n> Would this make it more similar to a shallow clone (in that none of the\n> have_object tests work for items beyond the cut point) ?\n\nCorrect.  After the server sends a promise instead of a tree object, the\nclient has no reason to try to access blobs pointed to by that tree, any\nmore than it has reason to try to access commits on a branch it has not\nfetched.  This means the client does not have to be aware of those blobs\nuntil it fetches the tree and associated blob promises.\n\n[...]\n> For the former case, if you just have a few omitted objects, then a\n> second round-trip to mget their sizes isn't that much work.\n\nFor the client, that is true.  For the server, decreasing the number\nof requests even when requests are small and fast can be valuable.\n\n[...]\n> I think for the latter, forcing a full promise-list on clone is just\n> too much data to send -- data that we likely won't ever need.\n\nWhat did you think of the suggestion to not send promises for objects\nthat are only referenced by objects that weren't sent?\n\n[...]\n>> What do you think of doing this:\n>>   * add a \"type\" field to the list of promised objects (formerly the list\n>>     of promised blobs)\n>>   * retain mandatory size for blobs\n>>   * retain single file containing list of promised objects (I don't feel\n>>     too strongly about this, but it has a slight simplicity and\n>>     in-between-GC performance advantage)\n>\n> The single promise-set is problematic.  I think it will grow too\n> large (in our case) and will need all the usual lock juggling\n> and merging.\n>\n> I still prefer my suggestion for a per-packfile promise-set for all\n> of the reasons I stated the other day.  This can be computed quickly\n> during index-pack, is (nearly) read-only, and doesn't require the\n> whole file rewrite lock file.  It also has the benefit of being\n> portable -- in that I can also copy the .promise file if I copy the\n> .pack and .idx file to another repo.\n\nOkay.\n\nThanks,\nJonathan\n"},{"id":"324627","messageId":"C299C45128634A21AF9D65E1B2B52C5B@PhilipOakley","threadId":"46359","inReplyTo":"cover.1499800530.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing blobs\")","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-07-16T15:23:06Z","receivedAt":"2017-07-16T15:23:16Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Jonathan Tan\" <jonathantanmy@google.com>\nSent: Tuesday, July 11, 2017 8:48 PM\n> These patches are part of a set of patches implementing partial clone,\n> as you can see here:\n>\n> https://github.com/jonathantanmy/git/tree/partialclone\n>\n> In that branch, clone with batch checkout works, as you can see in the\n> README. The code and tests are generally done, but some patches are\n> still missing documentation and commit messages.\n>\n> These 3 patches implement the foundational concept - formerly known as\n> \"missing blobs\" in the \"missing blob manifest\", I decided to call them\n> \"promised blobs\". The repo knows their object names and sizes. It also\n> does not have the blobs themselves, but can be configured to know how to\n> fetch them.\n>\nIf I understand correctly, this method doesn't give any direct user \nvisibility of missing blobs in the file system. Is that correct?\n\nI was hoping that eventually the various 'on demand' approaches would still \nallow users to continue to work as they go off-line such that they can see \ndirectly (in the FS) where the missing blobs (and trees) are located, so \nthat they can continue to commit new work on existing files.\n\nI had felt that some sort of 'gitlink' should be present (huma readable) as \na place holder for the missing blob/tree. e.g. 'gitblob: 1234abcd' (showing \nthe missing oid, jsut like sub-modules can do - it's no different really.\n\nI'm concerned that the various GVFS extensions haven't fully achieved a \nseparation of concerns surrounding the DVCS capability for on-line/off-line \nconversion as comms drop in and out. The GVFS looks great for a fully \nnetworked, always on, environment, but it would be good to also have the \nsepration for those who (will) have shallow/narrow clones that may also need \nto work with a local upstream that is also shallow/narrow.\n\n--\nPhilip\nI wanted to at least get my thoughts into the discussion before it all \npasses by.\n\n> An older version of these patches was sent as a single demonstration\n> patch in versions 1 to 3 of [1]. In there, Junio suggested that I have\n> only one file containing missing blob information. I have made that\n> suggested change in this version.\n>\n> One thing remaining is to add a repository extension [2] so that older\n> versions of Git fail immediately instead of trying to read missing\n> blobs, but I thought I'd send these first in order to get some initial\n> feedback.\n>\n> [1] \n> https://public-inbox.org/git/cover.1497035376.git.jonathantanmy@google.com/\n> [2] Documentation/technical/repository-version.txt\n>\n> Jonathan Tan (3):\n>  promised-blob, fsck: introduce promised blobs\n>  sha1-array: support appending unsigned char hash\n>  sha1_file: add promised blob hook support\n>\n> Documentation/config.txt               |   8 ++\n> Documentation/gitrepository-layout.txt |   8 ++\n> Makefile                               |   1 +\n> builtin/cat-file.c                     |   9 ++\n> builtin/fsck.c                         |  13 +++\n> promised-blob.c                        | 170 \n> +++++++++++++++++++++++++++++++++\n> promised-blob.h                        |  27 ++++++\n> sha1-array.c                           |   7 ++\n> sha1-array.h                           |   1 +\n> sha1_file.c                            |  44 ++++++---\n> t/t3907-promised-blob.sh               |  65 +++++++++++++\n> t/test-lib-functions.sh                |   6 ++\n> 12 files changed, 345 insertions(+), 14 deletions(-)\n> create mode 100644 promised-blob.c\n> create mode 100644 promised-blob.h\n> create mode 100755 t/t3907-promised-blob.sh\n>\n> -- \n> 2.13.2.932.g7449e964c-goog\n> \n\n"},{"id":"324652","messageId":"b928c073-6156-30f2-c850-993e59079ed1@gmail.com","threadId":"46359","inReplyTo":"C299C45128634A21AF9D65E1B2B52C5B@PhilipOakley","subject":"Re: [RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing blobs\")","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-17T17:43:42Z","receivedAt":"2017-07-17T17:43:51Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/16/2017 11:23 AM, Philip Oakley wrote:\n> From: \"Jonathan Tan\" <jonathantanmy@google.com>\n> Sent: Tuesday, July 11, 2017 8:48 PM\n>> These patches are part of a set of patches implementing partial clone,\n>> as you can see here:\n>>\n>> https://github.com/jonathantanmy/git/tree/partialclone\n>>\n>> In that branch, clone with batch checkout works, as you can see in the\n>> README. The code and tests are generally done, but some patches are\n>> still missing documentation and commit messages.\n>>\n>> These 3 patches implement the foundational concept - formerly known as\n>> \"missing blobs\" in the \"missing blob manifest\", I decided to call them\n>> \"promised blobs\". The repo knows their object names and sizes. It also\n>> does not have the blobs themselves, but can be configured to know how to\n>> fetch them.\n>>\n> If I understand correctly, this method doesn't give any direct user \n> visibility of missing blobs in the file system. Is that correct?\n\nThat is correct\n\n> \n> I was hoping that eventually the various 'on demand' approaches would \n> still allow users to continue to work as they go off-line such that they \n> can see directly (in the FS) where the missing blobs (and trees) are \n> located, so that they can continue to commit new work on existing files.\n> \n\nThis is a challenge as git assumes all objects are always available \n(that is a key design principal of a DVCS) so any missing object is \nconsidered a corruption that typically results in a call to \"die.\"\n\nThe GVFS solution gets around this by ensuring any missing object is \nretrieved on behalf of git so that it never sees it as missing.  The \nobvious tradeoff is that this requires a network connection so the \nobject can be retrieved.\n\n> I had felt that some sort of 'gitlink' should be present (huma readable) \n> as a place holder for the missing blob/tree. e.g. 'gitblob: 1234abcd' \n> (showing the missing oid, jsut like sub-modules can do - it's no \n> different really.\n> \n\nWe explored that option briefly but when you have a large number of \nfiles, even writing out some sort of place holder can take a very long \ntime.  In fact, since the typical source file is relatively small (a few \nkilobytes), writing out a placeholder doesn't save much time vs just \nwriting out the actual file contents.\n\nAnother challenge is that even if there is a placeholder written to \ndisk, you still need a network connection to retrieve the actual \ncontents if/when it is needed.\n\n> I'm concerned that the various GVFS extensions haven't fully achieved a \n> separation of concerns surrounding the DVCS capability for \n> on-line/off-line conversion as comms drop in and out. The GVFS looks \n> great for a fully networked, always on, environment, but it would be \n> good to also have the sepration for those who (will) have shallow/narrow \n> clones that may also need to work with a local upstream that is also \n> shallow/narrow.\n> \n\nYou are correct that this hasn't been tackled yet. It is a challenging \nproblem. I can envision something along the lines of what was done for \nthe shallow clone feature where there are distinct ways to change the \nset of objects that are available but that would hopefully come in some \nfuture patch series.\n\n> -- \n> Philip\n> I wanted to at least get my thoughts into the discussion before it all \n> passes by.\n> \n>> An older version of these patches was sent as a single demonstration\n>> patch in versions 1 to 3 of [1]. In there, Junio suggested that I have\n>> only one file containing missing blob information. I have made that\n>> suggested change in this version.\n>>\n>> One thing remaining is to add a repository extension [2] so that older\n>> versions of Git fail immediately instead of trying to read missing\n>> blobs, but I thought I'd send these first in order to get some initial\n>> feedback.\n>>\n>> [1] \n>> https://public-inbox.org/git/cover.1497035376.git.jonathantanmy@google.com/ \n>>\n>> [2] Documentation/technical/repository-version.txt\n>>\n>> Jonathan Tan (3):\n>>  promised-blob, fsck: introduce promised blobs\n>>  sha1-array: support appending unsigned char hash\n>>  sha1_file: add promised blob hook support\n>>\n>> Documentation/config.txt               |   8 ++\n>> Documentation/gitrepository-layout.txt |   8 ++\n>> Makefile                               |   1 +\n>> builtin/cat-file.c                     |   9 ++\n>> builtin/fsck.c                         |  13 +++\n>> promised-blob.c                        | 170 \n>> +++++++++++++++++++++++++++++++++\n>> promised-blob.h                        |  27 ++++++\n>> sha1-array.c                           |   7 ++\n>> sha1-array.h                           |   1 +\n>> sha1_file.c                            |  44 ++++++---\n>> t/t3907-promised-blob.sh               |  65 +++++++++++++\n>> t/test-lib-functions.sh                |   6 ++\n>> 12 files changed, 345 insertions(+), 14 deletions(-)\n>> create mode 100644 promised-blob.c\n>> create mode 100644 promised-blob.h\n>> create mode 100755 t/t3907-promised-blob.sh\n>>\n>> -- \n>> 2.13.2.932.g7449e964c-goog\n>>\n> \n"},{"id":"324656","messageId":"20170717180322.GM93855@aiede.mtv.corp.google.com","threadId":"46359","inReplyTo":"C299C45128634A21AF9D65E1B2B52C5B@PhilipOakley","subject":"Re: [RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing blobs\")","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-07-17T18:03:22Z","receivedAt":"2017-07-17T18:03:32Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi Philip,\n\nPhilip Oakley wrote:\n> From: \"Jonathan Tan\" <jonathantanmy@google.com>\n\n>> These patches are part of a set of patches implementing partial clone,\n>> as you can see here:\n>>\n>> https://github.com/jonathantanmy/git/tree/partialclone\n[...]\n> If I understand correctly, this method doesn't give any direct user\n> visibility of missing blobs in the file system. Is that correct?\n>\n> I was hoping that eventually the various 'on demand' approaches\n> would still allow users to continue to work as they go off-line such\n> that they can see directly (in the FS) where the missing blobs (and\n> trees) are located, so that they can continue to commit new work on\n> existing files.\n>\n> I had felt that some sort of 'gitlink' should be present (huma\n> readable) as a place holder for the missing blob/tree. e.g.\n> 'gitblob: 1234abcd' (showing the missing oid, jsut like sub-modules\n> can do - it's no different really.\n\nThat's a reasonable thing to want, but it's a little different from\nthe use cases that partial clone work so far has aimed to support.\nThey are:\n\n A. Avoiding downloading all blobs (and likely trees as well) that are\n    not needed in the current operation (e.g. checkout).  This blends\n    well with the sparse checkout feature, which allows the current\n    checkout to be fairly small in a large repository.\n\n    GVFS uses a trick that makes it a little easier to widen a sparse\n    checkout upon access of a directory.  But the same building blocks\n    should work fine with a sparse checkout that has been set up\n    explicitly.\n\n B. Avoiding downloading large blobs, except for those needed in the\n    current operation (e.g. checkout).\n\n    When not using sparse checkout, the main benefit out of the box is\n    avoiding downloading *historical versions* of large blobs.\n\nIt sounds like you are looking for a sort of placeholder outside the\nsparse checkout area.  In a way, that's orthogonal to these patches:\neven if you have all relevant blobs, you may want to avoid inflating\nthem to check them out and reading them to compare to the index (i.e.\nthe usual benefits of sparse checkout).  In a sparse checkout, you\nstill might like to be able to get a listing of files outside the\nsparse area (which you can get with \"git ls-tree\") and you may even\nwant to be able to get such a listing with plain \"ls\" (as with your\nproposal).\n\nThanks and hope that helps,\nJonathan\n"},{"id":"324777","messageId":"20170719163713.2d01f6e7@twelve2.svl.corp.google.com","threadId":"46359","inReplyTo":"CAGZ79kZ8i0Xqp2Yft19JKzPFMUdbzEDNJ01zER9RoM8ZzKPu8A@mail.gmail.com","subject":"Re: [RFC PATCH 1/3] promised-blob, fsck: introduce promised blobs","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-19T23:37:13Z","receivedAt":"2017-07-19T23:37:27Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 11 Jul 2017 15:02:09 -0700\nStefan Beller <sbeller@google.com> wrote:\n\n> Here I wondered what this file looks like, in a later patch you\n> add documentation:\n> \n>   +objects/promisedblob::\n>   +       This file records the sha1 object names and sizes of promised\n>   +       blobs.\n>   +\n> \n> but that leaves more fundamental questions:\n> * Is that a list of sha1s, separated by LF? (CRLF? How would Windows\n>   users interact with it, are they supposed to ever modify this file?)\n> * Similar to .git/packed-refs, would we want to have a first line\n>   that has some sort of comment?\n> * In the first question I assumed a linear list, will that be a sorted list,\n>   or do we want to have some fancy data structure here?\n>   (critbit tree, bloom filter)\n> * is the contents in ASCII or binary? (What are the separators?)\n> * In the future I presume we'd want to quickly answer \"Is X in the\n>   promised blobs list?\" so would it be possible (in the future) to\n>   improve the searching e.g. binary search?\n> * ... I'll read on to see my questions answered, but I'd guess\n>   others would prefer to see it in the docs. :)\n\nI'll write more documentation once the file format is finalized. At the\ntime this patch was written, this was a sorted binary file, where each\nentry consists of a 20-byte SHA-1 and an 8-byte size. Now each entry is\na 20-byte SHA-1, a 1-byte type, and an 8-byte size (I will send this\npatch out soon).\n\n> Similar to other files, would we want to prefix the file with\n> a 4 letter magic number and a version number?\n\nThat's a good idea.\n\n> Later down the road, do we want to have a\n> (plumbing) command to move an object from\n> standard blob to promised blob (as of now I'd think\n> it would perform this rm call as well as an insert into\n> the promised blobs file) ?\n> (Well, we'd also have to think about how to get objects\n> out of a pack)\n> \n> With such a command you can easily write your own custom\n> filter to free up blobs again.\n\nThis sounds reasonable, but probably not in the initial set of patches.\n\n> > +       test_must_fail git -C repo fsck\n> \n>     test_i18n_grep missing out ?\n> \n> maybe, too? (Maybe that is already tested elsewhere,\n> so no need for it)\n\nThis test is just meant to show that configuration can make fsck pass,\nnot the existing behavior of fsck when it fails (which is tested\nelsewhere - I guess that was your question).\n"},{"id":"324778","messageId":"20170719165622.78e463fd@twelve2.svl.corp.google.com","threadId":"46359","inReplyTo":"CAGZ79kZ9Ewxv=8fnRZZaQjpyAdv8df81a9aLRkJzEt5ZU0VcTA@mail.gmail.com","subject":"Re: [RFC PATCH 2/3] sha1-array: support appending unsigned char hash","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-19T23:56:22Z","receivedAt":"2017-07-19T23:56:35Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 11 Jul 2017 15:06:11 -0700\nStefan Beller <sbeller@google.com> wrote:\n\n> On Tue, Jul 11, 2017 at 12:48 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> > In a subsequent patch, sha1_file will need to append object names in the\n> > form of \"unsigned char *\" to oid arrays. Teach sha1-array support for\n> > that.\n> >\n> > Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> \n> This breaks the oid/sha1 barrier?\n\nNot sure what you mean by this. This patch is meant to be a change to\nmake \"unsigned char *\"-using code able to create/modify oid arrays,\nwhile we migrate away from \"unsigned char *\" - I don't know of any\nbarrier.\n\n> I would have expected the caller to do a\n> \n>   oid_array_append_oid(&array, sha1_to_oid(sha1));\n> \n> with sha1_to_oid working off some static memory, such that the\n> call to oid_array_append_oid (which we have as oid_array_append)\n> is just as cheap?\n\nThe solution you present would need 2 copies (one in sha1_to_oid, and\nanother in oid_array_append), but the solution in this patch only\nrequires one.\n\n> Could you say more in the commit message on why we collude\n> sha1 and oids here?\n\nI also don't understand this, but the answer to this is probably the\nsame as the answer to your first question.\n"},{"id":"324779","messageId":"CAGZ79kYCr_Ozt3sa5FffgY-kzwAnGc2rTFEQ1AzeruURXQsYmA@mail.gmail.com","threadId":"46359","inReplyTo":"20170719165622.78e463fd@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH 2/3] sha1-array: support appending unsigned char hash","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-20T00:06:24Z","receivedAt":"2017-07-20T00:06:30Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jul 19, 2017 at 4:56 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> On Tue, 11 Jul 2017 15:06:11 -0700\n> Stefan Beller <sbeller@google.com> wrote:\n>\n>> On Tue, Jul 11, 2017 at 12:48 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n>> > In a subsequent patch, sha1_file will need to append object names in the\n>> > form of \"unsigned char *\" to oid arrays. Teach sha1-array support for\n>> > that.\n>> >\n>> > Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n>>\n>> This breaks the oid/sha1 barrier?\n>\n> Not sure what you mean by this. This patch is meant to be a change to\n> make \"unsigned char *\"-using code able to create/modify oid arrays,\n> while we migrate away from \"unsigned char *\" - I don't know of any\n> barrier.\n\nIn my mind we have 2 states:\n* the current state where we have \"char *sha1\"\n* the future in which we have \"struct object_id oid\"\n\nIn both realms you can do things (and the current code\nbase has examples for both), but these 2 are connected\nvia only a couple well defined functions (sha1_to_hex,\noid_to_sha1) during the conversion phase.\n\nSo with that mindset I was slightly irritated to see a function\nthat uses both realms so interchangeably.\n\nRevisiting that it is actually not that bad, but also a good\nidea to have in the transition phase.\n\nSorry for the noise,\nStefan\n"},{"id":"324780","messageId":"cover.1500508695.git.jonathantanmy@google.com","threadId":"46359","inReplyTo":"cover.1499800530.git.jonathantanmy@google.com","subject":"[RFC PATCH v2 0/4] Partial clone: promised objects (not only blobs)","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-20T00:21:43Z","receivedAt":"2017-07-20T00:21:57Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Thanks for all your comments on the earlier version. This is a\nsubstantially different version. In particular:\n - Now supports all types (tag, commit, tree) of objects, not only blobs\n - fsck works\n - Incorporates Ben Peart's code that uses a long-living process\n   (lifetime of the Git invocation) to obtain objects\n - Implemented as a repository extension\n\nIf anyone would like to comment on the overall direction of this\napproach, that would be great. In particular, I'm not too sure on the\nnames of things. I know that I have some things to still work on\n(documentation and coding style, more improvements to and tests for\nfsck, maybe division of commits?) but I would like some early feedback\non the bigger picture first.\n\nIf you want to patch this in, this is built off my recent cleanup patch\n[1].\n\nAbout inability to scale if we have the list of promised blobs: In this\ndesign, we will have a promised blob entry only if we have a concrete\ntree, so in a repository in which many trees are omitted, there will not\nbe many promised blob entry objects. In fact, the minimal partial clone\nwill be one in which there will be one promised commit for each ref\n(assuming no duplicates), no promised trees, and no promised blobs.\n(I'm not planning to implement such a clone, but someone else could do\nso.)\n\nAbout having multiple promise lists: I have retained the single list,\nbut am still open to change. The support of all the object types might\nbe sufficient mitigation for the issues that caused us to investigate\nmultiple promise lists in the first place.\n\n[1] https://public-inbox.org/git/20170718222848.1453-1-jonathantanmy@google.com/\n\nJonathan Tan (4):\n  object: remove \"used\" field from struct object\n  promised-object, fsck: introduce promised objects\n  sha1-array: support appending unsigned char hash\n  sha1_file: support promised object hook\n\n Documentation/config.txt                         |   8 +\n Documentation/gitrepository-layout.txt           |   8 +\n Documentation/technical/read-object-protocol.txt | 102 +++++++\n Documentation/technical/repository-version.txt   |   6 +\n Makefile                                         |   1 +\n builtin/cat-file.c                               |   9 +\n builtin/fsck.c                                   |  42 ++-\n cache.h                                          |   4 +\n environment.c                                    |   1 +\n fsck.c                                           |   6 +-\n object.c                                         |  21 +-\n object.h                                         |  21 +-\n promised-object.c                                | 324 +++++++++++++++++++++++\n promised-object.h                                |  34 +++\n setup.c                                          |   7 +-\n sha1-array.c                                     |   7 +\n sha1-array.h                                     |   1 +\n sha1_file.c                                      |  44 ++-\n t/t3907-promised-object.sh                       |  73 +++++\n t/t3907/read-object                              | 114 ++++++++\n t/test-lib-functions.sh                          |   6 +\n 21 files changed, 808 insertions(+), 31 deletions(-)\n create mode 100644 Documentation/technical/read-object-protocol.txt\n create mode 100644 promised-object.c\n create mode 100644 promised-object.h\n create mode 100755 t/t3907-promised-object.sh\n create mode 100755 t/t3907/read-object\n\n-- \n2.14.0.rc0.284.gd933b75aa4-goog\n\n"},{"id":"324781","messageId":"0eb270f40095f132bf9383f230506e45d895e5e1.1500508695.git.jonathantanmy@google.com","threadId":"46359","inReplyTo":"cover.1500508695.git.jonathantanmy@google.com","subject":"[RFC PATCH v2 1/4] object: remove \"used\" field from struct object","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-20T00:21:44Z","receivedAt":"2017-07-20T00:22:01Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"The \"used\" field in struct object is only used by builtin/fsck. Remove\nthat field and modify builtin/fsck to use a flag instead.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/fsck.c | 24 ++++++++++++++----------\n object.c       |  1 -\n object.h       |  2 +-\n 3 files changed, 15 insertions(+), 12 deletions(-)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 4ba311cda..462b8643b 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -19,6 +19,8 @@\n #define REACHABLE 0x0001\n #define SEEN      0x0002\n #define HAS_OBJ   0x0004\n+/* This flag is set if something points to this object. */\n+#define USED      0x0008\n \n static int show_root;\n static int show_tags;\n@@ -195,7 +197,7 @@ static int mark_used(struct object *obj, int type, void *data, struct fsck_optio\n {\n \tif (!obj)\n \t\treturn 1;\n-\tobj->used = 1;\n+\tobj->flags |= USED;\n \treturn 0;\n }\n \n@@ -244,7 +246,7 @@ static void check_unreachable_object(struct object *obj)\n \t}\n \n \t/*\n-\t * \"!used\" means that nothing at all points to it, including\n+\t * \"!USED\" means that nothing at all points to it, including\n \t * other unreachable objects. In other words, it's the \"tip\"\n \t * of some set of unreachable objects, usually a commit that\n \t * got dropped.\n@@ -255,7 +257,7 @@ static void check_unreachable_object(struct object *obj)\n \t * deleted a branch by mistake, this is a prime candidate to\n \t * start looking at, for example.\n \t */\n-\tif (!obj->used) {\n+\tif (!(obj->flags & USED)) {\n \t\tif (show_dangling)\n \t\t\tprintf(\"dangling %s %s\\n\", printable_type(obj),\n \t\t\t       describe_object(obj));\n@@ -379,7 +381,8 @@ static int fsck_obj_buffer(const struct object_id *oid, enum object_type type,\n \t\terrors_found |= ERROR_OBJECT;\n \t\treturn error(\"%s: object corrupt or missing\", oid_to_hex(oid));\n \t}\n-\tobj->flags = HAS_OBJ;\n+\tobj->flags &= ~(REACHABLE | SEEN);\n+\tobj->flags |= HAS_OBJ;\n \treturn fsck_obj(obj);\n }\n \n@@ -397,7 +400,7 @@ static void fsck_handle_reflog_oid(const char *refname, struct object_id *oid,\n \t\t\t\tadd_decoration(fsck_walk_options.object_names,\n \t\t\t\t\tobj,\n \t\t\t\t\txstrfmt(\"%s@{%\"PRItime\"}\", refname, timestamp));\n-\t\t\tobj->used = 1;\n+\t\t\tobj->flags |= USED;\n \t\t\tmark_object_reachable(obj);\n \t\t} else {\n \t\t\terror(\"%s: invalid reflog entry %s\", refname, oid_to_hex(oid));\n@@ -445,7 +448,7 @@ static int fsck_handle_ref(const char *refname, const struct object_id *oid,\n \t\terrors_found |= ERROR_REFS;\n \t}\n \tdefault_refs++;\n-\tobj->used = 1;\n+\tobj->flags |= USED;\n \tif (name_objects)\n \t\tadd_decoration(fsck_walk_options.object_names,\n \t\t\tobj, xstrdup(refname));\n@@ -513,7 +516,8 @@ static int fsck_loose(const struct object_id *oid, const char *path, void *data)\n \t\treturn 0; /* keep checking other objects */\n \t}\n \n-\tobj->flags = HAS_OBJ;\n+\tobj->flags &= ~(REACHABLE | SEEN);\n+\tobj->flags |= HAS_OBJ;\n \tif (fsck_obj(obj))\n \t\terrors_found |= ERROR_OBJECT;\n \treturn 0;\n@@ -595,7 +599,7 @@ static int fsck_cache_tree(struct cache_tree *it)\n \t\t\terrors_found |= ERROR_REFS;\n \t\t\treturn 1;\n \t\t}\n-\t\tobj->used = 1;\n+\t\tobj->flags |= USED;\n \t\tif (name_objects)\n \t\t\tadd_decoration(fsck_walk_options.object_names,\n \t\t\t\tobj, xstrdup(\":\"));\n@@ -737,7 +741,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\t\t\tcontinue;\n \t\t\t}\n \n-\t\t\tobj->used = 1;\n+\t\t\tobj->flags |= USED;\n \t\t\tif (name_objects)\n \t\t\t\tadd_decoration(fsck_walk_options.object_names,\n \t\t\t\t\tobj, xstrdup(arg));\n@@ -774,7 +778,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\t\tif (!blob)\n \t\t\t\tcontinue;\n \t\t\tobj = &blob->object;\n-\t\t\tobj->used = 1;\n+\t\t\tobj->flags |= USED;\n \t\t\tif (name_objects)\n \t\t\t\tadd_decoration(fsck_walk_options.object_names,\n \t\t\t\t\tobj,\ndiff --git a/object.c b/object.c\nindex f81877741..321d7e920 100644\n--- a/object.c\n+++ b/object.c\n@@ -141,7 +141,6 @@ void *create_object(const unsigned char *sha1, void *o)\n \tstruct object *obj = o;\n \n \tobj->parsed = 0;\n-\tobj->used = 0;\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \ndiff --git a/object.h b/object.h\nindex 33e5cc994..0a419ba8d 100644\n--- a/object.h\n+++ b/object.h\n@@ -38,6 +38,7 @@ struct object_array {\n  * http-push.c:                            16-----19\n  * commit.c:                               16-----19\n  * sha1_name.c:                                     20\n+ * builtin/fsck.c:  0--3\n  */\n #define FLAG_BITS  27\n \n@@ -46,7 +47,6 @@ struct object_array {\n  */\n struct object {\n \tunsigned parsed : 1;\n-\tunsigned used : 1;\n \tunsigned type : TYPE_BITS;\n \tunsigned flags : FLAG_BITS;\n \tstruct object_id oid;\n-- \n2.14.0.rc0.284.gd933b75aa4-goog\n\n"},{"id":"324782","messageId":"3420d9ae9ef86b78af1abe721891233e3f5865a2.1500508695.git.jonathantanmy@google.com","threadId":"46359","inReplyTo":"cover.1500508695.git.jonathantanmy@google.com","subject":"[RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-20T00:21:45Z","receivedAt":"2017-07-20T00:22:05Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, Git does not support repos with very large numbers of objects\nor repos that wish to minimize manipulation of certain blobs (for\nexample, because they are very large) very well, even if the user\noperates mostly on part of the repo, because Git is designed on the\nassumption that every referenced object is available somewhere in the\nrepo storage.\n\nAs a first step to reducing this problem, introduce the concept of\npromised objects. Each Git repo can contain a list of promised objects\nand their sizes (if blobs) at $GIT_DIR/objects/promised. This patch\ncontains functions to query them; functions for creating and modifying\nthat file will be introduced in later patches.\n\nA repository that is missing an object but has that object promised is not\nconsidered to be in error, so also teach fsck this. As part of doing\nthis, object.{h,c} has been modified to generate \"struct object\" based\non only the information available to promised objects, without requiring\nthe object itself.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/technical/repository-version.txt |   6 ++\n Makefile                                       |   1 +\n builtin/fsck.c                                 |  18 +++-\n cache.h                                        |   2 +\n environment.c                                  |   1 +\n fsck.c                                         |   6 +-\n object.c                                       |  19 ++++\n object.h                                       |  19 ++++\n promised-object.c                              | 130 +++++++++++++++++++++++++\n promised-object.h                              |  22 +++++\n setup.c                                        |   7 +-\n t/t3907-promised-object.sh                     |  41 ++++++++\n t/test-lib-functions.sh                        |   6 ++\n 13 files changed, 273 insertions(+), 5 deletions(-)\n create mode 100644 promised-object.c\n create mode 100644 promised-object.h\n create mode 100755 t/t3907-promised-object.sh\n\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 00ad37986..f8b82c1c7 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -86,3 +86,9 @@ for testing format-1 compatibility.\n When the config key `extensions.preciousObjects` is set to `true`,\n objects in the repository MUST NOT be deleted (e.g., by `git-prune` or\n `git repack -d`).\n+\n+`promisedObjects`\n+~~~~~~~~~~~~~~~~~\n+\n+(Explain this - basically a string containing a command to be run\n+whenever a missing object needs to be fetched.)\ndiff --git a/Makefile b/Makefile\nindex 9c9c42f8f..c1446d5ef 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -828,6 +828,7 @@ LIB_OBJS += preload-index.o\n LIB_OBJS += pretty.o\n LIB_OBJS += prio-queue.o\n LIB_OBJS += progress.o\n+LIB_OBJS += promised-object.o\n LIB_OBJS += prompt.o\n LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 462b8643b..49e21f361 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -15,6 +15,7 @@\n #include \"progress.h\"\n #include \"streaming.h\"\n #include \"decorate.h\"\n+#include \"promised-object.h\"\n \n #define REACHABLE 0x0001\n #define SEEN      0x0002\n@@ -44,6 +45,7 @@ static int name_objects;\n #define ERROR_REACHABLE 02\n #define ERROR_PACK 04\n #define ERROR_REFS 010\n+#define ERROR_PROMISED_OBJECT 011\n \n static const char *describe_object(struct object *obj)\n {\n@@ -436,7 +438,7 @@ static int fsck_handle_ref(const char *refname, const struct object_id *oid,\n {\n \tstruct object *obj;\n \n-\tobj = parse_object(oid);\n+\tobj = parse_or_promise_object(oid);\n \tif (!obj) {\n \t\terror(\"%s: invalid sha1 pointer %s\", refname, oid_to_hex(oid));\n \t\terrors_found |= ERROR_REACHABLE;\n@@ -592,7 +594,7 @@ static int fsck_cache_tree(struct cache_tree *it)\n \t\tfprintf(stderr, \"Checking cache tree\\n\");\n \n \tif (0 <= it->entry_count) {\n-\t\tstruct object *obj = parse_object(&it->oid);\n+\t\tstruct object *obj = parse_or_promise_object(&it->oid);\n \t\tif (!obj) {\n \t\t\terror(\"%s: invalid sha1 pointer in cache-tree\",\n \t\t\t      oid_to_hex(&it->oid));\n@@ -635,6 +637,12 @@ static int mark_packed_for_connectivity(const struct object_id *oid,\n \treturn 0;\n }\n \n+static int mark_have_promised_object(const struct object_id *oid, void *data)\n+{\n+\tmark_object_for_connectivity(oid);\n+\treturn 0;\n+}\n+\n static char const * const fsck_usage[] = {\n \tN_(\"git fsck [<options>] [<object>...]\"),\n \tNULL\n@@ -690,6 +698,11 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \n \tgit_config(fsck_config, NULL);\n \n+\tif (fsck_promised_objects()) {\n+\t\terror(\"Errors found in promised object list\");\n+\t\terrors_found |= ERROR_PROMISED_OBJECT;\n+\t}\n+\n \tfsck_head_link();\n \tif (connectivity_only) {\n \t\tfor_each_loose_object(mark_loose_for_connectivity, NULL, 0);\n@@ -727,6 +740,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\t\tstop_progress(&progress);\n \t\t}\n \t}\n+\tfor_each_promised_object(mark_have_promised_object, NULL);\n \n \theads = 0;\n \tfor (i = 0; i < argc; i++) {\ndiff --git a/cache.h b/cache.h\nindex 71fe09264..dd94b5ffc 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -853,10 +853,12 @@ extern int grafts_replace_parents;\n #define GIT_REPO_VERSION 0\n #define GIT_REPO_VERSION_READ 1\n extern int repository_format_precious_objects;\n+extern char *repository_format_promised_objects;\n \n struct repository_format {\n \tint version;\n \tint precious_objects;\n+\tchar *promised_objects;\n \tint is_bare;\n \tchar *work_tree;\n \tstruct string_list unknown_extensions;\ndiff --git a/environment.c b/environment.c\nindex 3fd4b1084..82658470d 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -27,6 +27,7 @@ int warn_ambiguous_refs = 1;\n int warn_on_object_refname_ambiguity = 1;\n int ref_paranoia = -1;\n int repository_format_precious_objects;\n+char *repository_format_promised_objects;\n const char *git_commit_encoding;\n const char *git_log_output_encoding;\n const char *apply_default_whitespace;\ndiff --git a/fsck.c b/fsck.c\nindex b4204d772..f08ff0675 100644\n--- a/fsck.c\n+++ b/fsck.c\n@@ -460,8 +460,10 @@ int fsck_walk(struct object *obj, void *data, struct fsck_options *options)\n \tif (!obj)\n \t\treturn -1;\n \n-\tif (obj->type == OBJ_NONE)\n-\t\tparse_object(&obj->oid);\n+\tif (!obj->parsed && !obj->promised)\n+\t\tparse_or_promise_object(&obj->oid);\n+\tif (obj->promised)\n+\t\treturn 0;\n \n \tswitch (obj->type) {\n \tcase OBJ_BLOB:\ndiff --git a/object.c b/object.c\nindex 321d7e920..0aeb95084 100644\n--- a/object.c\n+++ b/object.c\n@@ -4,6 +4,7 @@\n #include \"tree.h\"\n #include \"commit.h\"\n #include \"tag.h\"\n+#include \"promised-object.h\"\n \n static struct object **obj_hash;\n static int nr_objs, obj_hash_size;\n@@ -141,6 +142,7 @@ void *create_object(const unsigned char *sha1, void *o)\n \tstruct object *obj = o;\n \n \tobj->parsed = 0;\n+\tobj->promised = 0;\n \tobj->flags = 0;\n \thashcpy(obj->oid.hash, sha1);\n \n@@ -279,6 +281,23 @@ struct object *parse_object(const struct object_id *oid)\n \treturn NULL;\n }\n \n+struct object *parse_or_promise_object(const struct object_id *oid)\n+{\n+\tenum object_type type;\n+\n+\tif (has_object_file(oid))\n+\t\treturn parse_object(oid);\n+\n+\tif (is_promised_object(oid, &type, NULL)) {\n+\t\tstruct object *obj = lookup_unknown_object(oid->hash);\n+\t\tobj->promised = 1;\n+\t\tobj->type = type;\n+\t\treturn obj;\n+\t}\n+\n+\treturn NULL;\n+}\n+\n struct object_list *object_list_insert(struct object *item,\n \t\t\t\t       struct object_list **list_p)\n {\ndiff --git a/object.h b/object.h\nindex 0a419ba8d..640c8bff1 100644\n--- a/object.h\n+++ b/object.h\n@@ -46,7 +46,17 @@ struct object_array {\n  * The object type is stored in 3 bits.\n  */\n struct object {\n+\t/*\n+\t * Set if this object is parsed. If set, \"type\" is populated and this\n+\t * object can be casted to \"struct commit\" or an equivalent.\n+\t */\n \tunsigned parsed : 1;\n+\t/*\n+\t * Set if this object is not in the repo but is promised. If set,\n+\t * \"type\" is populated, but this object cannot be casted to \"struct\n+\t * commit\" or an equivalent.\n+\t */\n+\tunsigned promised : 1;\n \tunsigned type : TYPE_BITS;\n \tunsigned flags : FLAG_BITS;\n \tstruct object_id oid;\n@@ -104,6 +114,15 @@ struct object *parse_object_or_die(const struct object_id *oid, const char *name\n  */\n struct object *parse_object_buffer(const struct object_id *oid, enum object_type type, unsigned long size, void *buffer, int *eaten_p);\n \n+/*\n+ * Returns the object, having parsed it if possible. In the returned object,\n+ * either \"parsed\" or \"promised\" will be set.\n+ *\n+ * Returns NULL if the object is missing and not promised, or if the object is\n+ * corrupt.\n+ */\n+struct object *parse_or_promise_object(const struct object_id *oid);\n+\n /** Returns the object, with potentially excess memory allocated. **/\n struct object *lookup_unknown_object(const unsigned  char *sha1);\n \ndiff --git a/promised-object.c b/promised-object.c\nnew file mode 100644\nindex 000000000..487ade437\n--- /dev/null\n+++ b/promised-object.c\n@@ -0,0 +1,130 @@\n+#include \"cache.h\"\n+#include \"promised-object.h\"\n+#include \"sha1-lookup.h\"\n+#include \"strbuf.h\"\n+\n+#define ENTRY_SIZE (GIT_SHA1_RAWSZ + 1 + 8)\n+/*\n+ * A mmap-ed byte array of size (promised_object_nr * ENTRY_SIZE). Each\n+ * ENTRY_SIZE-sized entry consists of the SHA-1 of the promised object, its\n+ * 8-bit object type, and its 64-bit size in network byte order. The entries\n+ * are sorted in ascending SHA-1 order.\n+ */\n+static char *promised_objects;\n+static int64_t promised_object_nr = -1;\n+\n+static void prepare_promised_objects(void)\n+{\n+\tchar *filename;\n+\tint fd;\n+\tstruct stat st;\n+\n+\tif (promised_object_nr >= 0)\n+\t\treturn;\n+\n+\tif (!repository_format_promised_objects ||\n+\t    getenv(\"GIT_IGNORE_PROMISED_OBJECTS\")) {\n+\t\tpromised_object_nr = 0;\n+\t\treturn;\n+\t}\n+\n+\tfilename = xstrfmt(\"%s/promised\", get_object_directory());\n+\tfd = git_open(filename);\n+\tif (fd < 0) {\n+\t\tif (errno == ENOENT) {\n+\t\t\tpromised_object_nr = 0;\n+\t\t\tgoto cleanup;\n+\t\t}\n+\t\tperror(\"prepare_promised_objects\");\n+\t\tdie(\"Could not open %s\", filename);\n+\t}\n+\tif (fstat(fd, &st)) {\n+\t\tperror(\"prepare_promised_objects\");\n+\t\tdie(\"Could not stat %s\", filename);\n+\t}\n+\tif (st.st_size == 0) {\n+\t\tpromised_object_nr = 0;\n+\t\tgoto cleanup;\n+\t}\n+\tif (st.st_size % ENTRY_SIZE) {\n+\t\tdie(\"Size of %s is not a multiple of %d\", filename, ENTRY_SIZE);\n+\t}\n+\n+\tpromised_objects = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n+\tpromised_object_nr = st.st_size / ENTRY_SIZE;\n+\n+cleanup:\n+\tfree(filename);\n+\tif (fd >= 0)\n+\t\tclose(fd);\n+}\n+\n+int is_promised_object(const struct object_id *oid, enum object_type *type,\n+\t\t       unsigned long *size)\n+{\n+\tint result;\n+\n+\tprepare_promised_objects();\n+\tresult = sha1_entry_pos(promised_objects, ENTRY_SIZE, 0, 0,\n+\t\t\t\tpromised_object_nr, promised_object_nr,\n+\t\t\t\toid->hash);\n+\tif (result >= 0) {\n+\t\tif (type) {\n+\t\t\tchar *ptr = promised_objects +\n+\t\t\t\t    result * ENTRY_SIZE + GIT_SHA1_RAWSZ;\n+\t\t\t*type = *ptr;\n+\t\t}\n+\t\tif (size) {\n+\t\t\tuint64_t size_nbo;\n+\t\t\tchar *ptr = promised_objects +\n+\t\t\t\t    result * ENTRY_SIZE + GIT_SHA1_RAWSZ + 1;\n+\t\t\tmemcpy(&size_nbo, ptr, sizeof(size_nbo));\n+\t\t\t*size = ntohll(size_nbo);\n+\t\t}\n+\t\treturn 1;\n+\t}\n+\treturn 0;\n+}\n+\n+int for_each_promised_object(each_promised_object_fn cb, void *data)\n+{\n+\tstruct object_id oid;\n+\tint i, r;\n+\n+\tprepare_promised_objects();\n+\tfor (i = 0; i < promised_object_nr; i++) {\n+\t\tmemcpy(oid.hash, &promised_objects[i * ENTRY_SIZE],\n+\t\t       GIT_SHA1_RAWSZ);\n+\t\tr = cb(&oid, data);\n+\t\tif (r)\n+\t\t\treturn r;\n+\t}\n+\treturn 0;\n+}\n+\n+int fsck_promised_objects(void)\n+{\n+\tint i;\n+\tprepare_promised_objects();\n+\tfor (i = 0; i < promised_object_nr; i++) {\n+\t\tenum object_type type;\n+\t\tif (i != 0 && memcmp(&promised_objects[(i - 1) * ENTRY_SIZE],\n+\t\t\t\t     &promised_objects[i * ENTRY_SIZE],\n+\t\t\t\t     GIT_SHA1_RAWSZ) >= 0)\n+\t\t\treturn error(\"Error in list of promised objects: not \"\n+\t\t\t\t     \"in ascending order of object name\");\n+\t\ttype = promised_objects[i * ENTRY_SIZE + GIT_SHA1_RAWSZ];\n+\t\tswitch (type) {\n+\t\t\tcase OBJ_BLOB:\n+\t\t\tcase OBJ_TREE:\n+\t\t\tcase OBJ_COMMIT:\n+\t\t\tcase OBJ_TAG:\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\t\treturn error(\"Error in list of promised \"\n+\t\t\t\t\t     \"objects: found object of type %d\",\n+\t\t\t\t\t     type);\n+\t\t}\n+\t}\n+\treturn 0;\n+}\ndiff --git a/promised-object.h b/promised-object.h\nnew file mode 100644\nindex 000000000..7eaedff17\n--- /dev/null\n+++ b/promised-object.h\n@@ -0,0 +1,22 @@\n+#ifndef PROMISED_OBJECT_H\n+#define PROMISED_OBJECT_H\n+\n+#include \"cache.h\"\n+\n+/*\n+ * Returns 1 if oid is the name of a promised object. For non-blobs, 0 is\n+ * reported as their size.\n+ */\n+int is_promised_object(const struct object_id *oid, enum object_type *type,\n+\t\t       unsigned long *size);\n+\n+typedef int each_promised_object_fn(const struct object_id *oid, void *data);\n+int for_each_promised_object(each_promised_object_fn, void *);\n+\n+/*\n+ * Returns 0 if there is no list of promised objects or if the list of promised\n+ * objects is valid.\n+ */\n+int fsck_promised_objects(void);\n+\n+#endif\ndiff --git a/setup.c b/setup.c\nindex 860507e1f..36022452f 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -425,7 +425,11 @@ static int check_repo_format(const char *var, const char *value, void *vdata)\n \t\t\t;\n \t\telse if (!strcmp(ext, \"preciousobjects\"))\n \t\t\tdata->precious_objects = git_config_bool(var, value);\n-\t\telse\n+\t\telse if (!strcmp(ext, \"promisedobjects\")) {\n+\t\t\tif (!value)\n+\t\t\t\treturn config_error_nonbool(var);\n+\t\t\tdata->promised_objects = xstrdup(value);\n+\t\t} else\n \t\t\tstring_list_append(&data->unknown_extensions, ext);\n \t} else if (strcmp(var, \"core.bare\") == 0) {\n \t\tdata->is_bare = git_config_bool(var, value);\n@@ -468,6 +472,7 @@ static int check_repository_format_gently(const char *gitdir, int *nongit_ok)\n \t}\n \n \trepository_format_precious_objects = candidate.precious_objects;\n+\trepository_format_promised_objects = candidate.promised_objects;\n \tstring_list_clear(&candidate.unknown_extensions, 0);\n \tif (!has_common) {\n \t\tif (candidate.is_bare != -1) {\ndiff --git a/t/t3907-promised-object.sh b/t/t3907-promised-object.sh\nnew file mode 100755\nindex 000000000..3e0caf4f9\n--- /dev/null\n+++ b/t/t3907-promised-object.sh\n@@ -0,0 +1,41 @@\n+#!/bin/sh\n+\n+test_description='promised objects'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'fsck fails on missing objects' '\n+\ttest_create_repo repo &&\n+\n+\ttest_commit -C repo 1 &&\n+\ttest_commit -C repo 2 &&\n+\ttest_commit -C repo 3 &&\n+\tgit -C repo tag -a annotated_tag -m \"annotated tag\" &&\n+\tC=$(git -C repo rev-parse 1) &&\n+\tT=$(git -C repo rev-parse 2^{tree}) &&\n+\tB=$(git hash-object repo/3.t) &&\n+\tAT=$(git -C repo rev-parse annotated_tag) &&\n+\n+\t# missing commit, tree, blob, and tag\n+\trm repo/.git/objects/$(echo $C | cut -c1-2)/$(echo $C | cut -c3-40) &&\n+\trm repo/.git/objects/$(echo $T | cut -c1-2)/$(echo $T | cut -c3-40) &&\n+\trm repo/.git/objects/$(echo $B | cut -c1-2)/$(echo $B | cut -c3-40) &&\n+\trm repo/.git/objects/$(echo $AT | cut -c1-2)/$(echo $AT | cut -c3-40) &&\n+\ttest_must_fail git -C repo fsck\n+'\n+\n+test_expect_success '...but succeeds if they are promised objects' '\n+\tprintf \"%s01%016x\\n%s02%016x\\n%s03%016x\\n%s04%016x\" \\\n+\t\t\"$C\" 0 \"$T\" 0 \"$B\" \"$(wc -c <repo/3.t)\" \"$AT\" 0 |\n+\t\tsort | tr -d \"\\n\" | hex_pack >repo/.git/objects/promised &&\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.promisedobjects \"arbitrary string\" &&\n+\tgit -C repo fsck\n+'\n+\n+test_expect_success '...but fails again with GIT_IGNORE_PROMISED_OBJECTS' '\n+\tGIT_IGNORE_PROMISED_OBJECTS=1 test_must_fail git -C repo fsck &&\n+\tunset GIT_IGNORE_PROMISED_OBJECTS\n+'\n+\n+test_done\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex db622c355..1ebdd2d04 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1018,3 +1018,9 @@ nongit () {\n \t\t\"$@\"\n \t)\n }\n+\n+# Converts big-endian pairs of hexadecimal digits into bytes. For example,\n+# \"printf 61620d0a | hex_pack\" results in \"ab\\r\\n\".\n+hex_pack () {\n+\tperl -e '$/ = undef; $input = <>; print pack(\"H*\", $input)'\n+}\n-- \n2.14.0.rc0.284.gd933b75aa4-goog\n\n"},{"id":"324783","messageId":"0451fd25052ee217df8eca31fe12a08886cd6c34.1500508695.git.jonathantanmy@google.com","threadId":"46359","inReplyTo":"cover.1500508695.git.jonathantanmy@google.com","subject":"[RFC PATCH v2 3/4] sha1-array: support appending unsigned char hash","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-20T00:21:46Z","receivedAt":"2017-07-20T00:22:07Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In a subsequent patch, sha1_file will need to append object names in the\nform of \"unsigned char *\" to oid arrays. Teach sha1-array support for\nthat.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n sha1-array.c | 7 +++++++\n sha1-array.h | 1 +\n 2 files changed, 8 insertions(+)\n\ndiff --git a/sha1-array.c b/sha1-array.c\nindex 838b3bf84..6e0e35391 100644\n--- a/sha1-array.c\n+++ b/sha1-array.c\n@@ -9,6 +9,13 @@ void oid_array_append(struct oid_array *array, const struct object_id *oid)\n \tarray->sorted = 0;\n }\n \n+void oid_array_append_sha1(struct oid_array *array, const unsigned char *sha1)\n+{\n+\tALLOC_GROW(array->oid, array->nr + 1, array->alloc);\n+\thashcpy(array->oid[array->nr++].hash, sha1);\n+\tarray->sorted = 0;\n+}\n+\n static int void_hashcmp(const void *a, const void *b)\n {\n \treturn oidcmp(a, b);\ndiff --git a/sha1-array.h b/sha1-array.h\nindex 04b075633..3479959e4 100644\n--- a/sha1-array.h\n+++ b/sha1-array.h\n@@ -11,6 +11,7 @@ struct oid_array {\n #define OID_ARRAY_INIT { NULL, 0, 0, 0 }\n \n void oid_array_append(struct oid_array *array, const struct object_id *oid);\n+void oid_array_append_sha1(struct oid_array *array, const unsigned char *sha1);\n int oid_array_lookup(struct oid_array *array, const struct object_id *oid);\n void oid_array_clear(struct oid_array *array);\n \n-- \n2.14.0.rc0.284.gd933b75aa4-goog\n\n"},{"id":"324784","messageId":"a7fa848672281e29bff01f8caf2abe9f6ce5f4ba.1500508695.git.jonathantanmy@google.com","threadId":"46359","inReplyTo":"cover.1500508695.git.jonathantanmy@google.com","subject":"[RFC PATCH v2 4/4] sha1_file: support promised object hook","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-20T00:21:47Z","receivedAt":"2017-07-20T00:22:12Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Teach sha1_file to invoke a hook whenever an object is requested and\nunavailable but is promised. The hook is a shell command that can be\nconfigured through \"git config\"; this hook takes in a list of hashes and\nwrites (if successful) the corresponding objects to the repo's local\nstorage.\n\nThe usage of the hook can be suppressed through a flag when invoking\nhas_object_file_with_flags() and other similar functions.\nparse_or_promise_object() in object.c requires this functionality, and\nhas been modified to use it.\n\nThis is meant as a temporary measure to ensure that all Git commands\nwork in such a situation. Future patches will update some commands to\neither tolerate promised objects (without invoking the hook) or be more\nefficient in invoking the promised objects hook.\n\nIn order to determine the code changes in sha1_file.c necessary, I\ninvestigated the following:\n (1) functions in sha1_file that take in a hash, without the user\n     regarding how the object is stored (loose or packed)\n (2) functions in sha1_file that operate on packed objects (because I\n     need to check callers that know about the loose/packed distinction\n     and operate on both differently, and ensure that they can handle\n     the concept of objects that are neither loose nor packed)\n\n(1) is handled by the modification to sha1_object_info_extended().\n\nFor (2), I looked at for_each_packed_object and at the packed-related\nfunctions that take in a hash. For for_each_packed_object, the callers\neither already work or are fixed in this patch:\n - reachable - only to find recent objects\n - builtin/fsck - already knows about promised objects\n - builtin/cat-file - fixed in this commit\n\nCallers of the other functions do not need to be changed:\n - parse_pack_index\n   - http - indirectly from http_get_info_packs\n - find_pack_entry_one\n   - this searches a single pack that is provided as an argument; the\n     caller already knows (through other means) that the sought object\n     is in a specific pack\n - find_sha1_pack\n   - fast-import - appears to be an optimization to not store a\n     file if it is already in a pack\n   - http-walker - to search through a struct alt_base\n   - http-push - to search through remote packs\n - has_sha1_pack\n   - builtin/fsck - already knows about promised objects\n   - builtin/count-objects - informational purposes only (check if loose\n     object is also packed)\n   - builtin/prune-packed - check if object to be pruned is packed (if\n     not, don't prune it)\n   - revision - used to exclude packed objects if requested by user\n   - diff - just for optimization\n\nAn alternative design that I considered but rejected:\n\n - Adding a hook whenever a packed object is requested, not on any\n   object.  That is, whenever we attempt to search the packfiles for an\n   object, if it is missing (from the packfiles and from the loose\n   object storage), to invoke the hook (which must then store it as a\n   packfile), open the packfile the hook generated, and report that the\n   object is found in that new packfile. This reduces the amount of\n   analysis needed (in that we only need to look at how packed objects\n   are handled), but requires that the hook generate packfiles (or for\n   sha1_file to pack whatever loose objects are generated), creating one\n   packfile for each missing object and potentially very many packfiles\n   that must be linearly searched. This may be tolerable now for repos\n   that only have a few missing objects (for example, repos that only\n   want to exclude large blobs), and might be tolerable in the future if\n   we have batching support for the most commonly used commands, but is\n   not tolerable now for repos that exclude a large amount of objects.\n\nHelped-by: Ben Peart <benpeart@microsoft.com>\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Documentation/config.txt                         |   8 +\n Documentation/gitrepository-layout.txt           |   8 +\n Documentation/technical/read-object-protocol.txt | 102 ++++++++++++\n builtin/cat-file.c                               |   9 ++\n cache.h                                          |   2 +\n object.c                                         |   3 +-\n promised-object.c                                | 194 +++++++++++++++++++++++\n promised-object.h                                |  12 ++\n sha1_file.c                                      |  44 +++--\n t/t3907-promised-object.sh                       |  32 ++++\n t/t3907/read-object                              | 114 +++++++++++++\n 11 files changed, 513 insertions(+), 15 deletions(-)\n create mode 100644 Documentation/technical/read-object-protocol.txt\n create mode 100755 t/t3907/read-object\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex d5c9c4cab..c293ac921 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -393,6 +393,14 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n will probe and set core.ignoreCase true if appropriate when the repository\n is created.\n \n+core.promisedBlobCommand::\n+\tIf set, whenever a blob in the local repo is attempted to be read, but\n+\tis both missing and a promised blob, invoke this shell command to\n+\tgenerate or obtain that blob before reporting an error. This shell\n+\tcommand should take one or more hashes, each terminated by a newline,\n+\tas standard input, and (if successful) should write the corresponding\n+\tobjects to the local repo (packed or loose).\n+\n core.precomposeUnicode::\n \tThis option is only used by Mac OS implementation of Git.\n \tWhen core.precomposeUnicode=true, Git reverts the unicode decomposition\ndiff --git a/Documentation/gitrepository-layout.txt b/Documentation/gitrepository-layout.txt\nindex f51ed4e37..7dea7fe6b 100644\n--- a/Documentation/gitrepository-layout.txt\n+++ b/Documentation/gitrepository-layout.txt\n@@ -47,6 +47,10 @@ use with dumb transports but otherwise is OK as long as\n `objects/info/alternates` points at the object stores it\n borrows from.\n +\n+. You could have objects that are merely promised by another source.\n+When Git requires those objects, it will invoke the command in the\n+`extensions.promisedObjects` configuration variable.\n++\n This directory is ignored if $GIT_COMMON_DIR is set and\n \"$GIT_COMMON_DIR/objects\" will be used instead.\n \n@@ -91,6 +95,10 @@ objects/info/http-alternates::\n \tthis object store borrows objects from, to be used when\n \tthe repository is fetched over HTTP.\n \n+objects/promised::\n+\tThis file records the sha1 object names, types, and sizes of\n+\tpromised objects.\n+\n refs::\n \tReferences are stored in subdirectories of this\n \tdirectory.  The 'git prune' command knows to preserve\ndiff --git a/Documentation/technical/read-object-protocol.txt b/Documentation/technical/read-object-protocol.txt\nnew file mode 100644\nindex 000000000..a893b46e7\n--- /dev/null\n+++ b/Documentation/technical/read-object-protocol.txt\n@@ -0,0 +1,102 @@\n+Read Object Process\n+^^^^^^^^^^^^^^^^^^^^^^^^^^^\n+\n+The read-object process enables Git to read all missing blobs with a\n+single process invocation for the entire life of a single Git command.\n+This is achieved by using a packet format (pkt-line, see technical/\n+protocol-common.txt) based protocol over standard input and standard\n+output as follows. All packets, except for the \"*CONTENT\" packets and\n+the \"0000\" flush packet, are considered text and therefore are\n+terminated by a LF.\n+\n+Git starts the process when it encounters the first missing object that\n+needs to be retrieved. After the process is started, Git sends a welcome\n+message (\"git-read-object-client\"), a list of supported protocol version\n+numbers, and a flush packet. Git expects to read a welcome response\n+message (\"git-read-object-server\"), exactly one protocol version number\n+from the previously sent list, and a flush packet. All further\n+communication will be based on the selected version.\n+\n+The remaining protocol description below documents \"version=1\". Please\n+note that \"version=42\" in the example below does not exist and is only\n+there to illustrate how the protocol would look with more than one\n+version.\n+\n+After the version negotiation Git sends a list of all capabilities that\n+it supports and a flush packet. Git expects to read a list of desired\n+capabilities, which must be a subset of the supported capabilities list,\n+and a flush packet as response:\n+------------------------\n+packet: git> git-read-object-client\n+packet: git> version=1\n+packet: git> version=42\n+packet: git> 0000\n+packet: git< git-read-object-server\n+packet: git< version=1\n+packet: git< 0000\n+packet: git> capability=get\n+packet: git> capability=have\n+packet: git> capability=put\n+packet: git> capability=not-yet-invented\n+packet: git> 0000\n+packet: git< capability=get\n+packet: git< 0000\n+------------------------\n+The only supported capability in version 1 is \"get\".\n+\n+Afterwards Git sends a list of \"key=value\" pairs terminated with a flush\n+packet. The list will contain at least the command (based on the\n+supported capabilities) and the sha1 of the object to retrieve. Please\n+note, that the process must not send any response before it received the\n+final flush packet.\n+\n+When the process receives the \"get\" command, it should make the requested\n+object available in the git object store and then return success. Git will\n+then check the object store again and this time find it and proceed.\n+------------------------\n+packet: git> command=get\n+packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n+packet: git> 0000\n+------------------------\n+\n+The process is expected to respond with a list of \"key=value\" pairs\n+terminated with a flush packet. If the process does not experience\n+problems then the list must contain a \"success\" status.\n+------------------------\n+packet: git< status=success\n+packet: git< 0000\n+------------------------\n+\n+In case the process cannot or does not want to process the content, it\n+is expected to respond with an \"error\" status.\n+------------------------\n+packet: git< status=error\n+packet: git< 0000\n+------------------------\n+\n+In case the process cannot or does not want to process the content as\n+well as any future content for the lifetime of the Git process, then it\n+is expected to respond with an \"abort\" status at any point in the\n+protocol.\n+------------------------\n+packet: git< status=abort\n+packet: git< 0000\n+------------------------\n+\n+Git neither stops nor restarts the process in case the \"error\"/\"abort\"\n+status is set.\n+\n+If the process dies during the communication or does not adhere to the\n+protocol then Git will stop the process and restart it with the next\n+object that needs to be processed.\n+\n+After the read-object process has processed an object it is expected to\n+wait for the next \"key=value\" list containing a command. Git will close\n+the command pipe on exit. The process is expected to detect EOF and exit\n+gracefully on its own. Git will wait until the process has stopped.\n+\n+A long running read-object process demo implementation can be found in\n+`contrib/long-running-read-object/example.pl` located in the Git core\n+repository. If you develop your own long running process then the\n+`GIT_TRACE_PACKET` environment variables can be very helpful for\n+debugging (see linkgit:git[1]).\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex 96b786e48..33f636926 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -12,6 +12,7 @@\n #include \"streaming.h\"\n #include \"tree-walk.h\"\n #include \"sha1-array.h\"\n+#include \"promised-object.h\"\n \n struct batch_options {\n \tint enabled;\n@@ -432,6 +433,13 @@ static int batch_packed_object(const struct object_id *oid,\n \treturn 0;\n }\n \n+static int batch_promised_object(const struct object_id *oid,\n+\t\t\t\t void *data)\n+{\n+\toid_array_append(data, oid);\n+\treturn 0;\n+}\n+\n static int batch_objects(struct batch_options *opt)\n {\n \tstruct strbuf buf = STRBUF_INIT;\n@@ -473,6 +481,7 @@ static int batch_objects(struct batch_options *opt)\n \n \t\tfor_each_loose_object(batch_loose_object, &sa, 0);\n \t\tfor_each_packed_object(batch_packed_object, &sa, 0);\n+\t\tfor_each_promised_object(batch_promised_object, &sa);\n \n \t\tcb.opt = opt;\n \t\tcb.expand = &data;\ndiff --git a/cache.h b/cache.h\nindex dd94b5ffc..75b71f38b 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1835,6 +1835,8 @@ struct object_info {\n #define OBJECT_INFO_SKIP_CACHED 4\n /* Do not retry packed storage after checking packed and loose storage */\n #define OBJECT_INFO_QUICK 8\n+/* Ignore list of promised objects */\n+#define OBJECT_INFO_IGNORE_PROMISES 16\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n \ndiff --git a/object.c b/object.c\nindex 0aeb95084..23f2a6cbc 100644\n--- a/object.c\n+++ b/object.c\n@@ -285,7 +285,8 @@ struct object *parse_or_promise_object(const struct object_id *oid)\n {\n \tenum object_type type;\n \n-\tif (has_object_file(oid))\n+\tif (has_object_file_with_flags(oid, OBJECT_INFO_SKIP_CACHED |\n+\t\t\t\t\t    OBJECT_INFO_IGNORE_PROMISES))\n \t\treturn parse_object(oid);\n \n \tif (is_promised_object(oid, &type, NULL)) {\ndiff --git a/promised-object.c b/promised-object.c\nindex 487ade437..d8d95ebb2 100644\n--- a/promised-object.c\n+++ b/promised-object.c\n@@ -2,6 +2,12 @@\n #include \"promised-object.h\"\n #include \"sha1-lookup.h\"\n #include \"strbuf.h\"\n+#include \"run-command.h\"\n+#include \"sha1-array.h\"\n+#include \"config.h\"\n+#include \"sigchain.h\"\n+#include \"sub-process.h\"\n+#include \"pkt-line.h\"\n \n #define ENTRY_SIZE (GIT_SHA1_RAWSZ + 1 + 8)\n /*\n@@ -128,3 +134,191 @@ int fsck_promised_objects(void)\n \t}\n \treturn 0;\n }\n+\n+#define CAP_GET    (1u<<0)\n+\n+static int subprocess_map_initialized;\n+static struct hashmap subprocess_map;\n+\n+struct read_object_process {\n+\tstruct subprocess_entry subprocess;\n+\tunsigned int supported_capabilities;\n+};\n+\n+int start_read_object_fn(struct subprocess_entry *subprocess)\n+{\n+\tint err;\n+\tstruct read_object_process *entry = (struct read_object_process *)subprocess;\n+\tstruct child_process *process;\n+\tstruct string_list cap_list = STRING_LIST_INIT_NODUP;\n+\tchar *cap_buf;\n+\tconst char *cap_name;\n+\n+\tprocess = subprocess_get_child_process(&entry->subprocess);\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\terr = packet_writel(process->in, \"git-read-object-client\", \"version=1\", NULL);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = strcmp(packet_read_line(process->out, NULL), \"git-read-object-server\");\n+\tif (err) {\n+\t\terror(\"external process '%s' does not support read-object protocol version 1\", subprocess->cmd);\n+\t\tgoto done;\n+\t}\n+\terr = strcmp(packet_read_line(process->out, NULL), \"version=1\");\n+\tif (err)\n+\t\tgoto done;\n+\terr = packet_read_line(process->out, NULL) != NULL;\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_writel(process->in, \"capability=get\", NULL);\n+\tif (err)\n+\t\tgoto done;\n+\n+\tfor (;;) {\n+\t\tcap_buf = packet_read_line(process->out, NULL);\n+\t\tif (!cap_buf)\n+\t\t\tbreak;\n+\t\tstring_list_split_in_place(&cap_list, cap_buf, '=', 1);\n+\n+\t\tif (cap_list.nr != 2 || strcmp(cap_list.items[0].string, \"capability\"))\n+\t\t\tcontinue;\n+\n+\t\tcap_name = cap_list.items[1].string;\n+\t\tif (!strcmp(cap_name, \"get\")) {\n+\t\t\tentry->supported_capabilities |= CAP_GET;\n+\t\t}\n+\t\telse {\n+\t\t\twarning(\n+\t\t\t\t\"external process '%s' requested unsupported read-object capability '%s'\",\n+\t\t\t\tsubprocess->cmd, cap_name\n+\t\t\t);\n+\t\t}\n+\n+\t\tstring_list_clear(&cap_list, 0);\n+\t}\n+\n+done:\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (err || errno == EPIPE)\n+\t\treturn err ? err : errno;\n+\n+\treturn 0;\n+}\n+\n+static int read_object_process(const unsigned char *sha1)\n+{\n+\tint err;\n+\tstruct read_object_process *entry;\n+\tstruct child_process *process;\n+\tstruct strbuf status = STRBUF_INIT;\n+\tuint64_t start;\n+\n+\tstart = getnanotime();\n+\n+\tif (!repository_format_promised_objects)\n+\t\tdie(\"BUG: if extensions.promisedObjects is not set, there \"\n+\t\t    \"should not be any promised objects\");\n+\n+\tif (!subprocess_map_initialized) {\n+\t\tsubprocess_map_initialized = 1;\n+\t\thashmap_init(&subprocess_map, (hashmap_cmp_fn)cmd2process_cmp, 0);\n+\t\tentry = NULL;\n+\t} else {\n+\t\tentry = (struct read_object_process *)subprocess_find_entry(&subprocess_map, repository_format_promised_objects);\n+\t}\n+\tif (!entry) {\n+\t\tentry = xmalloc(sizeof(*entry));\n+\t\tentry->supported_capabilities = 0;\n+\n+\t\tif (subprocess_start(&subprocess_map, &entry->subprocess, repository_format_promised_objects, start_read_object_fn)) {\n+\t\t\tfree(entry);\n+\t\t\treturn -1;\n+\t\t}\n+\t}\n+\tprocess = subprocess_get_child_process(&entry->subprocess);\n+\n+\tif (!(CAP_GET & entry->supported_capabilities))\n+\t\treturn -1;\n+\n+\tsigchain_push(SIGPIPE, SIG_IGN);\n+\n+\terr = packet_write_fmt_gently(process->in, \"command=get\\n\");\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = packet_flush_gently(process->in);\n+\tif (err)\n+\t\tgoto done;\n+\n+\terr = subprocess_read_status(process->out, &status);\n+\terr = err ? err : strcmp(status.buf, \"success\");\n+\n+done:\n+\tsigchain_pop(SIGPIPE);\n+\n+\tif (err || errno == EPIPE) {\n+\t\terr = err ? err : errno;\n+\t\tif (!strcmp(status.buf, \"error\")) {\n+\t\t\t/* The process signaled a problem with the file. */\n+\t\t}\n+\t\telse if (!strcmp(status.buf, \"abort\")) {\n+\t\t\t/*\n+\t\t\t * The process signaled a permanent problem. Don't try to read\n+\t\t\t * objects with the same command for the lifetime of the current\n+\t\t\t * Git process.\n+\t\t\t */\n+\t\t\tentry->supported_capabilities &= ~CAP_GET;\n+\t\t}\n+\t\telse {\n+\t\t\t/*\n+\t\t\t * Something went wrong with the read-object process.\n+\t\t\t * Force shutdown and restart if needed.\n+\t\t\t */\n+\t\t\terror(\"external process '%s' failed\", repository_format_promised_objects);\n+\t\t\tsubprocess_stop(&subprocess_map, (struct subprocess_entry *)entry);\n+\t\t\tfree(entry);\n+\t\t}\n+\t}\n+\n+\ttrace_performance_since(start, \"read_object_process\");\n+\n+\treturn err;\n+}\n+\n+int request_promised_objects(const struct oid_array *oids)\n+{\n+\tint oids_requested = 0;\n+\tint i;\n+\n+\tfor (i = 0; i < oids->nr; i++) {\n+\t\tif (is_promised_object(&oids->oid[i], NULL, NULL))\n+\t\t\tbreak;\n+\t}\n+\n+\tif (i == oids->nr)\n+\t\t/* Nothing to fetch */\n+\t\treturn 0;\n+\n+\tfor (; i < oids->nr; i++) {\n+\t\tif (is_promised_object(&oids->oid[i], NULL, NULL)) {\n+\t\t\tread_object_process(oids->oid[i].hash);\n+\t\t\toids_requested++;\n+\t\t}\n+\t}\n+\n+\t/*\n+\t * The command above may have updated packfiles, so update our record\n+\t * of them.\n+\t */\n+\treprepare_packed_git();\n+\treturn oids_requested;\n+}\ndiff --git a/promised-object.h b/promised-object.h\nindex 7eaedff17..8ad47aa4c 100644\n--- a/promised-object.h\n+++ b/promised-object.h\n@@ -2,6 +2,7 @@\n #define PROMISED_OBJECT_H\n \n #include \"cache.h\"\n+#include \"sha1-array.h\"\n \n /*\n  * Returns 1 if oid is the name of a promised object. For non-blobs, 0 is\n@@ -19,4 +20,15 @@ int for_each_promised_object(each_promised_object_fn, void *);\n  */\n int fsck_promised_objects(void);\n \n+/*\n+ * If any of the given objects are promised objects, invokes\n+ * core.promisedobjectcommand with those objects and returns the number of\n+ * objects requested. No check is made as to whether the invocation actually\n+ * populated the repository with the promised objects.\n+ *\n+ * If none of the given objects are promised objects, this function does not\n+ * invoke anything and returns 0.\n+ */\n+int request_promised_objects(const struct oid_array *oids);\n+\n #endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 5862386cd..ded0ef46b 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -28,6 +28,11 @@\n #include \"list.h\"\n #include \"mergesort.h\"\n #include \"quote.h\"\n+#include \"iterator.h\"\n+#include \"dir-iterator.h\"\n+#include \"sha1-lookup.h\"\n+#include \"promised-object.h\"\n+#include \"sha1-array.h\"\n \n #define SZ_FMT PRIuMAX\n static inline uintmax_t sz_fmt(size_t s) { return s; }\n@@ -2983,6 +2988,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n \t\t\t\t    lookup_replace_object(sha1) :\n \t\t\t\t    sha1;\n+\tint already_retried = 0;\n \n \tif (!oi)\n \t\toi = &blank_oi;\n@@ -3007,30 +3013,40 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n-\tif (!find_pack_entry(real, &e)) {\n-\t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, oi, flags)) {\n-\t\t\toi->whence = OI_LOOSE;\n-\t\t\treturn 0;\n-\t\t}\n+retry:\n+\tif (find_pack_entry(real, &e))\n+\t\tgoto found_packed;\n \n-\t\t/* Not a loose object; someone else may have just packed it. */\n-\t\tif (flags & OBJECT_INFO_QUICK) {\n-\t\t\treturn -1;\n-\t\t} else {\n-\t\t\treprepare_packed_git();\n-\t\t\tif (!find_pack_entry(real, &e))\n-\t\t\t\treturn -1;\n+\t/* Most likely it's a loose object. */\n+\tif (!sha1_loose_object_info(real, oi, flags)) {\n+\t\toi->whence = OI_LOOSE;\n+\t\treturn 0;\n+\t}\n+\n+\t/* Not a loose object; someone else may have just packed it. */\n+\treprepare_packed_git();\n+\tif (find_pack_entry(real, &e))\n+\t\tgoto found_packed;\n+\n+\t/* Check if it is a promised blob */\n+\tif (!already_retried && !(flags & OBJECT_INFO_IGNORE_PROMISES)) {\n+\t\tstruct oid_array promised = OID_ARRAY_INIT;\n+\t\toid_array_append_sha1(&promised, real);\n+\t\tif (request_promised_objects(&promised)) {\n+\t\t\talready_retried = 1;\n+\t\t\tgoto retry;\n \t\t}\n \t}\n \n+\treturn -1;\n+\n+found_packed:\n \tif (oi == &blank_oi)\n \t\t/*\n \t\t * We know that the caller doesn't actually need the\n \t\t * information below, so return early.\n \t\t */\n \t\treturn 0;\n-\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\ndiff --git a/t/t3907-promised-object.sh b/t/t3907-promised-object.sh\nindex 3e0caf4f9..d9e6a6486 100755\n--- a/t/t3907-promised-object.sh\n+++ b/t/t3907-promised-object.sh\n@@ -38,4 +38,36 @@ test_expect_success '...but fails again with GIT_IGNORE_PROMISED_OBJECTS' '\n \tunset GIT_IGNORE_PROMISED_OBJECTS\n '\n \n+test_expect_success 'sha1_object_info_extended (through git cat-file)' '\n+\ttest_create_repo server &&\n+\ttest_commit -C server 1 1.t abcdefgh &&\n+\tHASH=$(git hash-object server/1.t) &&\n+\n+\ttest_create_repo client &&\n+\ttest_must_fail git -C client cat-file -p \"$HASH\"\n+'\n+\n+test_expect_success '...succeeds if it is a promised object' '\n+\tprintf \"%s03%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\thex_pack >client/.git/objects/promised &&\n+\tgit -C client config core.repositoryformatversion 1 &&\n+\tgit -C client config extensions.promisedobjects \\\n+\t\t\"\\\"$TEST_DIRECTORY/t3907/read-object\\\" \\\"$(pwd)/server/.git\\\"\" &&\n+\tgit -C client cat-file -p \"$HASH\"\n+'\n+\n+test_expect_success 'cat-file --batch-all-objects with promised objects' '\n+\trm -rf client &&\n+\ttest_create_repo client &&\n+\tgit -C client config core.repositoryformatversion 1 &&\n+\tgit -C client config extensions.promisedobjects \\\n+\t\t\"\\\"$TEST_DIRECTORY/t3907/read-object\\\" \\\"$(pwd)/server/.git\\\"\" &&\n+\tprintf \"%s03%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n+\t\thex_pack >client/.git/objects/promised &&\n+\n+\t# Verify that the promised object is printed\n+\tgit -C client cat-file --batch --batch-all-objects | tee out |\n+\t\tgrep abcdefgh\n+'\n+\n test_done\ndiff --git a/t/t3907/read-object b/t/t3907/read-object\nnew file mode 100755\nindex 000000000..9666ad597\n--- /dev/null\n+++ b/t/t3907/read-object\n@@ -0,0 +1,114 @@\n+#!/usr/bin/perl\n+#\n+# Example implementation for the Git read-object protocol version 1\n+# See Documentation/technical/read-object-protocol.txt\n+#\n+# Allows you to test the ability for blobs to be pulled from a host git repo\n+# \"on demand.\"  Called when git needs a blob it couldn't find locally due to\n+# a lazy clone that only cloned the commits and trees.\n+#\n+# A lazy clone can be simulated via the following commands from the host repo\n+# you wish to create a lazy clone of:\n+#\n+# cd /host_repo\n+# git rev-parse HEAD\n+# git init /guest_repo\n+# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n+#\tcut -d' ' -f1 | git pack-objects /guest_repo/.git/objects/pack/noblobs\n+# cd /guest_repo\n+# git config core.virtualizeobjects true\n+# git reset --hard <sha from rev-parse call above>\n+#\n+# Please note, this sample is a minimal skeleton. No proper error handling\n+# was implemented.\n+#\n+\n+use strict;\n+use warnings;\n+\n+#\n+# Point $DIR to the folder where your host git repo is located so we can pull\n+# missing objects from it\n+#\n+my $DIR = $ARGV[0];\n+\n+sub packet_bin_read {\n+\tmy $buffer;\n+\tmy $bytes_read = read STDIN, $buffer, 4;\n+\tif ( $bytes_read == 0 ) {\n+\n+\t\t# EOF - Git stopped talking to us!\n+\t\texit();\n+\t}\n+\telsif ( $bytes_read != 4 ) {\n+\t\tdie \"invalid packet: '$buffer'\";\n+\t}\n+\tmy $pkt_size = hex($buffer);\n+\tif ( $pkt_size == 0 ) {\n+\t\treturn ( 1, \"\" );\n+\t}\n+\telsif ( $pkt_size > 4 ) {\n+\t\tmy $content_size = $pkt_size - 4;\n+\t\t$bytes_read = read STDIN, $buffer, $content_size;\n+\t\tif ( $bytes_read != $content_size ) {\n+\t\t\tdie \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n+\t\t}\n+\t\treturn ( 0, $buffer );\n+\t}\n+\telse {\n+\t\tdie \"invalid packet size: $pkt_size\";\n+\t}\n+}\n+\n+sub packet_txt_read {\n+\tmy ( $res, $buf ) = packet_bin_read();\n+\tunless ( $buf =~ s/\\n$// ) {\n+\t\tdie \"A non-binary line MUST be terminated by an LF.\";\n+\t}\n+\treturn ( $res, $buf );\n+}\n+\n+sub packet_bin_write {\n+\tmy $buf = shift;\n+\tprint STDOUT sprintf( \"%04x\", length($buf) + 4 );\n+\tprint STDOUT $buf;\n+\tSTDOUT->flush();\n+}\n+\n+sub packet_txt_write {\n+\tpacket_bin_write( $_[0] . \"\\n\" );\n+}\n+\n+sub packet_flush {\n+\tprint STDOUT sprintf( \"%04x\", 0 );\n+\tSTDOUT->flush();\n+}\n+\n+( packet_txt_read() eq ( 0, \"git-read-object-client\" ) ) || die \"bad initialize\";\n+( packet_txt_read() eq ( 0, \"version=1\" ) )\t\t\t\t || die \"bad version\";\n+( packet_bin_read() eq ( 1, \"\" ) )                       || die \"bad version end\";\n+\n+packet_txt_write(\"git-read-object-server\");\n+packet_txt_write(\"version=1\");\n+packet_flush();\n+\n+( packet_txt_read() eq ( 0, \"capability=get\" ) )    || die \"bad capability\";\n+( packet_bin_read() eq ( 1, \"\" ) )                  || die \"bad capability end\";\n+\n+packet_txt_write(\"capability=get\");\n+packet_flush();\n+\n+while (1) {\n+\tmy ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n+\n+\tif ( $command eq \"get\" ) {\n+\t\tmy ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n+\t\tpacket_bin_read();\n+\n+\t\tsystem ('git --git-dir=\"' . $DIR . '\" cat-file blob ' . $sha1 . ' | git -c core.virtualizeobjects=false hash-object -w --stdin >/dev/null 2>&1');\n+\t\tpacket_txt_write(($?) ? \"status=error\" : \"status=success\");\n+\t\tpacket_flush();\n+\t} else {\n+\t\tdie \"bad command '$command'\";\n+\t}\n+}\n-- \n2.14.0.rc0.284.gd933b75aa4-goog\n\n"},{"id":"324785","messageId":"CAGZ79kbYebKkSBtiFOMO5+uknbNMnhyT6P6uu=MMhmFyOZRE+w@mail.gmail.com","threadId":"46359","inReplyTo":"0eb270f40095f132bf9383f230506e45d895e5e1.1500508695.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH v2 1/4] object: remove \"used\" field from struct object","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-20T00:36:39Z","receivedAt":"2017-07-20T00:36:45Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jul 19, 2017 at 5:21 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> The \"used\" field in struct object is only used by builtin/fsck. Remove\n> that field and modify builtin/fsck to use a flag instead.\n\nThe patch looks good to me (I would even claim this could\ngo in as an independent cleanup, not tied to the RFCish nature\nof the later patches), though I have a question:\nHow did you select 0x0008 for USED, i.e. does it\ncollide with other flags (theoretically?), and if so\nhow do we make sure to avoid the collusion in\nthe future?\n"},{"id":"324788","messageId":"20170719175512.4974d1a2@twelve2.svl.corp.google.com","threadId":"46359","inReplyTo":"CAGZ79kbYebKkSBtiFOMO5+uknbNMnhyT6P6uu=MMhmFyOZRE+w@mail.gmail.com","subject":"Re: [RFC PATCH v2 1/4] object: remove \"used\" field from struct object","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-20T00:55:12Z","receivedAt":"2017-07-20T00:55:22Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 19 Jul 2017 17:36:39 -0700\nStefan Beller <sbeller@google.com> wrote:\n\n> On Wed, Jul 19, 2017 at 5:21 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> > The \"used\" field in struct object is only used by builtin/fsck. Remove\n> > that field and modify builtin/fsck to use a flag instead.\n> \n> The patch looks good to me (I would even claim this could\n> go in as an independent cleanup, not tied to the RFCish nature\n> of the later patches), though I have a question:\n> How did you select 0x0008 for USED, i.e. does it\n> collide with other flags (theoretically?), and if so\n> how do we make sure to avoid the collusion in\n> the future?\n\nThanks. 0x0008 was the next one in the series (as you can see in the\ncontext). As for whether it collides with other flags, that is what the\nchart in object.h is for (which I have added to in this patch), I\npresume. As far as I can tell, each component must make sure not to\noverlap with any other component running concurrently.\n"},{"id":"324815","messageId":"dfa6a97c-683c-66d6-61f1-cf7cde6f8b16@gmail.com","threadId":"46359","inReplyTo":"20170719175512.4974d1a2@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH v2 1/4] object: remove \"used\" field from struct object","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-20T17:44:05Z","receivedAt":"2017-07-20T17:44:16Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/19/2017 8:55 PM, Jonathan Tan wrote:\n> On Wed, 19 Jul 2017 17:36:39 -0700\n> Stefan Beller <sbeller@google.com> wrote:\n> \n>> On Wed, Jul 19, 2017 at 5:21 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n>>> The \"used\" field in struct object is only used by builtin/fsck. Remove\n>>> that field and modify builtin/fsck to use a flag instead.\n>>\n>> The patch looks good to me (I would even claim this could\n>> go in as an independent cleanup, not tied to the RFCish nature\n>> of the later patches), though I have a question:\n>> How did you select 0x0008 for USED, i.e. does it\n>> collide with other flags (theoretically?), and if so\n>> how do we make sure to avoid the collusion in\n>> the future?\n> \n> Thanks. 0x0008 was the next one in the series (as you can see in the\n> context). As for whether it collides with other flags, that is what the\n> chart in object.h is for (which I have added to in this patch), I\n> presume. As far as I can tell, each component must make sure not to\n> overlap with any other component running concurrently.\n> \n\nThis patch seems reasonable to me.  I agree it could go in separately as \na general cleanup.\n"},{"id":"324817","messageId":"CAGZ79kZ45Z3R9FbsKQwotXh_Ek=StUdSTHj4er6HMhx79VbV4g@mail.gmail.com","threadId":"46359","inReplyTo":"3420d9ae9ef86b78af1abe721891233e3f5865a2.1500508695.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-20T18:07:29Z","receivedAt":"2017-07-20T18:07:36Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jul 19, 2017 at 5:21 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> Currently, Git does not support repos with very large numbers of objects\n> or repos that wish to minimize manipulation of certain blobs (for\n> example, because they are very large) very well, even if the user\n> operates mostly on part of the repo, because Git is designed on the\n> assumption that every referenced object is available somewhere in the\n> repo storage.\n>\n> As a first step to reducing this problem, introduce the concept of\n> promised objects. Each Git repo can contain a list of promised objects\n> and their sizes (if blobs) at $GIT_DIR/objects/promised. This patch\n> contains functions to query them; functions for creating and modifying\n> that file will be introduced in later patches.\n>\n> A repository that is missing an object but has that object promised is not\n> considered to be in error, so also teach fsck this. As part of doing\n> this, object.{h,c} has been modified to generate \"struct object\" based\n> on only the information available to promised objects, without requiring\n> the object itself.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  Documentation/technical/repository-version.txt |   6 ++\n>  Makefile                                       |   1 +\n>  builtin/fsck.c                                 |  18 +++-\n>  cache.h                                        |   2 +\n>  environment.c                                  |   1 +\n>  fsck.c                                         |   6 +-\n>  object.c                                       |  19 ++++\n>  object.h                                       |  19 ++++\n>  promised-object.c                              | 130 +++++++++++++++++++++++++\n>  promised-object.h                              |  22 +++++\n>  setup.c                                        |   7 +-\n>  t/t3907-promised-object.sh                     |  41 ++++++++\n>  t/test-lib-functions.sh                        |   6 ++\n>  13 files changed, 273 insertions(+), 5 deletions(-)\n>  create mode 100644 promised-object.c\n>  create mode 100644 promised-object.h\n>  create mode 100755 t/t3907-promised-object.sh\n>\n> diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\n> index 00ad37986..f8b82c1c7 100644\n> --- a/Documentation/technical/repository-version.txt\n> +++ b/Documentation/technical/repository-version.txt\n> @@ -86,3 +86,9 @@ for testing format-1 compatibility.\n>  When the config key `extensions.preciousObjects` is set to `true`,\n>  objects in the repository MUST NOT be deleted (e.g., by `git-prune` or\n>  `git repack -d`).\n> +\n> +`promisedObjects`\n> +~~~~~~~~~~~~~~~~~\n> +\n> +(Explain this - basically a string containing a command to be run\n> +whenever a missing object needs to be fetched.)\n> diff --git a/Makefile b/Makefile\n> index 9c9c42f8f..c1446d5ef 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -828,6 +828,7 @@ LIB_OBJS += preload-index.o\n>  LIB_OBJS += pretty.o\n>  LIB_OBJS += prio-queue.o\n>  LIB_OBJS += progress.o\n> +LIB_OBJS += promised-object.o\n>  LIB_OBJS += prompt.o\n>  LIB_OBJS += quote.o\n>  LIB_OBJS += reachable.o\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index 462b8643b..49e21f361 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -15,6 +15,7 @@\n>  #include \"progress.h\"\n>  #include \"streaming.h\"\n>  #include \"decorate.h\"\n> +#include \"promised-object.h\"\n>\n>  #define REACHABLE 0x0001\n>  #define SEEN      0x0002\n> @@ -44,6 +45,7 @@ static int name_objects;\n>  #define ERROR_REACHABLE 02\n>  #define ERROR_PACK 04\n>  #define ERROR_REFS 010\n> +#define ERROR_PROMISED_OBJECT 011\n>\n>  static const char *describe_object(struct object *obj)\n>  {\n> @@ -436,7 +438,7 @@ static int fsck_handle_ref(const char *refname, const struct object_id *oid,\n>  {\n>         struct object *obj;\n>\n> -       obj = parse_object(oid);\n> +       obj = parse_or_promise_object(oid);\n>         if (!obj) {\n>                 error(\"%s: invalid sha1 pointer %s\", refname, oid_to_hex(oid));\n>                 errors_found |= ERROR_REACHABLE;\n> @@ -592,7 +594,7 @@ static int fsck_cache_tree(struct cache_tree *it)\n>                 fprintf(stderr, \"Checking cache tree\\n\");\n>\n>         if (0 <= it->entry_count) {\n> -               struct object *obj = parse_object(&it->oid);\n> +               struct object *obj = parse_or_promise_object(&it->oid);\n>                 if (!obj) {\n>                         error(\"%s: invalid sha1 pointer in cache-tree\",\n>                               oid_to_hex(&it->oid));\n> @@ -635,6 +637,12 @@ static int mark_packed_for_connectivity(const struct object_id *oid,\n>         return 0;\n>  }\n>\n> +static int mark_have_promised_object(const struct object_id *oid, void *data)\n> +{\n> +       mark_object_for_connectivity(oid);\n> +       return 0;\n> +}\n> +\n>  static char const * const fsck_usage[] = {\n>         N_(\"git fsck [<options>] [<object>...]\"),\n>         NULL\n> @@ -690,6 +698,11 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n>\n>         git_config(fsck_config, NULL);\n>\n> +       if (fsck_promised_objects()) {\n> +               error(\"Errors found in promised object list\");\n> +               errors_found |= ERROR_PROMISED_OBJECT;\n> +       }\n\nThis got me thinking: It is an error if we do not have an object\nand also do not promise it, but what about the other case:\nhaving and object and promising it, too?\nThat is not harmful to the operation, except that the promise\nmachinery may be slower due to its size. (Should that be a soft\nwarning then? Do we have warnings in fsck?)\n\n>   * The object type is stored in 3 bits.\n>   */\n\nWe may want to remove this comment while we're here as it\nsounds stale despite being technically correct.\n1974632c66 (Remove TYPE_* constant macros and use\nobject_type enums consistently., 2006-07-11)\n\n>  struct object {\n> +       /*\n> +        * Set if this object is parsed. If set, \"type\" is populated and this\n> +        * object can be casted to \"struct commit\" or an equivalent.\n> +        */\n>         unsigned parsed : 1;\n> +       /*\n> +        * Set if this object is not in the repo but is promised. If set,\n> +        * \"type\" is populated, but this object cannot be casted to \"struct\n> +        * commit\" or an equivalent.\n> +        */\n> +       unsigned promised : 1;\n\nWould it make sense to have a bit field instead:\n\n#define STATE_BITS 2\n#define STATE_PARSED (1<<0)\n#define STATE_PROMISED (1<<1)\n\n    unsigned state : STATE_BITS\n\nThis would be similar to the types and flags?\n\n\n> +test_expect_success 'fsck fails on missing objects' '\n> +       test_create_repo repo &&\n> +\n> +       test_commit -C repo 1 &&\n> +       test_commit -C repo 2 &&\n> +       test_commit -C repo 3 &&\n> +       git -C repo tag -a annotated_tag -m \"annotated tag\" &&\n> +       C=$(git -C repo rev-parse 1) &&\n> +       T=$(git -C repo rev-parse 2^{tree}) &&\n> +       B=$(git hash-object repo/3.t) &&\n> +       AT=$(git -C repo rev-parse annotated_tag) &&\n> +\n> +       # missing commit, tree, blob, and tag\n> +       rm repo/.git/objects/$(echo $C | cut -c1-2)/$(echo $C | cut -c3-40) &&\n> +       rm repo/.git/objects/$(echo $T | cut -c1-2)/$(echo $T | cut -c3-40) &&\n> +       rm repo/.git/objects/$(echo $B | cut -c1-2)/$(echo $B | cut -c3-40) &&\n> +       rm repo/.git/objects/$(echo $AT | cut -c1-2)/$(echo $AT | cut -c3-40) &&\n\nThis is a pretty cool test as it promises all sorts of objects\nfrom different parts of the graph.\n"},{"id":"324819","messageId":"CAGZ79ka6vcF4Douc7EizwL_+_xaorro=gVw=1hfJv56kvN+7oQ@mail.gmail.com","threadId":"46359","inReplyTo":"a7fa848672281e29bff01f8caf2abe9f6ce5f4ba.1500508695.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH v2 4/4] sha1_file: support promised object hook","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-07-20T18:23:02Z","receivedAt":"2017-07-20T18:23:11Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Jul 19, 2017 at 5:21 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n> Teach sha1_file to invoke a hook whenever an object is requested and\n> unavailable but is promised. The hook is a shell command that can be\n> configured through \"git config\"; this hook takes in a list of hashes and\n> writes (if successful) the corresponding objects to the repo's local\n> storage.\n>\n> The usage of the hook can be suppressed through a flag when invoking\n> has_object_file_with_flags() and other similar functions.\n> parse_or_promise_object() in object.c requires this functionality, and\n> has been modified to use it.\n>\n> This is meant as a temporary measure to ensure that all Git commands\n> work in such a situation. Future patches will update some commands to\n> either tolerate promised objects (without invoking the hook) or be more\n> efficient in invoking the promised objects hook.\n>\n> In order to determine the code changes in sha1_file.c necessary, I\n> investigated the following:\n>  (1) functions in sha1_file that take in a hash, without the user\n>      regarding how the object is stored (loose or packed)\n>  (2) functions in sha1_file that operate on packed objects (because I\n>      need to check callers that know about the loose/packed distinction\n>      and operate on both differently, and ensure that they can handle\n>      the concept of objects that are neither loose nor packed)\n>\n> (1) is handled by the modification to sha1_object_info_extended().\n>\n> For (2), I looked at for_each_packed_object and at the packed-related\n> functions that take in a hash. For for_each_packed_object, the callers\n> either already work or are fixed in this patch:\n>  - reachable - only to find recent objects\n>  - builtin/fsck - already knows about promised objects\n>  - builtin/cat-file - fixed in this commit\n>\n> Callers of the other functions do not need to be changed:\n>  - parse_pack_index\n>    - http - indirectly from http_get_info_packs\n>  - find_pack_entry_one\n>    - this searches a single pack that is provided as an argument; the\n>      caller already knows (through other means) that the sought object\n>      is in a specific pack\n>  - find_sha1_pack\n>    - fast-import - appears to be an optimization to not store a\n>      file if it is already in a pack\n>    - http-walker - to search through a struct alt_base\n>    - http-push - to search through remote packs\n>  - has_sha1_pack\n>    - builtin/fsck - already knows about promised objects\n>    - builtin/count-objects - informational purposes only (check if loose\n>      object is also packed)\n>    - builtin/prune-packed - check if object to be pruned is packed (if\n>      not, don't prune it)\n>    - revision - used to exclude packed objects if requested by user\n>    - diff - just for optimization\n>\n> An alternative design that I considered but rejected:\n>\n>  - Adding a hook whenever a packed object is requested, not on any\n>    object.  That is, whenever we attempt to search the packfiles for an\n>    object, if it is missing (from the packfiles and from the loose\n>    object storage), to invoke the hook (which must then store it as a\n>    packfile), open the packfile the hook generated, and report that the\n>    object is found in that new packfile. This reduces the amount of\n>    analysis needed (in that we only need to look at how packed objects\n>    are handled), but requires that the hook generate packfiles (or for\n>    sha1_file to pack whatever loose objects are generated), creating one\n>    packfile for each missing object and potentially very many packfiles\n>    that must be linearly searched. This may be tolerable now for repos\n>    that only have a few missing objects (for example, repos that only\n>    want to exclude large blobs), and might be tolerable in the future if\n>    we have batching support for the most commonly used commands, but is\n>    not tolerable now for repos that exclude a large amount of objects.\n>\n> Helped-by: Ben Peart <benpeart@microsoft.com>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  Documentation/config.txt                         |   8 +\n>  Documentation/gitrepository-layout.txt           |   8 +\n>  Documentation/technical/read-object-protocol.txt | 102 ++++++++++++\n>  builtin/cat-file.c                               |   9 ++\n>  cache.h                                          |   2 +\n>  object.c                                         |   3 +-\n>  promised-object.c                                | 194 +++++++++++++++++++++++\n>  promised-object.h                                |  12 ++\n>  sha1_file.c                                      |  44 +++--\n>  t/t3907-promised-object.sh                       |  32 ++++\n>  t/t3907/read-object                              | 114 +++++++++++++\n>  11 files changed, 513 insertions(+), 15 deletions(-)\n>  create mode 100644 Documentation/technical/read-object-protocol.txt\n>  create mode 100755 t/t3907/read-object\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index d5c9c4cab..c293ac921 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -393,6 +393,14 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n>  will probe and set core.ignoreCase true if appropriate when the repository\n>  is created.\n>\n> +core.promisedBlobCommand::\n> +       If set, whenever a blob in the local repo is attempted to be read, but\n> +       is both missing and a promised blob, invoke this shell command to\n> +       generate or obtain that blob before reporting an error. This shell\n> +       command should take one or more hashes, each terminated by a newline,\n> +       as standard input, and (if successful) should write the corresponding\n> +       objects to the local repo (packed or loose).\n> +\n>  core.precomposeUnicode::\n>         This option is only used by Mac OS implementation of Git.\n>         When core.precomposeUnicode=true, Git reverts the unicode decomposition\n> diff --git a/Documentation/gitrepository-layout.txt b/Documentation/gitrepository-layout.txt\n> index f51ed4e37..7dea7fe6b 100644\n> --- a/Documentation/gitrepository-layout.txt\n> +++ b/Documentation/gitrepository-layout.txt\n> @@ -47,6 +47,10 @@ use with dumb transports but otherwise is OK as long as\n>  `objects/info/alternates` points at the object stores it\n>  borrows from.\n>  +\n> +. You could have objects that are merely promised by another source.\n> +When Git requires those objects, it will invoke the command in the\n> +`extensions.promisedObjects` configuration variable.\n> ++\n>  This directory is ignored if $GIT_COMMON_DIR is set and\n>  \"$GIT_COMMON_DIR/objects\" will be used instead.\n>\n> @@ -91,6 +95,10 @@ objects/info/http-alternates::\n>         this object store borrows objects from, to be used when\n>         the repository is fetched over HTTP.\n>\n> +objects/promised::\n> +       This file records the sha1 object names, types, and sizes of\n> +       promised objects.\n> +\n>  refs::\n>         References are stored in subdirectories of this\n>         directory.  The 'git prune' command knows to preserve\n> diff --git a/Documentation/technical/read-object-protocol.txt b/Documentation/technical/read-object-protocol.txt\n> new file mode 100644\n> index 000000000..a893b46e7\n> --- /dev/null\n> +++ b/Documentation/technical/read-object-protocol.txt\n> @@ -0,0 +1,102 @@\n> +Read Object Process\n> +^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\nThis protocol reads very similar to the protocol that is used in\nclean/smudge filtering, designed by Lars (cc'd)\n\n> +\n> +The read-object process enables Git to read all missing blobs with a\n> +single process invocation for the entire life of a single Git command.\n> +This is achieved by using a packet format (pkt-line, see technical/\n> +protocol-common.txt) based protocol over standard input and standard\n> +output as follows. All packets, except for the \"*CONTENT\" packets and\n> +the \"0000\" flush packet, are considered text and therefore are\n> +terminated by a LF.\n> +\n> +Git starts the process when it encounters the first missing object that\n> +needs to be retrieved. After the process is started, Git sends a welcome\n> +message (\"git-read-object-client\"), a list of supported protocol version\n> +numbers, and a flush packet. Git expects to read a welcome response\n> +message (\"git-read-object-server\"), exactly one protocol version number\n> +from the previously sent list, and a flush packet. All further\n> +communication will be based on the selected version.\n> +\n> +The remaining protocol description below documents \"version=1\". Please\n> +note that \"version=42\" in the example below does not exist and is only\n> +there to illustrate how the protocol would look with more than one\n> +version.\n> +\n> +After the version negotiation Git sends a list of all capabilities that\n> +it supports and a flush packet. Git expects to read a list of desired\n> +capabilities, which must be a subset of the supported capabilities list,\n> +and a flush packet as response:\n> +------------------------\n> +packet: git> git-read-object-client\n> +packet: git> version=1\n> +packet: git> version=42\n> +packet: git> 0000\n> +packet: git< git-read-object-server\n> +packet: git< version=1\n> +packet: git< 0000\n> +packet: git> capability=get\n> +packet: git> capability=have\n> +packet: git> capability=put\n> +packet: git> capability=not-yet-invented\n> +packet: git> 0000\n> +packet: git< capability=get\n> +packet: git< 0000\n> +------------------------\n> +The only supported capability in version 1 is \"get\".\n> +\n> +Afterwards Git sends a list of \"key=value\" pairs terminated with a flush\n> +packet. The list will contain at least the command (based on the\n> +supported capabilities) and the sha1 of the object to retrieve. Please\n> +note, that the process must not send any response before it received the\n> +final flush packet.\n> +\n> +When the process receives the \"get\" command, it should make the requested\n> +object available in the git object store and then return success. Git will\n> +then check the object store again and this time find it and proceed.\n> +------------------------\n> +packet: git> command=get\n> +packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n> +packet: git> 0000\n> +------------------------\n> +\n> +The process is expected to respond with a list of \"key=value\" pairs\n> +terminated with a flush packet. If the process does not experience\n> +problems then the list must contain a \"success\" status.\n> +------------------------\n> +packet: git< status=success\n> +packet: git< 0000\n> +------------------------\n> +\n> +In case the process cannot or does not want to process the content, it\n> +is expected to respond with an \"error\" status.\n> +------------------------\n> +packet: git< status=error\n> +packet: git< 0000\n> +------------------------\n> +\n> +In case the process cannot or does not want to process the content as\n> +well as any future content for the lifetime of the Git process, then it\n> +is expected to respond with an \"abort\" status at any point in the\n> +protocol.\n> +------------------------\n> +packet: git< status=abort\n> +packet: git< 0000\n> +------------------------\n> +\n> +Git neither stops nor restarts the process in case the \"error\"/\"abort\"\n> +status is set.\n> +\n> +If the process dies during the communication or does not adhere to the\n> +protocol then Git will stop the process and restart it with the next\n> +object that needs to be processed.\n> +\n> +After the read-object process has processed an object it is expected to\n> +wait for the next \"key=value\" list containing a command. Git will close\n> +the command pipe on exit. The process is expected to detect EOF and exit\n> +gracefully on its own. Git will wait until the process has stopped.\n> +\n> +A long running read-object process demo implementation can be found in\n> +`contrib/long-running-read-object/example.pl` located in the Git core\n> +repository. If you develop your own long running process then the\n> +`GIT_TRACE_PACKET` environment variables can be very helpful for\n> +debugging (see linkgit:git[1]).\n> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n> index 96b786e48..33f636926 100644\n> --- a/builtin/cat-file.c\n> +++ b/builtin/cat-file.c\n> @@ -12,6 +12,7 @@\n>  #include \"streaming.h\"\n>  #include \"tree-walk.h\"\n>  #include \"sha1-array.h\"\n> +#include \"promised-object.h\"\n>\n>  struct batch_options {\n>         int enabled;\n> @@ -432,6 +433,13 @@ static int batch_packed_object(const struct object_id *oid,\n>         return 0;\n>  }\n>\n> +static int batch_promised_object(const struct object_id *oid,\n> +                                void *data)\n> +{\n> +       oid_array_append(data, oid);\n> +       return 0;\n> +}\n> +\n>  static int batch_objects(struct batch_options *opt)\n>  {\n>         struct strbuf buf = STRBUF_INIT;\n> @@ -473,6 +481,7 @@ static int batch_objects(struct batch_options *opt)\n>\n>                 for_each_loose_object(batch_loose_object, &sa, 0);\n>                 for_each_packed_object(batch_packed_object, &sa, 0);\n> +               for_each_promised_object(batch_promised_object, &sa);\n>\n>                 cb.opt = opt;\n>                 cb.expand = &data;\n> diff --git a/cache.h b/cache.h\n> index dd94b5ffc..75b71f38b 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -1835,6 +1835,8 @@ struct object_info {\n>  #define OBJECT_INFO_SKIP_CACHED 4\n>  /* Do not retry packed storage after checking packed and loose storage */\n>  #define OBJECT_INFO_QUICK 8\n> +/* Ignore list of promised objects */\n> +#define OBJECT_INFO_IGNORE_PROMISES 16\n>  extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n>  extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n>\n> diff --git a/object.c b/object.c\n> index 0aeb95084..23f2a6cbc 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -285,7 +285,8 @@ struct object *parse_or_promise_object(const struct object_id *oid)\n>  {\n>         enum object_type type;\n>\n> -       if (has_object_file(oid))\n> +       if (has_object_file_with_flags(oid, OBJECT_INFO_SKIP_CACHED |\n> +                                           OBJECT_INFO_IGNORE_PROMISES))\n>                 return parse_object(oid);\n>\n>         if (is_promised_object(oid, &type, NULL)) {\n> diff --git a/promised-object.c b/promised-object.c\n> index 487ade437..d8d95ebb2 100644\n> --- a/promised-object.c\n> +++ b/promised-object.c\n> @@ -2,6 +2,12 @@\n>  #include \"promised-object.h\"\n>  #include \"sha1-lookup.h\"\n>  #include \"strbuf.h\"\n> +#include \"run-command.h\"\n> +#include \"sha1-array.h\"\n> +#include \"config.h\"\n> +#include \"sigchain.h\"\n> +#include \"sub-process.h\"\n> +#include \"pkt-line.h\"\n>\n>  #define ENTRY_SIZE (GIT_SHA1_RAWSZ + 1 + 8)\n>  /*\n> @@ -128,3 +134,191 @@ int fsck_promised_objects(void)\n>         }\n>         return 0;\n>  }\n> +\n> +#define CAP_GET    (1u<<0)\n> +\n> +static int subprocess_map_initialized;\n> +static struct hashmap subprocess_map;\n> +\n> +struct read_object_process {\n> +       struct subprocess_entry subprocess;\n> +       unsigned int supported_capabilities;\n> +};\n> +\n> +int start_read_object_fn(struct subprocess_entry *subprocess)\n> +{\n> +       int err;\n> +       struct read_object_process *entry = (struct read_object_process *)subprocess;\n> +       struct child_process *process;\n> +       struct string_list cap_list = STRING_LIST_INIT_NODUP;\n> +       char *cap_buf;\n> +       const char *cap_name;\n> +\n> +       process = subprocess_get_child_process(&entry->subprocess);\n> +\n> +       sigchain_push(SIGPIPE, SIG_IGN);\n> +\n> +       err = packet_writel(process->in, \"git-read-object-client\", \"version=1\", NULL);\n> +       if (err)\n> +               goto done;\n> +\n> +       err = strcmp(packet_read_line(process->out, NULL), \"git-read-object-server\");\n> +       if (err) {\n> +               error(\"external process '%s' does not support read-object protocol version 1\", subprocess->cmd);\n> +               goto done;\n> +       }\n> +       err = strcmp(packet_read_line(process->out, NULL), \"version=1\");\n> +       if (err)\n> +               goto done;\n> +       err = packet_read_line(process->out, NULL) != NULL;\n> +       if (err)\n> +               goto done;\n> +\n> +       err = packet_writel(process->in, \"capability=get\", NULL);\n> +       if (err)\n> +               goto done;\n> +\n> +       for (;;) {\n> +               cap_buf = packet_read_line(process->out, NULL);\n> +               if (!cap_buf)\n> +                       break;\n> +               string_list_split_in_place(&cap_list, cap_buf, '=', 1);\n> +\n> +               if (cap_list.nr != 2 || strcmp(cap_list.items[0].string, \"capability\"))\n> +                       continue;\n> +\n> +               cap_name = cap_list.items[1].string;\n> +               if (!strcmp(cap_name, \"get\")) {\n> +                       entry->supported_capabilities |= CAP_GET;\n> +               }\n> +               else {\n> +                       warning(\n> +                               \"external process '%s' requested unsupported read-object capability '%s'\",\n> +                               subprocess->cmd, cap_name\n> +                       );\n> +               }\n> +\n> +               string_list_clear(&cap_list, 0);\n> +       }\n> +\n> +done:\n> +       sigchain_pop(SIGPIPE);\n> +\n> +       if (err || errno == EPIPE)\n> +               return err ? err : errno;\n> +\n> +       return 0;\n> +}\n> +\n> +static int read_object_process(const unsigned char *sha1)\n> +{\n> +       int err;\n> +       struct read_object_process *entry;\n> +       struct child_process *process;\n> +       struct strbuf status = STRBUF_INIT;\n> +       uint64_t start;\n> +\n> +       start = getnanotime();\n> +\n> +       if (!repository_format_promised_objects)\n> +               die(\"BUG: if extensions.promisedObjects is not set, there \"\n> +                   \"should not be any promised objects\");\n> +\n> +       if (!subprocess_map_initialized) {\n> +               subprocess_map_initialized = 1;\n> +               hashmap_init(&subprocess_map, (hashmap_cmp_fn)cmd2process_cmp, 0);\n> +               entry = NULL;\n> +       } else {\n> +               entry = (struct read_object_process *)subprocess_find_entry(&subprocess_map, repository_format_promised_objects);\n> +       }\n> +       if (!entry) {\n> +               entry = xmalloc(sizeof(*entry));\n> +               entry->supported_capabilities = 0;\n> +\n> +               if (subprocess_start(&subprocess_map, &entry->subprocess, repository_format_promised_objects, start_read_object_fn)) {\n> +                       free(entry);\n> +                       return -1;\n> +               }\n> +       }\n> +       process = subprocess_get_child_process(&entry->subprocess);\n> +\n> +       if (!(CAP_GET & entry->supported_capabilities))\n> +               return -1;\n> +\n> +       sigchain_push(SIGPIPE, SIG_IGN);\n> +\n> +       err = packet_write_fmt_gently(process->in, \"command=get\\n\");\n> +       if (err)\n> +               goto done;\n> +\n> +       err = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n> +       if (err)\n> +               goto done;\n> +\n> +       err = packet_flush_gently(process->in);\n> +       if (err)\n> +               goto done;\n> +\n> +       err = subprocess_read_status(process->out, &status);\n> +       err = err ? err : strcmp(status.buf, \"success\");\n> +\n> +done:\n> +       sigchain_pop(SIGPIPE);\n> +\n> +       if (err || errno == EPIPE) {\n> +               err = err ? err : errno;\n> +               if (!strcmp(status.buf, \"error\")) {\n> +                       /* The process signaled a problem with the file. */\n> +               }\n> +               else if (!strcmp(status.buf, \"abort\")) {\n> +                       /*\n> +                        * The process signaled a permanent problem. Don't try to read\n> +                        * objects with the same command for the lifetime of the current\n> +                        * Git process.\n> +                        */\n> +                       entry->supported_capabilities &= ~CAP_GET;\n> +               }\n> +               else {\n> +                       /*\n> +                        * Something went wrong with the read-object process.\n> +                        * Force shutdown and restart if needed.\n> +                        */\n> +                       error(\"external process '%s' failed\", repository_format_promised_objects);\n> +                       subprocess_stop(&subprocess_map, (struct subprocess_entry *)entry);\n> +                       free(entry);\n> +               }\n> +       }\n> +\n> +       trace_performance_since(start, \"read_object_process\");\n> +\n> +       return err;\n> +}\n> +\n> +int request_promised_objects(const struct oid_array *oids)\n> +{\n> +       int oids_requested = 0;\n> +       int i;\n> +\n> +       for (i = 0; i < oids->nr; i++) {\n> +               if (is_promised_object(&oids->oid[i], NULL, NULL))\n> +                       break;\n> +       }\n> +\n> +       if (i == oids->nr)\n> +               /* Nothing to fetch */\n> +               return 0;\n> +\n> +       for (; i < oids->nr; i++) {\n> +               if (is_promised_object(&oids->oid[i], NULL, NULL)) {\n> +                       read_object_process(oids->oid[i].hash);\n> +                       oids_requested++;\n> +               }\n> +       }\n> +\n> +       /*\n> +        * The command above may have updated packfiles, so update our record\n> +        * of them.\n> +        */\n> +       reprepare_packed_git();\n> +       return oids_requested;\n> +}\n> diff --git a/promised-object.h b/promised-object.h\n> index 7eaedff17..8ad47aa4c 100644\n> --- a/promised-object.h\n> +++ b/promised-object.h\n> @@ -2,6 +2,7 @@\n>  #define PROMISED_OBJECT_H\n>\n>  #include \"cache.h\"\n> +#include \"sha1-array.h\"\n>\n>  /*\n>   * Returns 1 if oid is the name of a promised object. For non-blobs, 0 is\n> @@ -19,4 +20,15 @@ int for_each_promised_object(each_promised_object_fn, void *);\n>   */\n>  int fsck_promised_objects(void);\n>\n> +/*\n> + * If any of the given objects are promised objects, invokes\n> + * core.promisedobjectcommand with those objects and returns the number of\n> + * objects requested. No check is made as to whether the invocation actually\n> + * populated the repository with the promised objects.\n> + *\n> + * If none of the given objects are promised objects, this function does not\n> + * invoke anything and returns 0.\n> + */\n> +int request_promised_objects(const struct oid_array *oids);\n> +\n>  #endif\n> diff --git a/sha1_file.c b/sha1_file.c\n> index 5862386cd..ded0ef46b 100644\n> --- a/sha1_file.c\n> +++ b/sha1_file.c\n> @@ -28,6 +28,11 @@\n>  #include \"list.h\"\n>  #include \"mergesort.h\"\n>  #include \"quote.h\"\n> +#include \"iterator.h\"\n> +#include \"dir-iterator.h\"\n> +#include \"sha1-lookup.h\"\n> +#include \"promised-object.h\"\n> +#include \"sha1-array.h\"\n>\n>  #define SZ_FMT PRIuMAX\n>  static inline uintmax_t sz_fmt(size_t s) { return s; }\n> @@ -2983,6 +2988,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>         const unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n>                                     lookup_replace_object(sha1) :\n>                                     sha1;\n> +       int already_retried = 0;\n>\n>         if (!oi)\n>                 oi = &blank_oi;\n> @@ -3007,30 +3013,40 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>                 }\n>         }\n>\n> -       if (!find_pack_entry(real, &e)) {\n> -               /* Most likely it's a loose object. */\n> -               if (!sha1_loose_object_info(real, oi, flags)) {\n> -                       oi->whence = OI_LOOSE;\n> -                       return 0;\n> -               }\n> +retry:\n> +       if (find_pack_entry(real, &e))\n> +               goto found_packed;\n>\n> -               /* Not a loose object; someone else may have just packed it. */\n> -               if (flags & OBJECT_INFO_QUICK) {\n> -                       return -1;\n> -               } else {\n> -                       reprepare_packed_git();\n> -                       if (!find_pack_entry(real, &e))\n> -                               return -1;\n> +       /* Most likely it's a loose object. */\n> +       if (!sha1_loose_object_info(real, oi, flags)) {\n> +               oi->whence = OI_LOOSE;\n> +               return 0;\n> +       }\n> +\n> +       /* Not a loose object; someone else may have just packed it. */\n> +       reprepare_packed_git();\n> +       if (find_pack_entry(real, &e))\n> +               goto found_packed;\n> +\n> +       /* Check if it is a promised blob */\n> +       if (!already_retried && !(flags & OBJECT_INFO_IGNORE_PROMISES)) {\n> +               struct oid_array promised = OID_ARRAY_INIT;\n> +               oid_array_append_sha1(&promised, real);\n> +               if (request_promised_objects(&promised)) {\n> +                       already_retried = 1;\n> +                       goto retry;\n>                 }\n>         }\n>\n> +       return -1;\n> +\n> +found_packed:\n>         if (oi == &blank_oi)\n>                 /*\n>                  * We know that the caller doesn't actually need the\n>                  * information below, so return early.\n>                  */\n>                 return 0;\n> -\n>         rtype = packed_object_info(e.p, e.offset, oi);\n>         if (rtype < 0) {\n>                 mark_bad_packed_object(e.p, real);\n> diff --git a/t/t3907-promised-object.sh b/t/t3907-promised-object.sh\n> index 3e0caf4f9..d9e6a6486 100755\n> --- a/t/t3907-promised-object.sh\n> +++ b/t/t3907-promised-object.sh\n> @@ -38,4 +38,36 @@ test_expect_success '...but fails again with GIT_IGNORE_PROMISED_OBJECTS' '\n>         unset GIT_IGNORE_PROMISED_OBJECTS\n>  '\n>\n> +test_expect_success 'sha1_object_info_extended (through git cat-file)' '\n> +       test_create_repo server &&\n> +       test_commit -C server 1 1.t abcdefgh &&\n> +       HASH=$(git hash-object server/1.t) &&\n> +\n> +       test_create_repo client &&\n> +       test_must_fail git -C client cat-file -p \"$HASH\"\n> +'\n> +\n> +test_expect_success '...succeeds if it is a promised object' '\n> +       printf \"%s03%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n> +               hex_pack >client/.git/objects/promised &&\n> +       git -C client config core.repositoryformatversion 1 &&\n> +       git -C client config extensions.promisedobjects \\\n> +               \"\\\"$TEST_DIRECTORY/t3907/read-object\\\" \\\"$(pwd)/server/.git\\\"\" &&\n> +       git -C client cat-file -p \"$HASH\"\n> +'\n> +\n> +test_expect_success 'cat-file --batch-all-objects with promised objects' '\n> +       rm -rf client &&\n> +       test_create_repo client &&\n> +       git -C client config core.repositoryformatversion 1 &&\n> +       git -C client config extensions.promisedobjects \\\n> +               \"\\\"$TEST_DIRECTORY/t3907/read-object\\\" \\\"$(pwd)/server/.git\\\"\" &&\n> +       printf \"%s03%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n> +               hex_pack >client/.git/objects/promised &&\n> +\n> +       # Verify that the promised object is printed\n> +       git -C client cat-file --batch --batch-all-objects | tee out |\n> +               grep abcdefgh\n> +'\n> +\n>  test_done\n> diff --git a/t/t3907/read-object b/t/t3907/read-object\n> new file mode 100755\n> index 000000000..9666ad597\n> --- /dev/null\n> +++ b/t/t3907/read-object\n> @@ -0,0 +1,114 @@\n> +#!/usr/bin/perl\n> +#\n> +# Example implementation for the Git read-object protocol version 1\n> +# See Documentation/technical/read-object-protocol.txt\n> +#\n> +# Allows you to test the ability for blobs to be pulled from a host git repo\n> +# \"on demand.\"  Called when git needs a blob it couldn't find locally due to\n> +# a lazy clone that only cloned the commits and trees.\n> +#\n> +# A lazy clone can be simulated via the following commands from the host repo\n> +# you wish to create a lazy clone of:\n> +#\n> +# cd /host_repo\n> +# git rev-parse HEAD\n> +# git init /guest_repo\n> +# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n> +#      cut -d' ' -f1 | git pack-objects /guest_repo/.git/objects/pack/noblobs\n> +# cd /guest_repo\n> +# git config core.virtualizeobjects true\n> +# git reset --hard <sha from rev-parse call above>\n> +#\n> +# Please note, this sample is a minimal skeleton. No proper error handling\n> +# was implemented.\n> +#\n> +\n> +use strict;\n> +use warnings;\n> +\n> +#\n> +# Point $DIR to the folder where your host git repo is located so we can pull\n> +# missing objects from it\n> +#\n> +my $DIR = $ARGV[0];\n> +\n> +sub packet_bin_read {\n> +       my $buffer;\n> +       my $bytes_read = read STDIN, $buffer, 4;\n> +       if ( $bytes_read == 0 ) {\n> +\n> +               # EOF - Git stopped talking to us!\n> +               exit();\n> +       }\n> +       elsif ( $bytes_read != 4 ) {\n> +               die \"invalid packet: '$buffer'\";\n> +       }\n> +       my $pkt_size = hex($buffer);\n> +       if ( $pkt_size == 0 ) {\n> +               return ( 1, \"\" );\n> +       }\n> +       elsif ( $pkt_size > 4 ) {\n> +               my $content_size = $pkt_size - 4;\n> +               $bytes_read = read STDIN, $buffer, $content_size;\n> +               if ( $bytes_read != $content_size ) {\n> +                       die \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n> +               }\n> +               return ( 0, $buffer );\n> +       }\n> +       else {\n> +               die \"invalid packet size: $pkt_size\";\n> +       }\n> +}\n> +\n> +sub packet_txt_read {\n> +       my ( $res, $buf ) = packet_bin_read();\n> +       unless ( $buf =~ s/\\n$// ) {\n> +               die \"A non-binary line MUST be terminated by an LF.\";\n> +       }\n> +       return ( $res, $buf );\n> +}\n> +\n> +sub packet_bin_write {\n> +       my $buf = shift;\n> +       print STDOUT sprintf( \"%04x\", length($buf) + 4 );\n> +       print STDOUT $buf;\n> +       STDOUT->flush();\n> +}\n> +\n> +sub packet_txt_write {\n> +       packet_bin_write( $_[0] . \"\\n\" );\n> +}\n> +\n> +sub packet_flush {\n> +       print STDOUT sprintf( \"%04x\", 0 );\n> +       STDOUT->flush();\n> +}\n> +\n> +( packet_txt_read() eq ( 0, \"git-read-object-client\" ) ) || die \"bad initialize\";\n> +( packet_txt_read() eq ( 0, \"version=1\" ) )                             || die \"bad version\";\n> +( packet_bin_read() eq ( 1, \"\" ) )                       || die \"bad version end\";\n> +\n> +packet_txt_write(\"git-read-object-server\");\n> +packet_txt_write(\"version=1\");\n> +packet_flush();\n> +\n> +( packet_txt_read() eq ( 0, \"capability=get\" ) )    || die \"bad capability\";\n> +( packet_bin_read() eq ( 1, \"\" ) )                  || die \"bad capability end\";\n> +\n> +packet_txt_write(\"capability=get\");\n> +packet_flush();\n> +\n> +while (1) {\n> +       my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n> +\n> +       if ( $command eq \"get\" ) {\n> +               my ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n> +               packet_bin_read();\n> +\n> +               system ('git --git-dir=\"' . $DIR . '\" cat-file blob ' . $sha1 . ' | git -c core.virtualizeobjects=false hash-object -w --stdin >/dev/null 2>&1');\n> +               packet_txt_write(($?) ? \"status=error\" : \"status=success\");\n> +               packet_flush();\n> +       } else {\n> +               die \"bad command '$command'\";\n> +       }\n> +}\n> --\n> 2.14.0.rc0.284.gd933b75aa4-goog\n>\n"},{"id":"324825","messageId":"20170720121720.76e0c06e@twelve2.svl.corp.google.com","threadId":"46359","inReplyTo":"CAGZ79kZ45Z3R9FbsKQwotXh_Ek=StUdSTHj4er6HMhx79VbV4g@mail.gmail.com","subject":"Re: [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-20T19:17:20Z","receivedAt":"2017-07-20T19:17:31Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 20 Jul 2017 11:07:29 -0700\nStefan Beller <sbeller@google.com> wrote:\n\n> > +       if (fsck_promised_objects()) {\n> > +               error(\"Errors found in promised object list\");\n> > +               errors_found |= ERROR_PROMISED_OBJECT;\n> > +       }\n> \n> This got me thinking: It is an error if we do not have an object\n> and also do not promise it, but what about the other case:\n> having and object and promising it, too?\n> That is not harmful to the operation, except that the promise\n> machinery may be slower due to its size. (Should that be a soft\n> warning then? Do we have warnings in fsck?)\n\nGood question - having an object and also having it promised is not an\nerror condition (and I don't think it's a good idea to make it so,\nbecause objects can appear quite easily from various sources). In the\nfuture, I expect \"git gc\" to be extended to remove such redundant lines\nfrom the \"promised\" list.\n\n> >   * The object type is stored in 3 bits.\n> >   */\n> \n> We may want to remove this comment while we're here as it\n> sounds stale despite being technically correct.\n> 1974632c66 (Remove TYPE_* constant macros and use\n> object_type enums consistently., 2006-07-11)\n\nI agree that the comment is unnecessary, but in this commit I didn't\nmodify anything to do with the type, so I left it there.\n\n> >  struct object {\n> > +       /*\n> > +        * Set if this object is parsed. If set, \"type\" is populated and this\n> > +        * object can be casted to \"struct commit\" or an equivalent.\n> > +        */\n> >         unsigned parsed : 1;\n> > +       /*\n> > +        * Set if this object is not in the repo but is promised. If set,\n> > +        * \"type\" is populated, but this object cannot be casted to \"struct\n> > +        * commit\" or an equivalent.\n> > +        */\n> > +       unsigned promised : 1;\n> \n> Would it make sense to have a bit field instead:\n> \n> #define STATE_BITS 2\n> #define STATE_PARSED (1<<0)\n> #define STATE_PROMISED (1<<1)\n> \n>     unsigned state : STATE_BITS\n> \n> This would be similar to the types and flags?\n\nBoth type and flag have to be bit fields (for different reasons), but\nsince we don't need such a combined field for \"parsed\" and \"promised\", I\nprefer separating them each into their own field.\n\n> > +test_expect_success 'fsck fails on missing objects' '\n> > +       test_create_repo repo &&\n> > +\n> > +       test_commit -C repo 1 &&\n> > +       test_commit -C repo 2 &&\n> > +       test_commit -C repo 3 &&\n> > +       git -C repo tag -a annotated_tag -m \"annotated tag\" &&\n> > +       C=$(git -C repo rev-parse 1) &&\n> > +       T=$(git -C repo rev-parse 2^{tree}) &&\n> > +       B=$(git hash-object repo/3.t) &&\n> > +       AT=$(git -C repo rev-parse annotated_tag) &&\n> > +\n> > +       # missing commit, tree, blob, and tag\n> > +       rm repo/.git/objects/$(echo $C | cut -c1-2)/$(echo $C | cut -c3-40) &&\n> > +       rm repo/.git/objects/$(echo $T | cut -c1-2)/$(echo $T | cut -c3-40) &&\n> > +       rm repo/.git/objects/$(echo $B | cut -c1-2)/$(echo $B | cut -c3-40) &&\n> > +       rm repo/.git/objects/$(echo $AT | cut -c1-2)/$(echo $AT | cut -c3-40) &&\n> \n> This is a pretty cool test as it promises all sorts of objects\n> from different parts of the graph.\n\nThanks.\n"},{"id":"324829","messageId":"d8beb406-6d03-d893-4f37-83a7bdd6b9ac@gmail.com","threadId":"46359","inReplyTo":"3420d9ae9ef86b78af1abe721891233e3f5865a2.1500508695.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-20T19:58:51Z","receivedAt":"2017-07-20T19:59:05Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/19/2017 8:21 PM, Jonathan Tan wrote:\n> Currently, Git does not support repos with very large numbers of objects\n> or repos that wish to minimize manipulation of certain blobs (for\n> example, because they are very large) very well, even if the user\n> operates mostly on part of the repo, because Git is designed on the\n> assumption that every referenced object is available somewhere in the\n> repo storage.\n> \n\nGreat to see this idea making progress. Making git able to gracefully \nhandle partial clones (beyond the existing shallow clone support) is a \nkey piece of dealing with very large objects and repos.\n\n> As a first step to reducing this problem, introduce the concept of\n> promised objects. Each Git repo can contain a list of promised objects\n> and their sizes (if blobs) at $GIT_DIR/objects/promised. This patch\n> contains functions to query them; functions for creating and modifying\n> that file will be introduced in later patches.\n\nIf I'm reading this patch correctly, for a repo to successfully pass \n\"git fsck\" either the object or a promise must exist for everything fsck \nchecks.  From the documentation for fsck it says \"git fsck defaults to \nusing the index file, all SHA-1 references in refs namespace, and all \nreflogs (unless --no-reflogs is given) as heads.\" Doesn't this then \nimply objects or promises must exist for all objects referenced by any \nof these locations?\n\nWe're currently in the hundreds of millions of objects on some of our \nrepos so even downloading the promises for all the objects in the index \nis unreasonable as it is gigabytes of data and growing.\n\nI think we should have a flag (off by default) that enables someone to \nsay that promised objects are optional. If the flag is set, \n\"is_promised_object\" will return success and pass the OBJ_ANY type and a \nsize of -1.\n\nNothing today is using the size and in the two places where the object \ntype is being checked for consistency (fsck_cache_tree and \nfsck_handle_ref) the test can add a test for OBJ_ANY as well.\n\nThis will enable very large numbers of objects to be omitted from the \nclone without triggering a download of the corresponding number of \npromised objects.\n\n> \n> A repository that is missing an object but has that object promised is not\n> considered to be in error, so also teach fsck this. As part of doing\n> this, object.{h,c} has been modified to generate \"struct object\" based\n> on only the information available to promised objects, without requiring\n> the object itself.\n\nIn your work on this, did you investigate if there are other commands \n(ie repack/gc) that will need to learn about promised objects? Have you \nhad a chance (or have plans) to hack up the test suite so that it runs \nall tests with promised objects and see what (if anything) breaks?\n\n> \n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>   Documentation/technical/repository-version.txt |   6 ++\n>   Makefile                                       |   1 +\n>   builtin/fsck.c                                 |  18 +++-\n>   cache.h                                        |   2 +\n>   environment.c                                  |   1 +\n>   fsck.c                                         |   6 +-\n>   object.c                                       |  19 ++++\n>   object.h                                       |  19 ++++\n>   promised-object.c                              | 130 +++++++++++++++++++++++++\n>   promised-object.h                              |  22 +++++\n>   setup.c                                        |   7 +-\n>   t/t3907-promised-object.sh                     |  41 ++++++++\n>   t/test-lib-functions.sh                        |   6 ++\n>   13 files changed, 273 insertions(+), 5 deletions(-)\n>   create mode 100644 promised-object.c\n>   create mode 100644 promised-object.h\n>   create mode 100755 t/t3907-promised-object.sh\n> \n> diff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\n> index 00ad37986..f8b82c1c7 100644\n> --- a/Documentation/technical/repository-version.txt\n> +++ b/Documentation/technical/repository-version.txt\n> @@ -86,3 +86,9 @@ for testing format-1 compatibility.\n>   When the config key `extensions.preciousObjects` is set to `true`,\n>   objects in the repository MUST NOT be deleted (e.g., by `git-prune` or\n>   `git repack -d`).\n> +\n> +`promisedObjects`\n> +~~~~~~~~~~~~~~~~~\n> +\n> +(Explain this - basically a string containing a command to be run\n> +whenever a missing object needs to be fetched.)\n> diff --git a/Makefile b/Makefile\n> index 9c9c42f8f..c1446d5ef 100644\n> --- a/Makefile\n> +++ b/Makefile\n> @@ -828,6 +828,7 @@ LIB_OBJS += preload-index.o\n>   LIB_OBJS += pretty.o\n>   LIB_OBJS += prio-queue.o\n>   LIB_OBJS += progress.o\n> +LIB_OBJS += promised-object.o\n>   LIB_OBJS += prompt.o\n>   LIB_OBJS += quote.o\n>   LIB_OBJS += reachable.o\n> diff --git a/builtin/fsck.c b/builtin/fsck.c\n> index 462b8643b..49e21f361 100644\n> --- a/builtin/fsck.c\n> +++ b/builtin/fsck.c\n> @@ -15,6 +15,7 @@\n>   #include \"progress.h\"\n>   #include \"streaming.h\"\n>   #include \"decorate.h\"\n> +#include \"promised-object.h\"\n>   \n>   #define REACHABLE 0x0001\n>   #define SEEN      0x0002\n> @@ -44,6 +45,7 @@ static int name_objects;\n>   #define ERROR_REACHABLE 02\n>   #define ERROR_PACK 04\n>   #define ERROR_REFS 010\n> +#define ERROR_PROMISED_OBJECT 011\n>   \n>   static const char *describe_object(struct object *obj)\n>   {\n> @@ -436,7 +438,7 @@ static int fsck_handle_ref(const char *refname, const struct object_id *oid,\n>   {\n>   \tstruct object *obj;\n>   \n> -\tobj = parse_object(oid);\n> +\tobj = parse_or_promise_object(oid);\n>   \tif (!obj) {\n>   \t\terror(\"%s: invalid sha1 pointer %s\", refname, oid_to_hex(oid));\n>   \t\terrors_found |= ERROR_REACHABLE;\n> @@ -592,7 +594,7 @@ static int fsck_cache_tree(struct cache_tree *it)\n>   \t\tfprintf(stderr, \"Checking cache tree\\n\");\n>   \n>   \tif (0 <= it->entry_count) {\n> -\t\tstruct object *obj = parse_object(&it->oid);\n> +\t\tstruct object *obj = parse_or_promise_object(&it->oid);\n>   \t\tif (!obj) {\n>   \t\t\terror(\"%s: invalid sha1 pointer in cache-tree\",\n>   \t\t\t      oid_to_hex(&it->oid));\n> @@ -635,6 +637,12 @@ static int mark_packed_for_connectivity(const struct object_id *oid,\n>   \treturn 0;\n>   }\n>   \n> +static int mark_have_promised_object(const struct object_id *oid, void *data)\n> +{\n> +\tmark_object_for_connectivity(oid);\n> +\treturn 0;\n> +}\n> +\n>   static char const * const fsck_usage[] = {\n>   \tN_(\"git fsck [<options>] [<object>...]\"),\n>   \tNULL\n> @@ -690,6 +698,11 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n>   \n>   \tgit_config(fsck_config, NULL);\n>   \n> +\tif (fsck_promised_objects()) {\n> +\t\terror(\"Errors found in promised object list\");\n> +\t\terrors_found |= ERROR_PROMISED_OBJECT;\n> +\t}\n> +\n>   \tfsck_head_link();\n>   \tif (connectivity_only) {\n>   \t\tfor_each_loose_object(mark_loose_for_connectivity, NULL, 0);\n> @@ -727,6 +740,7 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n>   \t\t\tstop_progress(&progress);\n>   \t\t}\n>   \t}\n> +\tfor_each_promised_object(mark_have_promised_object, NULL);\n>   \n>   \theads = 0;\n>   \tfor (i = 0; i < argc; i++) {\n> diff --git a/cache.h b/cache.h\n> index 71fe09264..dd94b5ffc 100644\n> --- a/cache.h\n> +++ b/cache.h\n> @@ -853,10 +853,12 @@ extern int grafts_replace_parents;\n>   #define GIT_REPO_VERSION 0\n>   #define GIT_REPO_VERSION_READ 1\n>   extern int repository_format_precious_objects;\n> +extern char *repository_format_promised_objects;\n>   \n>   struct repository_format {\n>   \tint version;\n>   \tint precious_objects;\n> +\tchar *promised_objects;\n>   \tint is_bare;\n>   \tchar *work_tree;\n>   \tstruct string_list unknown_extensions;\n> diff --git a/environment.c b/environment.c\n> index 3fd4b1084..82658470d 100644\n> --- a/environment.c\n> +++ b/environment.c\n> @@ -27,6 +27,7 @@ int warn_ambiguous_refs = 1;\n>   int warn_on_object_refname_ambiguity = 1;\n>   int ref_paranoia = -1;\n>   int repository_format_precious_objects;\n> +char *repository_format_promised_objects;\n>   const char *git_commit_encoding;\n>   const char *git_log_output_encoding;\n>   const char *apply_default_whitespace;\n> diff --git a/fsck.c b/fsck.c\n> index b4204d772..f08ff0675 100644\n> --- a/fsck.c\n> +++ b/fsck.c\n> @@ -460,8 +460,10 @@ int fsck_walk(struct object *obj, void *data, struct fsck_options *options)\n>   \tif (!obj)\n>   \t\treturn -1;\n>   \n> -\tif (obj->type == OBJ_NONE)\n> -\t\tparse_object(&obj->oid);\n> +\tif (!obj->parsed && !obj->promised)\n> +\t\tparse_or_promise_object(&obj->oid);\n> +\tif (obj->promised)\n> +\t\treturn 0;\n>   \n>   \tswitch (obj->type) {\n>   \tcase OBJ_BLOB:\n> diff --git a/object.c b/object.c\n> index 321d7e920..0aeb95084 100644\n> --- a/object.c\n> +++ b/object.c\n> @@ -4,6 +4,7 @@\n>   #include \"tree.h\"\n>   #include \"commit.h\"\n>   #include \"tag.h\"\n> +#include \"promised-object.h\"\n>   \n>   static struct object **obj_hash;\n>   static int nr_objs, obj_hash_size;\n> @@ -141,6 +142,7 @@ void *create_object(const unsigned char *sha1, void *o)\n>   \tstruct object *obj = o;\n>   \n>   \tobj->parsed = 0;\n> +\tobj->promised = 0;\n>   \tobj->flags = 0;\n>   \thashcpy(obj->oid.hash, sha1);\n>   \n> @@ -279,6 +281,23 @@ struct object *parse_object(const struct object_id *oid)\n>   \treturn NULL;\n>   }\n>   \n> +struct object *parse_or_promise_object(const struct object_id *oid)\n> +{\n> +\tenum object_type type;\n> +\n> +\tif (has_object_file(oid))\n> +\t\treturn parse_object(oid);\n> +\n> +\tif (is_promised_object(oid, &type, NULL)) {\n> +\t\tstruct object *obj = lookup_unknown_object(oid->hash);\n> +\t\tobj->promised = 1;\n> +\t\tobj->type = type;\n> +\t\treturn obj;\n> +\t}\n> +\n> +\treturn NULL;\n> +}\n> +\n>   struct object_list *object_list_insert(struct object *item,\n>   \t\t\t\t       struct object_list **list_p)\n>   {\n> diff --git a/object.h b/object.h\n> index 0a419ba8d..640c8bff1 100644\n> --- a/object.h\n> +++ b/object.h\n> @@ -46,7 +46,17 @@ struct object_array {\n>    * The object type is stored in 3 bits.\n>    */\n>   struct object {\n> +\t/*\n> +\t * Set if this object is parsed. If set, \"type\" is populated and this\n> +\t * object can be casted to \"struct commit\" or an equivalent.\n> +\t */\n>   \tunsigned parsed : 1;\n> +\t/*\n> +\t * Set if this object is not in the repo but is promised. If set,\n> +\t * \"type\" is populated, but this object cannot be casted to \"struct\n> +\t * commit\" or an equivalent.\n> +\t */\n> +\tunsigned promised : 1;\n>   \tunsigned type : TYPE_BITS;\n>   \tunsigned flags : FLAG_BITS;\n>   \tstruct object_id oid;\n> @@ -104,6 +114,15 @@ struct object *parse_object_or_die(const struct object_id *oid, const char *name\n>    */\n>   struct object *parse_object_buffer(const struct object_id *oid, enum object_type type, unsigned long size, void *buffer, int *eaten_p);\n>   \n> +/*\n> + * Returns the object, having parsed it if possible. In the returned object,\n> + * either \"parsed\" or \"promised\" will be set.\n> + *\n> + * Returns NULL if the object is missing and not promised, or if the object is\n> + * corrupt.\n> + */\n> +struct object *parse_or_promise_object(const struct object_id *oid);\n> +\n>   /** Returns the object, with potentially excess memory allocated. **/\n>   struct object *lookup_unknown_object(const unsigned  char *sha1);\n>   \n> diff --git a/promised-object.c b/promised-object.c\n> new file mode 100644\n> index 000000000..487ade437\n> --- /dev/null\n> +++ b/promised-object.c\n> @@ -0,0 +1,130 @@\n> +#include \"cache.h\"\n> +#include \"promised-object.h\"\n> +#include \"sha1-lookup.h\"\n> +#include \"strbuf.h\"\n> +\n> +#define ENTRY_SIZE (GIT_SHA1_RAWSZ + 1 + 8)\n> +/*\n> + * A mmap-ed byte array of size (promised_object_nr * ENTRY_SIZE). Each\n> + * ENTRY_SIZE-sized entry consists of the SHA-1 of the promised object, its\n> + * 8-bit object type, and its 64-bit size in network byte order. The entries\n> + * are sorted in ascending SHA-1 order.\n> + */\n> +static char *promised_objects;\n> +static int64_t promised_object_nr = -1;\n> +\n> +static void prepare_promised_objects(void)\n> +{\n> +\tchar *filename;\n> +\tint fd;\n> +\tstruct stat st;\n> +\n> +\tif (promised_object_nr >= 0)\n> +\t\treturn;\n> +\n> +\tif (!repository_format_promised_objects ||\n> +\t    getenv(\"GIT_IGNORE_PROMISED_OBJECTS\")) {\n> +\t\tpromised_object_nr = 0;\n> +\t\treturn;\n> +\t}\n> +\n> +\tfilename = xstrfmt(\"%s/promised\", get_object_directory());\n> +\tfd = git_open(filename);\n> +\tif (fd < 0) {\n> +\t\tif (errno == ENOENT) {\n> +\t\t\tpromised_object_nr = 0;\n> +\t\t\tgoto cleanup;\n> +\t\t}\n> +\t\tperror(\"prepare_promised_objects\");\n> +\t\tdie(\"Could not open %s\", filename);\n> +\t}\n> +\tif (fstat(fd, &st)) {\n> +\t\tperror(\"prepare_promised_objects\");\n> +\t\tdie(\"Could not stat %s\", filename);\n> +\t}\n> +\tif (st.st_size == 0) {\n> +\t\tpromised_object_nr = 0;\n> +\t\tgoto cleanup;\n> +\t}\n> +\tif (st.st_size % ENTRY_SIZE) {\n> +\t\tdie(\"Size of %s is not a multiple of %d\", filename, ENTRY_SIZE);\n> +\t}\n> +\n> +\tpromised_objects = xmmap(NULL, st.st_size, PROT_READ, MAP_PRIVATE, fd, 0);\n> +\tpromised_object_nr = st.st_size / ENTRY_SIZE;\n> +\n> +cleanup:\n> +\tfree(filename);\n> +\tif (fd >= 0)\n> +\t\tclose(fd);\n> +}\n> +\n> +int is_promised_object(const struct object_id *oid, enum object_type *type,\n> +\t\t       unsigned long *size)\n> +{\n> +\tint result;\n> +\n> +\tprepare_promised_objects();\n> +\tresult = sha1_entry_pos(promised_objects, ENTRY_SIZE, 0, 0,\n> +\t\t\t\tpromised_object_nr, promised_object_nr,\n> +\t\t\t\toid->hash);\n> +\tif (result >= 0) {\n> +\t\tif (type) {\n> +\t\t\tchar *ptr = promised_objects +\n> +\t\t\t\t    result * ENTRY_SIZE + GIT_SHA1_RAWSZ;\n> +\t\t\t*type = *ptr;\n> +\t\t}\n> +\t\tif (size) {\n> +\t\t\tuint64_t size_nbo;\n> +\t\t\tchar *ptr = promised_objects +\n> +\t\t\t\t    result * ENTRY_SIZE + GIT_SHA1_RAWSZ + 1;\n> +\t\t\tmemcpy(&size_nbo, ptr, sizeof(size_nbo));\n> +\t\t\t*size = ntohll(size_nbo);\n> +\t\t}\n> +\t\treturn 1;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +int for_each_promised_object(each_promised_object_fn cb, void *data)\n> +{\n> +\tstruct object_id oid;\n> +\tint i, r;\n> +\n> +\tprepare_promised_objects();\n> +\tfor (i = 0; i < promised_object_nr; i++) {\n> +\t\tmemcpy(oid.hash, &promised_objects[i * ENTRY_SIZE],\n> +\t\t       GIT_SHA1_RAWSZ);\n> +\t\tr = cb(&oid, data);\n> +\t\tif (r)\n> +\t\t\treturn r;\n> +\t}\n> +\treturn 0;\n> +}\n> +\n> +int fsck_promised_objects(void)\n> +{\n> +\tint i;\n> +\tprepare_promised_objects();\n> +\tfor (i = 0; i < promised_object_nr; i++) {\n> +\t\tenum object_type type;\n> +\t\tif (i != 0 && memcmp(&promised_objects[(i - 1) * ENTRY_SIZE],\n> +\t\t\t\t     &promised_objects[i * ENTRY_SIZE],\n> +\t\t\t\t     GIT_SHA1_RAWSZ) >= 0)\n> +\t\t\treturn error(\"Error in list of promised objects: not \"\n> +\t\t\t\t     \"in ascending order of object name\");\n> +\t\ttype = promised_objects[i * ENTRY_SIZE + GIT_SHA1_RAWSZ];\n> +\t\tswitch (type) {\n> +\t\t\tcase OBJ_BLOB:\n> +\t\t\tcase OBJ_TREE:\n> +\t\t\tcase OBJ_COMMIT:\n> +\t\t\tcase OBJ_TAG:\n> +\t\t\t\tbreak;\n> +\t\t\tdefault:\n> +\t\t\t\treturn error(\"Error in list of promised \"\n> +\t\t\t\t\t     \"objects: found object of type %d\",\n> +\t\t\t\t\t     type);\n> +\t\t}\n> +\t}\n> +\treturn 0;\n> +}\n> diff --git a/promised-object.h b/promised-object.h\n> new file mode 100644\n> index 000000000..7eaedff17\n> --- /dev/null\n> +++ b/promised-object.h\n> @@ -0,0 +1,22 @@\n> +#ifndef PROMISED_OBJECT_H\n> +#define PROMISED_OBJECT_H\n> +\n> +#include \"cache.h\"\n> +\n> +/*\n> + * Returns 1 if oid is the name of a promised object. For non-blobs, 0 is\n> + * reported as their size.\n> + */\n> +int is_promised_object(const struct object_id *oid, enum object_type *type,\n> +\t\t       unsigned long *size);\n> +\n> +typedef int each_promised_object_fn(const struct object_id *oid, void *data);\n> +int for_each_promised_object(each_promised_object_fn, void *);\n> +\n> +/*\n> + * Returns 0 if there is no list of promised objects or if the list of promised\n> + * objects is valid.\n> + */\n> +int fsck_promised_objects(void);\n> +\n> +#endif\n> diff --git a/setup.c b/setup.c\n> index 860507e1f..36022452f 100644\n> --- a/setup.c\n> +++ b/setup.c\n> @@ -425,7 +425,11 @@ static int check_repo_format(const char *var, const char *value, void *vdata)\n>   \t\t\t;\n>   \t\telse if (!strcmp(ext, \"preciousobjects\"))\n>   \t\t\tdata->precious_objects = git_config_bool(var, value);\n> -\t\telse\n> +\t\telse if (!strcmp(ext, \"promisedobjects\")) {\n> +\t\t\tif (!value)\n> +\t\t\t\treturn config_error_nonbool(var);\n> +\t\t\tdata->promised_objects = xstrdup(value);\n> +\t\t} else\n>   \t\t\tstring_list_append(&data->unknown_extensions, ext);\n>   \t} else if (strcmp(var, \"core.bare\") == 0) {\n>   \t\tdata->is_bare = git_config_bool(var, value);\n> @@ -468,6 +472,7 @@ static int check_repository_format_gently(const char *gitdir, int *nongit_ok)\n>   \t}\n>   \n>   \trepository_format_precious_objects = candidate.precious_objects;\n> +\trepository_format_promised_objects = candidate.promised_objects;\n>   \tstring_list_clear(&candidate.unknown_extensions, 0);\n>   \tif (!has_common) {\n>   \t\tif (candidate.is_bare != -1) {\n> diff --git a/t/t3907-promised-object.sh b/t/t3907-promised-object.sh\n> new file mode 100755\n> index 000000000..3e0caf4f9\n> --- /dev/null\n> +++ b/t/t3907-promised-object.sh\n> @@ -0,0 +1,41 @@\n> +#!/bin/sh\n> +\n> +test_description='promised objects'\n> +\n> +. ./test-lib.sh\n> +\n> +test_expect_success 'fsck fails on missing objects' '\n> +\ttest_create_repo repo &&\n> +\n> +\ttest_commit -C repo 1 &&\n> +\ttest_commit -C repo 2 &&\n> +\ttest_commit -C repo 3 &&\n> +\tgit -C repo tag -a annotated_tag -m \"annotated tag\" &&\n> +\tC=$(git -C repo rev-parse 1) &&\n> +\tT=$(git -C repo rev-parse 2^{tree}) &&\n> +\tB=$(git hash-object repo/3.t) &&\n> +\tAT=$(git -C repo rev-parse annotated_tag) &&\n> +\n> +\t# missing commit, tree, blob, and tag\n> +\trm repo/.git/objects/$(echo $C | cut -c1-2)/$(echo $C | cut -c3-40) &&\n> +\trm repo/.git/objects/$(echo $T | cut -c1-2)/$(echo $T | cut -c3-40) &&\n> +\trm repo/.git/objects/$(echo $B | cut -c1-2)/$(echo $B | cut -c3-40) &&\n> +\trm repo/.git/objects/$(echo $AT | cut -c1-2)/$(echo $AT | cut -c3-40) &&\n> +\ttest_must_fail git -C repo fsck\n> +'\n> +\n> +test_expect_success '...but succeeds if they are promised objects' '\n> +\tprintf \"%s01%016x\\n%s02%016x\\n%s03%016x\\n%s04%016x\" \\\n> +\t\t\"$C\" 0 \"$T\" 0 \"$B\" \"$(wc -c <repo/3.t)\" \"$AT\" 0 |\n> +\t\tsort | tr -d \"\\n\" | hex_pack >repo/.git/objects/promised &&\n> +\tgit -C repo config core.repositoryformatversion 1 &&\n> +\tgit -C repo config extensions.promisedobjects \"arbitrary string\" &&\n> +\tgit -C repo fsck\n> +'\n> +\n> +test_expect_success '...but fails again with GIT_IGNORE_PROMISED_OBJECTS' '\n> +\tGIT_IGNORE_PROMISED_OBJECTS=1 test_must_fail git -C repo fsck &&\n> +\tunset GIT_IGNORE_PROMISED_OBJECTS\n> +'\n> +\n> +test_done\n> diff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\n> index db622c355..1ebdd2d04 100644\n> --- a/t/test-lib-functions.sh\n> +++ b/t/test-lib-functions.sh\n> @@ -1018,3 +1018,9 @@ nongit () {\n>   \t\t\"$@\"\n>   \t)\n>   }\n> +\n> +# Converts big-endian pairs of hexadecimal digits into bytes. For example,\n> +# \"printf 61620d0a | hex_pack\" results in \"ab\\r\\n\".\n> +hex_pack () {\n> +\tperl -e '$/ = undef; $input = <>; print pack(\"H*\", $input)'\n> +}\n> \n"},{"id":"324838","messageId":"75d5c3cd-c1d7-f06a-fc7e-894cde95afa7@gmail.com","threadId":"46359","inReplyTo":"CAGZ79ka6vcF4Douc7EizwL_+_xaorro=gVw=1hfJv56kvN+7oQ@mail.gmail.com","subject":"Re: [RFC PATCH v2 4/4] sha1_file: support promised object hook","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-20T20:58:16Z","receivedAt":"2017-07-20T20:58:29Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/20/2017 2:23 PM, Stefan Beller wrote:\n> On Wed, Jul 19, 2017 at 5:21 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n>> Teach sha1_file to invoke a hook whenever an object is requested and\n>> unavailable but is promised. The hook is a shell command that can be\n>> configured through \"git config\"; this hook takes in a list of hashes and\n>> writes (if successful) the corresponding objects to the repo's local\n>> storage.\n>>\n>> The usage of the hook can be suppressed through a flag when invoking\n>> has_object_file_with_flags() and other similar functions.\n>> parse_or_promise_object() in object.c requires this functionality, and\n>> has been modified to use it.\n>>\n>> This is meant as a temporary measure to ensure that all Git commands\n>> work in such a situation. Future patches will update some commands to\n>> either tolerate promised objects (without invoking the hook) or be more\n>> efficient in invoking the promised objects hook.\n\nI agree that making git more tolerant of promised objects if possible \nand precomputing a list of promised objects required to complete a \nparticular command and downloading them with a single request are good \noptimizations to add over time.\n\n>>\n>> In order to determine the code changes in sha1_file.c necessary, I\n>> investigated the following:\n>>   (1) functions in sha1_file that take in a hash, without the user\n>>       regarding how the object is stored (loose or packed)\n>>   (2) functions in sha1_file that operate on packed objects (because I\n>>       need to check callers that know about the loose/packed distinction\n>>       and operate on both differently, and ensure that they can handle\n>>       the concept of objects that are neither loose nor packed)\n>>\n>> (1) is handled by the modification to sha1_object_info_extended().\n>>\n>> For (2), I looked at for_each_packed_object and at the packed-related\n>> functions that take in a hash. For for_each_packed_object, the callers\n>> either already work or are fixed in this patch:\n>>   - reachable - only to find recent objects\n>>   - builtin/fsck - already knows about promised objects\n>>   - builtin/cat-file - fixed in this commit\n>>\n>> Callers of the other functions do not need to be changed:\n>>   - parse_pack_index\n>>     - http - indirectly from http_get_info_packs\n>>   - find_pack_entry_one\n>>     - this searches a single pack that is provided as an argument; the\n>>       caller already knows (through other means) that the sought object\n>>       is in a specific pack\n>>   - find_sha1_pack\n>>     - fast-import - appears to be an optimization to not store a\n>>       file if it is already in a pack\n>>     - http-walker - to search through a struct alt_base\n>>     - http-push - to search through remote packs\n>>   - has_sha1_pack\n>>     - builtin/fsck - already knows about promised objects\n>>     - builtin/count-objects - informational purposes only (check if loose\n>>       object is also packed)\n>>     - builtin/prune-packed - check if object to be pruned is packed (if\n>>       not, don't prune it)\n>>     - revision - used to exclude packed objects if requested by user\n>>     - diff - just for optimization\n>>\n\nhas_sha1_file also takes a hash \"whether local or in an alternate object \ndatabase, and whether packed or loose\" but never calls \nsha1_object_info_extended.  As a result, we had to add support in \ncheck_and_freshen to download missing objects to get proper behavior in \nall cases.  I don't think this will work correctly without it.\n\n>> An alternative design that I considered but rejected:\n>>\n>>   - Adding a hook whenever a packed object is requested, not on any\n>>     object.  That is, whenever we attempt to search the packfiles for an\n>>     object, if it is missing (from the packfiles and from the loose\n>>     object storage), to invoke the hook (which must then store it as a\n>>     packfile), open the packfile the hook generated, and report that the\n>>     object is found in that new packfile. This reduces the amount of\n>>     analysis needed (in that we only need to look at how packed objects\n>>     are handled), but requires that the hook generate packfiles (or for\n>>     sha1_file to pack whatever loose objects are generated), creating one\n>>     packfile for each missing object and potentially very many packfiles\n>>     that must be linearly searched. This may be tolerable now for repos\n>>     that only have a few missing objects (for example, repos that only\n>>     want to exclude large blobs), and might be tolerable in the future if\n>>     we have batching support for the most commonly used commands, but is\n>>     not tolerable now for repos that exclude a large amount of objects.\n>>\n>> Helped-by: Ben Peart <benpeart@microsoft.com>\n>> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n>> ---\n>>   Documentation/config.txt                         |   8 +\n>>   Documentation/gitrepository-layout.txt           |   8 +\n>>   Documentation/technical/read-object-protocol.txt | 102 ++++++++++++\n>>   builtin/cat-file.c                               |   9 ++\n>>   cache.h                                          |   2 +\n>>   object.c                                         |   3 +-\n>>   promised-object.c                                | 194 +++++++++++++++++++++++\n>>   promised-object.h                                |  12 ++\n>>   sha1_file.c                                      |  44 +++--\n>>   t/t3907-promised-object.sh                       |  32 ++++\n>>   t/t3907/read-object                              | 114 +++++++++++++\n>>   11 files changed, 513 insertions(+), 15 deletions(-)\n>>   create mode 100644 Documentation/technical/read-object-protocol.txt\n>>   create mode 100755 t/t3907/read-object\n>>\n>> diff --git a/Documentation/config.txt b/Documentation/config.txt\n>> index d5c9c4cab..c293ac921 100644\n>> --- a/Documentation/config.txt\n>> +++ b/Documentation/config.txt\n>> @@ -393,6 +393,14 @@ The default is false, except linkgit:git-clone[1] or linkgit:git-init[1]\n>>   will probe and set core.ignoreCase true if appropriate when the repository\n>>   is created.\n>>\n>> +core.promisedBlobCommand::\n>> +       If set, whenever a blob in the local repo is attempted to be read, but\n>> +       is both missing and a promised blob, invoke this shell command to\n>> +       generate or obtain that blob before reporting an error. This shell\n>> +       command should take one or more hashes, each terminated by a newline,\n>> +       as standard input, and (if successful) should write the corresponding\n>> +       objects to the local repo (packed or loose).\n>> +\n>>   core.precomposeUnicode::\n>>          This option is only used by Mac OS implementation of Git.\n>>          When core.precomposeUnicode=true, Git reverts the unicode decomposition\n>> diff --git a/Documentation/gitrepository-layout.txt b/Documentation/gitrepository-layout.txt\n>> index f51ed4e37..7dea7fe6b 100644\n>> --- a/Documentation/gitrepository-layout.txt\n>> +++ b/Documentation/gitrepository-layout.txt\n>> @@ -47,6 +47,10 @@ use with dumb transports but otherwise is OK as long as\n>>   `objects/info/alternates` points at the object stores it\n>>   borrows from.\n>>   +\n>> +. You could have objects that are merely promised by another source.\n>> +When Git requires those objects, it will invoke the command in the\n>> +`extensions.promisedObjects` configuration variable.\n>> ++\n>>   This directory is ignored if $GIT_COMMON_DIR is set and\n>>   \"$GIT_COMMON_DIR/objects\" will be used instead.\n>>\n>> @@ -91,6 +95,10 @@ objects/info/http-alternates::\n>>          this object store borrows objects from, to be used when\n>>          the repository is fetched over HTTP.\n>>\n>> +objects/promised::\n>> +       This file records the sha1 object names, types, and sizes of\n>> +       promised objects.\n>> +\n>>   refs::\n>>          References are stored in subdirectories of this\n>>          directory.  The 'git prune' command knows to preserve\n>> diff --git a/Documentation/technical/read-object-protocol.txt b/Documentation/technical/read-object-protocol.txt\n>> new file mode 100644\n>> index 000000000..a893b46e7\n>> --- /dev/null\n>> +++ b/Documentation/technical/read-object-protocol.txt\n>> @@ -0,0 +1,102 @@\n>> +Read Object Process\n>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^\n> \n> This protocol reads very similar to the protocol that is used in\n> clean/smudge filtering, designed by Lars (cc'd)\n> \n>> +\n>> +The read-object process enables Git to read all missing blobs with a\n>> +single process invocation for the entire life of a single Git command.\n>> +This is achieved by using a packet format (pkt-line, see technical/\n>> +protocol-common.txt) based protocol over standard input and standard\n>> +output as follows. All packets, except for the \"*CONTENT\" packets and\n>> +the \"0000\" flush packet, are considered text and therefore are\n>> +terminated by a LF.\n>> +\n>> +Git starts the process when it encounters the first missing object that\n>> +needs to be retrieved. After the process is started, Git sends a welcome\n>> +message (\"git-read-object-client\"), a list of supported protocol version\n>> +numbers, and a flush packet. Git expects to read a welcome response\n>> +message (\"git-read-object-server\"), exactly one protocol version number\n>> +from the previously sent list, and a flush packet. All further\n>> +communication will be based on the selected version.\n>> +\n>> +The remaining protocol description below documents \"version=1\". Please\n>> +note that \"version=42\" in the example below does not exist and is only\n>> +there to illustrate how the protocol would look with more than one\n>> +version.\n>> +\n>> +After the version negotiation Git sends a list of all capabilities that\n>> +it supports and a flush packet. Git expects to read a list of desired\n>> +capabilities, which must be a subset of the supported capabilities list,\n>> +and a flush packet as response:\n>> +------------------------\n>> +packet: git> git-read-object-client\n>> +packet: git> version=1\n>> +packet: git> version=42\n>> +packet: git> 0000\n>> +packet: git< git-read-object-server\n>> +packet: git< version=1\n>> +packet: git< 0000\n>> +packet: git> capability=get\n>> +packet: git> capability=have\n>> +packet: git> capability=put\n>> +packet: git> capability=not-yet-invented\n>> +packet: git> 0000\n>> +packet: git< capability=get\n>> +packet: git< 0000\n>> +------------------------\n>> +The only supported capability in version 1 is \"get\".\n>> +\n>> +Afterwards Git sends a list of \"key=value\" pairs terminated with a flush\n>> +packet. The list will contain at least the command (based on the\n>> +supported capabilities) and the sha1 of the object to retrieve. Please\n>> +note, that the process must not send any response before it received the\n>> +final flush packet.\n>> +\n>> +When the process receives the \"get\" command, it should make the requested\n>> +object available in the git object store and then return success. Git will\n>> +then check the object store again and this time find it and proceed.\n>> +------------------------\n>> +packet: git> command=get\n>> +packet: git> sha1=0a214a649e1b3d5011e14a3dc227753f2bd2be05\n>> +packet: git> 0000\n>> +------------------------\n>> +\n>> +The process is expected to respond with a list of \"key=value\" pairs\n>> +terminated with a flush packet. If the process does not experience\n>> +problems then the list must contain a \"success\" status.\n>> +------------------------\n>> +packet: git< status=success\n>> +packet: git< 0000\n>> +------------------------\n>> +\n>> +In case the process cannot or does not want to process the content, it\n>> +is expected to respond with an \"error\" status.\n>> +------------------------\n>> +packet: git< status=error\n>> +packet: git< 0000\n>> +------------------------\n>> +\n>> +In case the process cannot or does not want to process the content as\n>> +well as any future content for the lifetime of the Git process, then it\n>> +is expected to respond with an \"abort\" status at any point in the\n>> +protocol.\n>> +------------------------\n>> +packet: git< status=abort\n>> +packet: git< 0000\n>> +------------------------\n>> +\n>> +Git neither stops nor restarts the process in case the \"error\"/\"abort\"\n>> +status is set.\n>> +\n>> +If the process dies during the communication or does not adhere to the\n>> +protocol then Git will stop the process and restart it with the next\n>> +object that needs to be processed.\n>> +\n>> +After the read-object process has processed an object it is expected to\n>> +wait for the next \"key=value\" list containing a command. Git will close\n>> +the command pipe on exit. The process is expected to detect EOF and exit\n>> +gracefully on its own. Git will wait until the process has stopped.\n>> +\n>> +A long running read-object process demo implementation can be found in\n>> +`contrib/long-running-read-object/example.pl` located in the Git core\n>> +repository. If you develop your own long running process then the\n>> +`GIT_TRACE_PACKET` environment variables can be very helpful for\n>> +debugging (see linkgit:git[1]).\n>> diff --git a/builtin/cat-file.c b/builtin/cat-file.c\n>> index 96b786e48..33f636926 100644\n>> --- a/builtin/cat-file.c\n>> +++ b/builtin/cat-file.c\n>> @@ -12,6 +12,7 @@\n>>   #include \"streaming.h\"\n>>   #include \"tree-walk.h\"\n>>   #include \"sha1-array.h\"\n>> +#include \"promised-object.h\"\n>>\n>>   struct batch_options {\n>>          int enabled;\n>> @@ -432,6 +433,13 @@ static int batch_packed_object(const struct object_id *oid,\n>>          return 0;\n>>   }\n>>\n>> +static int batch_promised_object(const struct object_id *oid,\n>> +                                void *data)\n>> +{\n>> +       oid_array_append(data, oid);\n>> +       return 0;\n>> +}\n>> +\n>>   static int batch_objects(struct batch_options *opt)\n>>   {\n>>          struct strbuf buf = STRBUF_INIT;\n>> @@ -473,6 +481,7 @@ static int batch_objects(struct batch_options *opt)\n>>\n>>                  for_each_loose_object(batch_loose_object, &sa, 0);\n>>                  for_each_packed_object(batch_packed_object, &sa, 0);\n>> +               for_each_promised_object(batch_promised_object, &sa);\n>>\n>>                  cb.opt = opt;\n>>                  cb.expand = &data;\n>> diff --git a/cache.h b/cache.h\n>> index dd94b5ffc..75b71f38b 100644\n>> --- a/cache.h\n>> +++ b/cache.h\n>> @@ -1835,6 +1835,8 @@ struct object_info {\n>>   #define OBJECT_INFO_SKIP_CACHED 4\n>>   /* Do not retry packed storage after checking packed and loose storage */\n>>   #define OBJECT_INFO_QUICK 8\n>> +/* Ignore list of promised objects */\n>> +#define OBJECT_INFO_IGNORE_PROMISES 16\n>>   extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n>>   extern int packed_object_info(struct packed_git *pack, off_t offset, struct object_info *);\n>>\n>> diff --git a/object.c b/object.c\n>> index 0aeb95084..23f2a6cbc 100644\n>> --- a/object.c\n>> +++ b/object.c\n>> @@ -285,7 +285,8 @@ struct object *parse_or_promise_object(const struct object_id *oid)\n>>   {\n>>          enum object_type type;\n>>\n>> -       if (has_object_file(oid))\n>> +       if (has_object_file_with_flags(oid, OBJECT_INFO_SKIP_CACHED |\n>> +                                           OBJECT_INFO_IGNORE_PROMISES))\n>>                  return parse_object(oid);\n>>\n>>          if (is_promised_object(oid, &type, NULL)) {\n>> diff --git a/promised-object.c b/promised-object.c\n>> index 487ade437..d8d95ebb2 100644\n>> --- a/promised-object.c\n>> +++ b/promised-object.c\n>> @@ -2,6 +2,12 @@\n>>   #include \"promised-object.h\"\n>>   #include \"sha1-lookup.h\"\n>>   #include \"strbuf.h\"\n>> +#include \"run-command.h\"\n>> +#include \"sha1-array.h\"\n>> +#include \"config.h\"\n>> +#include \"sigchain.h\"\n>> +#include \"sub-process.h\"\n>> +#include \"pkt-line.h\"\n>>\n>>   #define ENTRY_SIZE (GIT_SHA1_RAWSZ + 1 + 8)\n>>   /*\n>> @@ -128,3 +134,191 @@ int fsck_promised_objects(void)\n>>          }\n>>          return 0;\n>>   }\n>> +\n>> +#define CAP_GET    (1u<<0)\n>> +\n>> +static int subprocess_map_initialized;\n>> +static struct hashmap subprocess_map;\n>> +\n>> +struct read_object_process {\n>> +       struct subprocess_entry subprocess;\n>> +       unsigned int supported_capabilities;\n>> +};\n>> +\n>> +int start_read_object_fn(struct subprocess_entry *subprocess)\n>> +{\n>> +       int err;\n>> +       struct read_object_process *entry = (struct read_object_process *)subprocess;\n>> +       struct child_process *process;\n>> +       struct string_list cap_list = STRING_LIST_INIT_NODUP;\n>> +       char *cap_buf;\n>> +       const char *cap_name;\n>> +\n>> +       process = subprocess_get_child_process(&entry->subprocess);\n>> +\n>> +       sigchain_push(SIGPIPE, SIG_IGN);\n>> +\n>> +       err = packet_writel(process->in, \"git-read-object-client\", \"version=1\", NULL);\n>> +       if (err)\n>> +               goto done;\n>> +\n>> +       err = strcmp(packet_read_line(process->out, NULL), \"git-read-object-server\");\n>> +       if (err) {\n>> +               error(\"external process '%s' does not support read-object protocol version 1\", subprocess->cmd);\n>> +               goto done;\n>> +       }\n>> +       err = strcmp(packet_read_line(process->out, NULL), \"version=1\");\n>> +       if (err)\n>> +               goto done;\n>> +       err = packet_read_line(process->out, NULL) != NULL;\n>> +       if (err)\n>> +               goto done;\n>> +\n>> +       err = packet_writel(process->in, \"capability=get\", NULL);\n>> +       if (err)\n>> +               goto done;\n>> +\n>> +       for (;;) {\n>> +               cap_buf = packet_read_line(process->out, NULL);\n>> +               if (!cap_buf)\n>> +                       break;\n>> +               string_list_split_in_place(&cap_list, cap_buf, '=', 1);\n>> +\n>> +               if (cap_list.nr != 2 || strcmp(cap_list.items[0].string, \"capability\"))\n>> +                       continue;\n>> +\n>> +               cap_name = cap_list.items[1].string;\n>> +               if (!strcmp(cap_name, \"get\")) {\n>> +                       entry->supported_capabilities |= CAP_GET;\n>> +               }\n>> +               else {\n>> +                       warning(\n>> +                               \"external process '%s' requested unsupported read-object capability '%s'\",\n>> +                               subprocess->cmd, cap_name\n>> +                       );\n>> +               }\n>> +\n>> +               string_list_clear(&cap_list, 0);\n>> +       }\n>> +\n>> +done:\n>> +       sigchain_pop(SIGPIPE);\n>> +\n>> +       if (err || errno == EPIPE)\n>> +               return err ? err : errno;\n>> +\n>> +       return 0;\n>> +}\n>> +\n>> +static int read_object_process(const unsigned char *sha1)\n>> +{\n>> +       int err;\n>> +       struct read_object_process *entry;\n>> +       struct child_process *process;\n>> +       struct strbuf status = STRBUF_INIT;\n>> +       uint64_t start;\n>> +\n>> +       start = getnanotime();\n>> +\n>> +       if (!repository_format_promised_objects)\n>> +               die(\"BUG: if extensions.promisedObjects is not set, there \"\n>> +                   \"should not be any promised objects\");\n>> +\n>> +       if (!subprocess_map_initialized) {\n>> +               subprocess_map_initialized = 1;\n>> +               hashmap_init(&subprocess_map, (hashmap_cmp_fn)cmd2process_cmp, 0);\n>> +               entry = NULL;\n>> +       } else {\n>> +               entry = (struct read_object_process *)subprocess_find_entry(&subprocess_map, repository_format_promised_objects);\n>> +       }\n>> +       if (!entry) {\n>> +               entry = xmalloc(sizeof(*entry));\n>> +               entry->supported_capabilities = 0;\n>> +\n>> +               if (subprocess_start(&subprocess_map, &entry->subprocess, repository_format_promised_objects, start_read_object_fn)) {\n>> +                       free(entry);\n>> +                       return -1;\n>> +               }\n>> +       }\n>> +       process = subprocess_get_child_process(&entry->subprocess);\n>> +\n>> +       if (!(CAP_GET & entry->supported_capabilities))\n>> +               return -1;\n>> +\n>> +       sigchain_push(SIGPIPE, SIG_IGN);\n>> +\n>> +       err = packet_write_fmt_gently(process->in, \"command=get\\n\");\n>> +       if (err)\n>> +               goto done;\n>> +\n>> +       err = packet_write_fmt_gently(process->in, \"sha1=%s\\n\", sha1_to_hex(sha1));\n>> +       if (err)\n>> +               goto done;\n>> +\n>> +       err = packet_flush_gently(process->in);\n>> +       if (err)\n>> +               goto done;\n>> +\n>> +       err = subprocess_read_status(process->out, &status);\n>> +       err = err ? err : strcmp(status.buf, \"success\");\n>> +\n>> +done:\n>> +       sigchain_pop(SIGPIPE);\n>> +\n>> +       if (err || errno == EPIPE) {\n>> +               err = err ? err : errno;\n>> +               if (!strcmp(status.buf, \"error\")) {\n>> +                       /* The process signaled a problem with the file. */\n>> +               }\n>> +               else if (!strcmp(status.buf, \"abort\")) {\n>> +                       /*\n>> +                        * The process signaled a permanent problem. Don't try to read\n>> +                        * objects with the same command for the lifetime of the current\n>> +                        * Git process.\n>> +                        */\n>> +                       entry->supported_capabilities &= ~CAP_GET;\n>> +               }\n>> +               else {\n>> +                       /*\n>> +                        * Something went wrong with the read-object process.\n>> +                        * Force shutdown and restart if needed.\n>> +                        */\n>> +                       error(\"external process '%s' failed\", repository_format_promised_objects);\n>> +                       subprocess_stop(&subprocess_map, (struct subprocess_entry *)entry);\n>> +                       free(entry);\n>> +               }\n>> +       }\n>> +\n>> +       trace_performance_since(start, \"read_object_process\");\n>> +\n>> +       return err;\n>> +}\n>> +\n>> +int request_promised_objects(const struct oid_array *oids)\n>> +{\n>> +       int oids_requested = 0;\n>> +       int i;\n>> +\n>> +       for (i = 0; i < oids->nr; i++) {\n>> +               if (is_promised_object(&oids->oid[i], NULL, NULL))\n>> +                       break;\n>> +       }\n>> +\n>> +       if (i == oids->nr)\n>> +               /* Nothing to fetch */\n>> +               return 0;\n>> +\n>> +       for (; i < oids->nr; i++) {\n>> +               if (is_promised_object(&oids->oid[i], NULL, NULL)) {\n>> +                       read_object_process(oids->oid[i].hash);\n>> +                       oids_requested++;\n>> +               }\n>> +       }\n>> +\n>> +       /*\n>> +        * The command above may have updated packfiles, so update our record\n>> +        * of them.\n>> +        */\n>> +       reprepare_packed_git();\n>> +       return oids_requested;\n>> +}\n>> diff --git a/promised-object.h b/promised-object.h\n>> index 7eaedff17..8ad47aa4c 100644\n>> --- a/promised-object.h\n>> +++ b/promised-object.h\n>> @@ -2,6 +2,7 @@\n>>   #define PROMISED_OBJECT_H\n>>\n>>   #include \"cache.h\"\n>> +#include \"sha1-array.h\"\n>>\n>>   /*\n>>    * Returns 1 if oid is the name of a promised object. For non-blobs, 0 is\n>> @@ -19,4 +20,15 @@ int for_each_promised_object(each_promised_object_fn, void *);\n>>    */\n>>   int fsck_promised_objects(void);\n>>\n>> +/*\n>> + * If any of the given objects are promised objects, invokes\n>> + * core.promisedobjectcommand with those objects and returns the number of\n>> + * objects requested. No check is made as to whether the invocation actually\n>> + * populated the repository with the promised objects.\n>> + *\n>> + * If none of the given objects are promised objects, this function does not\n>> + * invoke anything and returns 0.\n>> + */\n>> +int request_promised_objects(const struct oid_array *oids);\n>> +\n>>   #endif\n>> diff --git a/sha1_file.c b/sha1_file.c\n>> index 5862386cd..ded0ef46b 100644\n>> --- a/sha1_file.c\n>> +++ b/sha1_file.c\n>> @@ -28,6 +28,11 @@\n>>   #include \"list.h\"\n>>   #include \"mergesort.h\"\n>>   #include \"quote.h\"\n>> +#include \"iterator.h\"\n>> +#include \"dir-iterator.h\"\n>> +#include \"sha1-lookup.h\"\n>> +#include \"promised-object.h\"\n>> +#include \"sha1-array.h\"\n>>\n>>   #define SZ_FMT PRIuMAX\n>>   static inline uintmax_t sz_fmt(size_t s) { return s; }\n>> @@ -2983,6 +2988,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>>          const unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n>>                                      lookup_replace_object(sha1) :\n>>                                      sha1;\n>> +       int already_retried = 0;\n>>\n>>          if (!oi)\n>>                  oi = &blank_oi;\n>> @@ -3007,30 +3013,40 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n>>                  }\n>>          }\n>>\n>> -       if (!find_pack_entry(real, &e)) {\n>> -               /* Most likely it's a loose object. */\n>> -               if (!sha1_loose_object_info(real, oi, flags)) {\n>> -                       oi->whence = OI_LOOSE;\n>> -                       return 0;\n>> -               }\n>> +retry:\n>> +       if (find_pack_entry(real, &e))\n>> +               goto found_packed;\n>>\n>> -               /* Not a loose object; someone else may have just packed it. */\n>> -               if (flags & OBJECT_INFO_QUICK) {\n>> -                       return -1;\n>> -               } else {\n>> -                       reprepare_packed_git();\n>> -                       if (!find_pack_entry(real, &e))\n>> -                               return -1;\n>> +       /* Most likely it's a loose object. */\n>> +       if (!sha1_loose_object_info(real, oi, flags)) {\n>> +               oi->whence = OI_LOOSE;\n>> +               return 0;\n>> +       }\n>> +\n>> +       /* Not a loose object; someone else may have just packed it. */\n>> +       reprepare_packed_git();\n>> +       if (find_pack_entry(real, &e))\n>> +               goto found_packed;\n>> +\n>> +       /* Check if it is a promised blob */\n>> +       if (!already_retried && !(flags & OBJECT_INFO_IGNORE_PROMISES)) {\n>> +               struct oid_array promised = OID_ARRAY_INIT;\n>> +               oid_array_append_sha1(&promised, real);\n>> +               if (request_promised_objects(&promised)) {\n>> +                       already_retried = 1;\n>> +                       goto retry;\n>>                  }\n>>          }\n>>\n>> +       return -1;\n>> +\n>> +found_packed:\n>>          if (oi == &blank_oi)\n>>                  /*\n>>                   * We know that the caller doesn't actually need the\n>>                   * information below, so return early.\n>>                   */\n>>                  return 0;\n>> -\n>>          rtype = packed_object_info(e.p, e.offset, oi);\n>>          if (rtype < 0) {\n>>                  mark_bad_packed_object(e.p, real);\n>> diff --git a/t/t3907-promised-object.sh b/t/t3907-promised-object.sh\n>> index 3e0caf4f9..d9e6a6486 100755\n>> --- a/t/t3907-promised-object.sh\n>> +++ b/t/t3907-promised-object.sh\n>> @@ -38,4 +38,36 @@ test_expect_success '...but fails again with GIT_IGNORE_PROMISED_OBJECTS' '\n>>          unset GIT_IGNORE_PROMISED_OBJECTS\n>>   '\n>>\n>> +test_expect_success 'sha1_object_info_extended (through git cat-file)' '\n>> +       test_create_repo server &&\n>> +       test_commit -C server 1 1.t abcdefgh &&\n>> +       HASH=$(git hash-object server/1.t) &&\n>> +\n>> +       test_create_repo client &&\n>> +       test_must_fail git -C client cat-file -p \"$HASH\"\n>> +'\n>> +\n>> +test_expect_success '...succeeds if it is a promised object' '\n>> +       printf \"%s03%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n>> +               hex_pack >client/.git/objects/promised &&\n>> +       git -C client config core.repositoryformatversion 1 &&\n>> +       git -C client config extensions.promisedobjects \\\n>> +               \"\\\"$TEST_DIRECTORY/t3907/read-object\\\" \\\"$(pwd)/server/.git\\\"\" &&\n>> +       git -C client cat-file -p \"$HASH\"\n>> +'\n>> +\n>> +test_expect_success 'cat-file --batch-all-objects with promised objects' '\n>> +       rm -rf client &&\n>> +       test_create_repo client &&\n>> +       git -C client config core.repositoryformatversion 1 &&\n>> +       git -C client config extensions.promisedobjects \\\n>> +               \"\\\"$TEST_DIRECTORY/t3907/read-object\\\" \\\"$(pwd)/server/.git\\\"\" &&\n>> +       printf \"%s03%016x\" \"$HASH\" \"$(wc -c <server/1.t)\" |\n>> +               hex_pack >client/.git/objects/promised &&\n>> +\n>> +       # Verify that the promised object is printed\n>> +       git -C client cat-file --batch --batch-all-objects | tee out |\n>> +               grep abcdefgh\n>> +'\n>> +\n>>   test_done\n>> diff --git a/t/t3907/read-object b/t/t3907/read-object\n>> new file mode 100755\n>> index 000000000..9666ad597\n>> --- /dev/null\n>> +++ b/t/t3907/read-object\n>> @@ -0,0 +1,114 @@\n>> +#!/usr/bin/perl\n>> +#\n>> +# Example implementation for the Git read-object protocol version 1\n>> +# See Documentation/technical/read-object-protocol.txt\n>> +#\n>> +# Allows you to test the ability for blobs to be pulled from a host git repo\n>> +# \"on demand.\"  Called when git needs a blob it couldn't find locally due to\n>> +# a lazy clone that only cloned the commits and trees.\n>> +#\n>> +# A lazy clone can be simulated via the following commands from the host repo\n>> +# you wish to create a lazy clone of:\n>> +#\n>> +# cd /host_repo\n>> +# git rev-parse HEAD\n>> +# git init /guest_repo\n>> +# git cat-file --batch-check --batch-all-objects | grep -v 'blob' |\n>> +#      cut -d' ' -f1 | git pack-objects /guest_repo/.git/objects/pack/noblobs\n>> +# cd /guest_repo\n>> +# git config core.virtualizeobjects true\n>> +# git reset --hard <sha from rev-parse call above>\n>> +#\n>> +# Please note, this sample is a minimal skeleton. No proper error handling\n>> +# was implemented.\n>> +#\n>> +\n>> +use strict;\n>> +use warnings;\n>> +\n>> +#\n>> +# Point $DIR to the folder where your host git repo is located so we can pull\n>> +# missing objects from it\n>> +#\n>> +my $DIR = $ARGV[0];\n>> +\n>> +sub packet_bin_read {\n>> +       my $buffer;\n>> +       my $bytes_read = read STDIN, $buffer, 4;\n>> +       if ( $bytes_read == 0 ) {\n>> +\n>> +               # EOF - Git stopped talking to us!\n>> +               exit();\n>> +       }\n>> +       elsif ( $bytes_read != 4 ) {\n>> +               die \"invalid packet: '$buffer'\";\n>> +       }\n>> +       my $pkt_size = hex($buffer);\n>> +       if ( $pkt_size == 0 ) {\n>> +               return ( 1, \"\" );\n>> +       }\n>> +       elsif ( $pkt_size > 4 ) {\n>> +               my $content_size = $pkt_size - 4;\n>> +               $bytes_read = read STDIN, $buffer, $content_size;\n>> +               if ( $bytes_read != $content_size ) {\n>> +                       die \"invalid packet ($content_size bytes expected; $bytes_read bytes read)\";\n>> +               }\n>> +               return ( 0, $buffer );\n>> +       }\n>> +       else {\n>> +               die \"invalid packet size: $pkt_size\";\n>> +       }\n>> +}\n>> +\n>> +sub packet_txt_read {\n>> +       my ( $res, $buf ) = packet_bin_read();\n>> +       unless ( $buf =~ s/\\n$// ) {\n>> +               die \"A non-binary line MUST be terminated by an LF.\";\n>> +       }\n>> +       return ( $res, $buf );\n>> +}\n>> +\n>> +sub packet_bin_write {\n>> +       my $buf = shift;\n>> +       print STDOUT sprintf( \"%04x\", length($buf) + 4 );\n>> +       print STDOUT $buf;\n>> +       STDOUT->flush();\n>> +}\n>> +\n>> +sub packet_txt_write {\n>> +       packet_bin_write( $_[0] . \"\\n\" );\n>> +}\n>> +\n>> +sub packet_flush {\n>> +       print STDOUT sprintf( \"%04x\", 0 );\n>> +       STDOUT->flush();\n>> +}\n>> +\n>> +( packet_txt_read() eq ( 0, \"git-read-object-client\" ) ) || die \"bad initialize\";\n>> +( packet_txt_read() eq ( 0, \"version=1\" ) )                             || die \"bad version\";\n>> +( packet_bin_read() eq ( 1, \"\" ) )                       || die \"bad version end\";\n>> +\n>> +packet_txt_write(\"git-read-object-server\");\n>> +packet_txt_write(\"version=1\");\n>> +packet_flush();\n>> +\n>> +( packet_txt_read() eq ( 0, \"capability=get\" ) )    || die \"bad capability\";\n>> +( packet_bin_read() eq ( 1, \"\" ) )                  || die \"bad capability end\";\n>> +\n>> +packet_txt_write(\"capability=get\");\n>> +packet_flush();\n>> +\n>> +while (1) {\n>> +       my ($command) = packet_txt_read() =~ /^command=([^=]+)$/;\n>> +\n>> +       if ( $command eq \"get\" ) {\n>> +               my ($sha1) = packet_txt_read() =~ /^sha1=([0-9a-f]{40})$/;\n>> +               packet_bin_read();\n>> +\n>> +               system ('git --git-dir=\"' . $DIR . '\" cat-file blob ' . $sha1 . ' | git -c core.virtualizeobjects=false hash-object -w --stdin >/dev/null 2>&1');\n>> +               packet_txt_write(($?) ? \"status=error\" : \"status=success\");\n>> +               packet_flush();\n>> +       } else {\n>> +               die \"bad command '$command'\";\n>> +       }\n>> +}\n>> --\n>> 2.14.0.rc0.284.gd933b75aa4-goog\n>>\n"},{"id":"324841","messageId":"20170720141342.6a89aace@twelve2.svl.corp.google.com","threadId":"46359","inReplyTo":"d8beb406-6d03-d893-4f37-83a7bdd6b9ac@gmail.com","subject":"Re: [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-20T21:13:42Z","receivedAt":"2017-07-20T21:13:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 20 Jul 2017 15:58:51 -0400\nBen Peart <peartben@gmail.com> wrote:\n\n> On 7/19/2017 8:21 PM, Jonathan Tan wrote:\n> > Currently, Git does not support repos with very large numbers of objects\n> > or repos that wish to minimize manipulation of certain blobs (for\n> > example, because they are very large) very well, even if the user\n> > operates mostly on part of the repo, because Git is designed on the\n> > assumption that every referenced object is available somewhere in the\n> > repo storage.\n> > \n> \n> Great to see this idea making progress. Making git able to gracefully \n> handle partial clones (beyond the existing shallow clone support) is a \n> key piece of dealing with very large objects and repos.\n\nThanks.\n\n> > As a first step to reducing this problem, introduce the concept of\n> > promised objects. Each Git repo can contain a list of promised objects\n> > and their sizes (if blobs) at $GIT_DIR/objects/promised. This patch\n> > contains functions to query them; functions for creating and modifying\n> > that file will be introduced in later patches.\n> \n> If I'm reading this patch correctly, for a repo to successfully pass \n> \"git fsck\" either the object or a promise must exist for everything fsck \n> checks.  From the documentation for fsck it says \"git fsck defaults to \n> using the index file, all SHA-1 references in refs namespace, and all \n> reflogs (unless --no-reflogs is given) as heads.\" Doesn't this then \n> imply objects or promises must exist for all objects referenced by any \n> of these locations?\n> \n> We're currently in the hundreds of millions of objects on some of our \n> repos so even downloading the promises for all the objects in the index \n> is unreasonable as it is gigabytes of data and growing.\n\nFor the index to contain all the files, the repo must already have\ndownloaded all the trees for HEAD (at least). The trees collectively\ncontain entries for all the relevant blobs. We need one promise for each\nblob, and the size of a promise is comparable to the size of a tree\nentry, so the size (of download and storage) needed would be just double\nof what we would need if we didn't need promises. This is still only\nlinear growth, unless you have found that the absolute numbers are too\nlarge?\n\nAlso, if the index is ever changed to not have one entry for every file,\nwe also wouldn't need one promise for every file.\n\n> I think we should have a flag (off by default) that enables someone to \n> say that promised objects are optional. If the flag is set, \n> \"is_promised_object\" will return success and pass the OBJ_ANY type and a \n> size of -1.\n> \n> Nothing today is using the size and in the two places where the object \n> type is being checked for consistency (fsck_cache_tree and \n> fsck_handle_ref) the test can add a test for OBJ_ANY as well.\n> \n> This will enable very large numbers of objects to be omitted from the \n> clone without triggering a download of the corresponding number of \n> promised objects.\n\nEventually I plan to use the size when implementing parameters for\nhistory-searching commands (e.g. \"git log -S\"), but it's true that\nthat's in the future.\n\nAllowing promised objects to be optional would indeed solve the issue of\ndownloading too many promises. It would make the code more complicated,\nbut I'm not sure by how much.\n\nFor example, in this fsck patch, the easiest way I could think of to\nhave promised objects was to introduce a 3rd state, called \"promised\",\nof \"struct object\" - one in which the type is known, but we don't have\naccess to the full \"struct commit\" or equivalent. And thus fsck could\nassume that if the \"struct object\" is \"parsed\" or \"promised\", the type\nis known. Having optional promised objects would require that we let\nthis \"promised\" state have a type of OBJ_UNKNOWN (or something like\nthat) - maybe that would be fine, but I haven't looked into this in\ndetail.\n\n> > A repository that is missing an object but has that object promised is not\n> > considered to be in error, so also teach fsck this. As part of doing\n> > this, object.{h,c} has been modified to generate \"struct object\" based\n> > on only the information available to promised objects, without requiring\n> > the object itself.\n> \n> In your work on this, did you investigate if there are other commands \n> (ie repack/gc) that will need to learn about promised objects? Have you \n> had a chance (or have plans) to hack up the test suite so that it runs \n> all tests with promised objects and see what (if anything) breaks?\n\nIn one of the subsequent patches, I tried to ensure that all\nobject-reading functions in sha1_file.c somewhat works (albeit slowly)\nin the presence of promised objects - that would cover the functionality\nof the other commands. As for hacking up the test suite to run with\npromised objects, that would be ideal, but I haven't figured out how to\ndo that yet.\n"},{"id":"324842","messageId":"20170720141804.22cdd2bf@twelve2.svl.corp.google.com","threadId":"46359","inReplyTo":"75d5c3cd-c1d7-f06a-fc7e-894cde95afa7@gmail.com","subject":"Re: [RFC PATCH v2 4/4] sha1_file: support promised object hook","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-20T21:18:04Z","receivedAt":"2017-07-20T21:18:13Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 20 Jul 2017 16:58:16 -0400\nBen Peart <peartben@gmail.com> wrote:\n\n> >> This is meant as a temporary measure to ensure that all Git commands\n> >> work in such a situation. Future patches will update some commands to\n> >> either tolerate promised objects (without invoking the hook) or be more\n> >> efficient in invoking the promised objects hook.\n> \n> I agree that making git more tolerant of promised objects if possible \n> and precomputing a list of promised objects required to complete a \n> particular command and downloading them with a single request are good \n> optimizations to add over time.\n\nThat's good to know!\n\n> has_sha1_file also takes a hash \"whether local or in an alternate object \n> database, and whether packed or loose\" but never calls \n> sha1_object_info_extended.  As a result, we had to add support in \n> check_and_freshen to download missing objects to get proper behavior in \n> all cases.  I don't think this will work correctly without it.\n\nThanks for the attention to detail. Is this before or after commit\ne83e71c (\"sha1_file: refactor has_sha1_file_with_flags\", 2017-06-26)?\n"},{"id":"324843","messageId":"xmqqk232kdoo.fsf@gitster.mtv.corp.google.com","threadId":"46359","inReplyTo":"0eb270f40095f132bf9383f230506e45d895e5e1.1500508695.git.jonathantanmy@google.com","subject":"Re: [RFC PATCH v2 1/4] object: remove \"used\" field from struct object","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-07-20T21:20:55Z","receivedAt":"2017-07-20T21:21:04Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan <jonathantanmy@google.com> writes:\n\n> The \"used\" field in struct object is only used by builtin/fsck. Remove\n> that field and modify builtin/fsck to use a flag instead.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  builtin/fsck.c | 24 ++++++++++++++----------\n>  object.c       |  1 -\n>  object.h       |  2 +-\n>  3 files changed, 15 insertions(+), 12 deletions(-)\n\nI vaguely recall trying to do this myself a few years ago.  We can\neasily spot places within this file that does things like this:\n\n> -\tobj->flags = HAS_OBJ;\n\nand correctly update it to this:\n\n> +\tobj->flags &= ~(REACHABLE | SEEN);\n> +\tobj->flags |= HAS_OBJ;\n\nbut I didn't do so because I was hesitant having to validate, and\nhaving to maintain the invariant forever, that anything called from\nthese codepaths is always careful not to clobber the USED bit.\n\nLooks like a reasonable preparatory clean-up that probably should be\ndoable and should be done way before the main part of the series to\nme.  Will queue, together with your other fsck change.\n\nThanks.\n"},{"id":"324871","messageId":"d014682f-66a5-c3c1-cf66-ef4bb28de076@gmail.com","threadId":"46359","inReplyTo":"20170720141342.6a89aace@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-21T16:24:52Z","receivedAt":"2017-07-21T16:25:02Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/20/2017 5:13 PM, Jonathan Tan wrote:\n> On Thu, 20 Jul 2017 15:58:51 -0400\n> Ben Peart <peartben@gmail.com> wrote:\n> \n>> On 7/19/2017 8:21 PM, Jonathan Tan wrote:\n>>> Currently, Git does not support repos with very large numbers of objects\n>>> or repos that wish to minimize manipulation of certain blobs (for\n>>> example, because they are very large) very well, even if the user\n>>> operates mostly on part of the repo, because Git is designed on the\n>>> assumption that every referenced object is available somewhere in the\n>>> repo storage.\n>>>\n>>\n>> Great to see this idea making progress. Making git able to gracefully\n>> handle partial clones (beyond the existing shallow clone support) is a\n>> key piece of dealing with very large objects and repos.\n> \n> Thanks.\n> \n>>> As a first step to reducing this problem, introduce the concept of\n>>> promised objects. Each Git repo can contain a list of promised objects\n>>> and their sizes (if blobs) at $GIT_DIR/objects/promised. This patch\n>>> contains functions to query them; functions for creating and modifying\n>>> that file will be introduced in later patches.\n>>\n>> If I'm reading this patch correctly, for a repo to successfully pass\n>> \"git fsck\" either the object or a promise must exist for everything fsck\n>> checks.  From the documentation for fsck it says \"git fsck defaults to\n>> using the index file, all SHA-1 references in refs namespace, and all\n>> reflogs (unless --no-reflogs is given) as heads.\" Doesn't this then\n>> imply objects or promises must exist for all objects referenced by any\n>> of these locations?\n>>\n>> We're currently in the hundreds of millions of objects on some of our\n>> repos so even downloading the promises for all the objects in the index\n>> is unreasonable as it is gigabytes of data and growing.\n> \n> For the index to contain all the files, the repo must already have\n> downloaded all the trees for HEAD (at least). The trees collectively\n> contain entries for all the relevant blobs. We need one promise for each\n> blob, and the size of a promise is comparable to the size of a tree\n> entry, so the size (of download and storage) needed would be just double\n> of what we would need if we didn't need promises. This is still only\n> linear growth, unless you have found that the absolute numbers are too\n> large?\n> \n\nToday we have 3.5 million objects * 30 bytes per entry = 105 MB of \npromises. Given the average developer only hydrates 56K files (2 MB \npromises) that is 103 MB to download that no one will ever need. We \nwould like to avoid that if possible as this would be a significant \nregression in clone times from where we are today.\n\nI'm also concerned about the performance of merging in promises given we \nhave 100M objects today and growing so the number of promises over time \ncould get pretty large.\n\n> Also, if the index is ever changed to not have one entry for every file,\n> we also wouldn't need one promise for every file.\n> \n>> I think we should have a flag (off by default) that enables someone to\n>> say that promised objects are optional. If the flag is set,\n>> \"is_promised_object\" will return success and pass the OBJ_ANY type and a\n>> size of -1.\n>>\n>> Nothing today is using the size and in the two places where the object\n>> type is being checked for consistency (fsck_cache_tree and\n>> fsck_handle_ref) the test can add a test for OBJ_ANY as well.\n>>\n>> This will enable very large numbers of objects to be omitted from the\n>> clone without triggering a download of the corresponding number of\n>> promised objects.\n> \n> Eventually I plan to use the size when implementing parameters for\n> history-searching commands (e.g. \"git log -S\"), but it's true that\n> that's in the future.\n> \n> Allowing promised objects to be optional would indeed solve the issue of\n> downloading too many promises. It would make the code more complicated,\n> but I'm not sure by how much.\n> \n> For example, in this fsck patch, the easiest way I could think of to\n> have promised objects was to introduce a 3rd state, called \"promised\",\n> of \"struct object\" - one in which the type is known, but we don't have\n> access to the full \"struct commit\" or equivalent. And thus fsck could\n> assume that if the \"struct object\" is \"parsed\" or \"promised\", the type\n> is known. Having optional promised objects would require that we let\n> this \"promised\" state have a type of OBJ_UNKNOWN (or something like\n> that) - maybe that would be fine, but I haven't looked into this in\n> detail.\n> \n\nCaveats apply as I only did a quick look but I only found the two \nlocations that were checking the object type for consistency.\n\n>>> A repository that is missing an object but has that object promised is not\n>>> considered to be in error, so also teach fsck this. As part of doing\n>>> this, object.{h,c} has been modified to generate \"struct object\" based\n>>> on only the information available to promised objects, without requiring\n>>> the object itself.\n>>\n>> In your work on this, did you investigate if there are other commands\n>> (ie repack/gc) that will need to learn about promised objects? Have you\n>> had a chance (or have plans) to hack up the test suite so that it runs\n>> all tests with promised objects and see what (if anything) breaks?\n> \n> In one of the subsequent patches, I tried to ensure that all\n> object-reading functions in sha1_file.c somewhat works (albeit slowly)\n> in the presence of promised objects - that would cover the functionality\n> of the other commands. As for hacking up the test suite to run with\n> promised objects, that would be ideal, but I haven't figured out how to\n> do that yet.\n> \n"},{"id":"324872","messageId":"a3d1d0b9-5d28-e643-4964-6d087b7b0ec2@gmail.com","threadId":"46359","inReplyTo":"20170720141804.22cdd2bf@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH v2 4/4] sha1_file: support promised object hook","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-21T16:27:03Z","receivedAt":"2017-07-21T16:27:10Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/20/2017 5:18 PM, Jonathan Tan wrote:\n> On Thu, 20 Jul 2017 16:58:16 -0400\n> Ben Peart <peartben@gmail.com> wrote:\n> \n>>>> This is meant as a temporary measure to ensure that all Git commands\n>>>> work in such a situation. Future patches will update some commands to\n>>>> either tolerate promised objects (without invoking the hook) or be more\n>>>> efficient in invoking the promised objects hook.\n>>\n>> I agree that making git more tolerant of promised objects if possible\n>> and precomputing a list of promised objects required to complete a\n>> particular command and downloading them with a single request are good\n>> optimizations to add over time.\n> \n> That's good to know!\n> \n>> has_sha1_file also takes a hash \"whether local or in an alternate object\n>> database, and whether packed or loose\" but never calls\n>> sha1_object_info_extended.  As a result, we had to add support in\n>> check_and_freshen to download missing objects to get proper behavior in\n>> all cases.  I don't think this will work correctly without it.\n> \n> Thanks for the attention to detail. Is this before or after commit\n> e83e71c (\"sha1_file: refactor has_sha1_file_with_flags\", 2017-06-26)?\n> \n\nSorry, my bad.  I missed the comment in the cover letter that said this \nneeded to be applied on top of your other patch series.\n"},{"id":"324885","messageId":"20170721133308.422e736d@twelve2.svl.corp.google.com","threadId":"46359","inReplyTo":"d014682f-66a5-c3c1-cf66-ef4bb28de076@gmail.com","subject":"Re: [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-07-21T20:33:08Z","receivedAt":"2017-07-21T20:33:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 21 Jul 2017 12:24:52 -0400\nBen Peart <peartben@gmail.com> wrote:\n\n> Today we have 3.5 million objects * 30 bytes per entry = 105 MB of \n> promises. Given the average developer only hydrates 56K files (2 MB \n> promises) that is 103 MB to download that no one will ever need. We \n> would like to avoid that if possible as this would be a significant \n> regression in clone times from where we are today.\n> \n> I'm also concerned about the performance of merging in promises given we \n> have 100M objects today and growing so the number of promises over time \n> could get pretty large.\n\nAfter some thought, maybe a hybrid solution is best, in which it is\npermissible but optional for some missing objects to have promises. In\nthat case, it is more of a \"size cache\" (which stores the type as well)\nrather than a true promise. When fetching, the client can optionally\nrequest for the sizes and types of missing objects.\n\nThis is good for the large-blob case, in which we can always have size\ninformation of missing blobs, and we can subsequently add blob-size\nfiltering (as a parameter) to \"git log -S\" and friends to avoid needing\nto resolve a missing object. And this is, as far as I can tell, also\ngood for the many-blob case - just have an empty size cache all the\ntime. (And in the future, use cases could come up that desire non-empty\nbut non-comprehensive caches - for example, a directory lister working\non a partial clone that only needs to cache the sizes of frequently\naccessed directories.)\n\nAnother option is to have a repo-wide option that toggles between\nmandatory entries in the \"size cache\" and prohibited entries. Switching\nto mandatory provides stricter fsck and negative lookups, but I think\nit's not worth it for both the developers and users of Git to have to\nknow about these two modes.\n\n> >> I think we should have a flag (off by default) that enables someone to\n> >> say that promised objects are optional. If the flag is set,\n> >> \"is_promised_object\" will return success and pass the OBJ_ANY type and a\n> >> size of -1.\n> >>\n> >> Nothing today is using the size and in the two places where the object\n> >> type is being checked for consistency (fsck_cache_tree and\n> >> fsck_handle_ref) the test can add a test for OBJ_ANY as well.\n> >>\n> >> This will enable very large numbers of objects to be omitted from the\n> >> clone without triggering a download of the corresponding number of\n> >> promised objects.\n> > \n> > Eventually I plan to use the size when implementing parameters for\n> > history-searching commands (e.g. \"git log -S\"), but it's true that\n> > that's in the future.\n> > \n> > Allowing promised objects to be optional would indeed solve the issue of\n> > downloading too many promises. It would make the code more complicated,\n> > but I'm not sure by how much.\n> > \n> > For example, in this fsck patch, the easiest way I could think of to\n> > have promised objects was to introduce a 3rd state, called \"promised\",\n> > of \"struct object\" - one in which the type is known, but we don't have\n> > access to the full \"struct commit\" or equivalent. And thus fsck could\n> > assume that if the \"struct object\" is \"parsed\" or \"promised\", the type\n> > is known. Having optional promised objects would require that we let\n> > this \"promised\" state have a type of OBJ_UNKNOWN (or something like\n> > that) - maybe that would be fine, but I haven't looked into this in\n> > detail.\n> > \n> \n> Caveats apply as I only did a quick look but I only found the two \n> locations that were checking the object type for consistency.\n\nI haven't looked into detail, but you are probably right.\n"},{"id":"325043","messageId":"96e8a347-d6b5-e21e-4129-2957fc26db19@gmail.com","threadId":"46359","inReplyTo":"20170721133308.422e736d@twelve2.svl.corp.google.com","subject":"Re: [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects","fromName":"Ben Peart","fromEmail":"peartben@gmail.com","sentAt":"2017-07-25T15:10:32Z","receivedAt":"2017-07-25T15:10:45Z","isPatch":true,"sender":{"key":"benpeart@microsoft.com","avatar":"https://avatars.githubusercontent.com/u/15252029?v=4"},"body":"\n\nOn 7/21/2017 4:33 PM, Jonathan Tan wrote:\n> On Fri, 21 Jul 2017 12:24:52 -0400\n> Ben Peart <peartben@gmail.com> wrote:\n> \n>> Today we have 3.5 million objects * 30 bytes per entry = 105 MB of\n>> promises. Given the average developer only hydrates 56K files (2 MB\n>> promises) that is 103 MB to download that no one will ever need. We\n>> would like to avoid that if possible as this would be a significant\n>> regression in clone times from where we are today.\n>>\n>> I'm also concerned about the performance of merging in promises given we\n>> have 100M objects today and growing so the number of promises over time\n>> could get pretty large.\n> \n> After some thought, maybe a hybrid solution is best, in which it is\n> permissible but optional for some missing objects to have promises. In\n> that case, it is more of a \"size cache\" (which stores the type as well)\n> rather than a true promise. When fetching, the client can optionally\n> request for the sizes and types of missing objects.\n> \n\nIn our GVFS solution today we do not download any size or object type \ninformation at clone as the number of objects and the resulting file \nwould be too large.  Instead, we have a new sizes endpoint \n(https://github.com/Microsoft/GVFS/blob/master/Protocol.md) that enables \nus to retrieve object sizes \"on demand\" much like we are enabling for \nthe actual object content.\n\nThis protocol could easily be extended to return both size and type so \nthat it could be used to retrieve \"promise\" data for objects as they are \nneeded. Having a way to \"cache\" that data locally so that both git and \nother code could share it would be great.\n\nAt a minimum, we should ensure the data stream passed back is the same \nwhether at clone time or when hitting a \"promises\" end point. I think it \nwould also be helpful to enable promises to be downloaded on demand much \nlike we are doing for the object data itself.\n\n> This is good for the large-blob case, in which we can always have size\n> information of missing blobs, and we can subsequently add blob-size\n> filtering (as a parameter) to \"git log -S\" and friends to avoid needing\n> to resolve a missing object. And this is, as far as I can tell, also\n> good for the many-blob case - just have an empty size cache all the\n> time. (And in the future, use cases could come up that desire non-empty\n> but non-comprehensive caches - for example, a directory lister working\n> on a partial clone that only needs to cache the sizes of frequently\n> accessed directories.)\n> \n> Another option is to have a repo-wide option that toggles between\n> mandatory entries in the \"size cache\" and prohibited entries. Switching\n> to mandatory provides stricter fsck and negative lookups, but I think\n> it's not worth it for both the developers and users of Git to have to\n> know about these two modes.\n> \n>>>> I think we should have a flag (off by default) that enables someone to\n>>>> say that promised objects are optional. If the flag is set,\n>>>> \"is_promised_object\" will return success and pass the OBJ_ANY type and a\n>>>> size of -1.\n>>>>\n>>>> Nothing today is using the size and in the two places where the object\n>>>> type is being checked for consistency (fsck_cache_tree and\n>>>> fsck_handle_ref) the test can add a test for OBJ_ANY as well.\n>>>>\n>>>> This will enable very large numbers of objects to be omitted from the\n>>>> clone without triggering a download of the corresponding number of\n>>>> promised objects.\n>>>\n>>> Eventually I plan to use the size when implementing parameters for\n>>> history-searching commands (e.g. \"git log -S\"), but it's true that\n>>> that's in the future.\n>>>\n>>> Allowing promised objects to be optional would indeed solve the issue of\n>>> downloading too many promises. It would make the code more complicated,\n>>> but I'm not sure by how much.\n>>>\n>>> For example, in this fsck patch, the easiest way I could think of to\n>>> have promised objects was to introduce a 3rd state, called \"promised\",\n>>> of \"struct object\" - one in which the type is known, but we don't have\n>>> access to the full \"struct commit\" or equivalent. And thus fsck could\n>>> assume that if the \"struct object\" is \"parsed\" or \"promised\", the type\n>>> is known. Having optional promised objects would require that we let\n>>> this \"promised\" state have a type of OBJ_UNKNOWN (or something like\n>>> that) - maybe that would be fine, but I haven't looked into this in\n>>> detail.\n>>>\n>>\n>> Caveats apply as I only did a quick look but I only found the two\n>> locations that were checking the object type for consistency.\n> \n> I haven't looked into detail, but you are probably right.\n> \n"},{"id":"325068","messageId":"BC1048A63B034E46A11A01758BC04855@PhilipOakley","threadId":"46359","inReplyTo":"b928c073-6156-30f2-c850-993e59079ed1@gmail.com","subject":"Re: [RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing blobs\")","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-07-25T20:48:46Z","receivedAt":"2017-07-25T20:49:05Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"Sorry for the delay - been away...\n\nFrom: \"Ben Peart\" <peartben@gmail.com>\nSent: Monday, July 17, 2017 6:43 PM\n>\n> On 7/16/2017 11:23 AM, Philip Oakley wrote:\n>> From: \"Jonathan Tan\" <jonathantanmy@google.com>\n>> Sent: Tuesday, July 11, 2017 8:48 PM\n>>> These patches are part of a set of patches implementing partial clone,\n>>> as you can see here:\n>>>\n>>> https://github.com/jonathantanmy/git/tree/partialclone\n>>>\n>>> In that branch, clone with batch checkout works, as you can see in the\n>>> README. The code and tests are generally done, but some patches are\n>>> still missing documentation and commit messages.\n>>>\n>>> These 3 patches implement the foundational concept - formerly known as\n>>> \"missing blobs\" in the \"missing blob manifest\", I decided to call them\n>>> \"promised blobs\". The repo knows their object names and sizes. It also\n>>> does not have the blobs themselves, but can be configured to know how to\n>>> fetch them.\n>>>\n>> If I understand correctly, this method doesn't give any direct user \n>> visibility of missing blobs in the file system. Is that correct?\n>\n> That is correct\n>\n>>\n>> I was hoping that eventually the various 'on demand' approaches would \n>> still allow users to continue to work as they go off-line such that they \n>> can see directly (in the FS) where the missing blobs (and trees) are \n>> located, so that they can continue to commit new work on existing files.\n>>\n>\n> This is a challenge as git assumes all objects are always available (that \n> is a key design principal of a DVCS) so any missing object is considered a \n> corruption that typically results in a call to \"die.\"\n\nMy view/concept was more based on the fact that Git is happy to have missing \n'trees', as long as they are submodules ;-), so I was hoping to massage that \nso git could carry on working as if the whole 'tree' (or blob when they were \nomitted) was still present in as 'unchanged', so the oid's would stay as \nthey were.\n\nI see that you don't omit the trees, which would be more common in my type \nof environment (defence/security). I expect in an idealised BigWin repo the \nsame would also be true - user only gets /Office/Excel if that's what they \nare working on ;-)\n\n>\n> The GVFS solution gets around this by ensuring any missing object is \n> retrieved on behalf of git so that it never sees it as missing.  The \n> obvious tradeoff is that this requires a network connection so the object \n> can be retrieved.\n\nIn my concept, the user would not have the opportunity to fetch the \ntree/blob, but could replace it in its entirety (we'd still have the meta \ndata of the tree/blob name and it's old oid, but couldn't do a diff)\n>\n>> I had felt that some sort of 'gitlink' should be present (huma readable) \n>> as a place holder for the missing blob/tree. e.g. 'gitblob: 1234abcd' \n>> (showing the missing oid, jsut like sub-modules can do - it's no \n>> different really.\n>>\n>\n> We explored that option briefly but when you have a large number of files, \n> even writing out some sort of place holder can take a very long time.  In \n> fact, since the typical source file is relatively small (a few kilobytes), \n> writing out a placeholder doesn't save much time vs just writing out the \n> actual file contents.\n>\n> Another challenge is that even if there is a placeholder written to disk, \n> you still need a network connection to retrieve the actual contents \n> if/when it is needed.\n\nI was viewing the 'missing' tree/blobs to be part of a narrow clone concept, \nso the user would need to explicitly widen the narrow clone to get missing \ntrees/blobs (which could have been omitted by age, size, name, style of a \n.gitNarrowIgnore spec etc)\n\n>\n>> I'm concerned that the various GVFS extensions haven't fully achieved a \n>> separation of concerns surrounding the DVCS capability for \n>> on-line/off-line conversion as comms drop in and out. The GVFS looks \n>> great for a fully networked, always on, environment, but it would be good \n>> to also have the sepration for those who (will) have shallow/narrow \n>> clones that may also need to work with a local upstream that is also \n>> shallow/narrow.\n>>\n>\n> You are correct that this hasn't been tackled yet. It is a challenging \n> problem. I can envision something along the lines of what was done for the \n> shallow clone feature where there are distinct ways to change the set of \n> objects that are available but that would hopefully come in some future \n> patch series.\n\nOK. That's good to know. If the GFVS could be expanded to create a type of \nNarrow Clone capability  so that the 'going off-line' problem easily \ntransitions between being just a neat VFS and then to being a neat narrow \nclone, and  that it may solve two problems in one.\n\nI had it in my mind that the missing blobs/trees could be simply stubbed out \nwithin the repo itself, as just the oid ref, or maybe the oid ref plus the \nlength (given that size is one of the common causes on not wanting the \ncontent just yet). The repo could still be packed etc, as long as the format \nis understood.\n\n--\nPhilip\n\n>\n>> -- \n>> Philip\n>> I wanted to at least get my thoughts into the discussion before it all \n>> passes by.\n>>\n>>> An older version of these patches was sent as a single demonstration\n>>> patch in versions 1 to 3 of [1]. In there, Junio suggested that I have\n>>> only one file containing missing blob information. I have made that\n>>> suggested change in this version.\n>>>\n>>> One thing remaining is to add a repository extension [2] so that older\n>>> versions of Git fail immediately instead of trying to read missing\n>>> blobs, but I thought I'd send these first in order to get some initial\n>>> feedback.\n>>>\n>>> [1] \n>>> https://public-inbox.org/git/cover.1497035376.git.jonathantanmy@google.com/\n>>> [2] Documentation/technical/repository-version.txt\n>>>\n>>> Jonathan Tan (3):\n>>>  promised-blob, fsck: introduce promised blobs\n>>>  sha1-array: support appending unsigned char hash\n>>>  sha1_file: add promised blob hook support\n>>>\n>>> Documentation/config.txt               |   8 ++\n>>> Documentation/gitrepository-layout.txt |   8 ++\n>>> Makefile                               |   1 +\n>>> builtin/cat-file.c                     |   9 ++\n>>> builtin/fsck.c                         |  13 +++\n>>> promised-blob.c                        | 170 \n>>> +++++++++++++++++++++++++++++++++\n>>> promised-blob.h                        |  27 ++++++\n>>> sha1-array.c                           |   7 ++\n>>> sha1-array.h                           |   1 +\n>>> sha1_file.c                            |  44 ++++++---\n>>> t/t3907-promised-blob.sh               |  65 +++++++++++++\n>>> t/test-lib-functions.sh                |   6 ++\n>>> 12 files changed, 345 insertions(+), 14 deletions(-)\n>>> create mode 100644 promised-blob.c\n>>> create mode 100644 promised-blob.h\n>>> create mode 100755 t/t3907-promised-blob.sh\n>>>\n>>> -- \n>>> 2.13.2.932.g7449e964c-goog\n>>>\n>> \n\n"},{"id":"325244","messageId":"8EE0108BA72B42EA9494B571DDE2005D@PhilipOakley","threadId":"46359","inReplyTo":"20170717180322.GM93855@aiede.mtv.corp.google.com","subject":"Re: [RFC PATCH 0/3] Partial clone: promised blobs (formerly \"missing blobs\")","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-07-29T12:51:16Z","receivedAt":"2017-07-29T12:51:22Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Jonathan Nieder\" <jrnieder@gmail.com>\nSent: Monday, July 17, 2017 7:03 PM\n> Hi Philip,\n>\n> Philip Oakley wrote:\n>> From: \"Jonathan Tan\" <jonathantanmy@google.com>\n>\n>>> These patches are part of a set of patches implementing partial clone,\n>>> as you can see here:\n>>>\n>>> https://github.com/jonathantanmy/git/tree/partialclone\n> [...]\n>> If I understand correctly, this method doesn't give any direct user\n>> visibility of missing blobs in the file system. Is that correct?\n>>\n>> I was hoping that eventually the various 'on demand' approaches\n>> would still allow users to continue to work as they go off-line such\n>> that they can see directly (in the FS) where the missing blobs (and\n>> trees) are located, so that they can continue to commit new work on\n>> existing files.\n>>\n>> I had felt that some sort of 'gitlink' should be present (huma\n>> readable) as a place holder for the missing blob/tree. e.g.\n>> 'gitblob: 1234abcd' (showing the missing oid, jsut like sub-modules\n>> can do - it's no different really.\n>\n> That's a reasonable thing to want, but it's a little different from\n> the use cases that partial clone work so far has aimed to support.\n> They are:\n>\n> A. Avoiding downloading all blobs (and likely trees as well) that are\n>    not needed in the current operation (e.g. checkout).  This blends\n>    well with the sparse checkout feature, which allows the current\n>    checkout to be fairly small in a large repository.\n\nTrue. In my case I was looking for a method that would allow a 'Narrow \nclone' such that the local repo would be smaller (have less content), but \nwould feel as if all the usefull files/directories were available, and there \nwould be place holders at the points where the trees were pruned, both in \nthe object store, and in the user's work-tree.\n\nAs you say, in some ways its conceptually orthogonal to the original sparse \nchecket (which has a full width object store / repo, and then omitted files \nfrom the checkout.\n>\n>    GVFS uses a trick that makes it a little easier to widen a sparse\n>    checkout upon access of a directory.  But the same building blocks\n>    should work fine with a sparse checkout that has been set up\n>    explicitly.\n>\n> B. Avoiding downloading large blobs, except for those needed in the\n>    current operation (e.g. checkout).\n>\n>    When not using sparse checkout, the main benefit out of the box is\n>    avoiding downloading *historical versions* of large blobs.\n>\n\n> It sounds like you are looking for a sort of placeholder outside the\n> sparse checkout area.\n\nTrue.\n\n> In a way, that's orthogonal to these patches:\n> even if you have all relevant blobs, you may want to avoid inflating\n> them to check them out and reading them to compare to the index (i.e.\n> the usual benefits of sparse checkout).\n\nIn my concept, it should be possible to create the ('sparse'/narrow) index \nfrom the content of the local object store, without any network connection \n(though that content is determined by the prior fetch/clone;-). The proper \ngit sparse checkout could proceed from there as being a further local \nrestriction on what is omitted from the worktree.\n\nThose missing from the narrow clone would still show as place holders with \ncontent \".gitnarrowtree 13a24b..<oid>\" (so we know what the hash oid of the \nfile/tree should be (so they can be moved/renamed etc!). The index would \nonly know the content/structure as far as the place holders (just like \nsub-modules are a break point in the tracking, with identical caveats)\n\n\nIt would be interesting to know from Ben the level of sparseness/narrowness \nhas been seen typically in the BigWin GVFS repo case.\n\n>  In a sparse checkout, you\n> still might like to be able to get a listing of files outside the\n> sparse area (which you can get with \"git ls-tree\") and you may even\n> want to be able to get such a listing with plain \"ls\" (as with your\n> proposal).\n>\n> Thanks and hope that helps,\n> Jonathan\n\nThanks, yes. It has help consolidate some of the parts of my concept that \nhas been in the back of my mind for a while now.\n\nPhilip \n\n"},{"id":"325245","messageId":"244AA0848E9D46F480E7CA407582A162@PhilipOakley","threadId":"46359","inReplyTo":"96e8a347-d6b5-e21e-4129-2957fc26db19@gmail.com","subject":"Re: [RFC PATCH v2 2/4] promised-object, fsck: introduce promised objects","fromName":"Philip Oakley","fromEmail":"philipoakley@iee.org","sentAt":"2017-07-29T13:26:52Z","receivedAt":"2017-07-29T13:26:59Z","isPatch":true,"sender":{"key":"philipoakley@iee.email","avatar":"https://avatars.githubusercontent.com/u/914343?v=4"},"body":"From: \"Ben Peart\" <peartben@gmail.com>\nSent: Tuesday, July 25, 2017 4:10 PM\n> On 7/21/2017 4:33 PM, Jonathan Tan wrote:\n>> On Fri, 21 Jul 2017 12:24:52 -0400\n>> Ben Peart <peartben@gmail.com> wrote:\n>>\n>>> Today we have 3.5 million objects * 30 bytes per entry = 105 MB of\n>>> promises. Given the average developer only hydrates 56K files (2 MB\n>>> promises) that is 103 MB to download that no one will ever need. We\n>>> would like to avoid that if possible as this would be a significant\n>>> regression in clone times from where we are today.\n>>>\n\nA question in the broader context of Narrow clones that I'd be interested \nin.\n\nHow narrow are the tree levels that contain the hydrated files? The question \nsplits a couple of ways:\n\nA. If one goes to the highest tree that contains all the 56K files, how many \nfiles and sub trees would be in that complete tree (i.e. what fraction of \nthat 'top' tree is inflated).\nA2. Is there some deep/wide metric that indicates how tightly together the \ninflated files tend to cluster?\n\nB. If instead we look at just the trees in the paths of those inflated \nfiles, those trees will also reference other trees/blobs that are not \ninflated, how big is that list (it would indicate the size of a narrow \nrepo's object store that holds the oid stubs)\n\nI would quess / expect that the typical inflation only has a few clustered \nareas of interest, but it maybe that in such a big reality (*) the inflated \nfiles are actually spread very widely. (*) as per various blog posts saying \nthere was no realistic way of partitioning the BigWin repo!\n\nI'd be interested in any such sparsity metric (apologies if I've missed \nprevious reports).\n--\nPhilip\n\n>>> I'm also concerned about the performance of merging in promises given we\n>>> have 100M objects today and growing so the number of promises over time\n>>> could get pretty large.\n>>\n>> After some thought, maybe a hybrid solution is best, in which it is\n>> permissible but optional for some missing objects to have promises. In\n>> that case, it is more of a \"size cache\" (which stores the type as well)\n>> rather than a true promise. When fetching, the client can optionally\n>> request for the sizes and types of missing objects.\n>>\n>\n> In our GVFS solution today we do not download any size or object type \n> information at clone as the number of objects and the resulting file would \n> be too large.  Instead, we have a new sizes endpoint \n> (https://github.com/Microsoft/GVFS/blob/master/Protocol.md) that enables \n> us to retrieve object sizes \"on demand\" much like we are enabling for the \n> actual object content.\n>\n> This protocol could easily be extended to return both size and type so \n> that it could be used to retrieve \"promise\" data for objects as they are \n> needed. Having a way to \"cache\" that data locally so that both git and \n> other code could share it would be great.\n>\n> At a minimum, we should ensure the data stream passed back is the same \n> whether at clone time or when hitting a \"promises\" end point. I think it \n> would also be helpful to enable promises to be downloaded on demand much \n> like we are doing for the object data itself.\n>\n>> This is good for the large-blob case, in which we can always have size\n>> information of missing blobs, and we can subsequently add blob-size\n>> filtering (as a parameter) to \"git log -S\" and friends to avoid needing\n>> to resolve a missing object. And this is, as far as I can tell, also\n>> good for the many-blob case - just have an empty size cache all the\n>> time. (And in the future, use cases could come up that desire non-empty\n>> but non-comprehensive caches - for example, a directory lister working\n>> on a partial clone that only needs to cache the sizes of frequently\n>> accessed directories.)\n>>\n>> Another option is to have a repo-wide option that toggles between\n>> mandatory entries in the \"size cache\" and prohibited entries. Switching\n>> to mandatory provides stricter fsck and negative lookups, but I think\n>> it's not worth it for both the developers and users of Git to have to\n>> know about these two modes.\n>>\n>>>>> I think we should have a flag (off by default) that enables someone to\n>>>>> say that promised objects are optional. If the flag is set,\n>>>>> \"is_promised_object\" will return success and pass the OBJ_ANY type and \n>>>>> a\n>>>>> size of -1.\n>>>>>\n>>>>> Nothing today is using the size and in the two places where the object\n>>>>> type is being checked for consistency (fsck_cache_tree and\n>>>>> fsck_handle_ref) the test can add a test for OBJ_ANY as well.\n>>>>>\n>>>>> This will enable very large numbers of objects to be omitted from the\n>>>>> clone without triggering a download of the corresponding number of\n>>>>> promised objects.\n>>>>\n>>>> Eventually I plan to use the size when implementing parameters for\n>>>> history-searching commands (e.g. \"git log -S\"), but it's true that\n>>>> that's in the future.\n>>>>\n>>>> Allowing promised objects to be optional would indeed solve the issue \n>>>> of\n>>>> downloading too many promises. It would make the code more complicated,\n>>>> but I'm not sure by how much.\n>>>>\n>>>> For example, in this fsck patch, the easiest way I could think of to\n>>>> have promised objects was to introduce a 3rd state, called \"promised\",\n>>>> of \"struct object\" - one in which the type is known, but we don't have\n>>>> access to the full \"struct commit\" or equivalent. And thus fsck could\n>>>> assume that if the \"struct object\" is \"parsed\" or \"promised\", the type\n>>>> is known. Having optional promised objects would require that we let\n>>>> this \"promised\" state have a type of OBJ_UNKNOWN (or something like\n>>>> that) - maybe that would be fine, but I haven't looked into this in\n>>>> detail.\n>>>>\n>>>\n>>> Caveats apply as I only did a quick look but I only found the two\n>>> locations that were checking the object type for consistency.\n>>\n>> I haven't looked into detail, but you are probably right.\n>> \n\n"}]}