{"thread":{"id":"47103","subject":"[PATCH 0/9] WIP Partial clone part 2: fsck and promisors","startedAt":"2017-11-02T20:21:13Z","lastAt":"2017-11-16T17:33:40Z","messageCount":19,"participants":["Jeff Hostetler","Jonathan Tan"],"isPatch":true,"patchVersion":1,"patchTotal":9},"messages":[{"id":"331700","messageId":"20171102202052.58762-1-git@jeffhostetler.com","threadId":"47103","inReplyTo":null,"subject":"[PATCH 0/9] WIP Partial clone part 2: fsck and promisors","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:20:43Z","receivedAt":"2017-11-02T20:21:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nThis is part 2 of a proposed 3 part sequence for partial clone.\nPart 2 assumes part 1 [1] is in place.\n\nPart 2 is concerned with fsck, gc, initial support for dynamic\nobject fetching, and tracking promisor objects. Jonathan Tan\noriginally developed this code. I have moved it on top of [1].\n\n[1] https://public-inbox.org/git/20171102124445.fbffd43521cd35f6a71e1851@google.com/T/\n[2] https://public-inbox.org/git/cover.1506714999.git.jonathantanmy@google.com/\n\n\nJeff Hostetler (1):\n extension.partialclone: introduce partial clone extension\n\nJonathan Tan (8):\n fsck: introduce partialclone extension\n fsck: support refs pointing to promisor objects\n fsck: support referenced promisor objects\n fsck: support promisor objects as CLI argument\n index-pack: refactor writing of .keep files\n introduce fetch-object: fetch one promisor object\n sha1_file: support lazily fetching missing objects\n gc: do not repack promisor packfiles\n\n Documentation/git-pack-objects.txt | 4 +\n Documentation/gitremote-helpers.txt | 2 +\n Documentation/technical/repository-version.txt | 22 ++\n Makefile | 2 +\n builtin/cat-file.c | 3 +\n builtin/fetch-pack.c | 10 +\n builtin/fsck.c | 26 +-\n builtin/gc.c | 4 +\n builtin/index-pack.c | 113 ++++----\n builtin/pack-objects.c | 14 +\n builtin/prune.c | 7 +\n builtin/repack.c | 12 +-\n builtin/rev-list.c | 35 ++-\n cache.h | 15 +-\n config.h | 3 +\n environment.c | 2 +\n fetch-object.c | 26 ++\n fetch-object.h | 6 +\n fetch-pack.c | 8 +-\n fetch-pack.h | 2 +\n list-objects.c | 8 +-\n object.c | 2 +-\n packfile.c | 78 +++++-\n packfile.h | 13 +\n partial-clone-utils.c | 78 ++++++\n partial-clone-utils.h | 34 +++\n remote-curl.c | 17 +-\n revision.c | 32 ++-\n revision.h | 5 +-\n setup.c | 12 +\n sha1_file.c | 39 ++-\n t/t0410-partial-clone.sh | 343 +++++++++++++++++++++++++\n transport.c | 8 +\n transport.h | 8 +\n 34 files changed, 917 insertions(+), 76 deletions(-)\n create mode 100644 fetch-object.c\n create mode 100644 fetch-object.h\n create mode 100644 partial-clone-utils.c\n create mode 100644 partial-clone-utils.h\n create mode 100755 t/t0410-partial-clone.sh\n\n-- \n2.9.3\n\n"},{"id":"331701","messageId":"20171102202052.58762-2-git@jeffhostetler.com","threadId":"47103","inReplyTo":"20171102202052.58762-1-git@jeffhostetler.com","subject":"[PATCH 1/9] extension.partialclone: introduce partial clone extension","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:20:44Z","receivedAt":"2017-11-02T20:21:17Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nIntroduce the ability to have missing objects in a repo. This\nfunctionality is guarded by new repository extension options:\n `extensions.partialcloneremote` and\n `extensions.partialclonefilter`.\n\nSee the update to Documentation/technical/repository-version.txt\nin this patch for more information.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n Documentation/technical/repository-version.txt | 22 ++++++++\n Makefile | 1 +\n cache.h | 4 ++\n config.h | 3 +\n environment.c | 2 +\n partial-clone-utils.c | 78 ++++++++++++++++++++++++++\n partial-clone-utils.h | 34 +++++++++++\n setup.c | 15 +++++\n 8 files changed, 159 insertions(+)\n create mode 100644 partial-clone-utils.c\n create mode 100644 partial-clone-utils.h\n\ndiff --git a/Documentation/technical/repository-version.txt b/Documentation/technical/repository-version.txt\nindex 00ad379..9d488db 100644\n--- a/Documentation/technical/repository-version.txt\n+++ b/Documentation/technical/repository-version.txt\n@@ -86,3 +86,25 @@ for testing format-1 compatibility.\n When the config key `extensions.preciousObjects` is set to `true`,\n objects in the repository MUST NOT be deleted (e.g., by `git-prune` or\n `git repack -d`).\n+\n+`partialcloneremote`\n+~~~~~~~~~~~~~~~~~~~~\n+\n+When the config key `extensions.partialcloneremote` is set, it indicates\n+that the repo was created with a partial clone (or later performed\n+a partial fetch) and that the remote may have omitted sending\n+certain unwanted objects. Such a remote is called a \"promisor remote\"\n+and it promises that all such omitted objects can be fetched from it\n+in the future.\n+\n+The value of this key is the name of the promisor remote.\n+\n+`partialclonefilter`\n+~~~~~~~~~~~~~~~~~~~~\n+\n+When the config key `extensions.partialclonefilter` is set, it gives\n+the initial filter expression used to create the partial clone.\n+This value becomed the default filter expression for subsequent\n+fetches (called \"partial fetches\") from the promisor remote. This\n+value may also be set by the first explicit partial fetch following a\n+normal clone.\ndiff --git a/Makefile b/Makefile\nindex ca378a4..12d141a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -838,6 +838,7 @@ LIB_OBJS += pack-write.o\n LIB_OBJS += pager.o\n LIB_OBJS += parse-options.o\n LIB_OBJS += parse-options-cb.o\n+LIB_OBJS += partial-clone-utils.o\n LIB_OBJS += patch-delta.o\n LIB_OBJS += patch-ids.o\n LIB_OBJS += path.o\ndiff --git a/cache.h b/cache.h\nindex 6440e2b..4b785c0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -860,12 +860,16 @@ extern int grafts_replace_parents;\n #define GIT_REPO_VERSION 0\n #define GIT_REPO_VERSION_READ 1\n extern int repository_format_precious_objects;\n+extern char *repository_format_partial_clone_remote;\n+extern char *repository_format_partial_clone_filter;\n \n struct repository_format {\n \tint version;\n \tint precious_objects;\n \tint is_bare;\n \tchar *work_tree;\n+\tchar *partial_clone_remote; /* value of extensions.partialcloneremote */\n+\tchar *partial_clone_filter; /* value of extensions.partialclonefilter */\n \tstruct string_list unknown_extensions;\n };\n \ndiff --git a/config.h b/config.h\nindex a49d264..90544ef 100644\n--- a/config.h\n+++ b/config.h\n@@ -34,6 +34,9 @@ struct config_options {\n \tconst char *git_dir;\n };\n \n+#define KEY_PARTIALCLONEREMOTE \"partialcloneremote\"\n+#define KEY_PARTIALCLONEFILTER \"partialclonefilter\"\n+\n typedef int (*config_fn_t)(const char *, const char *, void *);\n extern int git_default_config(const char *, const char *, void *);\n extern int git_config_from_file(config_fn_t fn, const char *, void *);\ndiff --git a/environment.c b/environment.c\nindex 8289c25..2fcf9bb 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -27,6 +27,8 @@ int warn_ambiguous_refs = 1;\n int warn_on_object_refname_ambiguity = 1;\n int ref_paranoia = -1;\n int repository_format_precious_objects;\n+char *repository_format_partial_clone_remote;\n+char *repository_format_partial_clone_filter;\n const char *git_commit_encoding;\n const char *git_log_output_encoding;\n const char *apply_default_whitespace;\ndiff --git a/partial-clone-utils.c b/partial-clone-utils.c\nnew file mode 100644\nindex 0000000..32cc20d\n--- /dev/null\n+++ b/partial-clone-utils.c\n@@ -0,0 +1,78 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"partial-clone-utils.h\"\n+\n+int is_partial_clone_registered(void)\n+{\n+\tif (repository_format_partial_clone_remote ||\n+\t repository_format_partial_clone_filter)\n+\t\treturn 1;\n+\n+\treturn 0;\n+}\n+\n+void partial_clone_utils_register(\n+\tconst struct list_objects_filter_options *filter_options,\n+\tconst char *remote,\n+\tconst char *cmd_name)\n+{\n+\tif (is_partial_clone_registered()) {\n+\t\t/*\n+\t\t * The original partial-clone or a previous partial-fetch\n+\t\t * already registered the partial-clone settings.\n+\t\t * If we get here, we are in a subsequent partial-* command\n+\t\t * (with explicit filter args on the command line).\n+\t\t *\n+\t\t * For now, we restrict subsequent commands to one\n+\t\t * consistent with the original request. We may relax\n+\t\t * this later after we get more experience with the\n+\t\t * partial-clone feature.\n+\t\t *\n+\t\t * [] Restrict to same remote because our dynamic\n+\t\t * object loading only knows how to fetch objects\n+\t\t * from 1 remote.\n+\t\t */\n+\t\tassert(filter_options && filter_options->choice);\n+\t\tassert(remote && *remote);\n+\n+\t\tif (strcmp(remote, repository_format_partial_clone_remote))\n+\t\t\tdie(\"%s --%s currently limited to remote '%s'\",\n+\t\t\t cmd_name, CL_ARG__FILTER,\n+\t\t\t repository_format_partial_clone_remote);\n+\n+\t\t/*\n+\t\t * Treat the (possibly new) filter-spec as transient;\n+\t\t * use it for the current command, but do not overwrite\n+\t\t * the default.\n+\t\t */\n+\t\treturn;\n+\t}\n+\n+\trepository_format_partial_clone_remote = xstrdup(remote);\n+\trepository_format_partial_clone_filter = xstrdup(filter_options->raw_value);\n+\n+\t/*\n+\t * Force repo version > 0 to enable extensions namespace.\n+\t *\n+\t * TODO if already set > 0, we should not overwrite it. \n+\t */\n+\tgit_config_set(\"core.repositoryformatversion\", \"1\");\n+\n+\t/*\n+\t * Use the \"extensions\" namespace in the config to record\n+\t * the name of the remote used in the partial clone.\n+\t * This will help us return to that server when we need\n+\t * to backfill missing objects.\n+\t *\n+\t * It is also used to indicate that there *MAY* be\n+\t * missing objects so that subsequent commands don't\n+\t * immediately die if they hit one.\n+\t *\n+\t * Also remember the initial filter settings used by\n+\t * clone as a default for future fetches.\n+\t */\n+\tgit_config_set(\"extensions.\" KEY_PARTIALCLONEREMOTE,\n+\t\t repository_format_partial_clone_remote);\n+\tgit_config_set(\"extensions.\" KEY_PARTIALCLONEFILTER,\n+\t\t repository_format_partial_clone_filter);\n+}\ndiff --git a/partial-clone-utils.h b/partial-clone-utils.h\nnew file mode 100644\nindex 0000000..b527570\n--- /dev/null\n+++ b/partial-clone-utils.h\n@@ -0,0 +1,34 @@\n+#ifndef PARTIAL_CLONE_UTILS_H\n+#define PARTIAL_CLONE_UTILS_H\n+\n+#include \"list-objects-filter-options.h\"\n+\n+/*\n+ * Register that partial-clone was used to create the repo and\n+ * update the config on disk.\n+ *\n+ * If nothing else, this indicates that the ODB may have missing\n+ * objects and that various commands should handle that gracefully.\n+ *\n+ * Record the remote used for the clone so that we know where\n+ * to get missing objects in the future.\n+ *\n+ * Also record the filter expression so that we know something\n+ * about the missing objects (e.g., size-limit vs sparse).\n+ *\n+ * May also be used by a partial-fetch following a normal clone\n+ * to turn on the above tracking.\n+ */ \n+extern void partial_clone_utils_register(\n+\tconst struct list_objects_filter_options *filter_options,\n+\tconst char *remote,\n+\tconst char *cmd_name);\n+\n+/*\n+ * Return 1 if partial-clone was used to create the repo\n+ * or a subsequent partial-fetch was used. This is an\n+ * indicator that there may be missing objects.\n+ */\n+extern int is_partial_clone_registered(void);\n+\n+#endif /* PARTIAL_CLONE_UTILS_H */\ndiff --git a/setup.c b/setup.c\nindex 03f51e0..bc4133d 100644\n--- a/setup.c\n+++ b/setup.c\n@@ -420,6 +420,19 @@ static int check_repo_format(const char *var, const char *value, void *vdata)\n \t\t\t;\n \t\telse if (!strcmp(ext, \"preciousobjects\"))\n \t\t\tdata->precious_objects = git_config_bool(var, value);\n+\n+\t\telse if (!strcmp(ext, KEY_PARTIALCLONEREMOTE))\n+\t\t\tif (!value)\n+\t\t\t\treturn config_error_nonbool(var);\n+\t\t\telse\n+\t\t\t\tdata->partial_clone_remote = xstrdup(value);\n+\n+\t\telse if (!strcmp(ext, KEY_PARTIALCLONEFILTER))\n+\t\t\tif (!value)\n+\t\t\t\treturn config_error_nonbool(var);\n+\t\t\telse\n+\t\t\t\tdata->partial_clone_filter = xstrdup(value);\n+\n \t\telse\n \t\t\tstring_list_append(&data->unknown_extensions, ext);\n \t} else if (strcmp(var, \"core.bare\") == 0) {\n@@ -463,6 +476,8 @@ static int check_repository_format_gently(const char *gitdir, int *nongit_ok)\n \t}\n \n \trepository_format_precious_objects = candidate.precious_objects;\n+\trepository_format_partial_clone_remote = candidate.partial_clone_remote;\n+\trepository_format_partial_clone_filter = candidate.partial_clone_filter;\n \tstring_list_clear(&candidate.unknown_extensions, 0);\n \tif (!has_common) {\n \t\tif (candidate.is_bare != -1) {\n-- \n2.9.3\n\n"},{"id":"331702","messageId":"20171102202052.58762-6-git@jeffhostetler.com","threadId":"47103","inReplyTo":"20171102202052.58762-1-git@jeffhostetler.com","subject":"[PATCH 5/9] fsck: support promisor objects as CLI argument","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:20:48Z","receivedAt":"2017-11-02T20:21:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nTeach fsck to not treat missing promisor objects provided on the CLI as\nan error when extensions.partialcloneremote is set.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n builtin/fsck.c | 2 ++\n t/t0410-partial-clone.sh | 13 +++++++++++++\n 2 files changed, 15 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 4c2a56d..578a7c8 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -750,6 +750,8 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \t\t\tstruct object *obj = lookup_object(oid.hash);\n \n \t\t\tif (!obj || !(obj->flags & HAS_OBJ)) {\n+\t\t\t\tif (is_promisor_object(&oid))\n+\t\t\t\t\tcontinue;\n \t\t\t\terror(\"%s: object missing\", oid_to_hex(&oid));\n \t\t\t\terrors_found |= ERROR_OBJECT;\n \t\t\t\tcontinue;\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex b1d404e..002e071 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -125,4 +125,17 @@ test_expect_success 'missing object, but promised, passes fsck' '\n \tgit -C repo fsck\n '\n \n+test_expect_success 'missing CLI object, but promised, passes fsck' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo my_commit &&\n+\n+\tA=$(git -C repo commit-tree -m a HEAD^{tree}) &&\n+\tpromise_and_delete \"$A\" &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C repo fsck \"$A\"\n+'\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"331703","messageId":"20171102202052.58762-9-git@jeffhostetler.com","threadId":"47103","inReplyTo":"20171102202052.58762-1-git@jeffhostetler.com","subject":"[PATCH 8/9] sha1_file: support lazily fetching missing objects","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:20:51Z","receivedAt":"2017-11-02T20:21:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nTeach sha1_file to fetch objects from the remote configured in\nextensions.partialcloneremote whenever an object is requested but missing.\n\nThe fetching of objects can be suppressed through a global variable.\nThis is used by fsck and index-pack.\n\nHowever, by default, such fetching is not suppressed. This is meant as a\ntemporary measure to ensure that all Git commands work in such a\nsituation. Future patches will update some commands to either tolerate\nmissing objects (without fetching them) or be more efficient in fetching\nthem.\n\nIn order to determine the code changes in sha1_file.c necessary, I\ninvestigated the following:\n (1) functions in sha1_file.c that take in a hash, without the user\n regarding how the object is stored (loose or packed)\n (2) functions in packfile.c (because I need to check callers that know\n about the loose/packed distinction and operate on both differently,\n and ensure that they can handle the concept of objects that are\n neither loose nor packed)\n\n(1) is handled by the modification to sha1_object_info_extended().\n\nFor (2), I looked at for_each_packed_object and others. For\nfor_each_packed_object, the callers either already work or are fixed in\nthis patch:\n - reachable - only to find recent objects\n - builtin/fsck - already knows about missing objects\n - builtin/cat-file - warning message added in this commit\n\nCallers of the other functions do not need to be changed:\n - parse_pack_index\n - http - indirectly from http_get_info_packs\n - find_pack_entry_one\n - this searches a single pack that is provided as an argument; the\n caller already knows (through other means) that the sought object\n is in a specific pack\n - find_sha1_pack\n - fast-import - appears to be an optimization to not store a file if\n it is already in a pack\n - http-walker - to search through a struct alt_base\n - http-push - to search through remote packs\n - has_sha1_pack\n - builtin/fsck - already knows about promisor objects\n - builtin/count-objects - informational purposes only (check if loose\n object is also packed)\n - builtin/prune-packed - check if object to be pruned is packed (if\n not, don't prune it)\n - revision - used to exclude packed objects if requested by user\n - diff - just for optimization\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n builtin/cat-file.c | 3 +\n builtin/fetch-pack.c | 2 +\n builtin/fsck.c | 3 +\n builtin/index-pack.c | 6 ++\n builtin/rev-list.c | 35 +++++++++--\n cache.h | 8 +++\n fetch-object.c | 3 +\n list-objects.c | 8 ++-\n object.c | 2 +-\n revision.c | 32 +++++++++-\n revision.h | 5 +-\n sha1_file.c | 39 ++++++++----\n t/t0410-partial-clone.sh | 152 +++++++++++++++++++++++++++++++++++++++++++++++\n 13 files changed, 277 insertions(+), 21 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex f5fa4fd..ba77b73 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -13,6 +13,7 @@\n #include \"tree-walk.h\"\n #include \"sha1-array.h\"\n #include \"packfile.h\"\n+#include \"partial-clone-utils.h\"\n \n struct batch_options {\n \tint enabled;\n@@ -475,6 +476,8 @@ static int batch_objects(struct batch_options *opt)\n \n \t\tfor_each_loose_object(batch_loose_object, &sa, 0);\n \t\tfor_each_packed_object(batch_packed_object, &sa, 0);\n+\t\tif (is_partial_clone_registered())\n+\t\t\twarning(\"This repository has partial clone enabled. Some objects may not be loaded.\");\n \n \t\tcb.opt = opt;\n \t\tcb.expand = &data;\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 9f303cf..9a7ebf6 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -53,6 +53,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tstruct oid_array shallow = OID_ARRAY_INIT;\n \tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n \n+\tfetch_if_missing = 0;\n+\n \tpacket_trace_identity(\"fetch-pack\");\n \n \tmemset(&args, 0, sizeof(args));\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 578a7c8..3b76c0e 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -678,6 +678,9 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \tint i;\n \tstruct alternate_object_database *alt;\n \n+\t/* fsck knows how to handle missing promisor objects */\n+\tfetch_if_missing = 0;\n+\n \terrors_found = 0;\n \tcheck_replace_refs = 0;\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 24c2f05..a0a35e6 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1657,6 +1657,12 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tunsigned foreign_nr = 1;\t/* zero is a \"good\" value, assume bad */\n \tint report_end_of_input = 0;\n \n+\t/*\n+\t * index-pack never needs to fetch missing objects, since it only\n+\t * accesses the repo to do hash collision checks\n+\t */\n+\tfetch_if_missing = 0;\n+\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage(index_pack_usage);\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex cc9fa40..21f0b4c 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -15,6 +15,7 @@\n #include \"progress.h\"\n #include \"reflog-walk.h\"\n #include \"oidset.h\"\n+#include \"packfile.h\"\n \n static const char rev_list_usage[] =\n \"git rev-list [OPTION] ... [ -- paths... ]\\n\"\n@@ -60,6 +61,7 @@ static unsigned progress_counter;\n static struct list_objects_filter_options filter_options;\n static struct oidset missing_objects;\n static struct oidset omitted_objects;\n+static int arg_exclude_promisor_objects;\n static int arg_print_missing;\n static int arg_print_omitted;\n static int arg_ignore_missing;\n@@ -190,27 +192,39 @@ static void finish_commit(struct commit *commit, void *data)\n \tfree_commit_buffer(commit);\n }\n \n-static void finish_object(struct object *obj, const char *name, void *cb_data)\n+static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n \tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n+\t\t/*\n+\t\t * If \"--exclude-promisor-objects\", we try to dynamically\n+\t\t * fetch missing objects. Whether we tried or not, we do\n+\t\t * not have the object. We can then either print, ignore,\n+\t\t * or conditionally ignore them.\n+\t\t */\n \t\tif (arg_print_missing) {\n \t\t\toidset_insert(&missing_objects, &obj->oid);\n-\t\t\treturn;\n+\t\t\treturn 1;\n \t\t}\n \t\tif (arg_ignore_missing)\n-\t\t\treturn;\n+\t\t\treturn 1;\n+\n+\t\tif (arg_exclude_promisor_objects &&\n+\t\t is_promisor_object(&obj->oid))\n+\t\t\treturn 1;\n \n \t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n \t}\n \tif (info->revs->verify_objects && !obj->parsed && obj->type != OBJ_COMMIT)\n \t\tparse_object(&obj->oid);\n+\treturn 0;\n }\n \n static void show_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tfinish_object(obj, name, cb_data);\n+\tif (finish_object(obj, name, cb_data))\n+\t\treturn;\n \tdisplay_progress(progress, ++progress_counter);\n \tif (info->flags & REV_LIST_QUIET)\n \t\treturn;\n@@ -307,6 +321,19 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\n+\t/*\n+\t * Scan the argument list before invoking setup_revisions(), so that we\n+\t * know if fetch_if_missing needs to be set to 0.\n+\t */\n+\tfor (i = 1; i < argc; i++) {\n+\t\tif (!strcmp(argv[i], \"--exclude-promisor-objects\")) {\n+\t\t\tfetch_if_missing = 0;\n+\t\t\targ_exclude_promisor_objects = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\n \targc = setup_revisions(argc, argv, &revs, NULL);\n \n \tmemset(&info, 0, sizeof(info));\ndiff --git a/cache.h b/cache.h\nindex 5f84103..360ff9e 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1729,6 +1729,14 @@ struct object_info {\n #define OBJECT_INFO_QUICK 8\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n \n+/*\n+ * Set this to 0 to prevent sha1_object_info_extended() from fetching missing\n+ * blobs. This has a difference only if is_partial_clone_registered() is true.\n+ *\n+ * Its default value is 1.\n+ */\n+extern int fetch_if_missing;\n+\n /* Dumb servers support */\n extern int update_server_info(int);\n \ndiff --git a/fetch-object.c b/fetch-object.c\nindex f89dbba..369b61c 100644\n--- a/fetch-object.c\n+++ b/fetch-object.c\n@@ -9,7 +9,9 @@ void fetch_object(const char *remote_name, const unsigned char *sha1)\n \tstruct remote *remote;\n \tstruct transport *transport;\n \tstruct ref *ref;\n+\tint original_fetch_if_missing = fetch_if_missing;\n \n+\tfetch_if_missing = 0;\n \tremote = remote_get(remote_name);\n \tif (!remote->url[0])\n \t\tdie(_(\"Remote with no URL\"));\n@@ -20,4 +22,5 @@ void fetch_object(const char *remote_name, const unsigned char *sha1)\n \ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n \ttransport_set_option(transport, TRANS_OPT_NO_HAVES, \"1\");\n \ttransport_fetch_refs(transport, ref);\n+\tfetch_if_missing = original_fetch_if_missing;\n }\ndiff --git a/list-objects.c b/list-objects.c\nindex 848b040..5390a74 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -9,6 +9,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"packfile.h\"\n \n static void process_blob(struct rev_info *revs,\n \t\t\t struct blob *blob,\n@@ -90,6 +91,8 @@ static void process_tree(struct rev_info *revs,\n \tenum interesting match = revs->diffopt.pathspec.nr == 0 ?\n \t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n+\tint gently = revs->ignore_missing_links ||\n+\t\trevs->exclude_promisor_objects;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_SHOW;\n \n \tif (!revs->tree_objects)\n@@ -98,9 +101,12 @@ static void process_tree(struct rev_info *revs,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, revs->ignore_missing_links) < 0) {\n+\tif (parse_tree_gently(tree, gently) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n+\t\tif (revs->exclude_promisor_objects &&\n+\t\t is_promisor_object(&obj->oid))\n+\t\t\treturn;\n \t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \ndiff --git a/object.c b/object.c\nindex b9a4a0e..4c222d6 100644\n--- a/object.c\n+++ b/object.c\n@@ -252,7 +252,7 @@ struct object *parse_object(const struct object_id *oid)\n \tif (obj && obj->parsed)\n \t\treturn obj;\n \n-\tif ((obj && obj->type == OBJ_BLOB) ||\n+\tif ((obj && obj->type == OBJ_BLOB && has_object_file(oid)) ||\n \t (!obj && has_object_file(oid) &&\n \t sha1_object_info(oid->hash, NULL) == OBJ_BLOB)) {\n \t\tif (check_sha1_signature(repl, NULL, 0, NULL) < 0) {\ndiff --git a/revision.c b/revision.c\nindex d167223..85265a4 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -198,6 +198,8 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n \tif (!object) {\n \t\tif (revs->ignore_missing)\n \t\t\treturn object;\n+\t\tif (revs->exclude_promisor_objects && is_promisor_object(oid))\n+\t\t\treturn NULL;\n \t\tdie(\"bad object %s\", name);\n \t}\n \tobject->flags |= flags;\n@@ -791,8 +793,17 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \tfor (parent = commit->parents; parent; parent = parent->next) {\n \t\tstruct commit *p = parent->item;\n \n-\t\tif (parse_commit_gently(p, revs->ignore_missing_links) < 0)\n+\t\tint gently = revs->ignore_missing_links ||\n+\t\t\trevs->exclude_promisor_objects;\n+\t\tif (parse_commit_gently(p, gently) < 0) {\n+\t\t\tif (revs->exclude_promisor_objects &&\n+\t\t\t is_promisor_object(&p->object.oid)) {\n+\t\t\t\tif (revs->first_parent_only)\n+\t\t\t\t\tbreak;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\treturn -1;\n+\t\t}\n \t\tif (revs->show_source && !p->util)\n \t\t\tp->util = commit->util;\n \t\tp->object.flags |= left_flag;\n@@ -2088,6 +2099,10 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t\trevs->limited = 1;\n \t} else if (!strcmp(arg, \"--ignore-missing\")) {\n \t\trevs->ignore_missing = 1;\n+\t}else if (!strcmp(arg, \"--exclude-promisor-objects\")) {\n+\t\tif (fetch_if_missing)\n+\t\t\tdie(\"BUG: --exclude-promisor-objects can only be used when fetch_if_missing is 0\");\n+\t\trevs->exclude_promisor_objects = 1;\n \t} else {\n \t\tint opts = diff_opt_parse(&revs->diffopt, argv, argc, revs->prefix);\n \t\tif (!opts)\n@@ -2830,6 +2845,16 @@ void reset_revision_walk(void)\n \tclear_object_flags(SEEN | ADDED | SHOWN);\n }\n \n+static int mark_uninteresting(const struct object_id *oid,\n+\t\t\t struct packed_git *pack,\n+\t\t\t uint32_t pos,\n+\t\t\t void *unused)\n+{\n+\tstruct object *o = parse_object(oid);\n+\to->flags |= UNINTERESTING | SEEN;\n+\treturn 0;\n+}\n+\n int prepare_revision_walk(struct rev_info *revs)\n {\n \tint i;\n@@ -2858,6 +2883,11 @@ int prepare_revision_walk(struct rev_info *revs)\n \t (revs->limited && limiting_can_increase_treesame(revs)))\n \t\trevs->treesame.name = \"treesame\";\n \n+\tif (revs->exclude_promisor_objects) {\n+\t\tfor_each_packed_object(mark_uninteresting, NULL,\n+\t\t\t\t FOR_EACH_OBJECT_PROMISOR_ONLY);\n+\t}\n+\n \tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\ndiff --git a/revision.h b/revision.h\nindex 5476120..5f9a49c 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -121,7 +121,10 @@ struct rev_info {\n \t\t\tbisect:1,\n \t\t\tancestry_path:1,\n \t\t\tfirst_parent_only:1,\n-\t\t\tline_level_traverse:1;\n+\t\t\tline_level_traverse:1,\n+\n+\t\t\t/* for internal use only */\n+\t\t\texclude_promisor_objects:1;\n \n \t/* Diff flags */\n \tunsigned int\tdiff:1,\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 10c3a00..88e92aa 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -29,6 +29,8 @@\n #include \"mergesort.h\"\n #include \"quote.h\"\n #include \"packfile.h\"\n+#include \"fetch-object.h\"\n+#include \"partial-clone-utils.h\"\n \n const unsigned char null_sha1[GIT_MAX_RAWSZ];\n const struct object_id null_oid;\n@@ -1144,6 +1146,8 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \treturn (status < 0) ? status : 0;\n }\n \n+int fetch_if_missing = 1;\n+\n int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n {\n \tstatic struct object_info blank_oi = OBJECT_INFO_INIT;\n@@ -1152,6 +1156,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n \t\t\t\t lookup_replace_object(sha1) :\n \t\t\t\t sha1;\n+\tint already_retried = 0;\n \n \tif (!oi)\n \t\toi = &blank_oi;\n@@ -1176,28 +1181,36 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n-\tif (!find_pack_entry(real, &e)) {\n-\t\t/* Most likely it's a loose object. */\n-\t\tif (!sha1_loose_object_info(real, oi, flags))\n-\t\t\treturn 0;\n+retry:\n+\tif (find_pack_entry(real, &e))\n+\t\tgoto found_packed;\n \n-\t\t/* Not a loose object; someone else may have just packed it. */\n-\t\tif (flags & OBJECT_INFO_QUICK) {\n-\t\t\treturn -1;\n-\t\t} else {\n-\t\t\treprepare_packed_git();\n-\t\t\tif (!find_pack_entry(real, &e))\n-\t\t\t\treturn -1;\n-\t\t}\n+\t/* Most likely it's a loose object. */\n+\tif (!sha1_loose_object_info(real, oi, flags))\n+\t\treturn 0;\n+\n+\t/* Not a loose object; someone else may have just packed it. */\n+\treprepare_packed_git();\n+\tif (find_pack_entry(real, &e))\n+\t\tgoto found_packed;\n+\n+\t/* Check if it is a missing object */\n+\tif (fetch_if_missing && repository_format_partial_clone_remote &&\n+\t !already_retried) {\n+\t\tfetch_object(repository_format_partial_clone_remote, real);\n+\t\talready_retried = 1;\n+\t\tgoto retry;\n \t}\n \n+\treturn -1;\n+\n+found_packed:\n \tif (oi == &blank_oi)\n \t\t/*\n \t\t * We know that the caller doesn't actually need the\n \t\t * information below, so return early.\n \t\t */\n \t\treturn 0;\n-\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 002e071..59de768 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -138,4 +138,156 @@ test_expect_success 'missing CLI object, but promised, passes fsck' '\n \tgit -C repo fsck \"$A\"\n '\n \n+test_expect_success 'fetching of missing objects' '\n+\trm -rf repo &&\n+\ttest_create_repo server &&\n+\ttest_commit -C server foo &&\n+\tgit -C server repack -a -d --write-bitmap-index &&\n+\n+\tgit clone \"file://$(pwd)/server\" repo &&\n+\tHASH=$(git -C repo rev-parse foo) &&\n+\trm -rf repo/.git/objects/* &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialcloneremote \"origin\" &&\n+\tgit -C repo cat-file -p \"$HASH\" &&\n+\n+\t# Ensure that the .promisor file is written, and check that its\n+\t# associated packfile contains the object\n+\tls repo/.git/objects/pack/pack-*.promisor >promisorlist &&\n+\ttest_line_count = 1 promisorlist &&\n+\tIDX=$(cat promisorlist | sed \"s/promisor$/idx/\") &&\n+\tgit verify-pack --verbose \"$IDX\" | grep \"$HASH\"\n+'\n+\n+test_expect_success 'rev-list stops traversal at missing and promised commit' '\n+\t\t rm -rf repo &&\n+\t\t test_create_repo repo &&\n+\t\t test_commit -C repo foo &&\n+\t\t test_commit -C repo bar &&\n+\n+\tFOO=$(git -C repo rev-parse foo) &&\n+\tpromise_and_delete \"$FOO\" &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects bar >out &&\n+\tgrep $(git -C repo rev-parse bar) out &&\n+\t! grep $FOO out\n+'\n+\n+test_expect_success 'rev-list stops traversal at missing and promised tree' '\n+\t\t rm -rf repo &&\n+\t\t test_create_repo repo &&\n+\t\t test_commit -C repo foo &&\n+\t\t mkdir repo/a_dir &&\n+\t\t echo something >repo/a_dir/something &&\n+\t\t git -C repo add a_dir/something &&\n+\t\t git -C repo commit -m bar &&\n+\n+\t# foo^{tree} (tree referenced from commit)\n+\tTREE=$(git -C repo rev-parse foo^{tree}) &&\n+\n+\t# a tree referenced by HEAD^{tree} (tree referenced from tree)\n+\tTREE2=$(git -C repo ls-tree HEAD^{tree} | grep \" tree \" | head -1 | cut -b13-52) &&\n+\n+\tpromise_and_delete \"$TREE\" &&\n+\tpromise_and_delete \"$TREE2\" &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n+\tgrep $(git -C repo rev-parse foo) out &&\n+\t! grep $TREE out &&\n+\tgrep $(git -C repo rev-parse HEAD) out &&\n+\t! grep $TREE2 out\n+'\n+\n+test_expect_success 'rev-list stops traversal at missing and promised blob' '\n+\t\t rm -rf repo &&\n+\t\t test_create_repo repo &&\n+\t\t echo something >repo/something &&\n+\t\t git -C repo add something &&\n+\t\t git -C repo commit -m foo &&\n+\n+\tBLOB=$(git -C repo hash-object -w something) &&\n+\tpromise_and_delete \"$BLOB\" &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n+\tgrep $(git -C repo rev-parse HEAD) out &&\n+\t! grep $BLOB out\n+'\n+\n+test_expect_success 'rev-list stops traversal at promisor commit, tree, and blob' '\n+\t\t rm -rf repo &&\n+\t\t test_create_repo repo &&\n+\t\t test_commit -C repo foo &&\n+\t\t test_commit -C repo bar &&\n+\t\t test_commit -C repo baz &&\n+\n+\tCOMMIT=$(git -C repo rev-parse foo) &&\n+\tTREE=$(git -C repo rev-parse bar^{tree}) &&\n+\tBLOB=$(git hash-object repo/baz.t) &&\n+\tprintf \"%s\\n%s\\n%s\\n\" $COMMIT $TREE $BLOB | pack_as_from_promisor &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n+\t! grep $COMMIT out &&\n+\t! grep $TREE out &&\n+\t! grep $BLOB out &&\n+\tgrep $(git -C repo rev-parse bar) out # sanity check that some walking was done\n+'\n+\n+test_expect_success 'rev-list accepts missing and promised objects on command line' '\n+\t\t rm -rf repo &&\n+\t\t test_create_repo repo &&\n+\t\t test_commit -C repo foo &&\n+\t\t test_commit -C repo bar &&\n+\t\t test_commit -C repo baz &&\n+\n+\tCOMMIT=$(git -C repo rev-parse foo) &&\n+\tTREE=$(git -C repo rev-parse bar^{tree}) &&\n+\tBLOB=$(git hash-object repo/baz.t) &&\n+\n+\tpromise_and_delete $COMMIT &&\n+\tpromise_and_delete $TREE &&\n+\tpromise_and_delete $BLOB &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects \"$COMMIT\" \"$TREE\" \"$BLOB\"\n+'\n+\n+LIB_HTTPD_PORT=12345 # default port, 410, cannot be used as non-root\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'fetching of missing objects from an HTTP server' '\n+\trm -rf repo &&\n+\tSERVER=\"$HTTPD_DOCUMENT_ROOT_PATH/server\" &&\n+\ttest_create_repo \"$SERVER\" &&\n+\ttest_commit -C \"$SERVER\" foo &&\n+\tgit -C \"$SERVER\" repack -a -d --write-bitmap-index &&\n+\n+\tgit clone $HTTPD_URL/smart/server repo &&\n+\tHASH=$(git -C repo rev-parse foo) &&\n+\trm -rf repo/.git/objects/* &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialcloneremoteremote \"origin\" &&\n+\tgit -C repo cat-file -p \"$HASH\" &&\n+\n+\t# Ensure that the .promisor file is written, and check that its\n+\t# associated packfile contains the object\n+\tls repo/.git/objects/pack/pack-*.promisor >promisorlist &&\n+\ttest_line_count = 1 promisorlist &&\n+\tIDX=$(cat promisorlist | sed \"s/promisor$/idx/\") &&\n+\tgit verify-pack --verbose \"$IDX\" | grep \"$HASH\"\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"331704","messageId":"20171102202052.58762-4-git@jeffhostetler.com","threadId":"47103","inReplyTo":"20171102202052.58762-1-git@jeffhostetler.com","subject":"[PATCH 3/9] fsck: support refs pointing to promisor objects","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:20:46Z","receivedAt":"2017-11-02T20:21:26Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nTeach fsck to not treat refs referring to missing promisor objects as an\nerror when extensions.partialclone is set.\n\nFor the purposes of warning about no default refs, such refs are still\ntreated as legitimate refs.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n builtin/fsck.c | 8 ++++++++\n t/t0410-partial-clone.sh | 24 ++++++++++++++++++++++++\n 2 files changed, 32 insertions(+)\n\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 2934299..ee937bb 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -434,6 +434,14 @@ static int fsck_handle_ref(const char *refname, const struct object_id *oid,\n \n \tobj = parse_object(oid);\n \tif (!obj) {\n+\t\tif (is_promisor_object(oid)) {\n+\t\t\t/*\n+\t\t\t * Increment default_refs anyway, because this is a\n+\t\t\t * valid ref.\n+\t\t\t */\n+\t\t\t default_refs++;\n+\t\t\t return 0;\n+\t\t}\n \t\terror(\"%s: invalid sha1 pointer %s\", refname, oid_to_hex(oid));\n \t\terrors_found |= ERROR_REACHABLE;\n \t\t/* We'll continue with the rest despite the error.. */\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 52347fb..5a03ead 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -13,6 +13,14 @@ pack_as_from_promisor () {\n \t>repo/.git/objects/pack/pack-$HASH.promisor\n }\n \n+promise_and_delete () {\n+\tHASH=$(git -C repo rev-parse \"$1\") &&\n+\tgit -C repo tag -a -m message my_annotated_tag \"$HASH\" &&\n+\tgit -C repo rev-parse my_annotated_tag | pack_as_from_promisor &&\n+\tgit -C repo tag -d my_annotated_tag &&\n+\tdelete_object repo \"$HASH\"\n+}\n+\n test_expect_success 'missing reflog object, but promised by a commit, passes fsck' '\n \ttest_create_repo repo &&\n \ttest_commit -C repo my_commit &&\n@@ -78,4 +86,20 @@ test_expect_success 'missing reflog object alone fails fsck, even with extension\n \ttest_must_fail git -C repo fsck\n '\n \n+test_expect_success 'missing ref object, but promised, passes fsck' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo my_commit &&\n+\n+\tA=$(git -C repo commit-tree -m a HEAD^{tree}) &&\n+\n+\t# Reference $A only from ref\n+\tgit -C repo branch my_branch \"$A\" &&\n+\tpromise_and_delete \"$A\" &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C repo fsck\n+'\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"331705","messageId":"20171102202052.58762-7-git@jeffhostetler.com","threadId":"47103","inReplyTo":"20171102202052.58762-1-git@jeffhostetler.com","subject":"[PATCH 6/9] index-pack: refactor writing of .keep files","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:20:49Z","receivedAt":"2017-11-02T20:21:28Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nIn a subsequent commit, index-pack will be taught to write \".promisor\"\nfiles which are similar to the \".keep\" files it knows how to write.\nRefactor the writing of \".keep\" files, so that the implementation of\nwriting \".promisor\" files becomes easier.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n builtin/index-pack.c | 99 ++++++++++++++++++++++++++++------------------------\n 1 file changed, 53 insertions(+), 46 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 8ec459f..4f305a7 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1389,15 +1389,58 @@ static void fix_unresolved_deltas(struct sha1file *f)\n \tfree(sorted_by_pos);\n }\n \n+static const char *derive_filename(const char *pack_name, const char *suffix,\n+\t\t\t\t struct strbuf *buf)\n+{\n+\tsize_t len;\n+\tif (!strip_suffix(pack_name, \".pack\", &len))\n+\t\tdie(_(\"packfile name '%s' does not end with '.pack'\"),\n+\t\t pack_name);\n+\tstrbuf_add(buf, pack_name, len);\n+\tstrbuf_addch(buf, '.');\n+\tstrbuf_addstr(buf, suffix);\n+\treturn buf->buf;\n+}\n+\n+static void write_special_file(const char *suffix, const char *msg,\n+\t\t\t const char *pack_name, const unsigned char *sha1,\n+\t\t\t const char **report)\n+{\n+\tstruct strbuf name_buf = STRBUF_INIT;\n+\tconst char *filename;\n+\tint fd;\n+\tint msg_len = strlen(msg);\n+\n+\tif (pack_name)\n+\t\tfilename = derive_filename(pack_name, suffix, &name_buf);\n+\telse\n+\t\tfilename = odb_pack_name(&name_buf, sha1, suffix);\n+\n+\tfd = odb_pack_keep(filename);\n+\tif (fd < 0) {\n+\t\tif (errno != EEXIST)\n+\t\t\tdie_errno(_(\"cannot write %s file '%s'\"),\n+\t\t\t\t suffix, filename);\n+\t} else {\n+\t\tif (msg_len > 0) {\n+\t\t\twrite_or_die(fd, msg, msg_len);\n+\t\t\twrite_or_die(fd, \"\\n\", 1);\n+\t\t}\n+\t\tif (close(fd) != 0)\n+\t\t\tdie_errno(_(\"cannot close written %s file '%s'\"),\n+\t\t\t\t suffix, filename);\n+\t\t*report = suffix;\n+\t}\n+\tstrbuf_release(&name_buf);\n+}\n+\n static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t const char *final_index_name, const char *curr_index_name,\n-\t\t const char *keep_name, const char *keep_msg,\n-\t\t unsigned char *sha1)\n+\t\t const char *keep_msg, unsigned char *sha1)\n {\n \tconst char *report = \"pack\";\n \tstruct strbuf pack_name = STRBUF_INIT;\n \tstruct strbuf index_name = STRBUF_INIT;\n-\tstruct strbuf keep_name_buf = STRBUF_INIT;\n \tint err;\n \n \tif (!from_stdin) {\n@@ -1409,28 +1452,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t\tdie_errno(_(\"error while closing pack file\"));\n \t}\n \n-\tif (keep_msg) {\n-\t\tint keep_fd, keep_msg_len = strlen(keep_msg);\n-\n-\t\tif (!keep_name)\n-\t\t\tkeep_name = odb_pack_name(&keep_name_buf, sha1, \"keep\");\n-\n-\t\tkeep_fd = odb_pack_keep(keep_name);\n-\t\tif (keep_fd < 0) {\n-\t\t\tif (errno != EEXIST)\n-\t\t\t\tdie_errno(_(\"cannot write keep file '%s'\"),\n-\t\t\t\t\t keep_name);\n-\t\t} else {\n-\t\t\tif (keep_msg_len > 0) {\n-\t\t\t\twrite_or_die(keep_fd, keep_msg, keep_msg_len);\n-\t\t\t\twrite_or_die(keep_fd, \"\\n\", 1);\n-\t\t\t}\n-\t\t\tif (close(keep_fd) != 0)\n-\t\t\t\tdie_errno(_(\"cannot close written keep file '%s'\"),\n-\t\t\t\t\t keep_name);\n-\t\t\treport = \"keep\";\n-\t\t}\n-\t}\n+\tif (keep_msg)\n+\t\twrite_special_file(\"keep\", keep_msg, final_pack_name, sha1,\n+\t\t\t\t &report);\n \n \tif (final_pack_name != curr_pack_name) {\n \t\tif (!final_pack_name)\n@@ -1472,7 +1496,6 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \n \tstrbuf_release(&index_name);\n \tstrbuf_release(&pack_name);\n-\tstrbuf_release(&keep_name_buf);\n }\n \n static int git_index_pack_config(const char *k, const char *v, void *cb)\n@@ -1615,26 +1638,13 @@ static void show_pack_info(int stat_only)\n \t}\n }\n \n-static const char *derive_filename(const char *pack_name, const char *suffix,\n-\t\t\t\t struct strbuf *buf)\n-{\n-\tsize_t len;\n-\tif (!strip_suffix(pack_name, \".pack\", &len))\n-\t\tdie(_(\"packfile name '%s' does not end with '.pack'\"),\n-\t\t pack_name);\n-\tstrbuf_add(buf, pack_name, len);\n-\tstrbuf_addstr(buf, suffix);\n-\treturn buf->buf;\n-}\n-\n int cmd_index_pack(int argc, const char **argv, const char *prefix)\n {\n \tint i, fix_thin_pack = 0, verify = 0, stat_only = 0;\n \tconst char *curr_index;\n \tconst char *index_name = NULL, *pack_name = NULL;\n-\tconst char *keep_name = NULL, *keep_msg = NULL;\n-\tstruct strbuf index_name_buf = STRBUF_INIT,\n-\t\t keep_name_buf = STRBUF_INIT;\n+\tconst char *keep_msg = NULL;\n+\tstruct strbuf index_name_buf = STRBUF_INIT;\n \tstruct pack_idx_entry **idx_objects;\n \tstruct pack_idx_option opts;\n \tunsigned char pack_sha1[20];\n@@ -1745,9 +1755,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (from_stdin && !startup_info->have_repository)\n \t\tdie(_(\"--stdin requires a git repository\"));\n \tif (!index_name && pack_name)\n-\t\tindex_name = derive_filename(pack_name, \".idx\", &index_name_buf);\n-\tif (keep_msg && !keep_name && pack_name)\n-\t\tkeep_name = derive_filename(pack_name, \".keep\", &keep_name_buf);\n+\t\tindex_name = derive_filename(pack_name, \"idx\", &index_name_buf);\n \n \tif (verify) {\n \t\tif (!index_name)\n@@ -1795,13 +1803,12 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (!verify)\n \t\tfinal(pack_name, curr_pack,\n \t\t index_name, curr_index,\n-\t\t keep_name, keep_msg,\n+\t\t keep_msg,\n \t\t pack_sha1);\n \telse\n \t\tclose(input_fd);\n \tfree(objects);\n \tstrbuf_release(&index_name_buf);\n-\tstrbuf_release(&keep_name_buf);\n \tif (pack_name == NULL)\n \t\tfree((void *) curr_pack);\n \tif (index_name == NULL)\n-- \n2.9.3\n\n"},{"id":"331706","messageId":"20171102202052.58762-8-git@jeffhostetler.com","threadId":"47103","inReplyTo":"20171102202052.58762-1-git@jeffhostetler.com","subject":"[PATCH 7/9] introduce fetch-object: fetch one promisor object","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:20:50Z","receivedAt":"2017-11-02T20:21:30Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nIntroduce fetch-object, providing the ability to fetch one object from a\npromisor remote.\n\nThis uses fetch-pack. To do this, the transport mechanism has been\nupdated with 2 flags, \"from-promisor\" to indicate that the resulting\npack comes from a promisor remote (and thus should be annotated as such\nby index-pack), and \"no-haves\" to suppress the sending of \"have\" lines.\n\nThis will be tested in a subsequent commit.\n\nNEEDSWORK: update this when we have more information about protocol v2,\nwhich should allow a way to suppress the ref advertisement and\nofficially allow any object type to be \"want\"-ed.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n Documentation/gitremote-helpers.txt | 2 ++\n Makefile | 1 +\n builtin/fetch-pack.c | 8 ++++++++\n builtin/index-pack.c | 16 +++++++++++++---\n fetch-object.c | 23 +++++++++++++++++++++++\n fetch-object.h | 6 ++++++\n fetch-pack.c | 8 ++++++--\n fetch-pack.h | 2 ++\n remote-curl.c | 17 ++++++++++++++++-\n transport.c | 8 ++++++++\n transport.h | 8 ++++++++\n 11 files changed, 93 insertions(+), 6 deletions(-)\n create mode 100644 fetch-object.c\n create mode 100644 fetch-object.h\n\ndiff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt\nindex 4a584f3..6da3f41 100644\n--- a/Documentation/gitremote-helpers.txt\n+++ b/Documentation/gitremote-helpers.txt\n@@ -466,6 +466,8 @@ set by Git if the remote helper has the 'option' capability.\n \tTransmit as a push option. As the push option\n \tmust not contain LF or NUL characters, the string is not encoded.\n \n+TODO document 'option from-promisor' and 'option no-haves' ?\n+\n SEE ALSO\n --------\n linkgit:git-remote[1]\ndiff --git a/Makefile b/Makefile\nindex 12d141a..7a0679a 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -792,6 +792,7 @@ LIB_OBJS += ewah/ewah_bitmap.o\n LIB_OBJS += ewah/ewah_io.o\n LIB_OBJS += ewah/ewah_rlw.o\n LIB_OBJS += exec_cmd.o\n+LIB_OBJS += fetch-object.o\n LIB_OBJS += fetch-pack.o\n LIB_OBJS += fsck.o\n LIB_OBJS += gettext.o\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 366b9d1..9f303cf 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -143,6 +143,14 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.update_shallow = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(\"--from-promisor\", arg)) {\n+\t\t\targs.from_promisor = 1;\n+\t\t\tcontinue;\n+\t\t}\n+\t\tif (!strcmp(\"--no-haves\", arg)) {\n+\t\t\targs.no_haves = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \tif (deepen_not.nr)\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 4f305a7..24c2f05 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1429,14 +1429,16 @@ static void write_special_file(const char *suffix, const char *msg,\n \t\tif (close(fd) != 0)\n \t\t\tdie_errno(_(\"cannot close written %s file '%s'\"),\n \t\t\t\t suffix, filename);\n-\t\t*report = suffix;\n+\t\tif (report)\n+\t\t\t*report = suffix;\n \t}\n \tstrbuf_release(&name_buf);\n }\n \n static void final(const char *final_pack_name, const char *curr_pack_name,\n \t\t const char *final_index_name, const char *curr_index_name,\n-\t\t const char *keep_msg, unsigned char *sha1)\n+\t\t const char *keep_msg, const char *promisor_msg,\n+\t\t unsigned char *sha1)\n {\n \tconst char *report = \"pack\";\n \tstruct strbuf pack_name = STRBUF_INIT;\n@@ -1455,6 +1457,9 @@ static void final(const char *final_pack_name, const char *curr_pack_name,\n \tif (keep_msg)\n \t\twrite_special_file(\"keep\", keep_msg, final_pack_name, sha1,\n \t\t\t\t &report);\n+\tif (promisor_msg)\n+\t\twrite_special_file(\"promisor\", promisor_msg, final_pack_name,\n+\t\t\t\t sha1, NULL);\n \n \tif (final_pack_name != curr_pack_name) {\n \t\tif (!final_pack_name)\n@@ -1644,6 +1649,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tconst char *curr_index;\n \tconst char *index_name = NULL, *pack_name = NULL;\n \tconst char *keep_msg = NULL;\n+\tconst char *promisor_msg = NULL;\n \tstruct strbuf index_name_buf = STRBUF_INIT;\n \tstruct pack_idx_entry **idx_objects;\n \tstruct pack_idx_option opts;\n@@ -1693,6 +1699,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tkeep_msg = \"\";\n \t\t\t} else if (starts_with(arg, \"--keep=\")) {\n \t\t\t\tkeep_msg = arg + 7;\n+\t\t\t} else if (!strcmp(arg, \"--promisor\")) {\n+\t\t\t\tpromisor_msg = \"\";\n+\t\t\t} else if (starts_with(arg, \"--promisor=\")) {\n+\t\t\t\tpromisor_msg = arg + strlen(\"--promisor=\");\n \t\t\t} else if (starts_with(arg, \"--threads=\")) {\n \t\t\t\tchar *end;\n \t\t\t\tnr_threads = strtoul(arg+10, &end, 0);\n@@ -1803,7 +1813,7 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tif (!verify)\n \t\tfinal(pack_name, curr_pack,\n \t\t index_name, curr_index,\n-\t\t keep_msg,\n+\t\t keep_msg, promisor_msg,\n \t\t pack_sha1);\n \telse\n \t\tclose(input_fd);\ndiff --git a/fetch-object.c b/fetch-object.c\nnew file mode 100644\nindex 0000000..f89dbba\n--- /dev/null\n+++ b/fetch-object.c\n@@ -0,0 +1,23 @@\n+#include \"cache.h\"\n+#include \"packfile.h\"\n+#include \"pkt-line.h\"\n+#include \"strbuf.h\"\n+#include \"transport.h\"\n+\n+void fetch_object(const char *remote_name, const unsigned char *sha1)\n+{\n+\tstruct remote *remote;\n+\tstruct transport *transport;\n+\tstruct ref *ref;\n+\n+\tremote = remote_get(remote_name);\n+\tif (!remote->url[0])\n+\t\tdie(_(\"Remote with no URL\"));\n+\ttransport = transport_get(remote, remote->url[0]);\n+\n+\tref = alloc_ref(sha1_to_hex(sha1));\n+\thashcpy(ref->old_oid.hash, sha1);\n+\ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n+\ttransport_set_option(transport, TRANS_OPT_NO_HAVES, \"1\");\n+\ttransport_fetch_refs(transport, ref);\n+}\ndiff --git a/fetch-object.h b/fetch-object.h\nnew file mode 100644\nindex 0000000..f371300\n--- /dev/null\n+++ b/fetch-object.h\n@@ -0,0 +1,6 @@\n+#ifndef FETCH_OBJECT_H\n+#define FETCH_OBJECT_H\n+\n+extern void fetch_object(const char *remote_name, const unsigned char *sha1);\n+\n+#endif\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 008b25d..4640b4e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -450,6 +450,8 @@ static int find_common(struct fetch_pack_args *args,\n \n \tflushes = 0;\n \tretval = -1;\n+\tif (args->no_haves)\n+\t\tgoto done;\n \twhile ((oid = get_rev())) {\n \t\tpacket_buf_write(&req_buf, \"have %s\\n\", oid_to_hex(oid));\n \t\tprint_verbose(args, \"have %s\", oid_to_hex(oid));\n@@ -832,7 +834,7 @@ static int get_pack(struct fetch_pack_args *args,\n \t\targv_array_push(&cmd.args, alternate_shallow_file);\n \t}\n \n-\tif (do_keep) {\n+\tif (do_keep || args->from_promisor) {\n \t\tif (pack_lockfile)\n \t\t\tcmd.out = -1;\n \t\tcmd_name = \"index-pack\";\n@@ -842,7 +844,7 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t\targv_array_push(&cmd.args, \"-v\");\n \t\tif (args->use_thin_pack)\n \t\t\targv_array_push(&cmd.args, \"--fix-thin\");\n-\t\tif (args->lock_pack || unpack_limit) {\n+\t\tif (do_keep && (args->lock_pack || unpack_limit)) {\n \t\t\tchar hostname[HOST_NAME_MAX + 1];\n \t\t\tif (xgethostname(hostname, sizeof(hostname)))\n \t\t\t\txsnprintf(hostname, sizeof(hostname), \"localhost\");\n@@ -852,6 +854,8 @@ static int get_pack(struct fetch_pack_args *args,\n \t\t}\n \t\tif (args->check_self_contained_and_connected)\n \t\t\targv_array_push(&cmd.args, \"--check-self-contained-and-connected\");\n+\t\tif (args->from_promisor)\n+\t\t\targv_array_push(&cmd.args, \"--promisor\");\n \t}\n \telse {\n \t\tcmd_name = \"unpack-objects\";\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex b6aeb43..84904c3 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -29,6 +29,8 @@ struct fetch_pack_args {\n \tunsigned cloning:1;\n \tunsigned update_shallow:1;\n \tunsigned deepen:1;\n+\tunsigned from_promisor:1;\n+\tunsigned no_haves:1;\n };\n \n /*\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 0053b09..41e8a42 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -33,7 +33,9 @@ struct options {\n \t\tthin : 1,\n \t\t/* One of the SEND_PACK_PUSH_CERT_* constants. */\n \t\tpush_cert : 2,\n-\t\tdeepen_relative : 1;\n+\t\tdeepen_relative : 1,\n+\t\tfrom_promisor : 1,\n+\t\tno_haves : 1;\n };\n static struct options options;\n static struct string_list cas_options = STRING_LIST_INIT_DUP;\n@@ -157,6 +159,15 @@ static int set_option(const char *name, const char *value)\n \t\t\treturn -1;\n \t\treturn 0;\n #endif /* LIBCURL_VERSION_NUM >= 0x070a08 */\n+\n+\t} else if (!strcmp(name, \"from-promisor\")) {\n+\t\toptions.from_promisor = 1;\n+\t\treturn 0;\n+\n+\t} else if (!strcmp(name, \"no-haves\")) {\n+\t\toptions.no_haves = 1;\n+\t\treturn 0;\n+\n \t} else {\n \t\treturn 1 /* unsupported */;\n \t}\n@@ -822,6 +833,10 @@ static int fetch_git(struct discovery *heads,\n \t\t\t\t options.deepen_not.items[i].string);\n \tif (options.deepen_relative && options.depth)\n \t\targv_array_push(&args, \"--deepen-relative\");\n+\tif (options.from_promisor)\n+\t\targv_array_push(&args, \"--from-promisor\");\n+\tif (options.no_haves)\n+\t\targv_array_push(&args, \"--no-haves\");\n \targv_array_push(&args, url.buf);\n \n \tfor (i = 0; i < nr_heads; i++) {\ndiff --git a/transport.c b/transport.c\nindex f1e2f61..8211f82 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -160,6 +160,12 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {\n \t\topts->deepen_relative = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_FROM_PROMISOR)) {\n+\t\topts->from_promisor = !!value;\n+\t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_NO_HAVES)) {\n+\t\topts->no_haves = !!value;\n+\t\treturn 0;\n \t}\n \treturn 1;\n }\n@@ -228,6 +234,8 @@ static int fetch_refs_via_pack(struct transport *transport,\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\n \targs.update_shallow = data->options.update_shallow;\n+\targs.from_promisor = data->options.from_promisor;\n+\targs.no_haves = data->options.no_haves;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0);\ndiff --git a/transport.h b/transport.h\nindex bc55715..67428f6 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -15,6 +15,8 @@ struct git_transport_options {\n \tunsigned self_contained_and_connected : 1;\n \tunsigned update_shallow : 1;\n \tunsigned deepen_relative : 1;\n+\tunsigned from_promisor : 1;\n+\tunsigned no_haves : 1;\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n@@ -210,6 +212,12 @@ void transport_check_allowed(const char *type);\n /* Send push certificates */\n #define TRANS_OPT_PUSH_CERT \"pushcert\"\n \n+/* Indicate that these objects are being fetched by a promisor */\n+#define TRANS_OPT_FROM_PROMISOR \"from-promisor\"\n+\n+/* Do not send \"have\" lines */\n+#define TRANS_OPT_NO_HAVES \"no-haves\"\n+\n /**\n * Returns 0 if the option was used, non-zero otherwise. Prints a\n * message to stderr if the option is not used.\n-- \n2.9.3\n\n"},{"id":"331707","messageId":"20171102202052.58762-10-git@jeffhostetler.com","threadId":"47103","inReplyTo":"20171102202052.58762-1-git@jeffhostetler.com","subject":"[PATCH 9/9] gc: do not repack promisor packfiles","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:20:52Z","receivedAt":"2017-11-02T20:21:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nTeach gc to stop traversal at promisor objects, and to leave promisor\npackfiles alone. This has the effect of only repacking non-promisor\npackfiles, and preserves the distinction between promisor packfiles and\nnon-promisor packfiles.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n Documentation/git-pack-objects.txt | 4 +++\n builtin/gc.c | 4 +++\n builtin/pack-objects.c | 14 ++++++++++\n builtin/prune.c | 7 +++++\n builtin/repack.c | 12 +++++++--\n t/t0410-partial-clone.sh | 54 ++++++++++++++++++++++++++++++++++++--\n 6 files changed, 91 insertions(+), 4 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex 6786351..ee462c6 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -246,6 +246,10 @@ So does `git bundle` (see linkgit:git-bundle[1]) when it creates a bundle.\n \tIgnore missing objects without error. This may be used with\n \tor without and of the above filtering.\n \n+--exclude-promisor-objects::\n+\tSilently omit referenced but missing objects from the packfile.\n+\tThis is used with partial clone.\n+\n SEE ALSO\n --------\n linkgit:git-rev-list[1]\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3c5eae0..a17806a 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -20,6 +20,7 @@\n #include \"argv-array.h\"\n #include \"commit.h\"\n #include \"packfile.h\"\n+#include \"partial-clone-utils.h\"\n \n #define FAILED_RUN \"failed to run %s\"\n \n@@ -458,6 +459,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\targv_array_push(&prune, prune_expire);\n \t\t\tif (quiet)\n \t\t\t\targv_array_push(&prune, \"--no-progress\");\n+\t\t\tif (is_partial_clone_registered())\n+\t\t\t\targv_array_push(&prune,\n+\t\t\t\t\t\t\"--exclude-promisor-objects\");\n \t\t\tif (run_command_v_opt(prune.argv, RUN_GIT_CMD))\n \t\t\t\treturn error(FAILED_RUN, prune.argv[0]);\n \t\t}\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex e16722f..957e459 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -83,6 +83,7 @@ static unsigned long window_memory_limit = 0;\n \n static struct list_objects_filter_options filter_options;\n static int arg_ignore_missing;\n+static int arg_exclude_promisor_objects;\n \n /*\n * stats\n@@ -2561,6 +2562,11 @@ static void show_object(struct object *obj, const char *name, void *data)\n \tif (arg_ignore_missing && !has_object_file(&obj->oid))\n \t\treturn;\n \n+\tif (arg_exclude_promisor_objects &&\n+\t !has_object_file(&obj->oid) &&\n+\t is_promisor_object(&obj->oid))\n+\t\treturn;\n+\n \tadd_preferred_base_object(name);\n \tadd_object_entry(obj->oid.hash, obj->type, name, 0);\n \tobj->flags |= OBJECT_ADDED;\n@@ -2972,6 +2978,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \t\tOPT_BOOL(0, \"filter-ignore-missing\", &arg_ignore_missing,\n \t\t\t N_(\"ignore and omit missing objects from packfile\")),\n+\t\tOPT_BOOL(0, \"exclude-promisor-objects\", &arg_exclude_promisor_objects,\n+\t\t\t N_(\"do not pack objects in promisor packfiles\")),\n \t\tOPT_END(),\n \t};\n \n@@ -3017,6 +3025,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\targv_array_push(&rp, \"--unpacked\");\n \t}\n \n+\tif (arg_exclude_promisor_objects) {\n+\t\tuse_internal_rev_list = 1;\n+\t\tfetch_if_missing = 0;\n+\t\targv_array_push(&rp, \"--exclude-promisor-objects\");\n+\t}\n+\n \tif (!reuse_object)\n \t\treuse_delta = 0;\n \tif (pack_compression_level == -1)\ndiff --git a/builtin/prune.c b/builtin/prune.c\nindex cddabf2..be34645 100644\n--- a/builtin/prune.c\n+++ b/builtin/prune.c\n@@ -101,12 +101,15 @@ int cmd_prune(int argc, const char **argv, const char *prefix)\n {\n \tstruct rev_info revs;\n \tstruct progress *progress = NULL;\n+\tint exclude_promisor_objects = 0;\n \tconst struct option options[] = {\n \t\tOPT__DRY_RUN(&show_only, N_(\"do not remove, show only\")),\n \t\tOPT__VERBOSE(&verbose, N_(\"report pruned objects\")),\n \t\tOPT_BOOL(0, \"progress\", &show_progress, N_(\"show progress\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire\", &expire,\n \t\t\t\tN_(\"expire objects older than