{"thread":{"id":"47104","subject":"[PATCH 00/14] WIP Partial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","startedAt":"2017-11-02T20:31:46Z","lastAt":"2017-11-16T17:55:06Z","messageCount":23,"participants":["Jeff Hostetler","Jonathan Tan","Junio C Hamano","Adam Dinwoodie"],"isPatch":true,"patchVersion":1,"patchTotal":14},"messages":[{"id":"331711","messageId":"20171102203129.59417-1-git@jeffhostetler.com","threadId":"47104","inReplyTo":null,"subject":"[PATCH 00/14] WIP Partial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:15Z","receivedAt":"2017-11-02T20:31:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nThis is part 3 of 3 for partial clone.\nIt assumes that part 1 [1] and part 2 [2] are in place.\n\nPart 3 is concerned with the commands: clone, fetch, upload-pack, fetch-pack,\nremote-curl, index-pack, and the pack-protocol.\n\nJonathan and I independently started on this task. This is a first\npass at merging those efforts. So there are several places that need\nrefactoring and cleanup. In particular, the test cases should be\nsquashed and new tests added.\n\n[1] https://public-inbox.org/git/20171102124445.fbffd43521cd35f6a71e1851@google.com/T/\n[2] TODO\n\n\nJeff Hostetler (5):\n upload-pack: add object filtering for partial clone\n clone, fetch-pack, index-pack, transport: partial clone\n fetch: add object filtering for partial fetch\n remote-curl: add object filtering for partial clone\n index-pack: silently assume missing objects are promisor\n\nJonathan Tan (9):\n fetch: refactor calculation of remote list\n pack-objects: test support for blob filtering\n fetch-pack: test support excluding large blobs\n fetch: add from_promisor and exclude-promisor-objects parameters\n t5500: add fetch-pack tests for partial clone\n t5601: test for partial clone\n t5500: more tests for partial clone and fetch\n unpack-trees: batch fetching of missing blobs\n fetch-pack: restore save_commit_buffer after use\n\n Documentation/config.txt | 4 +\n Documentation/gitremote-helpers.txt | 4 +\n Documentation/technical/pack-protocol.txt | 8 ++\n Documentation/technical/protocol-capabilities.txt | 8 ++\n builtin/clone.c | 24 ++++-\n builtin/fetch-pack.c | 4 +\n builtin/fetch.c | 83 ++++++++++++++--\n builtin/index-pack.c | 14 +++\n connected.c | 3 +\n fetch-object.c | 27 ++++-\n fetch-object.h | 5 +\n fetch-pack.c | 17 ++++\n fetch-pack.h | 2 +\n remote-curl.c | 10 +-\n t/t5300-pack-object.sh | 45 +++++++++\n t/t5500-fetch-pack.sh | 115 ++++++++++++++++++++++\n t/t5601-clone.sh | 101 +++++++++++++++++++\n t/test-lib-functions.sh | 12 +++\n transport-helper.c | 5 +\n transport.c | 4 +\n transport.h | 5 +\n unpack-trees.c | 22 +++++\n upload-pack.c | 20 +++-\n 23 files changed, 526 insertions(+), 16 deletions(-)\n\n-- \n2.9.3\n\n"},{"id":"331712","messageId":"20171102203129.59417-3-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 02/14] clone, fetch-pack, index-pack, transport: partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:17Z","receivedAt":"2017-11-02T20:31:49Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n builtin/clone.c | 9 +++++++++\n builtin/fetch-pack.c | 4 ++++\n builtin/index-pack.c | 10 ++++++++++\n fetch-pack.c | 13 +++++++++++++\n fetch-pack.h | 2 ++\n transport-helper.c | 5 +++++\n transport.c | 4 ++++\n transport.h | 5 +++++\n 8 files changed, 52 insertions(+)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex dbddd98..fceb9e7 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -26,6 +26,7 @@\n #include \"run-command.h\"\n #include \"connected.h\"\n #include \"packfile.h\"\n+#include \"list-objects-filter-options.h\"\n \n /*\n * Overall FIXMEs:\n@@ -60,6 +61,7 @@ static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;\n static int option_dissociate;\n static int max_jobs = -1;\n static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;\n+static struct list_objects_filter_options filter_options;\n \n static int recurse_submodules_cb(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -135,6 +137,7 @@ static struct option builtin_clone_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_END()\n };\n \n@@ -1073,6 +1076,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\twarning(_(\"--shallow-since is ignored in local clones; use file:// instead.\"));\n \t\tif (option_not.nr)\n \t\t\twarning(_(\"--shallow-exclude is ignored in local clones; use file:// instead.\"));\n+\t\tif (filter_options.choice)\n+\t\t\twarning(_(\"--filter is ignored in local clones; use file:// instead.\"));\n \t\tif (!access(mkpath(\"%s/shallow\", path), F_OK)) {\n \t\t\tif (option_local > 0)\n \t\t\t\twarning(_(\"source repository is shallow, ignoring --local\"));\n@@ -1104,6 +1109,10 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n+\tif (filter_options.choice)\n+\t\ttransport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n+\t\t\t\t filter_options.raw_value);\n+\n \tif (transport->smart_options && !deepen)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 9a7ebf6..d0fdaa8 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -153,6 +153,10 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.no_haves = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n+\t\t\tparse_list_objects_filter(&args.filter_options, arg);\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \tif (deepen_not.nr)\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex a0a35e6..31cd5ba 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -222,6 +222,16 @@ static unsigned check_object(struct object *obj)\n \tif (!(obj->flags & FLAG_CHECKED)) {\n \t\tunsigned long size;\n \t\tint type = sha1_object_info(obj->oid.hash, &size);\n+\n+\t\tif (type <= 0) {\n+\t\t\t/*\n+\t\t\t * TODO Use the promisor code to conditionally\n+\t\t\t * try to fetch this object -or- assume it is ok.\n+\t\t\t */\n+\t\t\tobj->flags |= FLAG_CHECKED;\n+\t\t\treturn 0;\n+\t\t}\n+\n \t\tif (type <= 0)\n \t\t\tdie(_(\"did not receive expected object %s\"),\n \t\t\t oid_to_hex(&obj->oid));\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 4640b4e..895e8f9 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -29,6 +29,7 @@ static int deepen_not_ok;\n static int fetch_fsck_objects = -1;\n static int transfer_fsck_objects = -1;\n static int agent_supported;\n+static int server_supports_filtering;\n static struct lock_file shallow_lock;\n static const char *alternate_shallow_file;\n \n@@ -379,6 +380,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tif (deepen_not_ok) strbuf_addstr(&c, \" deepen-not\");\n \t\t\tif (agent_supported) strbuf_addf(&c, \" agent=%s\",\n \t\t\t\t\t\t\t git_user_agent_sanitized());\n+\t\t\tif (args->filter_options.choice)\n+\t\t\t\tstrbuf_addstr(&c, \" filter\");\n \t\t\tpacket_buf_write(&req_buf, \"want %s%s\\n\", remote_hex, c.buf);\n \t\t\tstrbuf_release(&c);\n \t\t} else\n@@ -407,6 +410,9 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n+\tif (server_supports_filtering && args->filter_options.choice)\n+\t\tpacket_buf_write(&req_buf, \"filter %s\",\n+\t\t\t\t args->filter_options.raw_value);\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n@@ -967,6 +973,13 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \telse\n \t\tprefer_ofs_delta = 0;\n \n+\tif (server_supports(\"filter\")) {\n+\t\tserver_supports_filtering = 1;\n+\t\tprint_verbose(args, _(\"Server supports filter\"));\n+\t} else if (args->filter_options.choice) {\n+\t\twarning(\"filtering not recognized by server, ignoring\");\n+\t}\n+\n \tif ((agent_feature = server_feature_value(\"agent\", &agent_len))) {\n \t\tagent_supported = 1;\n \t\tif (agent_len)\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex 84904c3..64661b6 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -3,6 +3,7 @@\n \n #include \"string-list.h\"\n #include \"run-command.h\"\n+#include \"list-objects-filter-options.h\"\n \n struct oid_array;\n \n@@ -12,6 +13,7 @@ struct fetch_pack_args {\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n+\tstruct list_objects_filter_options filter_options;\n \tunsigned deepen_relative:1;\n \tunsigned quiet:1;\n \tunsigned keep_pack:1;\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c948d52..96823c7 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -671,6 +671,11 @@ static int fetch(struct transport *transport,\n \tif (data->transport_options.update_shallow)\n \t\tset_helper_option(transport, \"update-shallow\", \"true\");\n \n+\tif (data->transport_options.filter_options.choice)\n+\t\tset_helper_option(\n+\t\t\ttransport, \"filter\",\n+\t\t\tdata->transport_options.filter_options.raw_value);\n+\n \tif (data->fetch)\n \t\treturn fetch_with_fetch(transport, nr_heads, to_fetch);\n \ndiff --git a/transport.c b/transport.c\nindex 8211f82..d50c73b 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -166,6 +166,9 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_NO_HAVES)) {\n \t\topts->no_haves = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_LIST_OBJECTS_FILTER)) {\n+\t\tparse_list_objects_filter(&opts->filter_options, value);\n+\t\treturn 0;\n \t}\n \treturn 1;\n }\n@@ -236,6 +239,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.update_shallow = data->options.update_shallow;\n \targs.from_promisor = data->options.from_promisor;\n \targs.no_haves = data->options.no_haves;\n+\targs.filter_options = data->options.filter_options;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0);\ndiff --git a/transport.h b/transport.h\nindex 67428f6..f64aa3a 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -4,6 +4,7 @@\n #include \"cache.h\"\n #include \"run-command.h\"\n #include \"remote.h\"\n+#include \"list-objects-filter-options.h\"\n \n struct string_list;\n \n@@ -23,6 +24,7 @@ struct git_transport_options {\n \tconst char *uploadpack;\n \tconst char *receivepack;\n \tstruct push_cas_option *cas;\n+\tstruct list_objects_filter_options filter_options;\n };\n \n enum transport_family {\n@@ -218,6 +220,9 @@ void transport_check_allowed(const char *type);\n /* Do not send \"have\" lines */\n #define TRANS_OPT_NO_HAVES \"no-haves\"\n \n+/* Filter objects for partial clone and fetch */\n+#define TRANS_OPT_LIST_OBJECTS_FILTER \"filter\"\n+\n /**\n * Returns 0 if the option was used, non-zero otherwise. Prints a\n * message to stderr if the option is not used.\n-- \n2.9.3\n\n"},{"id":"331713","messageId":"20171102203129.59417-5-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 04/14] fetch: add object filtering for partial fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:19Z","receivedAt":"2017-11-02T20:31:51Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach fetch to use the list-objects filtering parameters\nto allow a \"partial fetch\" following a \"partial clone\".\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/fetch.c | 66 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++-\n 1 file changed, 65 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 1b1f039..150ca0a 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -18,6 +18,7 @@\n #include \"argv-array.h\"\n #include \"utf8.h\"\n #include \"packfile.h\"\n+#include \"list-objects-filter-options.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] [ [...]]\"),\n@@ -55,6 +56,7 @@ static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;\n static int shown_url = 0;\n static int refmap_alloc, refmap_nr;\n static const char **refmap_array;\n+static struct list_objects_filter_options filter_options;\n \n static int git_fetch_config(const char *k, const char *v, void *cb)\n {\n@@ -160,6 +162,7 @@ static struct option builtin_fetch_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_END()\n };\n \n@@ -754,6 +757,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \tconst char *filename = dry_run ? \"/dev/null\" : git_path_fetch_head();\n \tint want_status;\n \tint summary_width = transport_summary_width(ref_map);\n+\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n \n \tfp = fopen(filename, \"a\");\n \tif (!fp)\n@@ -765,7 +769,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n \t\turl = xstrdup(\"foreign\");\n \n \trm = ref_map;\n-\tif (check_connected(iterate_ref_map, &rm, NULL)) {\n+\tif (check_connected(iterate_ref_map, &rm, &opt)) {\n \t\trc = error(_(\"%s did not send all necessary objects\\n\"), url);\n \t\tgoto abort;\n \t}\n@@ -1044,6 +1048,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n+\tif (filter_options.choice)\n+\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n+\t\t\t filter_options.raw_value);\n \treturn transport;\n }\n \n@@ -1242,6 +1249,20 @@ static int fetch_multiple(struct string_list *list)\n \tint i, result = 0;\n \tstruct argv_array argv = ARGV_ARRAY_INIT;\n \n+\tif (filter_options.choice) {\n+\t\t/*\n+\t\t * We currently only support partial-fetches to the remote\n+\t\t * used for the partial-clone because we only support 1\n+\t\t * promisor remote, so we DO NOT allow explicit command\n+\t\t * line filter arguments.\n+\t\t *\n+\t\t * Note that the loop below will spawn background fetches\n+\t\t * for each remote and one of them MAY INHERIT the proper\n+\t\t * partial-fetch settings, so everything is consistent.\n+\t\t */\n+\t\tdie(_(\"partial-fetch is not supported on multiple remotes\"));\n+\t}\n+\n \tif (!append && !dry_run) {\n \t\tint errcode = truncate_fetch_head();\n \t\tif (errcode)\n@@ -1267,6 +1288,46 @@ static int fetch_multiple(struct string_list *list)\n \treturn result;\n }\n \n+static inline void partial_fetch_one_setup(struct remote *remote)\n+{\n+#if 0 /* TODO */\n+\tif (filter_options.choice) {\n+\t\t/*\n+\t\t * A partial-fetch was explicitly requested.\n+\t\t *\n+\t\t * If this is the first partial-* command on\n+\t\t * this repo, we must register the partial\n+\t\t * settings in the repository extension.\n+\t\t *\n+\t\t * If this follows a previous partial-* command\n+\t\t * we must ensure the args are consistent with\n+\t\t * the existing registration (because we don't\n+\t\t * currently support mixing-and-matching).\n+\t\t */\n+\t\tpartial_clone_utils_register(&filter_options,\n+\t\t\t\t\t remote->name, \"fetch\");\n+\t\treturn;\n+\t}\n+\n+\tif (is_partial_clone_registered() &&\n+\t !strcmp(remote->name, repository_format_partial_clone_remote)) {\n+\t\t/*\n+\t\t * If a partial-* command has already been used on\n+\t\t * this repo and it was to this remote, we should\n+\t\t * inherit the filter settings used previously.\n+\t\t * That is, if clone omitted very large blobs, then\n+\t\t * fetch should too.\n+\t\t *\n+\t\t * Use the cached filter-spec and create the filter\n+\t\t * settings.\n+\t\t */\n+\t\tparse_list_objects_filter(\n+\t\t\t&filter_options,\n+\t\t\trepository_format_partial_clone_filter);\n+\t}\n+#endif\n+}\n+\n static int fetch_one(struct remote *remote, int argc, const char **argv)\n {\n \tstatic const char **refs = NULL;\n@@ -1278,6 +1339,9 @@ static int fetch_one(struct remote *remote, int argc, const char **argv)\n \t\tdie(_(\"No remote repository specified. Please, specify either a URL or a\\n\"\n \t\t \"remote name from which new revisions should be fetched.\"));\n \n+\tpartial_fetch_one_setup(remote);\n+\n+\n \tgtransport = prepare_transport(remote, 1);\n \n \tif (prune < 0) {\n-- \n2.9.3\n\n"},{"id":"331714","messageId":"20171102203129.59417-6-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 05/14] remote-curl: add object filtering for partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:20Z","receivedAt":"2017-11-02T20:31:52Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/gitremote-helpers.txt | 4 ++++\n remote-curl.c | 10 ++++++++--\n 2 files changed, 12 insertions(+), 2 deletions(-)\n\ndiff --git a/Documentation/gitremote-helpers.txt b/Documentation/gitremote-helpers.txt\nindex 6da3f41..d46d561 100644\n--- a/Documentation/gitremote-helpers.txt\n+++ b/Documentation/gitremote-helpers.txt\n@@ -468,6 +468,10 @@ set by Git if the remote helper has the 'option' capability.\n \n TODO document 'option from-promisor' and 'option no-haves' ?\n \n+'option filter '::\n+\tAn object filter specification for partial clone or fetch\n+\tas described in rev-list.\n+\n SEE ALSO\n --------\n linkgit:git-remote[1]\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 41e8a42..840f3ce 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -13,6 +13,7 @@\n #include \"credential.h\"\n #include \"sha1-array.h\"\n #include \"send-pack.h\"\n+#include \"list-objects-filter-options.h\"\n \n static struct remote *remote;\n /* always ends with a trailing slash */\n@@ -22,6 +23,7 @@ struct options {\n \tint verbosity;\n \tunsigned long depth;\n \tchar *deepen_since;\n+\tchar *partial_clone_filter;\n \tstruct string_list deepen_not;\n \tstruct string_list push_options;\n \tunsigned progress : 1,\n@@ -163,11 +165,12 @@ static int set_option(const char *name, const char *value)\n \t} else if (!strcmp(name, \"from-promisor\")) {\n \t\toptions.from_promisor = 1;\n \t\treturn 0;\n-\n \t} else if (!strcmp(name, \"no-haves\")) {\n \t\toptions.no_haves = 1;\n \t\treturn 0;\n-\n+\t} else if (!strcmp(name, \"filter\")) {\n+\t\toptions.partial_clone_filter = xstrdup(value);\n+\t\treturn 0;\n \t} else {\n \t\treturn 1 /* unsupported */;\n \t}\n@@ -837,6 +840,9 @@ static int fetch_git(struct discovery *heads,\n \t\targv_array_push(&args, \"--from-promisor\");\n \tif (options.no_haves)\n \t\targv_array_push(&args, \"--no-haves\");\n+\tif (options.partial_clone_filter)\n+\t\targv_array_pushf(&args, \"--%s=%s\",\n+\t\t\t\t CL_ARG__FILTER, options.partial_clone_filter);\n \targv_array_push(&args, url.buf);\n \n \tfor (i = 0; i < nr_heads; i++) {\n-- \n2.9.3\n\n"},{"id":"331715","messageId":"20171102203129.59417-8-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 07/14] fetch-pack: test support excluding large blobs","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:22Z","receivedAt":"2017-11-02T20:31:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nCreated tests to verify fetch-pack and upload-pack support\nfor excluding large blobs using --filter=blobs:limit=\nparameter.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n t/t5500-fetch-pack.sh | 27 +++++++++++++++++++++++++++\n 1 file changed, 27 insertions(+)\n\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex 80a1a32..fdb98a8 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -755,4 +755,31 @@ test_expect_success 'fetching deepen' '\n \t)\n '\n \n+test_expect_success 'filtering by size' '\n+\trm -rf server client &&\n+\ttest_create_repo server &&\n+\ttest_commit -C server one &&\n+\ttest_config -C server uploadpack.allowfilter 1 &&\n+\n+\ttest_create_repo client &&\n+\tgit -C client fetch-pack --filter=blobs:limit=0 ../server HEAD &&\n+\n+\t# Ensure that object is not inadvertently fetched\n+\ttest_must_fail git -C client cat-file -e $(git hash-object server/one.t)\n+'\n+\n+test_expect_success 'filtering by size has no effect if support for it is not advertised' '\n+\trm -rf server client &&\n+\ttest_create_repo server &&\n+\ttest_commit -C server one &&\n+\n+\ttest_create_repo client &&\n+\tgit -C client fetch-pack --filter=blobs:limit=0 ../server HEAD 2> err &&\n+\n+\t# Ensure that object is fetched\n+\tgit -C client cat-file -e $(git hash-object server/one.t) &&\n+\n+\ttest_i18ngrep \"filtering not recognized by server\" err\n+'\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"331716","messageId":"20171102203129.59417-9-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 08/14] fetch: add from_promisor and exclude-promisor-objects parameters","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:23Z","receivedAt":"2017-11-02T20:31:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nTeach fetch to use from-promisor and exclude-promisor-objects\nparameters with sub-commands. Initialize fetch_if_missing\nglobal variable.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n builtin/fetch.c | 9 ++++++---\n connected.c | 3 +++\n 2 files changed, 9 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 150ca0a..ab53df3 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -19,6 +19,7 @@\n #include \"utf8.h\"\n #include \"packfile.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"partial-clone-utils.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [] [ [...]]\"),\n@@ -1048,9 +1049,11 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n-\tif (filter_options.choice)\n+\tif (filter_options.choice) {\n \t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n \t\t\t filter_options.raw_value);\n+\t\tset_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n+\t}\n \treturn transport;\n }\n \n@@ -1290,7 +1293,6 @@ static int fetch_multiple(struct string_list *list)\n \n static inline void partial_fetch_one_setup(struct remote *remote)\n {\n-#if 0 /* TODO */\n \tif (filter_options.choice) {\n \t\t/*\n \t\t * A partial-fetch was explicitly requested.\n@@ -1325,7 +1327,6 @@ static inline void partial_fetch_one_setup(struct remote *remote)\n \t\t\t&filter_options,\n \t\t\trepository_format_partial_clone_filter);\n \t}\n-#endif\n }\n \n static int fetch_one(struct remote *remote, int argc, const char **argv)\n@@ -1392,6 +1393,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \n \tpacket_trace_identity(\"fetch\");\n \n+\tfetch_if_missing = 0;\n+\n \t/* Record the command line for the reflog */\n \tstrbuf_addstr(&default_rla, \"fetch\");\n \tfor (i = 1; i < argc; i++)\ndiff --git a/connected.c b/connected.c\nindex f416b05..6015316 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -4,6 +4,7 @@\n #include \"connected.h\"\n #include \"transport.h\"\n #include \"packfile.h\"\n+#include \"partial-clone-utils.h\"\n \n /*\n * If we feed all the commits we want to verify to this command\n@@ -56,6 +57,8 @@ int check_connected(sha1_iterate_fn fn, void *cb_data,\n \targv_array_push(&rev_list.args,\"rev-list\");\n \targv_array_push(&rev_list.args, \"--objects\");\n \targv_array_push(&rev_list.args, \"--stdin\");\n+\tif (is_partial_clone_registered())\n+\t\targv_array_push(&rev_list.args, \"--exclude-promisor-objects\");\n \targv_array_push(&rev_list.args, \"--not\");\n \targv_array_push(&rev_list.args, \"--all\");\n \targv_array_push(&rev_list.args, \"--quiet\");\n-- \n2.9.3\n\n"},{"id":"331717","messageId":"20171102203129.59417-11-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 10/14] t5601: test for partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:25Z","receivedAt":"2017-11-02T20:32:00Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n builtin/clone.c | 17 ++++++++++++++---\n t/t5601-clone.sh | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 63 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex fceb9e7..08315d8 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -27,6 +27,7 @@\n #include \"connected.h\"\n #include \"packfile.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"partial-clone-utils.h\"\n \n /*\n * Overall FIXMEs:\n@@ -889,6 +890,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstruct refspec *refspec;\n \tconst char *fetch_pattern;\n \n+\tfetch_if_missing = 0;\n+\n \tpacket_trace_identity(\"clone\");\n \targc = parse_options(argc, argv, prefix, builtin_clone_options,\n \t\t\t builtin_clone_usage, 0);\n@@ -1109,11 +1112,13 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t option_upload_pack);\n \n-\tif (filter_options.choice)\n+\tif (filter_options.choice) {\n \t\ttransport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n \t\t\t\t filter_options.raw_value);\n+\t\ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n+\t}\n \n-\tif (transport->smart_options && !deepen)\n+\tif (transport->smart_options && !deepen && !filter_options.choice)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n \trefs = transport_get_remote_refs(transport);\n@@ -1173,13 +1178,18 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \twrite_refspec_config(src_ref_prefix, our_head_points_at,\n \t\t\tremote_head_points_at, &branch_top);\n \n+\tif (filter_options.choice)\n+\t\tpartial_clone_utils_register(&filter_options, \"origin\",\n+\t\t\t\t\t \"clone\");\n+\n \tif (is_local)\n \t\tclone_local(path, git_dir);\n \telse if (refs && complete_refs_before_fetch)\n \t\ttransport_fetch_refs(transport, mapped_refs);\n \n \tupdate_remote_refs(refs, mapped_refs, remote_head_points_at,\n-\t\t\t branch_top.buf, reflog_msg.buf, transport, !is_local);\n+\t\t\t branch_top.buf, reflog_msg.buf, transport,\n+\t\t\t !is_local && !filter_options.choice);\n \n \tupdate_head(our_head_points_at, remote_head, reflog_msg.buf);\n \n@@ -1200,6 +1210,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tjunk_mode = JUNK_LEAVE_REPO;\n+\tfetch_if_missing = 1;\n \terr = checkout(submodule_progress);\n \n \tstrbuf_release(&reflog_msg);\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 9c56f77..567161e 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -571,4 +571,53 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' '\n \tgit -C replay.git index-pack -v --stdin err &&\n+\n+\ttest_i18ngrep \"filtering not recognized by server\" err\n+'\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'partial clone using HTTP' '\n+\t\t partial_clone \"$HTTPD_DOCUMENT_ROOT_PATH/server\" \"$HTTPD_URL/smart/server\"\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"331718","messageId":"20171102203129.59417-14-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 13/14] fetch-pack: restore save_commit_buffer after use","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:28Z","receivedAt":"2017-11-02T20:32:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nIn fetch-pack, the global variable save_commit_buffer is set to 0, but\nnot restored to its original value after use.\n\nIn particular, if show_log() (in log-tree.c) is invoked after\nfetch_pack() in the same process, show_log() will return before printing\nout the commit message (because the invocation to\nget_cached_commit_buffer() returns NULL, because the commit buffer was\nnot saved). I discovered this when attempting to run \"git log -S\" in a\npartial clone, triggering the case where revision walking lazily loads\nmissing objects.\n\nTherefore, restore save_commit_buffer to its original value after use.\n\nAn alternative to solve the problem I had is to replace\nget_cached_commit_buffer() with get_commit_buffer(). That invocation was\nintroduced in commit a97934d (\"use get_cached_commit_buffer where\nappropriate\", 2014-06-13) to replace \"commit->buffer\" introduced in\ncommit 3131b71 (\"Add \"--show-all\" revision walker flag for debugging\",\n2008-02-13). In the latter commit, the commit author seems to be\ndeciding between not showing an unparsed commit at all and showing an\nunparsed commit without the message (which is what the commit does), and\ndid not mention parsing the unparsed commit, so I prefer to preserve the\nexisting behavior.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n fetch-pack.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 895e8f9..121f03e 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -717,6 +717,7 @@ static int everything_local(struct fetch_pack_args *args,\n {\n \tstruct ref *ref;\n \tint retval;\n+\tint old_save_commit_buffer = save_commit_buffer;\n \ttimestamp_t cutoff = 0;\n \n \tsave_commit_buffer = 0;\n@@ -784,6 +785,9 @@ static int everything_local(struct fetch_pack_args *args,\n \t\tprint_verbose(args, _(\"already have %s (%s)\"), oid_to_hex(remote),\n \t\t\t ref->name);\n \t}\n+\n+\tsave_commit_buffer = old_save_commit_buffer;\n+\n \treturn retval;\n }\n \n-- \n2.9.3\n\n"},{"id":"331719","messageId":"20171102203129.59417-12-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 11/14] t5500: more tests for partial clone and fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:26Z","receivedAt":"2017-11-02T20:32:04Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n t/t5500-fetch-pack.sh | 60 +++++++++++++++++++++++++++++++++++++++++++++++----\n 1 file changed, 56 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex 7c8339f..86cf653 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -782,7 +782,7 @@ test_expect_success 'filtering by size has no effect if support for it is not ad\n \ttest_i18ngrep \"filtering not recognized by server\" err\n '\n \n-fetch_blob_max_bytes () {\n+setup_blob_max_bytes () {\n \t\t SERVER=\"$1\"\n \t\t URL=\"$2\"\n \n@@ -794,7 +794,11 @@ fetch_blob_max_bytes () {\n \tgit clone \"$URL\" client &&\n \ttest_config -C client extensions.partialcloneremote origin &&\n \n-\ttest_commit -C \"$SERVER\" two &&\n+\ttest_commit -C \"$SERVER\" two\n+}\n+\n+do_blob_max_bytes() {\n+\tSERVER=\"$1\" &&\n \n \tgit -C client fetch --filter=blobs:limit=0 origin HEAD:somewhere &&\n \n@@ -805,14 +809,62 @@ fetch_blob_max_bytes () {\n }\n \n test_expect_success 'fetch with filtering' '\n-\t\t fetch_blob_max_bytes server server\n+\tsetup_blob_max_bytes server server &&\n+\tdo_blob_max_bytes server\n+'\n+\n+test_expect_success 'fetch respects configured filtering' '\n+\tsetup_blob_max_bytes server server &&\n+\n+\ttest_config -C client extensions.partialclonefilter blobs:limit=0 &&\n+\n+\tgit -C client fetch origin HEAD:somewhere &&\n+\n+\t# Ensure that commit is fetched, but blob is not\n+\ttest_config -C client extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C client cat-file -e $(git -C server rev-parse two) &&\n+\ttest_must_fail git -C client cat-file -e $(git hash-object server/two.t)\n+'\n+\n+test_expect_success 'pull respects configured filtering' '\n+\tsetup_blob_max_bytes server server &&\n+\n+\t# Hide two.t from tip so that client does not load it upon the\n+\t# automatic checkout that pull performs\n+\tgit -C server rm two.t &&\n+\ttest_commit -C server three &&\n+\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\ttest_config -C client extensions.partialclonefilter blobs:limit=0 &&\n+\n+\tgit -C client pull origin &&\n+\n+\t# Ensure that commit is fetched, but blob is not\n+\ttest_config -C client extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C client cat-file -e $(git -C server rev-parse two) &&\n+\ttest_must_fail git -C client cat-file -e $(git hash-object server/two.t)\n+'\n+\n+test_expect_success 'clone configures filtering' '\n+\trm -rf server client &&\n+\ttest_create_repo server &&\n+\ttest_commit -C server one &&\n+\ttest_commit -C server two &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\n+\tgit clone --filter=blobs:limit=12345 server client &&\n+\n+\t# Ensure that we can, for example, checkout HEAD^\n+\trm -rf client/.git/objects/* &&\n+\tgit -C client checkout HEAD^\n '\n \n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \n test_expect_success 'fetch with filtering and HTTP' '\n-\t\t fetch_blob_max_bytes \"$HTTPD_DOCUMENT_ROOT_PATH/server\" \"$HTTPD_URL/smart/server\"\n+\tsetup_blob_max_bytes \"$HTTPD_DOCUMENT_ROOT_PATH/server\" \"$HTTPD_URL/smart/server\" &&\n+\tdo_blob_max_bytes \"$HTTPD_DOCUMENT_ROOT_PATH/server\"\n '\n \n stop_httpd\n-- \n2.9.3\n\n"},{"id":"331720","messageId":"20171102203129.59417-15-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 14/14] index-pack: silently assume missing objects are promisor","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:29Z","receivedAt":"2017-11-02T20:32:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach index-pack to not complain about missing objects\nwhen the --promisor flag is given. The assumption is that\nindex-pack is currently building the idx and promisor data\nand the is_promisor_object() query would fail anyway.\n\nSigned-off-by: Jeff Hostetler \n---\n builtin/index-pack.c | 10 +++++++---\n 1 file changed, 7 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 31cd5ba..51693dc 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -82,6 +82,7 @@ static int verbose;\n static int show_resolving_progress;\n static int show_stat;\n static int check_self_contained_and_connected;\n+static int arg_promisor_given;\n \n static struct progress *progress;\n \n@@ -223,10 +224,11 @@ static unsigned check_object(struct object *obj)\n \t\tunsigned long size;\n \t\tint type = sha1_object_info(obj->oid.hash, &size);\n \n-\t\tif (type <= 0) {\n+\t\tif (type <= 0 && arg_promisor_given) {\n \t\t\t/*\n-\t\t\t * TODO Use the promisor code to conditionally\n-\t\t\t * try to fetch this object -or- assume it is ok.\n+\t\t\t * Assume this missing object is promised. We can't\n+\t\t\t * confirm it because we are indexing the packfile\n+\t\t\t * that omitted it.\n \t\t\t */\n \t\t\tobj->flags |= FLAG_CHECKED;\n \t\t\treturn 0;\n@@ -1717,8 +1719,10 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \t\t\t\tkeep_msg = arg + 7;\n \t\t\t} else if (!strcmp(arg, \"--promisor\")) {\n \t\t\t\tpromisor_msg = \"\";\n+\t\t\t\targ_promisor_given = 1;\n \t\t\t} else if (starts_with(arg, \"--promisor=\")) {\n \t\t\t\tpromisor_msg = arg + strlen(\"--promisor=\");\n+\t\t\t\targ_promisor_given = 1;\n \t\t\t} else if (starts_with(arg, \"--threads=\")) {\n \t\t\t\tchar *end;\n \t\t\t\tnr_threads = strtoul(arg+10, &end, 0);\n-- \n2.9.3\n\n"},{"id":"331721","messageId":"20171102203129.59417-13-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 12/14] unpack-trees: batch fetching of missing blobs","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:27Z","receivedAt":"2017-11-02T20:32:09Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nWhen running checkout, first prefetch all blobs that are to be updated\nbut are missing. This means that only one pack is downloaded during such\noperations, instead of one per missing blob.\n\nThis operates only on the blob level - if a repository has a missing\ntree, they are still fetched one at a time.\n\nThis does not use the delayed checkout mechanism introduced in commit\n2841e8f (\"convert: add \"status=delayed\" to filter process protocol\",\n2017-06-30) due to significant conceptual differences - in particular,\nfor partial clones, we already know what needs to be fetched based on\nthe contents of the local repo alone, whereas for status=delayed, it is\nthe filter process that tells us what needs to be checked in the end.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n fetch-object.c | 27 +++++++++++++++++++++++----\n fetch-object.h | 5 +++++\n t/t5601-clone.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n unpack-trees.c | 22 ++++++++++++++++++++++\n 4 files changed, 102 insertions(+), 4 deletions(-)\n\ndiff --git a/fetch-object.c b/fetch-object.c\nindex 369b61c..21b4dfa 100644\n--- a/fetch-object.c\n+++ b/fetch-object.c\n@@ -3,12 +3,12 @@\n #include \"pkt-line.h\"\n #include \"strbuf.h\"\n #include \"transport.h\"\n+#include \"fetch-object.h\"\n \n-void fetch_object(const char *remote_name, const unsigned char *sha1)\n+static void fetch_refs(const char *remote_name, struct ref *ref)\n {\n \tstruct remote *remote;\n \tstruct transport *transport;\n-\tstruct ref *ref;\n \tint original_fetch_if_missing = fetch_if_missing;\n \n \tfetch_if_missing = 0;\n@@ -17,10 +17,29 @@ void fetch_object(const char *remote_name, const unsigned char *sha1)\n \t\tdie(_(\"Remote with no URL\"));\n \ttransport = transport_get(remote, remote->url[0]);\n \n-\tref = alloc_ref(sha1_to_hex(sha1));\n-\thashcpy(ref->old_oid.hash, sha1);\n \ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n \ttransport_set_option(transport, TRANS_OPT_NO_HAVES, \"1\");\n \ttransport_fetch_refs(transport, ref);\n \tfetch_if_missing = original_fetch_if_missing;\n }\n+\n+void fetch_object(const char *remote_name, const unsigned char *sha1)\n+{\n+\tstruct ref *ref = alloc_ref(sha1_to_hex(sha1));\n+\thashcpy(ref->old_oid.hash, sha1);\n+\tfetch_refs(remote_name, ref);\n+}\n+\n+void fetch_objects(const char *remote_name, const struct oid_array *to_fetch)\n+{\n+\tstruct ref *ref = NULL;\n+\tint i;\n+\n+\tfor (i = 0; i < to_fetch->nr; i++) {\n+\t\tstruct ref *new_ref = alloc_ref(oid_to_hex(&to_fetch->oid[i]));\n+\t\toidcpy(&new_ref->old_oid, &to_fetch->oid[i]);\n+\t\tnew_ref->next = ref;\n+\t\tref = new_ref;\n+\t}\n+\tfetch_refs(remote_name, ref);\n+}\ndiff --git a/fetch-object.h b/fetch-object.h\nindex f371300..4b269d0 100644\n--- a/fetch-object.h\n+++ b/fetch-object.h\n@@ -1,6 +1,11 @@\n #ifndef FETCH_OBJECT_H\n #define FETCH_OBJECT_H\n \n+#include \"sha1-array.h\"\n+\n extern void fetch_object(const char *remote_name, const unsigned char *sha1);\n \n+extern void fetch_objects(const char *remote_name,\n+\t\t\t const struct oid_array *to_fetch);\n+\n #endif\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 567161e..3211f86 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -611,6 +611,58 @@ test_expect_success 'partial clone: warn if server does not support object filte\n \ttest_i18ngrep \"filtering not recognized by server\" err\n '\n \n+test_expect_success 'batch missing blob request during checkout' '\n+\trm -rf server client &&\n+\n+\ttest_create_repo server &&\n+\techo a >server/a &&\n+\techo b >server/b &&\n+\tgit -C server add a b &&\n+\n+\tgit -C server commit -m x &&\n+\techo aa >server/a &&\n+\techo bb >server/b &&\n+\tgit -C server add a b &&\n+\tgit -C server commit -m x &&\n+\n+\ttest_config -C server uploadpack.allowfilter 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\n+\tgit clone --filter=blobs:limit=0 \"file://$(pwd)/server\" client &&\n+\n+\t# Ensure that there is only one negotiation by checking that there is\n+\t# only \"done\" line sent. (\"done\" marks the end of negotiation.)\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client checkout HEAD^ &&\n+\tgrep \"git> done\" trace >done_lines &&\n+\ttest_line_count = 1 done_lines\n+'\n+\n+test_expect_success 'batch missing blob request does not inadvertently try to fetch gitlinks' '\n+\trm -rf server client &&\n+\n+\ttest_create_repo repo_for_submodule &&\n+\ttest_commit -C repo_for_submodule x &&\n+\n+\ttest_create_repo server &&\n+\techo a >server/a &&\n+\techo b >server/b &&\n+\tgit -C server add a b &&\n+\tgit -C server commit -m x &&\n+\n+\techo aa >server/a &&\n+\techo bb >server/b &&\n+\t# Also add a gitlink pointing to an arbitrary repository\n+\tgit -C server submodule add \"$(pwd)/repo_for_submodule\" c &&\n+\tgit -C server add a b c &&\n+\tgit -C server commit -m x &&\n+\n+\ttest_config -C server uploadpack.allowfilter 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\n+\t# Make sure that it succeeds\n+\tgit clone --filter=blobs:limit=0 \"file://$(pwd)/server\" client\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 71b70cc..c18c21a 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"fetch-object.h\"\n \n /*\n * Error messages expected by scripts out of plumbing commands such as\n@@ -369,6 +370,27 @@ static int check_updates(struct unpack_trees_options *o)\n \t\tload_gitmodules_file(index, &state);\n \n \tenable_delayed_checkout(&state);\n+\tif (repository_format_partial_clone_remote && o->update && !o->dry_run) {\n+\t\t/*\n+\t\t * Prefetch the objects that are to be checked out in the loop\n+\t\t * below.\n+\t\t */\n+\t\tstruct oid_array to_fetch = OID_ARRAY_INIT;\n+\t\tint fetch_if_missing_store = fetch_if_missing;\n+\t\tfetch_if_missing = 0;\n+\t\tfor (i = 0; i < index->cache_nr; i++) {\n+\t\t\tstruct cache_entry *ce = index->cache[i];\n+\t\t\tif ((ce->ce_flags & CE_UPDATE) &&\n+\t\t\t !S_ISGITLINK(ce->ce_mode)) {\n+\t\t\t\tif (!has_object_file(&ce->oid))\n+\t\t\t\t\toid_array_append(&to_fetch, &ce->oid);\n+\t\t\t}\n+\t\t}\n+\t\tif (to_fetch.nr)\n+\t\t\tfetch_objects(repository_format_partial_clone_remote,\n+\t\t\t\t &to_fetch);\n+\t\tfetch_if_missing = fetch_if_missing_store;\n+\t}\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n-- \n2.9.3\n\n"},{"id":"331722","messageId":"20171102203129.59417-10-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 09/14] t5500: add fetch-pack tests for partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:24Z","receivedAt":"2017-11-02T20:32:13Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n t/t5500-fetch-pack.sh | 36 ++++++++++++++++++++++++++++++++++++\n 1 file changed, 36 insertions(+)\n\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex fdb98a8..7c8339f 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -782,4 +782,40 @@ test_expect_success 'filtering by size has no effect if support for it is not ad\n \ttest_i18ngrep \"filtering not recognized by server\" err\n '\n \n+fetch_blob_max_bytes () {\n+\t\t SERVER=\"$1\"\n+\t\t URL=\"$2\"\n+\n+\trm -rf \"$SERVER\" client &&\n+\ttest_create_repo \"$SERVER\" &&\n+\ttest_commit -C \"$SERVER\" one &&\n+\ttest_config -C \"$SERVER\" uploadpack.allowfilter 1 &&\n+\n+\tgit clone \"$URL\" client &&\n+\ttest_config -C client extensions.partialcloneremote origin &&\n+\n+\ttest_commit -C \"$SERVER\" two &&\n+\n+\tgit -C client fetch --filter=blobs:limit=0 origin HEAD:somewhere &&\n+\n+\t# Ensure that commit is fetched, but blob is not\n+\ttest_config -C client extensions.partialcloneremote \"arbitrary string\" &&\n+\tgit -C client cat-file -e $(git -C \"$SERVER\" rev-parse two) &&\n+\ttest_must_fail git -C client cat-file -e $(git hash-object \"$SERVER/two.t\")\n+}\n+\n+test_expect_success 'fetch with filtering' '\n+\t\t fetch_blob_max_bytes server server\n+'\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'fetch with filtering and HTTP' '\n+\t\t fetch_blob_max_bytes \"$HTTPD_DOCUMENT_ROOT_PATH/server\" \"$HTTPD_URL/smart/server\"\n+'\n+\n+stop_httpd\n+\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"331723","messageId":"20171102203129.59417-7-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 06/14] pack-objects: test support for blob filtering","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:21Z","receivedAt":"2017-11-02T20:32:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nAs part of an effort to improve Git support for very large repositories\nin which clients typically have only a subset of all version-controlled\nblobs, test pack-objects support for --filter=blobs:limit=, packing only\nblobs not exceeding that size unless the blob corresponds to a file\nwhose name starts with \".git\". upload-pack will eventually be taught to\nuse this new parameter if needed to exclude certain blobs during a fetch\nor clone, potentially drastically reducing network consumption when\nserving these very large repositories.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Jeff Hostetler \n---\n t/t5300-pack-object.sh | 45 +++++++++++++++++++++++++++++++++++++++++++++\n t/test-lib-functions.sh | 12 ++++++++++++\n 2 files changed, 57 insertions(+)\n\ndiff --git a/t/t5300-pack-object.sh b/t/t5300-pack-object.sh\nindex 9c68b99..0739a07 100755\n--- a/t/t5300-pack-object.sh\n+++ b/t/t5300-pack-object.sh\n@@ -457,6 +457,51 @@ test_expect_success !PTHREADS,C_LOCALE_OUTPUT 'pack-objects --threads=N or pack.\n \tgrep -F \"no threads support, ignoring pack.threads\" err\n '\n \n+lcut () {\n+\tperl -e '$/ = undef; $_ = <>; s/^.{'$1'}//s; print $_'\n+}\n+\n+test_expect_success 'filtering by size works with multiple excluded' '\n+\trm -rf server &&\n+\tgit init server &&\n+\tprintf a > server/a &&\n+\tprintf b > server/b &&\n+\tprintf c-very-long-file > server/c &&\n+\tprintf d-very-long-file > server/d &&\n+\tgit -C server add a b c d &&\n+\tgit -C server commit -m x &&\n+\n+\tgit -C server rev-parse HEAD >objects &&\n+\tgit -C server pack-objects --revs --stdout --filter=blobs:limit=10 my.pack &&\n+\n+\t# Ensure that only the small blobs are in the packfile\n+\tgit index-pack my.pack &&\n+\tgit verify-pack -v my.idx >objectlist &&\n+\tgrep $(git hash-object server/a) objectlist &&\n+\tgrep $(git hash-object server/b) objectlist &&\n+\t! grep $(git hash-object server/c) objectlist &&\n+\t! grep $(git hash-object server/d) objectlist\n+'\n+\n+test_expect_success 'filtering by size never excludes special files' '\n+\trm -rf server &&\n+\tgit init server &&\n+\tprintf a-very-long-file > server/a &&\n+\tprintf a-very-long-file > server/.git-a &&\n+\tprintf b-very-long-file > server/b &&\n+\tgit -C server add a .git-a b &&\n+\tgit -C server commit -m x &&\n+\n+\tgit -C server rev-parse HEAD >objects &&\n+\tgit -C server pack-objects --revs --stdout --filter=blobs:limit=10 my.pack &&\n+\n+\t# Ensure that the .git-a blob is in the packfile, despite also\n+\t# appearing as a non-.git file\n+\tgit index-pack my.pack &&\n+\tgit verify-pack -v my.idx >objectlist &&\n+\tgrep $(git hash-object server/a) objectlist\n+'\n+\n #\n # WARNING!\n #\ndiff --git a/t/test-lib-functions.sh b/t/test-lib-functions.sh\nindex 1701fe2..07b79c7 100644\n--- a/t/test-lib-functions.sh\n+++ b/t/test-lib-functions.sh\n@@ -1020,3 +1020,15 @@ nongit () {\n \t\t\"$@\"\n \t)\n }\n+\n+# Converts big-endian pairs of hexadecimal digits into bytes. For example,\n+# \"printf 61620d0a | hex_pack\" results in \"ab\\r\\n\".\n+hex_pack () {\n+\tperl -e '$/ = undef; $input = <>; print pack(\"H*\", $input)'\n+}\n+\n+# Converts bytes into big-endian pairs of hexadecimal digits. For example,\n+# \"printf 'ab\\r\\n' | hex_unpack\" results in \"61620d0a\".\n+hex_unpack () {\n+\tperl -e '$/ = undef; $input = <>; print unpack(\"H2\" x length($input), $input)'\n+}\n-- \n2.9.3\n\n"},{"id":"331724","messageId":"20171102203129.59417-4-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 03/14] fetch: refactor calculation of remote list","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:18Z","receivedAt":"2017-11-02T20:32:21Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan \n\nSeparate out the calculation of remotes to be fetched from and the\nactual fetching. This will allow us to include an additional step before\nthe actual fetching in a subsequent commit.\n\nSigned-off-by: Jonathan Tan \n---\n builtin/fetch.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 225c734..1b1f039 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1322,7 +1322,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n-\tstruct remote *remote;\n+\tstruct remote *remote = NULL;\n \tint result = 0;\n \tstruct argv_array argv_gc_auto = ARGV_ARRAY_INIT;\n \n@@ -1367,17 +1367,14 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\telse if (argc > 1)\n \t\t\tdie(_(\"fetch --all does not make sense with refspecs\"));\n \t\t(void) for_each_remote(get_one_remote_for_fetch, &list);\n-\t\tresult = fetch_multiple(&list);\n \t} else if (argc == 0) {\n \t\t/* No arguments -- use default remote */\n \t\tremote = remote_get(NULL);\n-\t\tresult = fetch_one(remote, argc, argv);\n \t} else if (multiple) {\n \t\t/* All arguments are assumed to be remotes or groups */\n \t\tfor (i = 0; i < argc; i++)\n \t\t\tif (!add_remote_or_group(argv[i], &list))\n \t\t\t\tdie(_(\"No such remote or remote group: %s\"), argv[i]);\n-\t\tresult = fetch_multiple(&list);\n \t} else {\n \t\t/* Single remote or group */\n \t\t(void) add_remote_or_group(argv[0], &list);\n@@ -1385,14 +1382,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t\t/* More than one remote */\n \t\t\tif (argc > 1)\n \t\t\t\tdie(_(\"Fetching a group and specifying refspecs does not make sense\"));\n-\t\t\tresult = fetch_multiple(&list);\n \t\t} else {\n \t\t\t/* Zero or one remotes */\n \t\t\tremote = remote_get(argv[0]);\n-\t\t\tresult = fetch_one(remote, argc-1, argv+1);\n+\t\t\targc--;\n+\t\t\targv++;\n \t\t}\n \t}\n \n+\tif (remote)\n+\t\tresult = fetch_one(remote, argc, argv);\n+\telse\n+\t\tresult = fetch_multiple(&list);\n+\n \tif (!result && (recurse_submodules != RECURSE_SUBMODULES_OFF)) {\n \t\tstruct argv_array options = ARGV_ARRAY_INIT;\n \n-- \n2.9.3\n\n"},{"id":"331725","messageId":"20171102203129.59417-2-git@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"[PATCH 01/14] upload-pack: add object filtering for partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-02T20:31:16Z","receivedAt":"2017-11-02T20:32:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler \n\nTeach upload-pack to negotiate object filtering over the protocol and\nto send filter parameters to pack-objects. This is intended for partial\nclone and fetch.\n\nThe idea to make upload-pack configurable using uploadpack.allowFilter\ncomes from Jonathan Tan's work in [1].\n\n[1] https://public-inbox.org/git/f211093280b422c32cc1b7034130072f35c5ed51.1506714999.git.jonathantanmy@google.com/\n\nSigned-off-by: Jeff Hostetler \n---\n Documentation/config.txt | 4 ++++\n Documentation/technical/pack-protocol.txt | 8 ++++++++\n Documentation/technical/protocol-capabilities.txt | 8 ++++++++\n upload-pack.c | 20 +++++++++++++++++++-\n 4 files changed, 39 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 1ac0ae6..e528210 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -3268,6 +3268,10 @@ uploadpack.packObjectsHook::\n \twas run. I.e., `upload-pack` will feed input intended for\n \t`pack-objects` to the hook, and expects a completed packfile on\n \tstdout.\n+\n+uploadpack.allowFilter::\n+\tIf this option is set, `upload-pack` will advertise partial\n+\tclone and partial fetch object filtering.\n +\n Note that this configuration variable is ignored if it is seen in the\n repository-level config (this is a safety measure against fetching from\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex ed1eae8..a43a113 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.\n upload-request = want-list\n \t\t *shallow-line\n \t\t *1depth-request\n+\t\t [filter-request]\n \t\t flush-pkt\n \n want-list = first-want\n@@ -227,6 +228,8 @@ out of what the server said it could do with the first 'want' line.\n additional-want = PKT-LINE(\"want\" SP obj-id)\n \n depth = 1*DIGIT\n+\n+ filter-request = PKT-LINE(\"filter\" SP filter-spec)\n ----\n \n Clients MUST send all the obj-ids it wants from the reference\n@@ -249,6 +252,11 @@ complete those commits. Commits whose parents are not received as a\n result are defined as shallow and marked as such in the server. This\n information is sent back to the client in the next step.\n \n+The client can optionally request that pack-objects omit various\n+objects from the packfile using one of several filtering techniques.\n+These are intended for use with partial clone and partial fetch\n+operations. See `rev-list` for possible \"filter-spec\" values.\n+\n Once all the 'want's and 'shallow's (and optional 'deepen') are\n transferred, clients MUST send a flush-pkt, to tell the server side\n that it is done sending the list.\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 26dcc6f..332d209 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -309,3 +309,11 @@ to accept a signed push certificate, and asks the to be\n included in the push certificate. A send-pack client MUST NOT\n send a push-cert packet unless the receive-pack server advertises\n this capability.\n+\n+filter\n+------\n+\n+If the upload-pack server advertises the 'filter' capability,\n+fetch-pack may send \"filter\" commands to request a partial clone\n+or partial fetch and request that the server omit various objects\n+from the packfile.\ndiff --git a/upload-pack.c b/upload-pack.c\nindex e25f725..64a57a4 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -10,6 +10,8 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"list-objects-filter.h\"\n+#include \"list-objects-filter-options.h\"\n #include \"run-command.h\"\n #include \"connect.h\"\n #include \"sigchain.h\"\n@@ -64,6 +66,10 @@ static int advertise_refs;\n static int stateless_rpc;\n static const char *pack_objects_hook;\n \n+static int filter_capability_requested;\n+static int filter_advertise;\n+static struct list_objects_filter_options filter_options;\n+\n static void reset_timeout(void)\n {\n \talarm(timeout);\n@@ -131,6 +137,7 @@ static void create_pack_file(void)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n+\targ_format_list_objects_filter(&pack_objects.args, &filter_options);\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n@@ -794,6 +801,12 @@ static void receive_needs(void)\n \t\t\tdeepen_rev_list = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(line, \"filter \", &arg)) {\n+\t\t\tif (!filter_capability_requested)\n+\t\t\t\tdie(\"git upload-pack: filtering capability not negotiated\");\n+\t\t\tparse_list_objects_filter(&filter_options, arg);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!skip_prefix(line, \"want \", &arg) ||\n \t\t get_oid_hex(arg, &oid_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n@@ -821,6 +834,8 @@ static void receive_needs(void)\n \t\t\tno_progress = 1;\n \t\tif (parse_feature_request(features, \"include-tag\"))\n \t\t\tuse_include_tag = 1;\n+\t\tif (parse_feature_request(features, \"filter\"))\n+\t\t\tfilter_capability_requested = 1;\n \n \t\to = parse_object(&oid_buf);\n \t\tif (!o) {\n@@ -940,7 +955,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\tstruct strbuf symref_info = STRBUF_INIT;\n \n \t\tformat_symref_info(&symref_info, cb_data);\n-\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s agent=%s\\n\",\n+\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s agent=%s\\n\",\n \t\t\t oid_to_hex(oid), refname_nons,\n \t\t\t 0, capabilities,\n \t\t\t (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?\n@@ -949,6 +964,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\t\t\t \" allow-reachable-sha1-in-want\" : \"\",\n \t\t\t stateless_rpc ? \" no-done\" : \"\",\n \t\t\t symref_info.buf,\n+\t\t\t filter_advertise ? \" filter\" : \"\",\n \t\t\t git_user_agent_sanitized());\n \t\tstrbuf_release(&symref_info);\n \t} else {\n@@ -1027,6 +1043,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused)\n \t} else if (current_config_scope() != CONFIG_SCOPE_REPO) {\n \t\tif (!strcmp(\"uploadpack.packobjectshook\", var))\n \t\t\treturn git_config_string(&pack_objects_hook, var, value);\n+\t} else if (!strcmp(\"uploadpack.allowfilter\", var)) {\n+\t\tfilter_advertise = git_config_bool(var, value);\n \t}\n \treturn parse_hide_refs_config(var, value, \"uploadpack\");\n }\n-- \n2.9.3\n\n"},{"id":"331748","messageId":"20171102164135.435c05e07f71dcaf34149c2e@google.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"Re: [PATCH 00/14] WIP Partial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-11-02T23:41:35Z","receivedAt":"2017-11-02T23:41:44Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 2 Nov 2017 20:31:15 +0000\nJeff Hostetler wrote:\n\n> From: Jeff Hostetler \n> \n> This is part 3 of 3 for partial clone.\n> It assumes that part 1 [1] and part 2 [2] are in place.\n> \n> Part 3 is concerned with the commands: clone, fetch, upload-pack, fetch-pack,\n> remote-curl, index-pack, and the pack-protocol.\n> \n> Jonathan and I independently started on this task. This is a first\n> pass at merging those efforts. So there are several places that need\n> refactoring and cleanup. In particular, the test cases should be\n> squashed and new tests added.\n\nThanks. What are your future plans with this patch set? In particular, the\ntests don't pass at HEAD^.\n\nI took a quick glance to see if there were any issues that I could\nimmediately spot, but couldn't find any. I thought of fetch_if_missing,\nbut it seems that it is indeed used in this patch set (as expected).\n\nI'll look at it more thorougly, and feel free to let me know if there is\nanything in particular you would like comments on.\n"},{"id":"331781","messageId":"7fcb5464-d638-74f7-db5d-3b015e88ec2d@jeffhostetler.com","threadId":"47104","inReplyTo":"20171102164135.435c05e07f71dcaf34149c2e@google.com","subject":"Re: [PATCH 00/14] WIP Partial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-03T14:22:43Z","receivedAt":"2017-11-03T14:22:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/2/2017 7:41 PM, Jonathan Tan wrote:\n> On Thu, 2 Nov 2017 20:31:15 +0000\n> Jeff Hostetler wrote:\n> \n>> From: Jeff Hostetler \n>>\n>> This is part 3 of 3 for partial clone.\n>> It assumes that part 1 [1] and part 2 [2] are in place.\n>>\n>> Part 3 is concerned with the commands: clone, fetch, upload-pack, fetch-pack,\n>> remote-curl, index-pack, and the pack-protocol.\n>>\n>> Jonathan and I independently started on this task. This is a first\n>> pass at merging those efforts. So there are several places that need\n>> refactoring and cleanup. In particular, the test cases should be\n>> squashed and new tests added.\n> \n> Thanks. What are your future plans with this patch set? In particular, the\n> tests don't pass at HEAD^.\n> \nPatch 14/14 fixed 2 existing tests. I think I want to merge that with\npatch 2/14 as part of the cleanup.\n\nBigger picture, I would like squash all this down. But first I wanted\nyou to see if there was anything I missed during the merge.\n\n> I took a quick glance to see if there were any issues that I could\n> immediately spot, but couldn't find any. I thought of fetch_if_missing,\n> but it seems that it is indeed used in this patch set (as expected).\n> \n> I'll look at it more thorougly, and feel free to let me know if there is\n> anything in particular you would like comments on.\n> \n\nThanks, will do.\nJeff\n\n"},{"id":"331785","messageId":"xmqqy3nno0py.fsf@gitster.mtv.corp.google.com","threadId":"47104","inReplyTo":"20171102203129.59417-1-git@jeffhostetler.com","subject":"Re: [PATCH 00/14] WIP Partial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-03T15:12:25Z","receivedAt":"2017-11-03T15:12:33Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler writes:\n\n> From: Jeff Hostetler \n>\n> This is part 3 of 3 for partial clone.\n> It assumes that part 1 [1] and part 2 [2] are in place.\n\nThanks. As planned these replaced the partial clone with size\nfilter thing from Jonathan. The resulting integration passed the\ntests locally so I pushed it out.\n\nBy the way, the enhancement in this series made to list-objects.c\nhad a bit of interaction with the last round of Stefan's \"describe\nblob\" topic when both were merged to 'pu'. I think I resolved it\ncorrectly, but the merge resolution can use extra sets of eyes.\n\ndiff --cc list-objects.c\nindex 5390a7440d,07a92f35fe..e05de01af1\n--- a/list-objects.c\n+++ b/list-objects.c\n@@@ -220,32 -183,20 +220,22 @@@ static void add_pending_tree(struct rev\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n- static void do_traverse(struct rev_info *revs,\n- \t\t\tshow_commit_fn show_commit,\n- \t\t\tshow_object_fn show_object,\n- \t\t\tvoid *show_data,\n- \t\t\tfilter_object_fn filter_fn,\n- \t\t\tvoid *filter_data)\n+ static void traverse_trees_and_blobs(struct rev_info *revs,\n+ \t\t\t\t struct strbuf *base,\n+ \t\t\t\t show_object_fn show_object,\n -\t\t\t\t void *data)\n++\t\t\t\t void *show_data,\n++\t\t\t\t filter_object_fn filter_fn,\n++\t\t\t\t void *filter_data)\n {\n \tint i;\n- \tstruct commit *commit;\n- \tstruct strbuf base;\n \n- \tstrbuf_init(&base, PATH_MAX);\n- \twhile ((commit = get_revision(revs)) != NULL) {\n- \t\t/*\n- \t\t * an uninteresting boundary commit may not have its tree\n- \t\t * parsed yet, but we are not going to show them anyway\n- \t\t */\n- \t\tif (commit->tree)\n- \t\t\tadd_pending_tree(revs, commit->tree);\n- \t\tshow_commit(commit, show_data);\n- \t}\n -\tassert(base->len == 0);\n -\n++\tassert(!base->len);\n \tfor (i = 0; i < revs->pending.nr; i++) {\n \t\tstruct object_array_entry *pending = revs->pending.objects + i;\n \t\tstruct object *obj = pending->item;\n \t\tconst char *name = pending->name;\n \t\tconst char *path = pending->path;\n++\n \t\tif (obj->flags & (UNINTERESTING | SEEN))\n \t\t\tcontinue;\n \t\tif (obj->type == OBJ_TAG) {\n@@@ -257,47 -208,41 +247,76 @@@\n \t\t\tpath = \"\";\n \t\tif (obj->type == OBJ_TREE) {\n \t\t\tprocess_tree(revs, (struct tree *)obj, show_object,\n- \t\t\t\t &base, path, show_data,\n -\t\t\t\t base, path, data);\n++\t\t\t\t base, path, show_data,\n +\t\t\t\t filter_fn, filter_data);\n \t\t\tcontinue;\n \t\t}\n \t\tif (obj->type == OBJ_BLOB) {\n \t\t\tprocess_blob(revs, (struct blob *)obj, show_object,\n- \t\t\t\t &base, path, show_data,\n -\t\t\t\t base, path, data);\n++\t\t\t\t base, path, show_data,\n +\t\t\t\t filter_fn, filter_data);\n \t\t\tcontinue;\n \t\t}\n \t\tdie(\"unknown pending object %s (%s)\",\n \t\t oid_to_hex(&obj->oid), name);\n \t}\n \tobject_array_clear(&revs->pending);\n- \tstrbuf_release(&base);\n+ }\n+ \n -void traverse_commit_list(struct rev_info *revs,\n -\t\t\t show_commit_fn show_commit,\n -\t\t\t show_object_fn show_object,\n -\t\t\t void *data)\n++static void do_traverse(struct rev_info *revs,\n++\t\t\tshow_commit_fn show_commit,\n++\t\t\tshow_object_fn show_object,\n++\t\t\tvoid *show_data,\n++\t\t\tfilter_object_fn filter_fn,\n++\t\t\tvoid *filter_data)\n+ {\n+ \tstruct commit *commit;\n+ \tstruct strbuf csp; /* callee's scratch pad */\n -\tstrbuf_init(&csp, PATH_MAX);\n+ \n++\tstrbuf_init(&csp, PATH_MAX);\n+ \twhile ((commit = get_revision(revs)) != NULL) {\n+ \t\t/*\n+ \t\t * an uninteresting boundary commit may not have its tree\n+ \t\t * parsed yet, but we are not going to show them anyway\n+ \t\t */\n+ \t\tif (commit->tree)\n+ \t\t\tadd_pending_tree(revs, commit->tree);\n -\t\tshow_commit(commit, data);\n++\t\tshow_commit(commit, show_data);\n+ \t\tif (revs->tree_blobs_in_commit_order)\n -\t\t\ttraverse_trees_and_blobs(revs, &csp, show_object, data);\n++\t\t\ttraverse_trees_and_blobs(revs, &csp,\n++\t\t\t\t\t\t show_object, show_data,\n++\t\t\t\t\t\t filter_fn, filter_data);\n+ \t}\n -\ttraverse_trees_and_blobs(revs, &csp, show_object, data);\n -\n++\ttraverse_trees_and_blobs(revs, &csp,\n++\t\t\t\t show_object, show_data,\n++\t\t\t\t filter_fn, filter_data);\n+ \tstrbuf_release(&csp);\n }\n +\n +void traverse_commit_list(struct rev_info *revs,\n +\t\t\t show_commit_fn show_commit,\n +\t\t\t show_object_fn show_object,\n +\t\t\t void *show_data)\n +{\n +\tdo_traverse(revs, show_commit, show_object, show_data, NULL, NULL);\n +}\n +\n +void traverse_commit_list_filtered(\n +\tstruct list_objects_filter_options *filter_options,\n +\tstruct rev_info *revs,\n +\tshow_commit_fn show_commit,\n +\tshow_object_fn show_object,\n +\tvoid *show_data,\n +\tstruct oidset *omitted)\n +{\n +\tfilter_object_fn filter_fn = NULL;\n +\tfilter_free_fn filter_free_fn = NULL;\n +\tvoid *filter_data = NULL;\n +\n +\tfilter_data = list_objects_filter__init(omitted, filter_options,\n +\t\t\t\t\t\t&filter_fn, &filter_free_fn);\n +\tdo_traverse(revs, show_commit, show_object, show_data,\n +\t\t filter_fn, filter_data);\n +\tif (filter_data && filter_free_fn)\n +\t\tfilter_free_fn(filter_data);\n +}\n"},{"id":"331804","messageId":"20171103133229.1612a60560469af53c380a76@google.com","threadId":"47104","inReplyTo":"20171102203129.59417-3-git@jeffhostetler.com","subject":"Re: [PATCH 02/14] clone, fetch-pack, index-pack, transport: partial clone","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-11-03T20:32:29Z","receivedAt":"2017-11-03T20:32:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Thu, 2 Nov 2017 20:31:17 +0000\nJeff Hostetler wrote:\n\n> From: Jeff Hostetler \n> \n> Signed-off-by: Jeff Hostetler \n> ---\n> builtin/clone.c | 9 +++++++++\n> builtin/fetch-pack.c | 4 ++++\n> builtin/index-pack.c | 10 ++++++++++\n> fetch-pack.c | 13 +++++++++++++\n> fetch-pack.h | 2 ++\n> transport-helper.c | 5 +++++\n> transport.c | 4 ++++\n> transport.h | 5 +++++\n> 8 files changed, 52 insertions(+)\n\nI managed to take a look at some of these patches. Firstly, consider\nseparating out the clone part, since it will not be tested until a few\npatches later.\n\n> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> index a0a35e6..31cd5ba 100644\n> --- a/builtin/index-pack.c\n> +++ b/builtin/index-pack.c\n> @@ -222,6 +222,16 @@ static unsigned check_object(struct object *obj)\n> \tif (!(obj->flags & FLAG_CHECKED)) {\n> \t\tunsigned long size;\n> \t\tint type = sha1_object_info(obj->oid.hash, &size);\n> +\n> +\t\tif (type <= 0) {\n> +\t\t\t/*\n> +\t\t\t * TODO Use the promisor code to conditionally\n> +\t\t\t * try to fetch this object -or- assume it is ok.\n> +\t\t\t */\n> +\t\t\tobj->flags |= FLAG_CHECKED;\n> +\t\t\treturn 0;\n> +\t\t}\n> +\n> \t\tif (type <= 0)\n> \t\t\tdie(_(\"did not receive expected object %s\"),\n> \t\t\t oid_to_hex(&obj->oid));\n\nThis causes some repo corruption tests to fail.\n\nIf I remove this and rebase the fetch-pack tests on top [1], the tests\npass, so this might not be necessary (for now, at least).\n\n[1] https://github.com/jonathantanmy/git/commits/pc20171103\n"},{"id":"331805","messageId":"20171103133845.b270485dd04c6c6c1b47d42a@google.com","threadId":"47104","inReplyTo":"20171102203129.59417-5-git@jeffhostetler.com","subject":"Re: [PATCH 04/14] fetch: add object filtering for partial fetch","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-11-03T20:38:45Z","receivedAt":"2017-11-03T20:38:52Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"I did some of my own investigation and have a working (i.e. passing\ntests) version of this patch here:\n\nhttps://github.com/jonathantanmy/git/commits/pc20171103\n\nIf you want, you can use that, or incorporate the changes therein here.\nI'll also remark on my findings inline.\n\nOn Thu, 2 Nov 2017 20:31:19 +0000\nJeff Hostetler wrote:\n\n> @@ -754,6 +757,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n> \tconst char *filename = dry_run ? \"/dev/null\" : git_path_fetch_head();\n> \tint want_status;\n> \tint summary_width = transport_summary_width(ref_map);\n> +\tstruct check_connected_options opt = CHECK_CONNECTED_INIT;\n> \n> \tfp = fopen(filename, \"a\");\n> \tif (!fp)\n> @@ -765,7 +769,7 @@ static int store_updated_refs(const char *raw_url, const char *remote_name,\n> \t\turl = xstrdup(\"foreign\");\n> \n> \trm = ref_map;\n> -\tif (check_connected(iterate_ref_map, &rm, NULL)) {\n> +\tif (check_connected(iterate_ref_map, &rm, &opt)) {\n\nopt here is unchanged from CHECK_CONNECTED_INIT, so this change is unnecessary.\n\n> \t\trc = error(_(\"%s did not send all necessary objects\\n\"), url);\n> \t\tgoto abort;\n> \t}\n> @@ -1044,6 +1048,9 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n> \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n> \tif (update_shallow)\n> \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n> +\tif (filter_options.choice)\n> +\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n> +\t\t\t filter_options.raw_value);\n\nYou'll also need to set TRANS_OPT_FROM_PROMISOR.\n\n> @@ -1242,6 +1249,20 @@ static int fetch_multiple(struct string_list *list)\n> \tint i, result = 0;\n> \tstruct argv_array argv = ARGV_ARRAY_INIT;\n> \n> +\tif (filter_options.choice) {\n> +\t\t/*\n> +\t\t * We currently only support partial-fetches to the remote\n> +\t\t * used for the partial-clone because we only support 1\n> +\t\t * promisor remote, so we DO NOT allow explicit command\n> +\t\t * line filter arguments.\n> +\t\t *\n> +\t\t * Note that the loop below will spawn background fetches\n> +\t\t * for each remote and one of them MAY INHERIT the proper\n> +\t\t * partial-fetch settings, so everything is consistent.\n> +\t\t */\n> +\t\tdie(_(\"partial-fetch is not supported on multiple remotes\"));\n> +\t}\n> +\n> \tif (!append && !dry_run) {\n> \t\tint errcode = truncate_fetch_head();\n> \t\tif (errcode)\n\nMy intention in doing the \"fetch: refactor calculation of remote list\"\npatch is so that the interaction between the provided list of remotes\nand the specification of the filter can be handled using the following\ndiff:\n\n -\tif (remote)\n +\tif (remote) {\n +\t\tif (filter_options.choice &&\n +\t\t strcmp(remote->name, repository_format_partial_clone_remote))\n +\t\t\tdie(_(\"--blob-max-bytes can only be used with the remote configured in core.partialClone\"));\n \t\tresult = fetch_one(remote, argc, argv);\n -\telse\n +\t} else {\n +\t\tif (filter_options.choice)\n +\t\t\tdie(_(\"--blob-max-bytes can only be used with the remote configured in core.partialClone\"));\n \t\tresult = fetch_multiple(&list);\n +\t}\n\n(Ignore the \"blob-max-bytes\" in the error message - that needs to be\nupdated.)\n\nThe GitHub link I provided above has this diff, and it seems to work.\n"},{"id":"332086","messageId":"20171108180127.GG20681@dinwoodie.org","threadId":"47104","inReplyTo":"20171103133229.1612a60560469af53c380a76@google.com","subject":"Re: [PATCH 02/14] clone, fetch-pack, index-pack, transport: partial clone","fromName":"Adam Dinwoodie","fromEmail":"adam@dinwoodie.org","sentAt":"2017-11-08T18:01:27Z","receivedAt":"2017-11-08T18:01:36Z","isPatch":true,"sender":{"key":"adam@dinwoodie.org","avatar":"https://avatars.githubusercontent.com/u/1397507?v=4"},"body":"On Friday 03 November 2017 at 01:32 pm -0700, Jonathan Tan wrote:\n> On Thu, 2 Nov 2017 20:31:17 +0000\n> Jeff Hostetler wrote:\n> > diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n> > index a0a35e6..31cd5ba 100644\n> > --- a/builtin/index-pack.c\n> > +++ b/builtin/index-pack.c\n> > @@ -222,6 +222,16 @@ static unsigned check_object(struct object *obj)\n> > \tif (!(obj->flags & FLAG_CHECKED)) {\n> > \t\tunsigned long size;\n> > \t\tint type = sha1_object_info(obj->oid.hash, &size);\n> > +\n> > +\t\tif (type <= 0) {\n> > +\t\t\t/*\n> > +\t\t\t * TODO Use the promisor code to conditionally\n> > +\t\t\t * try to fetch this object -or- assume it is ok.\n> > +\t\t\t */\n> > +\t\t\tobj->flags |= FLAG_CHECKED;\n> > +\t\t\treturn 0;\n> > +\t\t}\n> > +\n> > \t\tif (type <= 0)\n> > \t\t\tdie(_(\"did not receive expected object %s\"),\n> > \t\t\t oid_to_hex(&obj->oid));\n> \n> This causes some repo corruption tests to fail.\n\nConfirmed: I see this patch, or at least f7e0dbc38 (\"clone, fetch-pack,\nindex-pack, transport: partial clone\", 2017-11-02), causing t5300.26 to\nfail on 64-bit Cygwin.\n\nFor the sake of anyone trying to reproduce this, I needed to cherry pick\n66d4c7a58 (\"fixup! upload-pack: add object filtering for partial clone\",\n2017-11-08) onto that commit before I was able to get it to compile.\n\nAdam\n"},{"id":"332681","messageId":"18ed2934-bf90-dc40-5d94-9de058d8f7f1@jeffhostetler.com","threadId":"47104","inReplyTo":"20171108180127.GG20681@dinwoodie.org","subject":"Re: [PATCH 02/14] clone, fetch-pack, index-pack, transport: partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-16T17:43:44Z","receivedAt":"2017-11-16T17:44:03Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/8/2017 1:01 PM, Adam Dinwoodie wrote:\n> On Friday 03 November 2017 at 01:32 pm -0700, Jonathan Tan wrote:\n>> On Thu, 2 Nov 2017 20:31:17 +0000\n>> Jeff Hostetler wrote:\n>>> diff --git a/builtin/index-pack.c b/builtin/index-pack.c\n>>> index a0a35e6..31cd5ba 100644\n>>> --- a/builtin/index-pack.c\n>>> +++ b/builtin/index-pack.c\n>>> @@ -222,6 +222,16 @@ static unsigned check_object(struct object *obj)\n>>> \tif (!(obj->flags & FLAG_CHECKED)) {\n>>> \t\tunsigned long size;\n>>> \t\tint type = sha1_object_info(obj->oid.hash, &size);\n>>> +\n>>> +\t\tif (type <= 0) {\n>>> +\t\t\t/*\n>>> +\t\t\t * TODO Use the promisor code to conditionally\n>>> +\t\t\t * try to fetch this object -or- assume it is ok.\n>>> +\t\t\t */\n>>> +\t\t\tobj->flags |= FLAG_CHECKED;\n>>> +\t\t\treturn 0;\n>>> +\t\t}\n>>> +\n>>> \t\tif (type <= 0)\n>>> \t\t\tdie(_(\"did not receive expected object %s\"),\n>>> \t\t\t oid_to_hex(&obj->oid));\n>>\n>> This causes some repo corruption tests to fail.\n> \n> Confirmed: I see this patch, or at least f7e0dbc38 (\"clone, fetch-pack,\n> index-pack, transport: partial clone\", 2017-11-02), causing t5300.26 to\n> fail on 64-bit Cygwin.\n> \n> For the sake of anyone trying to reproduce this, I needed to cherry pick\n> 66d4c7a58 (\"fixup! upload-pack: add object filtering for partial clone\",\n> 2017-11-08) onto that commit before I was able to get it to compile.\n> \n> Adam\n> \n\nThanks. I've removed this from my next version. I think it was\nleft over from a pre-promisor version.\n\nJeff\n"},{"id":"332682","messageId":"7eb6b165-77e5-78b9-e3a9-151f6a53a39a@jeffhostetler.com","threadId":"47104","inReplyTo":"20171103133845.b270485dd04c6c6c1b47d42a@google.com","subject":"Re: [PATCH 04/14] fetch: add object filtering for partial fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-11-16T17:53:27Z","receivedAt":"2017-11-16T17:55:06Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 11/3/2017 4:38 PM, Jonathan Tan wrote:\n>> @@ -1242,6 +1249,20 @@ static int fetch_multiple(struct string_list *list)\n>> \tint i, result = 0;\n>> \tstruct argv_array argv = ARGV_ARRAY_INIT;\n>> \n>> +\tif (filter_options.choice) {\n>> +\t\t/*\n>> +\t\t * We currently only support partial-fetches to the remote\n>> +\t\t * used for the partial-clone because we only support 1\n>> +\t\t * promisor remote, so we DO NOT allow explicit command\n>> +\t\t * line filter arguments.\n>> +\t\t *\n>> +\t\t * Note that the loop below will spawn background fetches\n>> +\t\t * for each remote and one of them MAY INHERIT the proper\n>> +\t\t * partial-fetch settings, so everything is consistent.\n>> +\t\t */\n>> +\t\tdie(_(\"partial-fetch is not supported on multiple remotes\"));\n>> +\t}\n>> +\n>> \tif (!append && !dry_run) {\n>> \t\tint errcode = truncate_fetch_head();\n>> \t\tif (errcode)\n> \n> My intention in doing the \"fetch: refactor calculation of remote list\"\n> patch is so that the interaction between the provided list of remotes\n> and the specification of the filter can be handled using the following\n> diff:\n> \n> -\tif (remote)\n> +\tif (remote) {\n> +\t\tif (filter_options.choice &&\n> +\t\t strcmp(remote->name, repository_format_partial_clone_remote))\n> +\t\t\tdie(_(\"--blob-max-bytes can only be used with the remote configured in core.partialClone\"));\n> \t\tresult = fetch_one(remote, argc, argv);\n> -\telse\n> +\t} else {\n> +\t\tif (filter_options.choice)\n> +\t\t\tdie(_(\"--blob-max-bytes can only be used with the remote configured in core.partialClone\"));\n> \t\tresult = fetch_multiple(&list);\n> +\t}\n> \n> (Ignore the \"blob-max-bytes\" in the error message - that needs to be\n> updated.)\n> \n> The GitHub link I provided above has this diff, and it seems to work.\n> \n\nI put the filter_options.choice tests inside the fetch_{one,multiple}\nroutines because the former needs to be able to register partial clone\nwith the config and/or inherit the default filter-spec for the\npromisor remote and that took more code that what can neatly fit inline\nhere. This will be more apparent in my next patch series.\n\nJeff\n"}]}