{"thread":{"id":"47411","subject":"[PATCH v7 00/16] Parial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","startedAt":"2017-12-08T15:59:16Z","lastAt":"2018-08-20T19:42:36Z","messageCount":24,"participants":["Jeff Hostetler","Junio C Hamano","Brandon Williams","Jonathan Tan","Duy Nguyen"],"isPatch":true,"patchVersion":7,"patchTotal":16},"messages":[{"id":"334434","messageId":"20171208155851.855-1-git@jeffhostetler.com","threadId":"47411","inReplyTo":null,"subject":"[PATCH v7 00/16] Parial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:35Z","receivedAt":"2017-12-08T15:59:16Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nThis is V7 of part 3 of partial clone.  It builds upon V7 of part 2\n(which builds upon V6 of part 1).\n\nThis version adds additional tests, fixes test errors on the MAC version,\nand squashes some fixup commits.\n\nIt also restores functionality accidentally dropped from the V6 series\nfor \"git fetch\" to automatically inherit the partial-clone filter-spec\nwhen appropriate.  This version extends the --no-filter argument to\noverride this inheritance.\n\nJeff Hostetler (8):\n  upload-pack: add object filtering for partial clone\n  fetch-pack, index-pack, transport: partial clone\n  fetch-pack: add --no-filter\n  fetch: support filters\n  partial-clone: define partial clone settings in config\n  t5616: end-to-end tests for partial clone\n  fetch: inherit filter-spec from partial clone\n  t5616: test bulk prefetch after partial fetch\n\nJonathan Tan (8):\n  sha1_file: support lazily fetching missing objects\n  rev-list: support termination at promisor objects\n  gc: do not repack promisor packfiles\n  fetch-pack: test support excluding large blobs\n  fetch: refactor calculation of remote list\n  clone: partial clone\n  unpack-trees: batch fetching of missing blobs\n  fetch-pack: restore save_commit_buffer after use\n\n Documentation/config.txt                          |   4 +\n Documentation/git-pack-objects.txt                |  11 ++\n Documentation/rev-list-options.txt                |  11 ++\n Documentation/technical/pack-protocol.txt         |   8 +\n Documentation/technical/protocol-capabilities.txt |   8 +\n builtin/cat-file.c                                |   2 +\n builtin/clone.c                                   |  22 ++-\n builtin/fetch-pack.c                              |  10 ++\n builtin/fetch.c                                   |  83 ++++++++-\n builtin/fsck.c                                    |   3 +\n builtin/gc.c                                      |   3 +\n builtin/index-pack.c                              |   6 +\n builtin/pack-objects.c                            |  37 +++-\n builtin/prune.c                                   |   7 +\n builtin/repack.c                                  |   8 +-\n builtin/rev-list.c                                |  73 +++++++-\n cache.h                                           |   9 +\n config.c                                          |   5 +\n connected.c                                       |   2 +\n environment.c                                     |   1 +\n fetch-object.c                                    |  29 ++-\n fetch-object.h                                    |   5 +\n fetch-pack.c                                      |  17 ++\n fetch-pack.h                                      |   2 +\n list-objects-filter-options.c                     |  92 ++++++++--\n list-objects-filter-options.h                     |  18 ++\n list-objects.c                                    |  29 ++-\n object.c                                          |   2 +-\n remote-curl.c                                     |   6 +\n revision.c                                        |  33 +++-\n revision.h                                        |   5 +-\n sha1_file.c                                       |  32 +++-\n t/t0410-partial-clone.sh                          | 206 +++++++++++++++++++++-\n t/t5500-fetch-pack.sh                             |  63 +++++++\n t/t5601-clone.sh                                  | 101 +++++++++++\n t/t5616-partial-clone.sh                          | 146 +++++++++++++++\n transport-helper.c                                |   5 +\n transport.c                                       |   4 +\n transport.h                                       |   5 +\n unpack-trees.c                                    |  22 +++\n upload-pack.c                                     |  31 +++-\n 41 files changed, 1110 insertions(+), 56 deletions(-)\n create mode 100755 t/t5616-partial-clone.sh\n\n-- \n2.9.3\n\n"},{"id":"334435","messageId":"20171208155851.855-4-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 03/16] gc: do not repack promisor packfiles","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:38Z","receivedAt":"2017-12-08T15:59:19Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nTeach gc to stop traversal at promisor objects, and to leave promisor\npackfiles alone. This has the effect of only repacking non-promisor\npackfiles, and preserves the distinction between promisor packfiles and\nnon-promisor packfiles.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/git-pack-objects.txt | 11 ++++++++\n builtin/gc.c                       |  3 +++\n builtin/pack-objects.c             | 37 ++++++++++++++++++++++++--\n builtin/prune.c                    |  7 +++++\n builtin/repack.c                   |  8 ++++--\n t/t0410-partial-clone.sh           | 54 ++++++++++++++++++++++++++++++++++++--\n 6 files changed, 114 insertions(+), 6 deletions(-)\n\ndiff --git a/Documentation/git-pack-objects.txt b/Documentation/git-pack-objects.txt\nindex aa403d0..81bc490 100644\n--- a/Documentation/git-pack-objects.txt\n+++ b/Documentation/git-pack-objects.txt\n@@ -255,6 +255,17 @@ a missing object is encountered.  This is the default action.\n The form '--missing=allow-any' will allow object traversal to continue\n if a missing object is encountered.  Missing objects will silently be\n omitted from the results.\n++\n+The form '--missing=allow-promisor' is like 'allow-any', but will only\n+allow object traversal to continue for EXPECTED promisor missing objects.\n+Unexpected missing object will raise an error.\n+\n+--exclude-promisor-objects::\n+\tOmit objects that are known to be in the promisor remote.  (This\n+\toption has the purpose of operating only on locally created objects,\n+\tso that when we repack, we still maintain a distinction between\n+\tlocally created objects [without .promisor] and objects from the\n+\tpromisor remote [with .promisor].)  This is used with partial clone.\n \n SEE ALSO\n --------\ndiff --git a/builtin/gc.c b/builtin/gc.c\nindex 3c5eae0..77fa720 100644\n--- a/builtin/gc.c\n+++ b/builtin/gc.c\n@@ -458,6 +458,9 @@ int cmd_gc(int argc, const char **argv, const char *prefix)\n \t\t\targv_array_push(&prune, prune_expire);\n \t\t\tif (quiet)\n \t\t\t\targv_array_push(&prune, \"--no-progress\");\n+\t\t\tif (repository_format_partial_clone)\n+\t\t\t\targv_array_push(&prune,\n+\t\t\t\t\t\t\"--exclude-promisor-objects\");\n \t\t\tif (run_command_v_opt(prune.argv, RUN_GIT_CMD))\n \t\t\t\treturn error(FAILED_RUN, prune.argv[0]);\n \t\t}\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex 45ad35d..f5fc401 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -75,6 +75,8 @@ static int use_bitmap_index = -1;\n static int write_bitmap_index;\n static uint16_t write_bitmap_options;\n \n+static int exclude_promisor_objects;\n+\n static unsigned long delta_cache_size = 0;\n static unsigned long max_delta_cache_size = 256 * 1024 * 1024;\n static unsigned long cache_max_small_delta_size = 1000;\n@@ -84,8 +86,9 @@ static unsigned long window_memory_limit = 0;\n static struct list_objects_filter_options filter_options;\n \n enum missing_action {\n-\tMA_ERROR = 0,    /* fail if any missing objects are encountered */\n-\tMA_ALLOW_ANY,    /* silently allow ALL missing objects */\n+\tMA_ERROR = 0,      /* fail if any missing objects are encountered */\n+\tMA_ALLOW_ANY,      /* silently allow ALL missing objects */\n+\tMA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */\n };\n static enum missing_action arg_missing_action;\n static show_object_fn fn_show_object;\n@@ -2577,6 +2580,20 @@ static void show_object__ma_allow_any(struct object *obj, const char *name, void\n \tshow_object(obj, name, data);\n }\n \n+static void show_object__ma_allow_promisor(struct object *obj, const char *name, void *data)\n+{\n+\tassert(arg_missing_action == MA_ALLOW_PROMISOR);\n+\n+\t/*\n+\t * Quietly ignore EXPECTED missing objects.  This avoids problems with\n+\t * staging them now and getting an odd error later.\n+\t */\n+\tif (!has_object_file(&obj->oid) && is_promisor_object(&obj->oid))\n+\t\treturn;\n+\n+\tshow_object(obj, name, data);\n+}\n+\n static int option_parse_missing_action(const struct option *opt,\n \t\t\t\t       const char *arg, int unset)\n {\n@@ -2591,10 +2608,18 @@ static int option_parse_missing_action(const struct option *opt,\n \n \tif (!strcmp(arg, \"allow-any\")) {\n \t\targ_missing_action = MA_ALLOW_ANY;\n+\t\tfetch_if_missing = 0;\n \t\tfn_show_object = show_object__ma_allow_any;\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(arg, \"allow-promisor\")) {\n+\t\targ_missing_action = MA_ALLOW_PROMISOR;\n+\t\tfetch_if_missing = 0;\n+\t\tfn_show_object = show_object__ma_allow_promisor;\n+\t\treturn 0;\n+\t}\n+\n \tdie(_(\"invalid value for --missing\"));\n \treturn 0;\n }\n@@ -3008,6 +3033,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t{ OPTION_CALLBACK, 0, \"missing\", NULL, N_(\"action\"),\n \t\t  N_(\"handling for missing objects\"), PARSE_OPT_NONEG,\n \t\t  option_parse_missing_action },\n+\t\tOPT_BOOL(0, \"exclude-promisor-objects\", &exclude_promisor_objects,\n+\t\t\t N_(\"do not pack objects in promisor packfiles\")),\n \t\tOPT_END(),\n \t};\n \n@@ -3053,6 +3080,12 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\targv_array_push(&rp, \"--unpacked\");\n \t}\n \n+\tif (exclude_promisor_objects) {\n+\t\tuse_internal_rev_list = 1;\n+\t\tfetch_if_missing = 0;\n+\t\targv_array_push(&rp, \"--exclude-promisor-objects\");\n+\t}\n+\n \tif (!reuse_object)\n \t\treuse_delta = 0;\n \tif (pack_compression_level == -1)\ndiff --git a/builtin/prune.c b/builtin/prune.c\nindex cddabf2..be34645 100644\n--- a/builtin/prune.c\n+++ b/builtin/prune.c\n@@ -101,12 +101,15 @@ int cmd_prune(int argc, const char **argv, const char *prefix)\n {\n \tstruct rev_info revs;\n \tstruct progress *progress = NULL;\n+\tint exclude_promisor_objects = 0;\n \tconst struct option options[] = {\n \t\tOPT__DRY_RUN(&show_only, N_(\"do not remove, show only\")),\n \t\tOPT__VERBOSE(&verbose, N_(\"report pruned objects\")),\n \t\tOPT_BOOL(0, \"progress\", &show_progress, N_(\"show progress\")),\n \t\tOPT_EXPIRY_DATE(0, \"expire\", &expire,\n \t\t\t\tN_(\"expire objects older than <time>\")),\n+\t\tOPT_BOOL(0, \"exclude-promisor-objects\", &exclude_promisor_objects,\n+\t\t\t N_(\"limit traversal to objects outside promisor packfiles\")),\n \t\tOPT_END()\n \t};\n \tchar *s;\n@@ -139,6 +142,10 @@ int cmd_prune(int argc, const char **argv, const char *prefix)\n \t\tshow_progress = isatty(2);\n \tif (show_progress)\n \t\tprogress = start_delayed_progress(_(\"Checking connectivity\"), 0);\n+\tif (exclude_promisor_objects) {\n+\t\tfetch_if_missing = 0;\n+\t\trevs.exclude_promisor_objects = 1;\n+\t}\n \n \tmark_reachable_objects(&revs, 1, expire, progress);\n \tstop_progress(&progress);\ndiff --git a/builtin/repack.c b/builtin/repack.c\nindex f17a68a..7bdb401 100644\n--- a/builtin/repack.c\n+++ b/builtin/repack.c\n@@ -83,7 +83,8 @@ static void remove_pack_on_signal(int signo)\n \n /*\n  * Adds all packs hex strings to the fname list, which do not\n- * have a corresponding .keep file.\n+ * have a corresponding .keep or .promisor file. These packs are not to\n+ * be kept if we are going to pack everything into one file.\n  */\n static void get_non_kept_pack_filenames(struct string_list *fname_list)\n {\n@@ -101,7 +102,8 @@ static void get_non_kept_pack_filenames(struct string_list *fname_list)\n \n \t\tfname = xmemdupz(e->d_name, len);\n \n-\t\tif (!file_exists(mkpath(\"%s/%s.keep\", packdir, fname)))\n+\t\tif (!file_exists(mkpath(\"%s/%s.keep\", packdir, fname)) &&\n+\t\t    !file_exists(mkpath(\"%s/%s.promisor\", packdir, fname)))\n \t\t\tstring_list_append_nodup(fname_list, fname);\n \t\telse\n \t\t\tfree(fname);\n@@ -232,6 +234,8 @@ int cmd_repack(int argc, const char **argv, const char *prefix)\n \targv_array_push(&cmd.args, \"--all\");\n \targv_array_push(&cmd.args, \"--reflog\");\n \targv_array_push(&cmd.args, \"--indexed-objects\");\n+\tif (repository_format_partial_clone)\n+\t\targv_array_push(&cmd.args, \"--exclude-promisor-objects\");\n \tif (window)\n \t\targv_array_pushf(&cmd.args, \"--window=%s\", window);\n \tif (window_memory)\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 3ca6af5..cc18b75 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -10,14 +10,16 @@ delete_object () {\n \n pack_as_from_promisor () {\n \tHASH=$(git -C repo pack-objects .git/objects/pack/pack) &&\n-\t>repo/.git/objects/pack/pack-$HASH.promisor\n+\t>repo/.git/objects/pack/pack-$HASH.promisor &&\n+\techo $HASH\n }\n \n promise_and_delete () {\n \tHASH=$(git -C repo rev-parse \"$1\") &&\n \tgit -C repo tag -a -m message my_annotated_tag \"$HASH\" &&\n \tgit -C repo rev-parse my_annotated_tag | pack_as_from_promisor &&\n-\tgit -C repo tag -d my_annotated_tag &&\n+\t# tag -d prints a message to stdout, so redirect it\n+\tgit -C repo tag -d my_annotated_tag >/dev/null &&\n \tdelete_object repo \"$HASH\"\n }\n \n@@ -261,6 +263,54 @@ test_expect_success 'rev-list accepts missing and promised objects on command li\n \tgit -C repo rev-list --exclude-promisor-objects --objects \"$COMMIT\" \"$TREE\" \"$BLOB\"\n '\n \n+test_expect_success 'gc does not repack promisor objects' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo my_commit &&\n+\n+\tTREE_HASH=$(git -C repo rev-parse HEAD^{tree}) &&\n+\tHASH=$(printf \"$TREE_HASH\\n\" | pack_as_from_promisor) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo gc &&\n+\n+\t# Ensure that the promisor packfile still exists, and remove it\n+\ttest -e repo/.git/objects/pack/pack-$HASH.pack &&\n+\trm repo/.git/objects/pack/pack-$HASH.* &&\n+\n+\t# Ensure that the single other pack contains the commit, but not the tree\n+\tls repo/.git/objects/pack/pack-*.pack >packlist &&\n+\ttest_line_count = 1 packlist &&\n+\tgit verify-pack repo/.git/objects/pack/pack-*.pack -v >out &&\n+\tgrep \"$(git -C repo rev-parse HEAD)\" out &&\n+\t! grep \"$TREE_HASH\" out\n+'\n+\n+test_expect_success 'gc stops traversal when a missing but promised object is reached' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo my_commit &&\n+\n+\tTREE_HASH=$(git -C repo rev-parse HEAD^{tree}) &&\n+\tHASH=$(promise_and_delete $TREE_HASH) &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo gc &&\n+\n+\t# Ensure that the promisor packfile still exists, and remove it\n+\ttest -e repo/.git/objects/pack/pack-$HASH.pack &&\n+\trm repo/.git/objects/pack/pack-$HASH.* &&\n+\n+\t# Ensure that the single other pack contains the commit, but not the tree\n+\tls repo/.git/objects/pack/pack-*.pack >packlist &&\n+\ttest_line_count = 1 packlist &&\n+\tgit verify-pack repo/.git/objects/pack/pack-*.pack -v >out &&\n+\tgrep \"$(git -C repo rev-parse HEAD)\" out &&\n+\t! grep \"$TREE_HASH\" out\n+'\n+\n LIB_HTTPD_PORT=12345  # default port, 410, cannot be used as non-root\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n-- \n2.9.3\n\n"},{"id":"334436","messageId":"20171208155851.855-10-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 09/16] fetch: support filters","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:44Z","receivedAt":"2017-12-08T15:59:22Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nTeach fetch to support filters. This is only allowed for the remote\nconfigured in extensions.partialcloneremote.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/fetch.c       | 23 +++++++++++++++++++++--\n connected.c           |  2 ++\n remote-curl.c         |  6 ++++++\n t/t5500-fetch-pack.sh | 36 ++++++++++++++++++++++++++++++++++++\n 4 files changed, 65 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 1b1f039..14aab71 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -18,6 +18,7 @@\n #include \"argv-array.h\"\n #include \"utf8.h\"\n #include \"packfile.h\"\n+#include \"list-objects-filter-options.h\"\n \n static const char * const builtin_fetch_usage[] = {\n \tN_(\"git fetch [<options>] [<repository> [<refspec>...]]\"),\n@@ -55,6 +56,7 @@ static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;\n static int shown_url = 0;\n static int refmap_alloc, refmap_nr;\n static const char **refmap_array;\n+static struct list_objects_filter_options filter_options;\n \n static int git_fetch_config(const char *k, const char *v, void *cb)\n {\n@@ -160,6 +162,7 @@ static struct option builtin_fetch_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_END()\n };\n \n@@ -1044,6 +1047,11 @@ static struct transport *prepare_transport(struct remote *remote, int deepen)\n \t\tset_option(transport, TRANS_OPT_DEEPEN_RELATIVE, \"yes\");\n \tif (update_shallow)\n \t\tset_option(transport, TRANS_OPT_UPDATE_SHALLOW, \"yes\");\n+\tif (filter_options.choice) {\n+\t\tset_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n+\t\t\t   filter_options.filter_spec);\n+\t\tset_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n+\t}\n \treturn transport;\n }\n \n@@ -1328,6 +1336,8 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \n \tpacket_trace_identity(\"fetch\");\n \n+\tfetch_if_missing = 0;\n+\n \t/* Record the command line for the reflog */\n \tstrbuf_addstr(&default_rla, \"fetch\");\n \tfor (i = 1; i < argc; i++)\n@@ -1361,6 +1371,9 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \tif (depth || deepen_since || deepen_not.nr)\n \t\tdeepen = 1;\n \n+\tif (filter_options.choice && !repository_format_partial_clone)\n+\t\tdie(\"--filter can only be used when extensions.partialClone is set\");\n+\n \tif (all) {\n \t\tif (argc == 1)\n \t\t\tdie(_(\"fetch --all does not take a repository argument\"));\n@@ -1390,10 +1403,16 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t}\n \t}\n \n-\tif (remote)\n+\tif (remote) {\n+\t\tif (filter_options.choice &&\n+\t\t    strcmp(remote->name, repository_format_partial_clone))\n+\t\t\tdie(_(\"--filter can only be used with the remote configured in core.partialClone\"));\n \t\tresult = fetch_one(remote, argc, argv);\n-\telse\n+\t} else {\n+\t\tif (filter_options.choice)\n+\t\t\tdie(_(\"--filter can only be used with the remote configured in core.partialClone\"));\n \t\tresult = fetch_multiple(&list);\n+\t}\n \n \tif (!result && (recurse_submodules != RECURSE_SUBMODULES_OFF)) {\n \t\tstruct argv_array options = ARGV_ARRAY_INIT;\ndiff --git a/connected.c b/connected.c\nindex f416b05..3a5bd67 100644\n--- a/connected.c\n+++ b/connected.c\n@@ -56,6 +56,8 @@ int check_connected(sha1_iterate_fn fn, void *cb_data,\n \targv_array_push(&rev_list.args,\"rev-list\");\n \targv_array_push(&rev_list.args, \"--objects\");\n \targv_array_push(&rev_list.args, \"--stdin\");\n+\tif (repository_format_partial_clone)\n+\t\targv_array_push(&rev_list.args, \"--exclude-promisor-objects\");\n \targv_array_push(&rev_list.args, \"--not\");\n \targv_array_push(&rev_list.args, \"--all\");\n \targv_array_push(&rev_list.args, \"--quiet\");\ndiff --git a/remote-curl.c b/remote-curl.c\nindex 4318391..6ec5352 100644\n--- a/remote-curl.c\n+++ b/remote-curl.c\n@@ -24,6 +24,7 @@ struct options {\n \tchar *deepen_since;\n \tstruct string_list deepen_not;\n \tstruct string_list push_options;\n+\tchar *filter;\n \tunsigned progress : 1,\n \t\tcheck_self_contained_and_connected : 1,\n \t\tcloning : 1,\n@@ -165,6 +166,9 @@ static int set_option(const char *name, const char *value)\n \t} else if (!strcmp(name, \"no-dependents\")) {\n \t\toptions.no_dependents = 1;\n \t\treturn 0;\n+\t} else if (!strcmp(name, \"filter\")) {\n+\t\toptions.filter = xstrdup(value);;\n+\t\treturn 0;\n \t} else {\n \t\treturn 1 /* unsupported */;\n \t}\n@@ -834,6 +838,8 @@ static int fetch_git(struct discovery *heads,\n \t\targv_array_push(&args, \"--from-promisor\");\n \tif (options.no_dependents)\n \t\targv_array_push(&args, \"--no-dependents\");\n+\tif (options.filter)\n+\t\targv_array_pushf(&args, \"--filter=%s\", options.filter);\n \targv_array_push(&args, url.buf);\n \n \tfor (i = 0; i < nr_heads; i++) {\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex c57916b..ec9ba9b 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -782,4 +782,40 @@ test_expect_success 'filtering by size has no effect if support for it is not ad\n \ttest_i18ngrep \"filtering not recognized by server\" err\n '\n \n+fetch_filter_blob_limit_zero () {\n+\tSERVER=\"$1\"\n+\tURL=\"$2\"\n+\n+\trm -rf \"$SERVER\" client &&\n+\ttest_create_repo \"$SERVER\" &&\n+\ttest_commit -C \"$SERVER\" one &&\n+\ttest_config -C \"$SERVER\" uploadpack.allowfilter 1 &&\n+\n+\tgit clone \"$URL\" client &&\n+\ttest_config -C client extensions.partialclone origin &&\n+\n+\ttest_commit -C \"$SERVER\" two &&\n+\n+\tgit -C client fetch --filter=blob:limit=0 origin HEAD:somewhere &&\n+\n+\t# Ensure that commit is fetched, but blob is not\n+\ttest_config -C client extensions.partialclone \"arbitrary string\" &&\n+\tgit -C client cat-file -e $(git -C \"$SERVER\" rev-parse two) &&\n+\ttest_must_fail git -C client cat-file -e $(git hash-object \"$SERVER/two.t\")\n+}\n+\n+test_expect_success 'fetch with --filter=blob:limit=0' '\n+\tfetch_filter_blob_limit_zero server server\n+'\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'fetch with --filter=blob:limit=0 and HTTP' '\n+\tfetch_filter_blob_limit_zero \"$HTTPD_DOCUMENT_ROOT_PATH/server\" \"$HTTPD_URL/smart/server\"\n+'\n+\n+stop_httpd\n+\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"334437","messageId":"20171208155851.855-14-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 13/16] fetch-pack: restore save_commit_buffer after use","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:48Z","receivedAt":"2017-12-08T15:59:25Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nIn fetch-pack, the global variable save_commit_buffer is set to 0, but\nnot restored to its original value after use.\n\nIn particular, if show_log() (in log-tree.c) is invoked after\nfetch_pack() in the same process, show_log() will return before printing\nout the commit message (because the invocation to\nget_cached_commit_buffer() returns NULL, because the commit buffer was\nnot saved). I discovered this when attempting to run \"git log -S\" in a\npartial clone, triggering the case where revision walking lazily loads\nmissing objects.\n\nTherefore, restore save_commit_buffer to its original value after use.\n\nAn alternative to solve the problem I had is to replace\nget_cached_commit_buffer() with get_commit_buffer(). That invocation was\nintroduced in commit a97934d (\"use get_cached_commit_buffer where\nappropriate\", 2014-06-13) to replace \"commit->buffer\" introduced in\ncommit 3131b71 (\"Add \"--show-all\" revision walker flag for debugging\",\n2008-02-13). In the latter commit, the commit author seems to be\ndeciding between not showing an unparsed commit at all and showing an\nunparsed commit without the message (which is what the commit does), and\ndid not mention parsing the unparsed commit, so I prefer to preserve the\nexisting behavior.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n fetch-pack.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 3c5f064..773453c 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -717,6 +717,7 @@ static int everything_local(struct fetch_pack_args *args,\n {\n \tstruct ref *ref;\n \tint retval;\n+\tint old_save_commit_buffer = save_commit_buffer;\n \ttimestamp_t cutoff = 0;\n \n \tsave_commit_buffer = 0;\n@@ -786,6 +787,9 @@ static int everything_local(struct fetch_pack_args *args,\n \t\tprint_verbose(args, _(\"already have %s (%s)\"), oid_to_hex(remote),\n \t\t\t      ref->name);\n \t}\n+\n+\tsave_commit_buffer = old_save_commit_buffer;\n+\n \treturn retval;\n }\n \n-- \n2.9.3\n\n"},{"id":"334438","messageId":"20171208155851.855-13-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 12/16] unpack-trees: batch fetching of missing blobs","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:47Z","receivedAt":"2017-12-08T15:59:27Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nWhen running checkout, first prefetch all blobs that are to be updated\nbut are missing. This means that only one pack is downloaded during such\noperations, instead of one per missing blob.\n\nThis operates only on the blob level - if a repository has a missing\ntree, they are still fetched one at a time.\n\nThis does not use the delayed checkout mechanism introduced in commit\n2841e8f (\"convert: add \"status=delayed\" to filter process protocol\",\n2017-06-30) due to significant conceptual differences - in particular,\nfor partial clones, we already know what needs to be fetched based on\nthe contents of the local repo alone, whereas for status=delayed, it is\nthe filter process that tells us what needs to be checked in the end.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n fetch-object.c   | 26 ++++++++++++++++++++++----\n fetch-object.h   |  5 +++++\n t/t5601-clone.sh | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++++\n unpack-trees.c   | 22 ++++++++++++++++++++++\n 4 files changed, 101 insertions(+), 4 deletions(-)\n\ndiff --git a/fetch-object.c b/fetch-object.c\nindex 258fcfa..853624f 100644\n--- a/fetch-object.c\n+++ b/fetch-object.c\n@@ -5,11 +5,10 @@\n #include \"transport.h\"\n #include \"fetch-object.h\"\n \n-void fetch_object(const char *remote_name, const unsigned char *sha1)\n+static void fetch_refs(const char *remote_name, struct ref *ref)\n {\n \tstruct remote *remote;\n \tstruct transport *transport;\n-\tstruct ref *ref;\n \tint original_fetch_if_missing = fetch_if_missing;\n \n \tfetch_if_missing = 0;\n@@ -18,10 +17,29 @@ void fetch_object(const char *remote_name, const unsigned char *sha1)\n \t\tdie(_(\"Remote with no URL\"));\n \ttransport = transport_get(remote, remote->url[0]);\n \n-\tref = alloc_ref(sha1_to_hex(sha1));\n-\thashcpy(ref->old_oid.hash, sha1);\n \ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n \ttransport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, \"1\");\n \ttransport_fetch_refs(transport, ref);\n \tfetch_if_missing = original_fetch_if_missing;\n }\n+\n+void fetch_object(const char *remote_name, const unsigned char *sha1)\n+{\n+\tstruct ref *ref = alloc_ref(sha1_to_hex(sha1));\n+\thashcpy(ref->old_oid.hash, sha1);\n+\tfetch_refs(remote_name, ref);\n+}\n+\n+void fetch_objects(const char *remote_name, const struct oid_array *to_fetch)\n+{\n+\tstruct ref *ref = NULL;\n+\tint i;\n+\n+\tfor (i = 0; i < to_fetch->nr; i++) {\n+\t\tstruct ref *new_ref = alloc_ref(oid_to_hex(&to_fetch->oid[i]));\n+\t\toidcpy(&new_ref->old_oid, &to_fetch->oid[i]);\n+\t\tnew_ref->next = ref;\n+\t\tref = new_ref;\n+\t}\n+\tfetch_refs(remote_name, ref);\n+}\ndiff --git a/fetch-object.h b/fetch-object.h\nindex f371300..4b269d0 100644\n--- a/fetch-object.h\n+++ b/fetch-object.h\n@@ -1,6 +1,11 @@\n #ifndef FETCH_OBJECT_H\n #define FETCH_OBJECT_H\n \n+#include \"sha1-array.h\"\n+\n extern void fetch_object(const char *remote_name, const unsigned char *sha1);\n \n+extern void fetch_objects(const char *remote_name,\n+\t\t\t  const struct oid_array *to_fetch);\n+\n #endif\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 6d37c6d..13610b7 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -611,6 +611,58 @@ test_expect_success 'partial clone: warn if server does not support object filte\n \ttest_i18ngrep \"filtering not recognized by server\" err\n '\n \n+test_expect_success 'batch missing blob request during checkout' '\n+\trm -rf server client &&\n+\n+\ttest_create_repo server &&\n+\techo a >server/a &&\n+\techo b >server/b &&\n+\tgit -C server add a b &&\n+\n+\tgit -C server commit -m x &&\n+\techo aa >server/a &&\n+\techo bb >server/b &&\n+\tgit -C server add a b &&\n+\tgit -C server commit -m x &&\n+\n+\ttest_config -C server uploadpack.allowfilter 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\n+\tgit clone --filter=blob:limit=0 \"file://$(pwd)/server\" client &&\n+\n+\t# Ensure that there is only one negotiation by checking that there is\n+\t# only \"done\" line sent. (\"done\" marks the end of negotiation.)\n+\tGIT_TRACE_PACKET=\"$(pwd)/trace\" git -C client checkout HEAD^ &&\n+\tgrep \"git> done\" trace >done_lines &&\n+\ttest_line_count = 1 done_lines\n+'\n+\n+test_expect_success 'batch missing blob request does not inadvertently try to fetch gitlinks' '\n+\trm -rf server client &&\n+\n+\ttest_create_repo repo_for_submodule &&\n+\ttest_commit -C repo_for_submodule x &&\n+\n+\ttest_create_repo server &&\n+\techo a >server/a &&\n+\techo b >server/b &&\n+\tgit -C server add a b &&\n+\tgit -C server commit -m x &&\n+\n+\techo aa >server/a &&\n+\techo bb >server/b &&\n+\t# Also add a gitlink pointing to an arbitrary repository\n+\tgit -C server submodule add \"$(pwd)/repo_for_submodule\" c &&\n+\tgit -C server add a b c &&\n+\tgit -C server commit -m x &&\n+\n+\ttest_config -C server uploadpack.allowfilter 1 &&\n+\ttest_config -C server uploadpack.allowanysha1inwant 1 &&\n+\n+\t# Make sure that it succeeds\n+\tgit clone --filter=blob:limit=0 \"file://$(pwd)/server\" client\n+'\n+\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n \ndiff --git a/unpack-trees.c b/unpack-trees.c\nindex 71b70cc..73a1cdb 100644\n--- a/unpack-trees.c\n+++ b/unpack-trees.c\n@@ -14,6 +14,7 @@\n #include \"dir.h\"\n #include \"submodule.h\"\n #include \"submodule-config.h\"\n+#include \"fetch-object.h\"\n \n /*\n  * Error messages expected by scripts out of plumbing commands such as\n@@ -369,6 +370,27 @@ static int check_updates(struct unpack_trees_options *o)\n \t\tload_gitmodules_file(index, &state);\n \n \tenable_delayed_checkout(&state);\n+\tif (repository_format_partial_clone && o->update && !o->dry_run) {\n+\t\t/*\n+\t\t * Prefetch the objects that are to be checked out in the loop\n+\t\t * below.\n+\t\t */\n+\t\tstruct oid_array to_fetch = OID_ARRAY_INIT;\n+\t\tint fetch_if_missing_store = fetch_if_missing;\n+\t\tfetch_if_missing = 0;\n+\t\tfor (i = 0; i < index->cache_nr; i++) {\n+\t\t\tstruct cache_entry *ce = index->cache[i];\n+\t\t\tif ((ce->ce_flags & CE_UPDATE) &&\n+\t\t\t    !S_ISGITLINK(ce->ce_mode)) {\n+\t\t\t\tif (!has_object_file(&ce->oid))\n+\t\t\t\t\toid_array_append(&to_fetch, &ce->oid);\n+\t\t\t}\n+\t\t}\n+\t\tif (to_fetch.nr)\n+\t\t\tfetch_objects(repository_format_partial_clone,\n+\t\t\t\t      &to_fetch);\n+\t\tfetch_if_missing = fetch_if_missing_store;\n+\t}\n \tfor (i = 0; i < index->cache_nr; i++) {\n \t\tstruct cache_entry *ce = index->cache[i];\n \n-- \n2.9.3\n\n"},{"id":"334439","messageId":"20171208155851.855-12-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 11/16] clone: partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:46Z","receivedAt":"2017-12-08T15:59:32Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n builtin/clone.c  | 22 ++++++++++++++++++++--\n t/t5601-clone.sh | 49 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 69 insertions(+), 2 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex dbddd98..f519bd4 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -26,6 +26,7 @@\n #include \"run-command.h\"\n #include \"connected.h\"\n #include \"packfile.h\"\n+#include \"list-objects-filter-options.h\"\n \n /*\n  * Overall FIXMEs:\n@@ -60,6 +61,7 @@ static struct string_list option_optional_reference = STRING_LIST_INIT_NODUP;\n static int option_dissociate;\n static int max_jobs = -1;\n static struct string_list option_recurse_submodules = STRING_LIST_INIT_NODUP;\n+static struct list_objects_filter_options filter_options;\n \n static int recurse_submodules_cb(const struct option *opt,\n \t\t\t\t const char *arg, int unset)\n@@ -135,6 +137,7 @@ static struct option builtin_clone_options[] = {\n \t\t\tTRANSPORT_FAMILY_IPV4),\n \tOPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n \t\t\tTRANSPORT_FAMILY_IPV6),\n+\tOPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n \tOPT_END()\n };\n \n@@ -886,6 +889,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tstruct refspec *refspec;\n \tconst char *fetch_pattern;\n \n+\tfetch_if_missing = 0;\n+\n \tpacket_trace_identity(\"clone\");\n \targc = parse_options(argc, argv, prefix, builtin_clone_options,\n \t\t\t     builtin_clone_usage, 0);\n@@ -1073,6 +1078,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\t\twarning(_(\"--shallow-since is ignored in local clones; use file:// instead.\"));\n \t\tif (option_not.nr)\n \t\t\twarning(_(\"--shallow-exclude is ignored in local clones; use file:// instead.\"));\n+\t\tif (filter_options.choice)\n+\t\t\twarning(_(\"--filter is ignored in local clones; use file:// instead.\"));\n \t\tif (!access(mkpath(\"%s/shallow\", path), F_OK)) {\n \t\t\tif (option_local > 0)\n \t\t\t\twarning(_(\"source repository is shallow, ignoring --local\"));\n@@ -1104,7 +1111,13 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_set_option(transport, TRANS_OPT_UPLOADPACK,\n \t\t\t\t     option_upload_pack);\n \n-\tif (transport->smart_options && !deepen)\n+\tif (filter_options.choice) {\n+\t\ttransport_set_option(transport, TRANS_OPT_LIST_OBJECTS_FILTER,\n+\t\t\t\t     filter_options.filter_spec);\n+\t\ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n+\t}\n+\n+\tif (transport->smart_options && !deepen && !filter_options.choice)\n \t\ttransport->smart_options->check_self_contained_and_connected = 1;\n \n \trefs = transport_get_remote_refs(transport);\n@@ -1164,13 +1177,17 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \twrite_refspec_config(src_ref_prefix, our_head_points_at,\n \t\t\tremote_head_points_at, &branch_top);\n \n+\tif (filter_options.choice)\n+\t\tpartial_clone_register(\"origin\", &filter_options);\n+\n \tif (is_local)\n \t\tclone_local(path, git_dir);\n \telse if (refs && complete_refs_before_fetch)\n \t\ttransport_fetch_refs(transport, mapped_refs);\n \n \tupdate_remote_refs(refs, mapped_refs, remote_head_points_at,\n-\t\t\t   branch_top.buf, reflog_msg.buf, transport, !is_local);\n+\t\t\t   branch_top.buf, reflog_msg.buf, transport,\n+\t\t\t   !is_local && !filter_options.choice);\n \n \tupdate_head(our_head_points_at, remote_head, reflog_msg.buf);\n \n@@ -1191,6 +1208,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t}\n \n \tjunk_mode = JUNK_LEAVE_REPO;\n+\tfetch_if_missing = 1;\n \terr = checkout(submodule_progress);\n \n \tstrbuf_release(&reflog_msg);\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 9c56f77..6d37c6d 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -571,4 +571,53 @@ test_expect_success 'GIT_TRACE_PACKFILE produces a usable pack' '\n \tgit -C replay.git index-pack -v --stdin <tmp.pack\n '\n \n+partial_clone () {\n+\t       SERVER=\"$1\" &&\n+\t       URL=\"$2\" &&\n+\n+\trm -rf \"$SERVER\" client &&\n+\ttest_create_repo \"$SERVER\" &&\n+\ttest_commit -C \"$SERVER\" one &&\n+\tHASH1=$(git hash-object \"$SERVER/one.t\") &&\n+\tgit -C \"$SERVER\" revert HEAD &&\n+\ttest_commit -C \"$SERVER\" two &&\n+\tHASH2=$(git hash-object \"$SERVER/two.t\") &&\n+\ttest_config -C \"$SERVER\" uploadpack.allowfilter 1 &&\n+\ttest_config -C \"$SERVER\" uploadpack.allowanysha1inwant 1 &&\n+\n+\tgit clone --filter=blob:limit=0 \"$URL\" client &&\n+\n+\tgit -C client fsck &&\n+\n+\t# Ensure that unneeded blobs are not inadvertently fetched.\n+\ttest_config -C client extensions.partialclone \"not a remote\" &&\n+\ttest_must_fail git -C client cat-file -e \"$HASH1\" &&\n+\n+\t# But this blob was fetched, because clone performs an initial checkout\n+\tgit -C client cat-file -e \"$HASH2\"\n+}\n+\n+test_expect_success 'partial clone' '\n+\tpartial_clone server \"file://$(pwd)/server\"\n+'\n+\n+test_expect_success 'partial clone: warn if server does not support object filtering' '\n+\trm -rf server client &&\n+\ttest_create_repo server &&\n+\ttest_commit -C server one &&\n+\n+\tgit clone --filter=blob:limit=0 \"file://$(pwd)/server\" client 2> err &&\n+\n+\ttest_i18ngrep \"filtering not recognized by server\" err\n+'\n+\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'partial clone using HTTP' '\n+\tpartial_clone \"$HTTPD_DOCUMENT_ROOT_PATH/server\" \"$HTTPD_URL/smart/server\"\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"334440","messageId":"20171208155851.855-16-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 15/16] fetch: inherit filter-spec from partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:50Z","receivedAt":"2017-12-08T15:59:35Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nTeach (partial) fetch to inherit the filter-spec used by\nthe partial clone.  Extend --no-filter to override this\ninheritance.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n builtin/fetch-pack.c          |  2 +-\n builtin/fetch.c               | 56 ++++++++++++++++++++++++++++++++++++++++---\n builtin/rev-list.c            |  2 +-\n list-objects-filter-options.c |  2 +-\n list-objects-filter-options.h | 12 ++++++++++\n t/t5616-partial-clone.sh      | 22 ++++++++++++++++-\n 6 files changed, 89 insertions(+), 7 deletions(-)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex cbf5035..a7bc136 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -158,7 +158,7 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tlist_objects_filter_release(&args.filter_options);\n+\t\t\tlist_objects_filter_set_no_filter(&args.filter_options);\n \t\t\tcontinue;\n \t\t}\n \t\tusage(fetch_pack_usage);\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 14aab71..79c866c 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1275,6 +1275,56 @@ static int fetch_multiple(struct string_list *list)\n \treturn result;\n }\n \n+/*\n+ * Fetching from the promisor remote should use the given filter-spec\n+ * or inherit the default filter-spec from the config.\n+ */\n+static inline void fetch_one_setup_partial(struct remote *remote)\n+{\n+\t/*\n+\t * Explicit --no-filter argument overrides everything, regardless\n+\t * of any prior partial clones and fetches.\n+\t */\n+\tif (filter_options.no_filter)\n+\t\treturn;\n+\n+\t/*\n+\t * If no prior partial clone/fetch and the current fetch DID NOT\n+\t * request a partial-fetch, do a normal fetch.\n+\t */\n+\tif (!repository_format_partial_clone && !filter_options.choice)\n+\t\treturn;\n+\n+\t/*\n+\t * If this is the FIRST partial-fetch request, we enable partial\n+\t * on this repo and remember the given filter-spec as the default\n+\t * for subsequent fetches to this remote.\n+\t */\n+\tif (!repository_format_partial_clone && filter_options.choice) {\n+\t\tpartial_clone_register(remote->name, &filter_options);\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * We are currently limited to only ONE promisor remote and only\n+\t * allow partial-fetches from the promisor remote.\n+\t */\n+\tif (strcmp(remote->name, repository_format_partial_clone)) {\n+\t\tif (filter_options.choice)\n+\t\t\tdie(_(\"--filter can only be used with the remote configured in core.partialClone\"));\n+\t\treturn;\n+\t}\n+\n+\t/*\n+\t * Do a partial-fetch from the promisor remote using either the\n+\t * explicitly given filter-spec or inherit the filter-spec from\n+\t * the config.\n+\t */\n+\tif (!filter_options.choice)\n+\t\tpartial_clone_get_default_filter_spec(&filter_options);\n+\treturn;\n+}\n+\n static int fetch_one(struct remote *remote, int argc, const char **argv)\n {\n \tstatic const char **refs = NULL;\n@@ -1404,13 +1454,13 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t}\n \n \tif (remote) {\n-\t\tif (filter_options.choice &&\n-\t\t    strcmp(remote->name, repository_format_partial_clone))\n-\t\t\tdie(_(\"--filter can only be used with the remote configured in core.partialClone\"));\n+\t\tif (filter_options.choice || repository_format_partial_clone)\n+\t\t\tfetch_one_setup_partial(remote);\n \t\tresult = fetch_one(remote, argc, argv);\n \t} else {\n \t\tif (filter_options.choice)\n \t\t\tdie(_(\"--filter can only be used with the remote configured in core.partialClone\"));\n+\t\t/* TODO should this also die if we have a previous partial-clone? */\n \t\tresult = fetch_multiple(&list);\n \t}\n \ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 48f922d..8503dea 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -460,7 +460,7 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n-\t\t\tlist_objects_filter_release(&filter_options);\n+\t\t\tlist_objects_filter_set_no_filter(&filter_options);\n \t\t\tcontinue;\n \t\t}\n \t\tif (!strcmp(arg, \"--filter-print-omitted\")) {\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 5c47e2b..6a3cc98 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -94,7 +94,7 @@ int opt_parse_list_objects_filter(const struct option *opt,\n \tstruct list_objects_filter_options *filter_options = opt->value;\n \n \tif (unset || !arg) {\n-\t\tlist_objects_filter_release(filter_options);\n+\t\tlist_objects_filter_set_no_filter(filter_options);\n \t\treturn 0;\n \t}\n \ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex 1143539..0000a61 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -31,6 +31,11 @@ struct list_objects_filter_options {\n \tenum list_objects_filter_choice choice;\n \n \t/*\n+\t * Choice is LOFC_DISABLED because \"--no-filter\" was requested.\n+\t */\n+\tunsigned int no_filter : 1;\n+\n+\t/*\n \t * Parsed values (fields) from within the filter-spec.  These are\n \t * choice-specific; not all values will be defined for any given\n \t * choice.\n@@ -58,6 +63,13 @@ int opt_parse_list_objects_filter(const struct option *opt,\n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options);\n \n+static inline void list_objects_filter_set_no_filter(\n+\tstruct list_objects_filter_options *filter_options)\n+{\n+\tlist_objects_filter_release(filter_options);\n+\tfilter_options->no_filter = 1;\n+}\n+\n void partial_clone_register(\n \tconst char *remote,\n \tconst struct list_objects_filter_options *filter_options);\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 3b8cc0b..433e07e 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -59,7 +59,7 @@ test_expect_success 'push new commits to server' '\n \tgit -C src remote add srv \"file://$(pwd)/srv.bare\" &&\n \tfor x in a b c d e\n \tdo\n-\t\techo \"Mod $x\" >>src/file.1.txt\n+\t\techo \"Mod file.1.txt $x\" >>src/file.1.txt\n \t\tgit -C src add file.1.txt\n \t\tgit -C src commit -m \"mod $x\"\n \tdone &&\n@@ -93,4 +93,24 @@ test_expect_success 'verify blame causes dynamic object fetch' '\n \ttest_line_count = 0 observed\n '\n \n+# create new commits in \"src\" repo to establish a history on file.2.txt\n+# and push to \"srv.bare\".\n+test_expect_success 'push new commits to server for file.2.txt' '\n+\tfor x in a b c d e f\n+\tdo\n+\t\techo \"Mod file.2.txt $x\" >>src/file.2.txt\n+\t\tgit -C src add file.2.txt\n+\t\tgit -C src commit -m \"mod $x\"\n+\tdone &&\n+\tgit -C src push -u srv master\n+'\n+\n+# Do FULL fetch by disabling filter-spec using --no-filter.\n+# Verify we have all the new blobs.\n+test_expect_success 'override inherited filter-spec using --no-filter' '\n+\tgit -C pc1 fetch --no-filter origin &&\n+\tgit -C pc1 rev-list master..origin/master --quiet --objects --missing=print >observed &&\n+\ttest_line_count = 0 observed\n+'\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"334441","messageId":"20171208155851.855-17-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 16/16] t5616: test bulk prefetch after partial fetch","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:51Z","receivedAt":"2017-12-08T15:59:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdd test to t5616 to bulk fetch missing objects following\na partial fetch.  A technique like this could be used in\na pre-command hook for example.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n t/t5616-partial-clone.sh | 32 +++++++++++++++++++++++++++++++-\n 1 file changed, 31 insertions(+), 1 deletion(-)\n\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nindex 433e07e..29d8631 100755\n--- a/t/t5616-partial-clone.sh\n+++ b/t/t5616-partial-clone.sh\n@@ -105,7 +105,7 @@ test_expect_success 'push new commits to server for file.2.txt' '\n \tgit -C src push -u srv master\n '\n \n-# Do FULL fetch by disabling filter-spec using --no-filter.\n+# Do FULL fetch by disabling inherited filter-spec using --no-filter.\n # Verify we have all the new blobs.\n test_expect_success 'override inherited filter-spec using --no-filter' '\n \tgit -C pc1 fetch --no-filter origin &&\n@@ -113,4 +113,34 @@ test_expect_success 'override inherited filter-spec using --no-filter' '\n \ttest_line_count = 0 observed\n '\n \n+# create new commits in \"src\" repo to establish a history on file.3.txt\n+# and push to \"srv.bare\".\n+test_expect_success 'push new commits to server for file.3.txt' '\n+\tfor x in a b c d e f\n+\tdo\n+\t\techo \"Mod file.3.txt $x\" >>src/file.3.txt\n+\t\tgit -C src add file.3.txt\n+\t\tgit -C src commit -m \"mod $x\"\n+\tdone &&\n+\tgit -C src push -u srv master\n+'\n+\n+# Do a partial fetch and then try to manually fetch the missing objects.\n+# This can be used as the basis of a pre-command hook to bulk fetch objects\n+# perhaps combined with a command in dry-run mode.\n+test_expect_success 'manual prefetch of missing objects' '\n+\tgit -C pc1 fetch --filter=blob:none origin &&\n+\tgit -C pc1 rev-list master..origin/master --quiet --objects --missing=print \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/?//\" \\\n+\t\t| sort >observed.oids &&\n+\ttest_line_count = 6 observed.oids &&\n+\tgit -C pc1 fetch-pack --stdin \"file://$(pwd)/srv.bare\" <observed.oids &&\n+\tgit -C pc1 rev-list master..origin/master --quiet --objects --missing=print \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/?//\" \\\n+\t\t| sort >observed.oids &&\n+\ttest_line_count = 0 observed.oids\n+'\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"334442","messageId":"20171208155851.855-11-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 10/16] partial-clone: define partial clone settings in config","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:45Z","receivedAt":"2017-12-08T15:59:39Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nCreate get and set routines for \"partial clone\" config settings.\nThese will be used in a future commit by clone and fetch to\nremember the promisor remote and the default filter-spec.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n cache.h                       |  1 +\n config.c                      |  5 +++\n environment.c                 |  1 +\n list-objects-filter-options.c | 90 +++++++++++++++++++++++++++++++++++--------\n list-objects-filter-options.h |  6 +++\n 5 files changed, 88 insertions(+), 15 deletions(-)\n\ndiff --git a/cache.h b/cache.h\nindex 6980072..bccc510 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -861,6 +861,7 @@ extern int grafts_replace_parents;\n #define GIT_REPO_VERSION_READ 1\n extern int repository_format_precious_objects;\n extern char *repository_format_partial_clone;\n+extern const char *core_partial_clone_filter_default;\n \n struct repository_format {\n \tint version;\ndiff --git a/config.c b/config.c\nindex adb7d7a..adeee04 100644\n--- a/config.c\n+++ b/config.c\n@@ -1241,6 +1241,11 @@ static int git_default_core_config(const char *var, const char *value)\n \t\treturn 0;\n \t}\n \n+\tif (!strcmp(var, \"core.partialclonefilter\")) {\n+\t\treturn git_config_string(&core_partial_clone_filter_default,\n+\t\t\t\t\t var, value);\n+\t}\n+\n \t/* Add other config variables here and to Documentation/config.txt. */\n \treturn 0;\n }\ndiff --git a/environment.c b/environment.c\nindex e52aab3..7537565 100644\n--- a/environment.c\n+++ b/environment.c\n@@ -28,6 +28,7 @@ int warn_on_object_refname_ambiguity = 1;\n int ref_paranoia = -1;\n int repository_format_precious_objects;\n char *repository_format_partial_clone;\n+const char *core_partial_clone_filter_default;\n const char *git_commit_encoding;\n const char *git_log_output_encoding;\n const char *apply_default_whitespace;\ndiff --git a/list-objects-filter-options.c b/list-objects-filter-options.c\nindex 4c5b34e..5c47e2b 100644\n--- a/list-objects-filter-options.c\n+++ b/list-objects-filter-options.c\n@@ -21,29 +21,36 @@\n  * subordinate commands when necessary.  We also \"intern\" the arg for\n  * the convenience of the current command.\n  */\n-int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n-\t\t\t      const char *arg)\n+static int gently_parse_list_objects_filter(\n+\tstruct list_objects_filter_options *filter_options,\n+\tconst char *arg,\n+\tstruct strbuf *errbuf)\n {\n \tconst char *v0;\n \n-\tif (filter_options->choice)\n-\t\tdie(_(\"multiple object filter types cannot be combined\"));\n+\tif (filter_options->choice) {\n+\t\tif (errbuf) {\n+\t\t\tstrbuf_init(errbuf, 0);\n+\t\t\tstrbuf_addstr(\n+\t\t\t\terrbuf,\n+\t\t\t\t_(\"multiple filter-specs cannot be combined\"));\n+\t\t}\n+\t\treturn 1;\n+\t}\n \n \tfilter_options->filter_spec = strdup(arg);\n \n \tif (!strcmp(arg, \"blob:none\")) {\n \t\tfilter_options->choice = LOFC_BLOB_NONE;\n \t\treturn 0;\n-\t}\n \n-\tif (skip_prefix(arg, \"blob:limit=\", &v0)) {\n-\t\tif (!git_parse_ulong(v0, &filter_options->blob_limit_value))\n-\t\t\tdie(_(\"invalid filter-spec expression '%s'\"), arg);\n-\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n-\t\treturn 0;\n-\t}\n+\t} else if (skip_prefix(arg, \"blob:limit=\", &v0)) {\n+\t\tif (git_parse_ulong(v0, &filter_options->blob_limit_value)) {\n+\t\t\tfilter_options->choice = LOFC_BLOB_LIMIT;\n+\t\t\treturn 0;\n+\t\t}\n \n-\tif (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n+\t} else if (skip_prefix(arg, \"sparse:oid=\", &v0)) {\n \t\tstruct object_context oc;\n \t\tstruct object_id sparse_oid;\n \n@@ -57,15 +64,27 @@ int parse_list_objects_filter(struct list_objects_filter_options *filter_options\n \t\t\tfilter_options->sparse_oid_value = oiddup(&sparse_oid);\n \t\tfilter_options->choice = LOFC_SPARSE_OID;\n \t\treturn 0;\n-\t}\n \n-\tif (skip_prefix(arg, \"sparse:path=\", &v0)) {\n+\t} else if (skip_prefix(arg, \"sparse:path=\", &v0)) {\n \t\tfilter_options->choice = LOFC_SPARSE_PATH;\n \t\tfilter_options->sparse_path_value = strdup(v0);\n \t\treturn 0;\n \t}\n \n-\tdie(_(\"invalid filter-spec expression '%s'\"), arg);\n+\tif (errbuf) {\n+\t\tstrbuf_init(errbuf, 0);\n+\t\tstrbuf_addf(errbuf, \"invalid filter-spec '%s'\", arg);\n+\t}\n+\tmemset(filter_options, 0, sizeof(*filter_options));\n+\treturn 1;\n+}\n+\n+int parse_list_objects_filter(struct list_objects_filter_options *filter_options,\n+\t\t\t      const char *arg)\n+{\n+\tstruct strbuf buf = STRBUF_INIT;\n+\tif (gently_parse_list_objects_filter(filter_options, arg, &buf))\n+\t\tdie(\"%s\", buf.buf);\n \treturn 0;\n }\n \n@@ -90,3 +109,44 @@ void list_objects_filter_release(\n \tfree(filter_options->sparse_path_value);\n \tmemset(filter_options, 0, sizeof(*filter_options));\n }\n+\n+void partial_clone_register(\n+\tconst char *remote,\n+\tconst struct list_objects_filter_options *filter_options)\n+{\n+\t/*\n+\t * Record the name of the partial clone remote in the\n+\t * config and in the global variable -- the latter is\n+\t * used throughout to indicate that partial clone is\n+\t * enabled and to expect missing objects.\n+\t */\n+\tif (repository_format_partial_clone &&\n+\t    *repository_format_partial_clone &&\n+\t    strcmp(remote, repository_format_partial_clone))\n+\t\tdie(_(\"cannot change partial clone promisor remote\"));\n+\n+\tgit_config_set(\"core.repositoryformatversion\", \"1\");\n+\tgit_config_set(\"extensions.partialclone\", remote);\n+\n+\trepository_format_partial_clone = xstrdup(remote);\n+\n+\t/*\n+\t * Record the initial filter-spec in the config as\n+\t * the default for subsequent fetches from this remote.\n+\t */\n+\tcore_partial_clone_filter_default =\n+\t\txstrdup(filter_options->filter_spec);\n+\tgit_config_set(\"core.partialclonefilter\",\n+\t\t       core_partial_clone_filter_default);\n+}\n+\n+void partial_clone_get_default_filter_spec(\n+\tstruct list_objects_filter_options *filter_options)\n+{\n+\t/*\n+\t * Parse default value, but silently ignore it if it is invalid.\n+\t */\n+\tgently_parse_list_objects_filter(filter_options,\n+\t\t\t\t\t core_partial_clone_filter_default,\n+\t\t\t\t\t NULL);\n+}\ndiff --git a/list-objects-filter-options.h b/list-objects-filter-options.h\nindex eea44a1..1143539 100644\n--- a/list-objects-filter-options.h\n+++ b/list-objects-filter-options.h\n@@ -58,4 +58,10 @@ int opt_parse_list_objects_filter(const struct option *opt,\n void list_objects_filter_release(\n \tstruct list_objects_filter_options *filter_options);\n \n+void partial_clone_register(\n+\tconst char *remote,\n+\tconst struct list_objects_filter_options *filter_options);\n+void partial_clone_get_default_filter_spec(\n+\tstruct list_objects_filter_options *filter_options);\n+\n #endif /* LIST_OBJECTS_FILTER_OPTIONS_H */\n-- \n2.9.3\n\n"},{"id":"334443","messageId":"20171208155851.855-15-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 14/16] t5616: end-to-end tests for partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:49Z","receivedAt":"2017-12-08T15:59:43Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nAdditional end-to-end tests for partial clone.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n t/t5616-partial-clone.sh | 96 ++++++++++++++++++++++++++++++++++++++++++++++++\n 1 file changed, 96 insertions(+)\n create mode 100755 t/t5616-partial-clone.sh\n\ndiff --git a/t/t5616-partial-clone.sh b/t/t5616-partial-clone.sh\nnew file mode 100755\nindex 0000000..3b8cc0b\n--- /dev/null\n+++ b/t/t5616-partial-clone.sh\n@@ -0,0 +1,96 @@\n+#!/bin/sh\n+\n+test_description='git partial clone'\n+\n+. ./test-lib.sh\n+\n+# create a normal \"src\" repo where we can later create new commits.\n+# expect_1.oids will contain a list of the OIDs of all blobs.\n+test_expect_success 'setup normal src repo' '\n+\techo \"{print \\$1}\" >print_1.awk &&\n+\techo \"{print \\$2}\" >print_2.awk &&\n+\n+\tgit init src &&\n+\tfor n in 1 2 3 4\n+\tdo\n+\t\techo \"This is file: $n\" > src/file.$n.txt\n+\t\tgit -C src add file.$n.txt\n+\t\tgit -C src commit -m \"file $n\"\n+\t\tgit -C src ls-files -s file.$n.txt >>temp\n+\tdone &&\n+\tawk -f print_2.awk <temp | sort >expect_1.oids &&\n+\ttest_line_count = 4 expect_1.oids\n+'\n+\n+# bare clone \"src\" giving \"srv.bare\" for use as our server.\n+test_expect_success 'setup bare clone for server' '\n+\tgit clone --bare \"file://$(pwd)/src\" srv.bare &&\n+\tgit -C srv.bare config --local uploadpack.allowfilter 1 &&\n+\tgit -C srv.bare config --local uploadpack.allowanysha1inwant 1\n+'\n+\n+# do basic partial clone from \"srv.bare\"\n+# confirm we are missing all of the known blobs.\n+# confirm partial clone was registered in the local config.\n+test_expect_success 'do partial clone 1' '\n+\tgit clone --no-checkout --filter=blob:none \"file://$(pwd)/srv.bare\" pc1 &&\n+\tgit -C pc1 rev-list HEAD --quiet --objects --missing=print \\\n+\t\t| awk -f print_1.awk \\\n+\t\t| sed \"s/?//\" \\\n+\t\t| sort >observed.oids &&\n+\ttest_cmp expect_1.oids observed.oids &&\n+\ttest \"$(git -C pc1 config --local core.repositoryformatversion)\" = \"1\" &&\n+\ttest \"$(git -C pc1 config --local extensions.partialclone)\" = \"origin\" &&\n+\ttest \"$(git -C pc1 config --local core.partialclonefilter)\" = \"blob:none\"\n+'\n+\n+# checkout master to force dynamic object fetch of blobs at HEAD.\n+test_expect_success 'verify checkout with dynamic object fetch' '\n+\tgit -C pc1 rev-list HEAD --quiet --objects --missing=print >observed &&\n+\ttest_line_count = 4 observed &&\n+\tgit -C pc1 checkout master &&\n+\tgit -C pc1 rev-list HEAD --quiet --objects --missing=print >observed &&\n+\ttest_line_count = 0 observed\n+'\n+\n+# create new commits in \"src\" repo to establish a blame history on file.1.txt\n+# and push to \"srv.bare\".\n+test_expect_success 'push new commits to server' '\n+\tgit -C src remote add srv \"file://$(pwd)/srv.bare\" &&\n+\tfor x in a b c d e\n+\tdo\n+\t\techo \"Mod $x\" >>src/file.1.txt\n+\t\tgit -C src add file.1.txt\n+\t\tgit -C src commit -m \"mod $x\"\n+\tdone &&\n+\tgit -C src blame master -- file.1.txt >expect.blame &&\n+\tgit -C src push -u srv master\n+'\n+\n+# (partial) fetch in the partial clone repo from the promisor remote.\n+# verify that fetch inherited the filter-spec from the config and DOES NOT\n+# have the new blobs.\n+test_expect_success 'partial fetch inherits filter settings' '\n+\tgit -C pc1 fetch origin &&\n+\tgit -C pc1 rev-list master..origin/master --quiet --objects --missing=print >observed &&\n+\ttest_line_count = 5 observed\n+'\n+\n+# force dynamic object fetch using diff.\n+# we should only get 1 new blob (for the file in origin/master).\n+test_expect_success 'verify diff causes dynamic object fetch' '\n+\tgit -C pc1 diff master..origin/master -- file.1.txt &&\n+\tgit -C pc1 rev-list master..origin/master --quiet --objects --missing=print >observed &&\n+\ttest_line_count = 4 observed\n+'\n+\n+# force full dynamic object fetch of the file's history using blame.\n+# we should get the intermediate blobs for the file.\n+test_expect_success 'verify blame causes dynamic object fetch' '\n+\tgit -C pc1 blame origin/master -- file.1.txt >observed.blame &&\n+\ttest_cmp expect.blame observed.blame &&\n+\tgit -C pc1 rev-list master..origin/master --quiet --objects --missing=print >observed &&\n+\ttest_line_count = 0 observed\n+'\n+\n+test_done\n-- \n2.9.3\n\n"},{"id":"334444","messageId":"20171208155851.855-3-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 02/16] rev-list: support termination at promisor objects","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:37Z","receivedAt":"2017-12-08T15:59:46Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nTeach rev-list to support termination of an object traversal at any\nobject from a promisor remote (whether one that the local repo also has,\nor one that the local repo knows about because it has another promisor\nobject that references it).\n\nThis will be used subsequently in gc and in the connectivity check used\nby fetch.\n\nFor efficiency, if an object is referenced by a promisor object, and is\nin the local repo only as a non-promisor object, object traversal will\nnot stop there. This is to avoid building the list of promisor object\nreferences.\n\n(In list-objects.c, the case where obj is NULL in process_blob() and\nprocess_tree() do not need to be changed because those happen only when\nthere is a conflict between the expected type and the existing object.\nIf the object doesn't exist, an object will be synthesized, which is\nfine.)\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/rev-list-options.txt |  11 ++++\n builtin/rev-list.c                 |  71 +++++++++++++++++++++++---\n list-objects.c                     |  29 ++++++++++-\n object.c                           |   2 +-\n revision.c                         |  33 +++++++++++-\n revision.h                         |   5 +-\n t/t0410-partial-clone.sh           | 101 +++++++++++++++++++++++++++++++++++++\n 7 files changed, 240 insertions(+), 12 deletions(-)\n\ndiff --git a/Documentation/rev-list-options.txt b/Documentation/rev-list-options.txt\nindex 8d8b7f4..0ce8ccd 100644\n--- a/Documentation/rev-list-options.txt\n+++ b/Documentation/rev-list-options.txt\n@@ -745,10 +745,21 @@ The form '--missing=allow-any' will allow object traversal to continue\n if a missing object is encountered.  Missing objects will silently be\n omitted from the results.\n +\n+The form '--missing=allow-promisor' is like 'allow-any', but will only\n+allow object traversal to continue for EXPECTED promisor missing objects.\n+Unexpected missing objects will raise an error.\n++\n The form '--missing=print' is like 'allow-any', but will also print a\n list of the missing objects.  Object IDs are prefixed with a ``?'' character.\n endif::git-rev-list[]\n \n+--exclude-promisor-objects::\n+\t(For internal use only.)  Prefilter object traversal at\n+\tpromisor boundary.  This is used with partial clone.  This is\n+\tstronger than `--missing=allow-promisor` because it limits the\n+\ttraversal, rather than just silencing errors about missing\n+\tobjects.\n+\n --no-walk[=(sorted|unsorted)]::\n \tOnly show the given commits, but do not traverse their ancestors.\n \tThis has no effect if a range is specified. If the argument\ndiff --git a/builtin/rev-list.c b/builtin/rev-list.c\nindex 547a3e0..48f922d 100644\n--- a/builtin/rev-list.c\n+++ b/builtin/rev-list.c\n@@ -15,6 +15,7 @@\n #include \"progress.h\"\n #include \"reflog-walk.h\"\n #include \"oidset.h\"\n+#include \"packfile.h\"\n \n static const char rev_list_usage[] =\n \"git rev-list [OPTION] <commit-id>... [ -- paths... ]\\n\"\n@@ -67,6 +68,7 @@ enum missing_action {\n \tMA_ERROR = 0,    /* fail if any missing objects are encountered */\n \tMA_ALLOW_ANY,    /* silently allow ALL missing objects */\n \tMA_PRINT,        /* print ALL missing objects in special section */\n+\tMA_ALLOW_PROMISOR, /* silently allow all missing PROMISOR objects */\n };\n static enum missing_action arg_missing_action;\n \n@@ -197,6 +199,12 @@ static void finish_commit(struct commit *commit, void *data)\n \n static inline void finish_object__ma(struct object *obj)\n {\n+\t/*\n+\t * Whether or not we try to dynamically fetch missing objects\n+\t * from the server, we currently DO NOT have the object.  We\n+\t * can either print, allow (ignore), or conditionally allow\n+\t * (ignore) them.\n+\t */\n \tswitch (arg_missing_action) {\n \tcase MA_ERROR:\n \t\tdie(\"missing blob object '%s'\", oid_to_hex(&obj->oid));\n@@ -209,25 +217,36 @@ static inline void finish_object__ma(struct object *obj)\n \t\toidset_insert(&missing_objects, &obj->oid);\n \t\treturn;\n \n+\tcase MA_ALLOW_PROMISOR:\n+\t\tif (is_promisor_object(&obj->oid))\n+\t\t\treturn;\n+\t\tdie(\"unexpected missing blob object '%s'\",\n+\t\t    oid_to_hex(&obj->oid));\n+\t\treturn;\n+\n \tdefault:\n \t\tBUG(\"unhandled missing_action\");\n \t\treturn;\n \t}\n }\n \n-static void finish_object(struct object *obj, const char *name, void *cb_data)\n+static int finish_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid))\n+\tif (obj->type == OBJ_BLOB && !has_object_file(&obj->oid)) {\n \t\tfinish_object__ma(obj);\n+\t\treturn 1;\n+\t}\n \tif (info->revs->verify_objects && !obj->parsed && obj->type != OBJ_COMMIT)\n \t\tparse_object(&obj->oid);\n+\treturn 0;\n }\n \n static void show_object(struct object *obj, const char *name, void *cb_data)\n {\n \tstruct rev_list_info *info = cb_data;\n-\tfinish_object(obj, name, cb_data);\n+\tif (finish_object(obj, name, cb_data))\n+\t\treturn;\n \tdisplay_progress(progress, ++progress_counter);\n \tif (info->flags & REV_LIST_QUIET)\n \t\treturn;\n@@ -315,11 +334,19 @@ static inline int parse_missing_action_value(const char *value)\n \n \tif (!strcmp(value, \"allow-any\")) {\n \t\targ_missing_action = MA_ALLOW_ANY;\n+\t\tfetch_if_missing = 0;\n \t\treturn 1;\n \t}\n \n \tif (!strcmp(value, \"print\")) {\n \t\targ_missing_action = MA_PRINT;\n+\t\tfetch_if_missing = 0;\n+\t\treturn 1;\n+\t}\n+\n+\tif (!strcmp(value, \"allow-promisor\")) {\n+\t\targ_missing_action = MA_ALLOW_PROMISOR;\n+\t\tfetch_if_missing = 0;\n \t\treturn 1;\n \t}\n \n@@ -344,6 +371,35 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \tinit_revisions(&revs, prefix);\n \trevs.abbrev = DEFAULT_ABBREV;\n \trevs.commit_format = CMIT_FMT_UNSPECIFIED;\n+\n+\t/*\n+\t * Scan the argument list before invoking setup_revisions(), so that we\n+\t * know if fetch_if_missing needs to be set to 0.\n+\t *\n+\t * \"--exclude-promisor-objects\" acts as a pre-filter on missing objects\n+\t * by not crossing the boundary from realized objects to promisor\n+\t * objects.\n+\t *\n+\t * Let \"--missing\" to conditionally set fetch_if_missing.\n+\t */\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\t\tif (!strcmp(arg, \"--exclude-promisor-objects\")) {\n+\t\t\tfetch_if_missing = 0;\n+\t\t\trevs.exclude_promisor_objects = 1;\n+\t\t\tbreak;\n+\t\t}\n+\t}\n+\tfor (i = 1; i < argc; i++) {\n+\t\tconst char *arg = argv[i];\n+\t\tif (skip_prefix(arg, \"--missing=\", &arg)) {\n+\t\t\tif (revs.exclude_promisor_objects)\n+\t\t\t\tdie(_(\"cannot combine --exclude-promisor-objects and --missing\"));\n+\t\t\tif (parse_missing_action_value(arg))\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\n \targc = setup_revisions(argc, argv, &revs, NULL);\n \n \tmemset(&info, 0, sizeof(info));\n@@ -412,10 +468,11 @@ int cmd_rev_list(int argc, const char **argv, const char *prefix)\n \t\t\tcontinue;\n \t\t}\n \n-\t\tif (skip_prefix(arg, \"--missing=\", &arg) &&\n-\t\t    parse_missing_action_value(arg))\n-\t\t\tcontinue;\n-\t\t\n+\t\tif (!strcmp(arg, \"--exclude-promisor-objects\"))\n+\t\t\tcontinue; /* already handled above */\n+\t\tif (skip_prefix(arg, \"--missing=\", &arg))\n+\t\t\tcontinue; /* already handled above */\n+\n \t\tusage(rev_list_usage);\n \n \t}\ndiff --git a/list-objects.c b/list-objects.c\nindex d9e83d0..58621fc 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -9,6 +9,7 @@\n #include \"list-objects.h\"\n #include \"list-objects-filter.h\"\n #include \"list-objects-filter-options.h\"\n+#include \"packfile.h\"\n \n static void process_blob(struct rev_info *revs,\n \t\t\t struct blob *blob,\n@@ -30,6 +31,20 @@ static void process_blob(struct rev_info *revs,\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n \n+\t/*\n+\t * Pre-filter known-missing objects when explicitly requested.\n+\t * Otherwise, a missing object error message may be reported\n+\t * later (depending on other filtering criteria).\n+\t *\n+\t * Note that this \"--exclude-promisor-objects\" pre-filtering\n+\t * may cause the actual filter to report an incomplete list\n+\t * of missing objects.\n+\t */\n+\tif (revs->exclude_promisor_objects &&\n+\t    !has_object_file(&obj->oid) &&\n+\t    is_promisor_object(&obj->oid))\n+\t\treturn;\n+\n \tpathlen = path->len;\n \tstrbuf_addstr(path, name);\n \tif (filter_fn)\n@@ -91,6 +106,8 @@ static void process_tree(struct rev_info *revs,\n \t\tall_entries_interesting: entry_not_interesting;\n \tint baselen = base->len;\n \tenum list_objects_filter_result r = LOFR_MARK_SEEN | LOFR_DO_SHOW;\n+\tint gently = revs->ignore_missing_links ||\n+\t\t     revs->exclude_promisor_objects;\n \n \tif (!revs->tree_objects)\n \t\treturn;\n@@ -98,9 +115,19 @@ static void process_tree(struct rev_info *revs,\n \t\tdie(\"bad tree object\");\n \tif (obj->flags & (UNINTERESTING | SEEN))\n \t\treturn;\n-\tif (parse_tree_gently(tree, revs->ignore_missing_links) < 0) {\n+\tif (parse_tree_gently(tree, gently) < 0) {\n \t\tif (revs->ignore_missing_links)\n \t\t\treturn;\n+\n+\t\t/*\n+\t\t * Pre-filter known-missing tree objects when explicitly\n+\t\t * requested.  This may cause the actual filter to report\n+\t\t * an incomplete list of missing objects.\n+\t\t */\n+\t\tif (revs->exclude_promisor_objects &&\n+\t\t    is_promisor_object(&obj->oid))\n+\t\t\treturn;\n+\n \t\tdie(\"bad tree object %s\", oid_to_hex(&obj->oid));\n \t}\n \ndiff --git a/object.c b/object.c\nindex b9a4a0e..4c222d6 100644\n--- a/object.c\n+++ b/object.c\n@@ -252,7 +252,7 @@ struct object *parse_object(const struct object_id *oid)\n \tif (obj && obj->parsed)\n \t\treturn obj;\n \n-\tif ((obj && obj->type == OBJ_BLOB) ||\n+\tif ((obj && obj->type == OBJ_BLOB && has_object_file(oid)) ||\n \t    (!obj && has_object_file(oid) &&\n \t     sha1_object_info(oid->hash, NULL) == OBJ_BLOB)) {\n \t\tif (check_sha1_signature(repl, NULL, 0, NULL) < 0) {\ndiff --git a/revision.c b/revision.c\nindex d167223..05a7aac 100644\n--- a/revision.c\n+++ b/revision.c\n@@ -198,6 +198,8 @@ static struct object *get_reference(struct rev_info *revs, const char *name,\n \tif (!object) {\n \t\tif (revs->ignore_missing)\n \t\t\treturn object;\n+\t\tif (revs->exclude_promisor_objects && is_promisor_object(oid))\n+\t\t\treturn NULL;\n \t\tdie(\"bad object %s\", name);\n \t}\n \tobject->flags |= flags;\n@@ -790,9 +792,17 @@ static int add_parents_to_list(struct rev_info *revs, struct commit *commit,\n \n \tfor (parent = commit->parents; parent; parent = parent->next) {\n \t\tstruct commit *p = parent->item;\n-\n-\t\tif (parse_commit_gently(p, revs->ignore_missing_links) < 0)\n+\t\tint gently = revs->ignore_missing_links ||\n+\t\t\t     revs->exclude_promisor_objects;\n+\t\tif (parse_commit_gently(p, gently) < 0) {\n+\t\t\tif (revs->exclude_promisor_objects &&\n+\t\t\t    is_promisor_object(&p->object.oid)) {\n+\t\t\t\tif (revs->first_parent_only)\n+\t\t\t\t\tbreak;\n+\t\t\t\tcontinue;\n+\t\t\t}\n \t\t\treturn -1;\n+\t\t}\n \t\tif (revs->show_source && !p->util)\n \t\t\tp->util = commit->util;\n \t\tp->object.flags |= left_flag;\n@@ -2088,6 +2098,10 @@ static int handle_revision_opt(struct rev_info *revs, int argc, const char **arg\n \t\trevs->limited = 1;\n \t} else if (!strcmp(arg, \"--ignore-missing\")) {\n \t\trevs->ignore_missing = 1;\n+\t} else if (!strcmp(arg, \"--exclude-promisor-objects\")) {\n+\t\tif (fetch_if_missing)\n+\t\t\tdie(\"BUG: exclude_promisor_objects can only be used when fetch_if_missing is 0\");\n+\t\trevs->exclude_promisor_objects = 1;\n \t} else {\n \t\tint opts = diff_opt_parse(&revs->diffopt, argv, argc, revs->prefix);\n \t\tif (!opts)\n@@ -2830,6 +2844,16 @@ void reset_revision_walk(void)\n \tclear_object_flags(SEEN | ADDED | SHOWN);\n }\n \n+static int mark_uninteresting(const struct object_id *oid,\n+\t\t\t      struct packed_git *pack,\n+\t\t\t      uint32_t pos,\n+\t\t\t      void *unused)\n+{\n+\tstruct object *o = parse_object(oid);\n+\to->flags |= UNINTERESTING | SEEN;\n+\treturn 0;\n+}\n+\n int prepare_revision_walk(struct rev_info *revs)\n {\n \tint i;\n@@ -2858,6 +2882,11 @@ int prepare_revision_walk(struct rev_info *revs)\n \t    (revs->limited && limiting_can_increase_treesame(revs)))\n \t\trevs->treesame.name = \"treesame\";\n \n+\tif (revs->exclude_promisor_objects) {\n+\t\tfor_each_packed_object(mark_uninteresting, NULL,\n+\t\t\t\t       FOR_EACH_OBJECT_PROMISOR_ONLY);\n+\t}\n+\n \tif (revs->no_walk != REVISION_WALK_NO_WALK_UNSORTED)\n \t\tcommit_list_sort_by_date(&revs->commits);\n \tif (revs->no_walk)\ndiff --git a/revision.h b/revision.h\nindex 5476120..5f9a49c 100644\n--- a/revision.h\n+++ b/revision.h\n@@ -121,7 +121,10 @@ struct rev_info {\n \t\t\tbisect:1,\n \t\t\tancestry_path:1,\n \t\t\tfirst_parent_only:1,\n-\t\t\tline_level_traverse:1;\n+\t\t\tline_level_traverse:1,\n+\n+\t\t\t/* for internal use only */\n+\t\t\texclude_promisor_objects:1;\n \n \t/* Diff flags */\n \tunsigned int\tdiff:1,\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex 8a90f6a..3ca6af5 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -160,6 +160,107 @@ test_expect_success 'fetching of missing objects' '\n \tgit verify-pack --verbose \"$IDX\" | grep \"$HASH\"\n '\n \n+test_expect_success 'rev-list stops traversal at missing and promised commit' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\n+\tFOO=$(git -C repo rev-parse foo) &&\n+\tpromise_and_delete \"$FOO\" &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects bar >out &&\n+\tgrep $(git -C repo rev-parse bar) out &&\n+\t! grep $FOO out\n+'\n+\n+test_expect_success 'rev-list stops traversal at missing and promised tree' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\tmkdir repo/a_dir &&\n+\techo something >repo/a_dir/something &&\n+\tgit -C repo add a_dir/something &&\n+\tgit -C repo commit -m bar &&\n+\n+\t# foo^{tree} (tree referenced from commit)\n+\tTREE=$(git -C repo rev-parse foo^{tree}) &&\n+\n+\t# a tree referenced by HEAD^{tree} (tree referenced from tree)\n+\tTREE2=$(git -C repo ls-tree HEAD^{tree} | grep \" tree \" | head -1 | cut -b13-52) &&\n+\n+\tpromise_and_delete \"$TREE\" &&\n+\tpromise_and_delete \"$TREE2\" &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n+\tgrep $(git -C repo rev-parse foo) out &&\n+\t! grep $TREE out &&\n+\tgrep $(git -C repo rev-parse HEAD) out &&\n+\t! grep $TREE2 out\n+'\n+\n+test_expect_success 'rev-list stops traversal at missing and promised blob' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\techo something >repo/something &&\n+\tgit -C repo add something &&\n+\tgit -C repo commit -m foo &&\n+\n+\tBLOB=$(git -C repo hash-object -w something) &&\n+\tpromise_and_delete \"$BLOB\" &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n+\tgrep $(git -C repo rev-parse HEAD) out &&\n+\t! grep $BLOB out\n+'\n+\n+test_expect_success 'rev-list stops traversal at promisor commit, tree, and blob' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tCOMMIT=$(git -C repo rev-parse foo) &&\n+\tTREE=$(git -C repo rev-parse bar^{tree}) &&\n+\tBLOB=$(git hash-object repo/baz.t) &&\n+\tprintf \"%s\\n%s\\n%s\\n\" $COMMIT $TREE $BLOB | pack_as_from_promisor &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects HEAD >out &&\n+\t! grep $COMMIT out &&\n+\t! grep $TREE out &&\n+\t! grep $BLOB out &&\n+\tgrep $(git -C repo rev-parse bar) out  # sanity check that some walking was done\n+'\n+\n+test_expect_success 'rev-list accepts missing and promised objects on command line' '\n+\trm -rf repo &&\n+\ttest_create_repo repo &&\n+\ttest_commit -C repo foo &&\n+\ttest_commit -C repo bar &&\n+\ttest_commit -C repo baz &&\n+\n+\tCOMMIT=$(git -C repo rev-parse foo) &&\n+\tTREE=$(git -C repo rev-parse bar^{tree}) &&\n+\tBLOB=$(git hash-object repo/baz.t) &&\n+\n+\tpromise_and_delete $COMMIT &&\n+\tpromise_and_delete $TREE &&\n+\tpromise_and_delete $BLOB &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"arbitrary string\" &&\n+\tgit -C repo rev-list --exclude-promisor-objects --objects \"$COMMIT\" \"$TREE\" \"$BLOB\"\n+'\n+\n LIB_HTTPD_PORT=12345  # default port, 410, cannot be used as non-root\n . \"$TEST_DIRECTORY\"/lib-httpd.sh\n start_httpd\n-- \n2.9.3\n\n"},{"id":"334445","messageId":"20171208155851.855-9-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 08/16] fetch: refactor calculation of remote list","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:43Z","receivedAt":"2017-12-08T15:59:48Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nSeparate out the calculation of remotes to be fetched from and the\nactual fetching. This will allow us to include an additional step before\nthe actual fetching in a subsequent commit.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/fetch.c | 14 ++++++++------\n 1 file changed, 8 insertions(+), 6 deletions(-)\n\ndiff --git a/builtin/fetch.c b/builtin/fetch.c\nindex 225c734..1b1f039 100644\n--- a/builtin/fetch.c\n+++ b/builtin/fetch.c\n@@ -1322,7 +1322,7 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n {\n \tint i;\n \tstruct string_list list = STRING_LIST_INIT_DUP;\n-\tstruct remote *remote;\n+\tstruct remote *remote = NULL;\n \tint result = 0;\n \tstruct argv_array argv_gc_auto = ARGV_ARRAY_INIT;\n \n@@ -1367,17 +1367,14 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\telse if (argc > 1)\n \t\t\tdie(_(\"fetch --all does not make sense with refspecs\"));\n \t\t(void) for_each_remote(get_one_remote_for_fetch, &list);\n-\t\tresult = fetch_multiple(&list);\n \t} else if (argc == 0) {\n \t\t/* No arguments -- use default remote */\n \t\tremote = remote_get(NULL);\n-\t\tresult = fetch_one(remote, argc, argv);\n \t} else if (multiple) {\n \t\t/* All arguments are assumed to be remotes or groups */\n \t\tfor (i = 0; i < argc; i++)\n \t\t\tif (!add_remote_or_group(argv[i], &list))\n \t\t\t\tdie(_(\"No such remote or remote group: %s\"), argv[i]);\n-\t\tresult = fetch_multiple(&list);\n \t} else {\n \t\t/* Single remote or group */\n \t\t(void) add_remote_or_group(argv[0], &list);\n@@ -1385,14 +1382,19 @@ int cmd_fetch(int argc, const char **argv, const char *prefix)\n \t\t\t/* More than one remote */\n \t\t\tif (argc > 1)\n \t\t\t\tdie(_(\"Fetching a group and specifying refspecs does not make sense\"));\n-\t\t\tresult = fetch_multiple(&list);\n \t\t} else {\n \t\t\t/* Zero or one remotes */\n \t\t\tremote = remote_get(argv[0]);\n-\t\t\tresult = fetch_one(remote, argc-1, argv+1);\n+\t\t\targc--;\n+\t\t\targv++;\n \t\t}\n \t}\n \n+\tif (remote)\n+\t\tresult = fetch_one(remote, argc, argv);\n+\telse\n+\t\tresult = fetch_multiple(&list);\n+\n \tif (!result && (recurse_submodules != RECURSE_SUBMODULES_OFF)) {\n \t\tstruct argv_array options = ARGV_ARRAY_INIT;\n \n-- \n2.9.3\n\n"},{"id":"334446","messageId":"20171208155851.855-8-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 07/16] fetch-pack: test support excluding large blobs","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:42Z","receivedAt":"2017-12-08T15:59:50Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nCreated tests to verify fetch-pack and upload-pack support\nfor excluding large blobs using --filter=blobs:limit=<n>\nparameter.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n t/t5500-fetch-pack.sh | 27 +++++++++++++++++++++++++++\n upload-pack.c         | 13 +++++++++----\n 2 files changed, 36 insertions(+), 4 deletions(-)\n\ndiff --git a/t/t5500-fetch-pack.sh b/t/t5500-fetch-pack.sh\nindex 80a1a32..c57916b 100755\n--- a/t/t5500-fetch-pack.sh\n+++ b/t/t5500-fetch-pack.sh\n@@ -755,4 +755,31 @@ test_expect_success 'fetching deepen' '\n \t)\n '\n \n+test_expect_success 'filtering by size' '\n+\trm -rf server client &&\n+\ttest_create_repo server &&\n+\ttest_commit -C server one &&\n+\ttest_config -C server uploadpack.allowfilter 1 &&\n+\n+\ttest_create_repo client &&\n+\tgit -C client fetch-pack --filter=blob:limit=0 ../server HEAD &&\n+\n+\t# Ensure that object is not inadvertently fetched\n+\ttest_must_fail git -C client cat-file -e $(git hash-object server/one.t)\n+'\n+\n+test_expect_success 'filtering by size has no effect if support for it is not advertised' '\n+\trm -rf server client &&\n+\ttest_create_repo server &&\n+\ttest_commit -C server one &&\n+\n+\ttest_create_repo client &&\n+\tgit -C client fetch-pack --filter=blob:limit=0 ../server HEAD 2> err &&\n+\n+\t# Ensure that object is fetched\n+\tgit -C client cat-file -e $(git hash-object server/one.t) &&\n+\n+\ttest_i18ngrep \"filtering not recognized by server\" err\n+'\n+\n test_done\ndiff --git a/upload-pack.c b/upload-pack.c\nindex e6d38b9..15b6605 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -139,10 +139,15 @@ static void create_pack_file(void)\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n \tif (filter_options.filter_spec) {\n-\t\tstruct strbuf buf = STRBUF_INIT;\n-\t\tsq_quote_buf(&buf, filter_options.filter_spec);\n-\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\", buf.buf);\n-\t\tstrbuf_release(&buf);\n+\t\tif (pack_objects.use_shell) {\n+\t\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\t\tsq_quote_buf(&buf, filter_options.filter_spec);\n+\t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\", buf.buf);\n+\t\t\tstrbuf_release(&buf);\n+\t\t} else {\n+\t\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\",\n+\t\t\t\t\t filter_options.filter_spec);\n+\t\t}\n \t}\n \n \tpack_objects.in = -1;\n-- \n2.9.3\n\n"},{"id":"334447","messageId":"20171208155851.855-2-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 01/16] sha1_file: support lazily fetching missing objects","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:36Z","receivedAt":"2017-12-08T15:59:54Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jonathan Tan <jonathantanmy@google.com>\n\nTeach sha1_file to fetch objects from the remote configured in\nextensions.partialclone whenever an object is requested but missing.\n\nThe fetching of objects can be suppressed through a global variable.\nThis is used by fsck and index-pack.\n\nHowever, by default, such fetching is not suppressed. This is meant as a\ntemporary measure to ensure that all Git commands work in such a\nsituation. Future patches will update some commands to either tolerate\nmissing objects (without fetching them) or be more efficient in fetching\nthem.\n\nIn order to determine the code changes in sha1_file.c necessary, I\ninvestigated the following:\n (1) functions in sha1_file.c that take in a hash, without the user\n     regarding how the object is stored (loose or packed)\n (2) functions in packfile.c (because I need to check callers that know\n     about the loose/packed distinction and operate on both differently,\n     and ensure that they can handle the concept of objects that are\n     neither loose nor packed)\n\n(1) is handled by the modification to sha1_object_info_extended().\n\nFor (2), I looked at for_each_packed_object and others.  For\nfor_each_packed_object, the callers either already work or are fixed in\nthis patch:\n - reachable - only to find recent objects\n - builtin/fsck - already knows about missing objects\n - builtin/cat-file - warning message added in this commit\n\nCallers of the other functions do not need to be changed:\n - parse_pack_index\n   - http - indirectly from http_get_info_packs\n   - find_pack_entry_one\n     - this searches a single pack that is provided as an argument; the\n       caller already knows (through other means) that the sought object\n       is in a specific pack\n - find_sha1_pack\n   - fast-import - appears to be an optimization to not store a file if\n     it is already in a pack\n   - http-walker - to search through a struct alt_base\n   - http-push - to search through remote packs\n - has_sha1_pack\n   - builtin/fsck - already knows about promisor objects\n   - builtin/count-objects - informational purposes only (check if loose\n     object is also packed)\n   - builtin/prune-packed - check if object to be pruned is packed (if\n     not, don't prune it)\n   - revision - used to exclude packed objects if requested by user\n   - diff - just for optimization\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n builtin/cat-file.c       |  2 ++\n builtin/fetch-pack.c     |  2 ++\n builtin/fsck.c           |  3 +++\n builtin/index-pack.c     |  6 ++++++\n cache.h                  |  8 ++++++++\n fetch-object.c           |  3 +++\n sha1_file.c              | 32 ++++++++++++++++++++++--------\n t/t0410-partial-clone.sh | 51 ++++++++++++++++++++++++++++++++++++++++++++++++\n 8 files changed, 99 insertions(+), 8 deletions(-)\n\ndiff --git a/builtin/cat-file.c b/builtin/cat-file.c\nindex f5fa4fd..cf9ea5c 100644\n--- a/builtin/cat-file.c\n+++ b/builtin/cat-file.c\n@@ -475,6 +475,8 @@ static int batch_objects(struct batch_options *opt)\n \n \t\tfor_each_loose_object(batch_loose_object, &sa, 0);\n \t\tfor_each_packed_object(batch_packed_object, &sa, 0);\n+\t\tif (repository_format_partial_clone)\n+\t\t\twarning(\"This repository has extensions.partialClone set. Some objects may not be loaded.\");\n \n \t\tcb.opt = opt;\n \t\tcb.expand = &data;\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 02abe72..15eeed7 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -53,6 +53,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \tstruct oid_array shallow = OID_ARRAY_INIT;\n \tstruct string_list deepen_not = STRING_LIST_INIT_DUP;\n \n+\tfetch_if_missing = 0;\n+\n \tpacket_trace_identity(\"fetch-pack\");\n \n \tmemset(&args, 0, sizeof(args));\ndiff --git a/builtin/fsck.c b/builtin/fsck.c\nindex 578a7c8..3b76c0e 100644\n--- a/builtin/fsck.c\n+++ b/builtin/fsck.c\n@@ -678,6 +678,9 @@ int cmd_fsck(int argc, const char **argv, const char *prefix)\n \tint i;\n \tstruct alternate_object_database *alt;\n \n+\t/* fsck knows how to handle missing promisor objects */\n+\tfetch_if_missing = 0;\n+\n \terrors_found = 0;\n \tcheck_replace_refs = 0;\n \ndiff --git a/builtin/index-pack.c b/builtin/index-pack.c\nindex 24c2f05..a0a35e6 100644\n--- a/builtin/index-pack.c\n+++ b/builtin/index-pack.c\n@@ -1657,6 +1657,12 @@ int cmd_index_pack(int argc, const char **argv, const char *prefix)\n \tunsigned foreign_nr = 1;\t/* zero is a \"good\" value, assume bad */\n \tint report_end_of_input = 0;\n \n+\t/*\n+\t * index-pack never needs to fetch missing objects, since it only\n+\t * accesses the repo to do hash collision checks\n+\t */\n+\tfetch_if_missing = 0;\n+\n \tif (argc == 2 && !strcmp(argv[1], \"-h\"))\n \t\tusage(index_pack_usage);\n \ndiff --git a/cache.h b/cache.h\nindex c76f2e9..6980072 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1727,6 +1727,14 @@ struct object_info {\n #define OBJECT_INFO_QUICK 8\n extern int sha1_object_info_extended(const unsigned char *, struct object_info *, unsigned flags);\n \n+/*\n+ * Set this to 0 to prevent sha1_object_info_extended() from fetching missing\n+ * blobs. This has a difference only if extensions.partialClone is set.\n+ *\n+ * Its default value is 1.\n+ */\n+extern int fetch_if_missing;\n+\n /* Dumb servers support */\n extern int update_server_info(int);\n \ndiff --git a/fetch-object.c b/fetch-object.c\nindex 08e91ce..258fcfa 100644\n--- a/fetch-object.c\n+++ b/fetch-object.c\n@@ -10,7 +10,9 @@ void fetch_object(const char *remote_name, const unsigned char *sha1)\n \tstruct remote *remote;\n \tstruct transport *transport;\n \tstruct ref *ref;\n+\tint original_fetch_if_missing = fetch_if_missing;\n \n+\tfetch_if_missing = 0;\n \tremote = remote_get(remote_name);\n \tif (!remote->url[0])\n \t\tdie(_(\"Remote with no URL\"));\n@@ -21,4 +23,5 @@ void fetch_object(const char *remote_name, const unsigned char *sha1)\n \ttransport_set_option(transport, TRANS_OPT_FROM_PROMISOR, \"1\");\n \ttransport_set_option(transport, TRANS_OPT_NO_DEPENDENTS, \"1\");\n \ttransport_fetch_refs(transport, ref);\n+\tfetch_if_missing = original_fetch_if_missing;\n }\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 10c3a00..dd956e2 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -29,6 +29,7 @@\n #include \"mergesort.h\"\n #include \"quote.h\"\n #include \"packfile.h\"\n+#include \"fetch-object.h\"\n \n const unsigned char null_sha1[GIT_MAX_RAWSZ];\n const struct object_id null_oid;\n@@ -1144,6 +1145,8 @@ static int sha1_loose_object_info(const unsigned char *sha1,\n \treturn (status < 0) ? status : 0;\n }\n \n+int fetch_if_missing = 1;\n+\n int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi, unsigned flags)\n {\n \tstatic struct object_info blank_oi = OBJECT_INFO_INIT;\n@@ -1152,6 +1155,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \tconst unsigned char *real = (flags & OBJECT_INFO_LOOKUP_REPLACE) ?\n \t\t\t\t    lookup_replace_object(sha1) :\n \t\t\t\t    sha1;\n+\tint already_retried = 0;\n \n \tif (!oi)\n \t\toi = &blank_oi;\n@@ -1176,19 +1180,32 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t}\n \t}\n \n-\tif (!find_pack_entry(real, &e)) {\n+\twhile (1) {\n+\t\tif (find_pack_entry(real, &e))\n+\t\t\tbreak;\n+\n \t\t/* Most likely it's a loose object. */\n \t\tif (!sha1_loose_object_info(real, oi, flags))\n \t\t\treturn 0;\n \n \t\t/* Not a loose object; someone else may have just packed it. */\n-\t\tif (flags & OBJECT_INFO_QUICK) {\n-\t\t\treturn -1;\n-\t\t} else {\n-\t\t\treprepare_packed_git();\n-\t\t\tif (!find_pack_entry(real, &e))\n-\t\t\t\treturn -1;\n+\t\treprepare_packed_git();\n+\t\tif (find_pack_entry(real, &e))\n+\t\t\tbreak;\n+\n+\t\t/* Check if it is a missing object */\n+\t\tif (fetch_if_missing && repository_format_partial_clone &&\n+\t\t    !already_retried) {\n+\t\t\t/*\n+\t\t\t * TODO Investigate haveing fetch_object() return\n+\t\t\t * TODO error/success and stopping the music here.\n+\t\t\t */\n+\t\t\tfetch_object(repository_format_partial_clone, real);\n+\t\t\talready_retried = 1;\n+\t\t\tcontinue;\n \t\t}\n+\n+\t\treturn -1;\n \t}\n \n \tif (oi == &blank_oi)\n@@ -1197,7 +1214,6 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t * information below, so return early.\n \t\t */\n \t\treturn 0;\n-\n \trtype = packed_object_info(e.p, e.offset, oi);\n \tif (rtype < 0) {\n \t\tmark_bad_packed_object(e.p, real);\ndiff --git a/t/t0410-partial-clone.sh b/t/t0410-partial-clone.sh\nindex e96f436..8a90f6a 100755\n--- a/t/t0410-partial-clone.sh\n+++ b/t/t0410-partial-clone.sh\n@@ -138,4 +138,55 @@ test_expect_success 'missing CLI object, but promised, passes fsck' '\n \tgit -C repo fsck \"$A\"\n '\n \n+test_expect_success 'fetching of missing objects' '\n+\trm -rf repo &&\n+\ttest_create_repo server &&\n+\ttest_commit -C server foo &&\n+\tgit -C server repack -a -d --write-bitmap-index &&\n+\n+\tgit clone \"file://$(pwd)/server\" repo &&\n+\tHASH=$(git -C repo rev-parse foo) &&\n+\trm -rf repo/.git/objects/* &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"origin\" &&\n+\tgit -C repo cat-file -p \"$HASH\" &&\n+\n+\t# Ensure that the .promisor file is written, and check that its\n+\t# associated packfile contains the object\n+\tls repo/.git/objects/pack/pack-*.promisor >promisorlist &&\n+\ttest_line_count = 1 promisorlist &&\n+\tIDX=$(cat promisorlist | sed \"s/promisor$/idx/\") &&\n+\tgit verify-pack --verbose \"$IDX\" | grep \"$HASH\"\n+'\n+\n+LIB_HTTPD_PORT=12345  # default port, 410, cannot be used as non-root\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'fetching of missing objects from an HTTP server' '\n+\trm -rf repo &&\n+\tSERVER=\"$HTTPD_DOCUMENT_ROOT_PATH/server\" &&\n+\ttest_create_repo \"$SERVER\" &&\n+\ttest_commit -C \"$SERVER\" foo &&\n+\tgit -C \"$SERVER\" repack -a -d --write-bitmap-index &&\n+\n+\tgit clone $HTTPD_URL/smart/server repo &&\n+\tHASH=$(git -C repo rev-parse foo) &&\n+\trm -rf repo/.git/objects/* &&\n+\n+\tgit -C repo config core.repositoryformatversion 1 &&\n+\tgit -C repo config extensions.partialclone \"origin\" &&\n+\tgit -C repo cat-file -p \"$HASH\" &&\n+\n+\t# Ensure that the .promisor file is written, and check that its\n+\t# associated packfile contains the object\n+\tls repo/.git/objects/pack/pack-*.promisor >promisorlist &&\n+\ttest_line_count = 1 promisorlist &&\n+\tIDX=$(cat promisorlist | sed \"s/promisor$/idx/\") &&\n+\tgit verify-pack --verbose \"$IDX\" | grep \"$HASH\"\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.9.3\n\n"},{"id":"334448","messageId":"20171208155851.855-6-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 05/16] fetch-pack, index-pack, transport: partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:40Z","receivedAt":"2017-12-08T15:59:57Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n builtin/fetch-pack.c |  4 ++++\n fetch-pack.c         | 13 +++++++++++++\n fetch-pack.h         |  2 ++\n transport-helper.c   |  5 +++++\n transport.c          |  4 ++++\n transport.h          |  5 +++++\n 6 files changed, 33 insertions(+)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 15eeed7..7957807 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -153,6 +153,10 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.no_dependents = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(arg, (\"--\" CL_ARG__FILTER \"=\"), &arg)) {\n+\t\t\tparse_list_objects_filter(&args.filter_options, arg);\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \tif (deepen_not.nr)\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 0798e0b..3c5f064 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -29,6 +29,7 @@ static int deepen_not_ok;\n static int fetch_fsck_objects = -1;\n static int transfer_fsck_objects = -1;\n static int agent_supported;\n+static int server_supports_filtering;\n static struct lock_file shallow_lock;\n static const char *alternate_shallow_file;\n \n@@ -379,6 +380,8 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tif (deepen_not_ok)      strbuf_addstr(&c, \" deepen-not\");\n \t\t\tif (agent_supported)    strbuf_addf(&c, \" agent=%s\",\n \t\t\t\t\t\t\t    git_user_agent_sanitized());\n+\t\t\tif (args->filter_options.choice)\n+\t\t\t\tstrbuf_addstr(&c, \" filter\");\n \t\t\tpacket_buf_write(&req_buf, \"want %s%s\\n\", remote_hex, c.buf);\n \t\t\tstrbuf_release(&c);\n \t\t} else\n@@ -407,6 +410,9 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tpacket_buf_write(&req_buf, \"deepen-not %s\", s->string);\n \t\t}\n \t}\n+\tif (server_supports_filtering && args->filter_options.choice)\n+\t\tpacket_buf_write(&req_buf, \"filter %s\",\n+\t\t\t\t args->filter_options.filter_spec);\n \tpacket_buf_flush(&req_buf);\n \tstate_len = req_buf.len;\n \n@@ -969,6 +975,13 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \telse\n \t\tprefer_ofs_delta = 0;\n \n+\tif (server_supports(\"filter\")) {\n+\t\tserver_supports_filtering = 1;\n+\t\tprint_verbose(args, _(\"Server supports filter\"));\n+\t} else if (args->filter_options.choice) {\n+\t\twarning(\"filtering not recognized by server, ignoring\");\n+\t}\n+\n \tif ((agent_feature = server_feature_value(\"agent\", &agent_len))) {\n \t\tagent_supported = 1;\n \t\tif (agent_len)\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex aeac152..3e224a1 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -3,6 +3,7 @@\n \n #include \"string-list.h\"\n #include \"run-command.h\"\n+#include \"list-objects-filter-options.h\"\n \n struct oid_array;\n \n@@ -12,6 +13,7 @@ struct fetch_pack_args {\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n+\tstruct list_objects_filter_options filter_options;\n \tunsigned deepen_relative:1;\n \tunsigned quiet:1;\n \tunsigned keep_pack:1;\ndiff --git a/transport-helper.c b/transport-helper.c\nindex c948d52..0650ca0 100644\n--- a/transport-helper.c\n+++ b/transport-helper.c\n@@ -671,6 +671,11 @@ static int fetch(struct transport *transport,\n \tif (data->transport_options.update_shallow)\n \t\tset_helper_option(transport, \"update-shallow\", \"true\");\n \n+\tif (data->transport_options.filter_options.choice)\n+\t\tset_helper_option(\n+\t\t\ttransport, \"filter\",\n+\t\t\tdata->transport_options.filter_options.filter_spec);\n+\n \tif (data->fetch)\n \t\treturn fetch_with_fetch(transport, nr_heads, to_fetch);\n \ndiff --git a/transport.c b/transport.c\nindex f2fbc6f..cce50f5 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -166,6 +166,9 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_NO_DEPENDENTS)) {\n \t\topts->no_dependents = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_LIST_OBJECTS_FILTER)) {\n+\t\tparse_list_objects_filter(&opts->filter_options, value);\n+\t\treturn 0;\n \t}\n \treturn 1;\n }\n@@ -236,6 +239,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.update_shallow = data->options.update_shallow;\n \targs.from_promisor = data->options.from_promisor;\n \targs.no_dependents = data->options.no_dependents;\n+\targs.filter_options = data->options.filter_options;\n \n \tif (!data->got_remote_heads) {\n \t\tconnect_setup(transport, 0);\ndiff --git a/transport.h b/transport.h\nindex c49a8ed..31b1936 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -4,6 +4,7 @@\n #include \"cache.h\"\n #include \"run-command.h\"\n #include \"remote.h\"\n+#include \"list-objects-filter-options.h\"\n \n struct string_list;\n \n@@ -23,6 +24,7 @@ struct git_transport_options {\n \tconst char *uploadpack;\n \tconst char *receivepack;\n \tstruct push_cas_option *cas;\n+\tstruct list_objects_filter_options filter_options;\n };\n \n enum transport_family {\n@@ -221,6 +223,9 @@ void transport_check_allowed(const char *type);\n  */\n #define TRANS_OPT_NO_DEPENDENTS \"no-dependents\"\n \n+/* Filter objects for partial clone and fetch */\n+#define TRANS_OPT_LIST_OBJECTS_FILTER \"filter\"\n+\n /**\n  * Returns 0 if the option was used, non-zero otherwise. Prints a\n  * message to stderr if the option is not used.\n-- \n2.9.3\n\n"},{"id":"334449","messageId":"20171208155851.855-7-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 06/16] fetch-pack: add --no-filter","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:41Z","receivedAt":"2017-12-08T15:59:59Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nFixup fetch-pack to accept --no-filter to be consistent with\nrev-list and pack-objects.\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n builtin/fetch-pack.c | 4 ++++\n 1 file changed, 4 insertions(+)\n\ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 7957807..cbf5035 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -157,6 +157,10 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\tparse_list_objects_filter(&args.filter_options, arg);\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(arg, (\"--no-\" CL_ARG__FILTER))) {\n+\t\t\tlist_objects_filter_release(&args.filter_options);\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \tif (deepen_not.nr)\n-- \n2.9.3\n\n"},{"id":"334450","messageId":"20171208155851.855-5-git@jeffhostetler.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"[PATCH v7 04/16] upload-pack: add object filtering for partial clone","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T15:58:39Z","receivedAt":"2017-12-08T16:00:02Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"From: Jeff Hostetler <jeffhost@microsoft.com>\n\nTeach upload-pack to negotiate object filtering over the protocol and\nto send filter parameters to pack-objects.  This is intended for partial\nclone and fetch.\n\nThe idea to make upload-pack configurable using uploadpack.allowFilter\ncomes from Jonathan Tan's work in [1].\n\n[1] https://public-inbox.org/git/f211093280b422c32cc1b7034130072f35c5ed51.1506714999.git.jonathantanmy@google.com/\n\nSigned-off-by: Jeff Hostetler <jeffhost@microsoft.com>\n---\n Documentation/config.txt                          |  4 ++++\n Documentation/technical/pack-protocol.txt         |  8 +++++++\n Documentation/technical/protocol-capabilities.txt |  8 +++++++\n upload-pack.c                                     | 26 ++++++++++++++++++++++-\n 4 files changed, 45 insertions(+), 1 deletion(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 1ac0ae6..e528210 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -3268,6 +3268,10 @@ uploadpack.packObjectsHook::\n \twas run. I.e., `upload-pack` will feed input intended for\n \t`pack-objects` to the hook, and expects a completed packfile on\n \tstdout.\n+\n+uploadpack.allowFilter::\n+\tIf this option is set, `upload-pack` will advertise partial\n+\tclone and partial fetch object filtering.\n +\n Note that this configuration variable is ignored if it is seen in the\n repository-level config (this is a safety measure against fetching from\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex ed1eae8..a43a113 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -212,6 +212,7 @@ out of what the server said it could do with the first 'want' line.\n   upload-request    =  want-list\n \t\t       *shallow-line\n \t\t       *1depth-request\n+\t\t       [filter-request]\n \t\t       flush-pkt\n \n   want-list         =  first-want\n@@ -227,6 +228,8 @@ out of what the server said it could do with the first 'want' line.\n   additional-want   =  PKT-LINE(\"want\" SP obj-id)\n \n   depth             =  1*DIGIT\n+\n+  filter-request    =  PKT-LINE(\"filter\" SP filter-spec)\n ----\n \n Clients MUST send all the obj-ids it wants from the reference\n@@ -249,6 +252,11 @@ complete those commits. Commits whose parents are not received as a\n result are defined as shallow and marked as such in the server. This\n information is sent back to the client in the next step.\n \n+The client can optionally request that pack-objects omit various\n+objects from the packfile using one of several filtering techniques.\n+These are intended for use with partial clone and partial fetch\n+operations.  See `rev-list` for possible \"filter-spec\" values.\n+\n Once all the 'want's and 'shallow's (and optional 'deepen') are\n transferred, clients MUST send a flush-pkt, to tell the server side\n that it is done sending the list.\ndiff --git a/Documentation/technical/protocol-capabilities.txt b/Documentation/technical/protocol-capabilities.txt\nindex 26dcc6f..332d209 100644\n--- a/Documentation/technical/protocol-capabilities.txt\n+++ b/Documentation/technical/protocol-capabilities.txt\n@@ -309,3 +309,11 @@ to accept a signed push certificate, and asks the <nonce> to be\n included in the push certificate.  A send-pack client MUST NOT\n send a push-cert packet unless the receive-pack server advertises\n this capability.\n+\n+filter\n+------\n+\n+If the upload-pack server advertises the 'filter' capability,\n+fetch-pack may send \"filter\" commands to request a partial clone\n+or partial fetch and request that the server omit various objects\n+from the packfile.\ndiff --git a/upload-pack.c b/upload-pack.c\nindex e25f725..e6d38b9 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -10,6 +10,8 @@\n #include \"diff.h\"\n #include \"revision.h\"\n #include \"list-objects.h\"\n+#include \"list-objects-filter.h\"\n+#include \"list-objects-filter-options.h\"\n #include \"run-command.h\"\n #include \"connect.h\"\n #include \"sigchain.h\"\n@@ -18,6 +20,7 @@\n #include \"parse-options.h\"\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n+#include \"quote.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [<options>] <dir>\"),\n@@ -64,6 +67,10 @@ static int advertise_refs;\n static int stateless_rpc;\n static const char *pack_objects_hook;\n \n+static int filter_capability_requested;\n+static int filter_advertise;\n+static struct list_objects_filter_options filter_options;\n+\n static void reset_timeout(void)\n {\n \talarm(timeout);\n@@ -131,6 +138,12 @@ static void create_pack_file(void)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n+\tif (filter_options.filter_spec) {\n+\t\tstruct strbuf buf = STRBUF_INIT;\n+\t\tsq_quote_buf(&buf, filter_options.filter_spec);\n+\t\targv_array_pushf(&pack_objects.args, \"--filter=%s\", buf.buf);\n+\t\tstrbuf_release(&buf);\n+\t}\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n@@ -794,6 +807,12 @@ static void receive_needs(void)\n \t\t\tdeepen_rev_list = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (skip_prefix(line, \"filter \", &arg)) {\n+\t\t\tif (!filter_capability_requested)\n+\t\t\t\tdie(\"git upload-pack: filtering capability not negotiated\");\n+\t\t\tparse_list_objects_filter(&filter_options, arg);\n+\t\t\tcontinue;\n+\t\t}\n \t\tif (!skip_prefix(line, \"want \", &arg) ||\n \t\t    get_oid_hex(arg, &oid_buf))\n \t\t\tdie(\"git upload-pack: protocol error, \"\n@@ -821,6 +840,8 @@ static void receive_needs(void)\n \t\t\tno_progress = 1;\n \t\tif (parse_feature_request(features, \"include-tag\"))\n \t\t\tuse_include_tag = 1;\n+\t\tif (parse_feature_request(features, \"filter\"))\n+\t\t\tfilter_capability_requested = 1;\n \n \t\to = parse_object(&oid_buf);\n \t\tif (!o) {\n@@ -940,7 +961,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\tstruct strbuf symref_info = STRBUF_INIT;\n \n \t\tformat_symref_info(&symref_info, cb_data);\n-\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s agent=%s\\n\",\n+\t\tpacket_write_fmt(1, \"%s %s%c%s%s%s%s%s%s agent=%s\\n\",\n \t\t\t     oid_to_hex(oid), refname_nons,\n \t\t\t     0, capabilities,\n \t\t\t     (allow_unadvertised_object_request & ALLOW_TIP_SHA1) ?\n@@ -949,6 +970,7 @@ static int send_ref(const char *refname, const struct object_id *oid,\n \t\t\t\t     \" allow-reachable-sha1-in-want\" : \"\",\n \t\t\t     stateless_rpc ? \" no-done\" : \"\",\n \t\t\t     symref_info.buf,\n+\t\t\t     filter_advertise ? \" filter\" : \"\",\n \t\t\t     git_user_agent_sanitized());\n \t\tstrbuf_release(&symref_info);\n \t} else {\n@@ -1027,6 +1049,8 @@ static int upload_pack_config(const char *var, const char *value, void *unused)\n \t} else if (current_config_scope() != CONFIG_SCOPE_REPO) {\n \t\tif (!strcmp(\"uploadpack.packobjectshook\", var))\n \t\t\treturn git_config_string(&pack_objects_hook, var, value);\n+\t} else if (!strcmp(\"uploadpack.allowfilter\", var)) {\n+\t\tfilter_advertise = git_config_bool(var, value);\n \t}\n \treturn parse_hide_refs_config(var, value, \"uploadpack\");\n }\n-- \n2.9.3\n\n"},{"id":"334469","messageId":"xmqqtvx185kq.fsf@gitster.mtv.corp.google.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"Re: [PATCH v7 00/16] Parial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-12-08T17:58:29Z","receivedAt":"2017-12-08T17:58:37Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler <git@jeffhostetler.com> writes:\n\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> This is V7 of part 3 of partial clone.  It builds upon V7 of part 2\n> (which builds upon V6 of part 1).\n\nAren't the three patches at the bottom sort-of duplicate from the\npart 2 series?\n\n"},{"id":"334472","messageId":"67f80ac5-6530-094f-bd10-01ef8b5a6b97@jeffhostetler.com","threadId":"47411","inReplyTo":"xmqqtvx185kq.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v7 00/16] Parial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2017-12-08T18:10:39Z","receivedAt":"2017-12-08T18:10:45Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 12/8/2017 12:58 PM, Junio C Hamano wrote:\n> Jeff Hostetler <git@jeffhostetler.com> writes:\n> \n>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>\n>> This is V7 of part 3 of partial clone.  It builds upon V7 of part 2\n>> (which builds upon V6 of part 1).\n> \n> Aren't the three patches at the bottom sort-of duplicate from the\n> part 2 series?\n> \n\noops.  yes, you're right.  it looks like i selected pc*6*_p2..pc7_p3\nrather than pc*7*_p2..pc7_p3.  sorry for the typo.\n\nand since the only changes in p2 were to squash those 2 commits near\nthe tip of p2, only those 3 commits changed SHAs in v7 over v6.\n\nso, please disregard the duplicates.\n\nwould you like me to send a corrected V8 for p3 ?\n\nJeff\n"},{"id":"334477","messageId":"xmqqh8t184ff.fsf@gitster.mtv.corp.google.com","threadId":"47411","inReplyTo":"67f80ac5-6530-094f-bd10-01ef8b5a6b97@jeffhostetler.com","subject":"Re: [PATCH v7 00/16] Parial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-12-08T18:23:16Z","receivedAt":"2017-12-08T18:23:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jeff Hostetler <git@jeffhostetler.com> writes:\n\n> On 12/8/2017 12:58 PM, Junio C Hamano wrote:\n>> Jeff Hostetler <git@jeffhostetler.com> writes:\n>>\n>>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>>\n>>> This is V7 of part 3 of partial clone.  It builds upon V7 of part 2\n>>> (which builds upon V6 of part 1).\n>>\n>> Aren't the three patches at the bottom sort-of duplicate from the\n>> part 2 series?\n>>\n>\n> oops.  yes, you're right.  it looks like i selected pc*6*_p2..pc7_p3\n> rather than pc*7*_p2..pc7_p3.  sorry for the typo.\n>\n> and since the only changes in p2 were to squash those 2 commits near\n> the tip of p2, only those 3 commits changed SHAs in v7 over v6.\n>\n> so, please disregard the duplicates.\n>\n> would you like me to send a corrected V8 for p3 ?\n\nNah.  I just wanted to make sure that I am discarding the right ones\n(i.e. 1-3/16 of partial-clone, not 8-10/10 of fsck-promisors).\n\nThanks for an update.\n\n"},{"id":"334512","messageId":"20171208223010.GF140529@google.com","threadId":"47411","inReplyTo":"20171208155851.855-1-git@jeffhostetler.com","subject":"Re: [PATCH v7 00/16] Parial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-12-08T22:30:10Z","receivedAt":"2017-12-08T22:30:20Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 12/08, Jeff Hostetler wrote:\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n> \n> This is V7 of part 3 of partial clone.  It builds upon V7 of part 2\n> (which builds upon V6 of part 1).\n> \n> This version adds additional tests, fixes test errors on the MAC version,\n> and squashes some fixup commits.\n> \n> It also restores functionality accidentally dropped from the V6 series\n> for \"git fetch\" to automatically inherit the partial-clone filter-spec\n> when appropriate.  This version extends the --no-filter argument to\n> override this inheritance.\n> \n\nI just finished reading through parts 1-3.  Overall I like the series.\nThere are a few point's that I'm not a big fan of but i wasn't able to\ncome up with a better alternative.  One of these being the need for a\nglobal variable to tell the fetch-object logic to not go to the server\nto try and fetch a missing object.\n\nOne other thing i noticed was it looks like when you discover that you\nare missing a blob you you'll try to fault it in from the server without\nfirst checking its an object the server would even have.  Shouldn't you\nfirst do a check to verify that the object in question is a promised\nobject before you go out to contact the server to request it?  You may\nhave already ruled this out for some reason I'm not aware of (maybe its\ntoo costly to compute?).\n\n\n-- \nBrandon Williams\n"},{"id":"334620","messageId":"20171211111916.b3ea2deacba67f6e8416d285@google.com","threadId":"47411","inReplyTo":"20171208223010.GF140529@google.com","subject":"Re: [PATCH v7 00/16] Parial clone part 3: clone, fetch, fetch-pack, upload-pack, and tests","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-12-11T19:19:16Z","receivedAt":"2017-12-11T19:19:24Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 8 Dec 2017 14:30:10 -0800\nBrandon Williams <bmwill@google.com> wrote:\n\n> I just finished reading through parts 1-3.  Overall I like the series.\n> There are a few point's that I'm not a big fan of but i wasn't able to\n> come up with a better alternative.  One of these being the need for a\n> global variable to tell the fetch-object logic to not go to the server\n> to try and fetch a missing object.\n\nI didn't really like that approach too but I went with that because,\nlike you, I couldn't come up with a better one. The main issue is that\ntoo many functions (e.g. parse_commit() in commit.c) indirectly read\nobjects, and I couldn't find a better way to control them all. Ideally,\nwe should have a \"struct object_store\" (or maybe \"struct repository\"\ncould do this too) on which we can set \"fetch_if_missing\", and have all\nobject-reading functions take a pointer to this struct. Or completely\nseparate the object-reading and object-parsing code (e.g. commit.c\nshould not be able to read objects at all). Or both.\n\nAny of these would be major undertakings, though, and there are good\nreasons for why the same function does the reading and parsing (for\nexample, parse_commit() does not perform any reading if the object has\nbeen already parsed).\n\n> One other thing i noticed was it looks like when you discover that you\n> are missing a blob you you'll try to fault it in from the server without\n> first checking its an object the server would even have.  Shouldn't you\n> first do a check to verify that the object in question is a promised\n> object before you go out to contact the server to request it?  You may\n> have already ruled this out for some reason I'm not aware of (maybe its\n> too costly to compute?).\n\nIt is quite costly to compute - in the worst case, we would need to read\nevery object in every promisor packfile of one or more certain types\n(e.g. if we know that we're fetching a blob, we need to read every tree)\nto find out if the object we want is a promisor object.\n\nSuch a check would be better at surfacing mistakes (e.g. the user giving\nthe wrong SHA-1) early, but beyond that, I don't think that having the\ncheck is very important. Consider these two very common situations:\n\n (1) Fetching a single branch by its tip's SHA-1. A naive implementation\n     will first check if we have that SHA-1, which triggers the dynamic\n     fetch (since it is an object read), and assuming success, notice\n     that we indeed have that tip, and not fetch anything else. The\n     check you describe will avoid this situation.\n (2) Dynamically fetching a missing blob by its SHA-1. A naive\n     implementation will first check if we have that SHA-1, which\n     triggers the dynamic fetch, and that fetch will first check if we\n     have that SHA-1, and so on (thus, an infinite loop). The check you\n     describe will not avoid that situation.\n\nThe check solves (1), but we still need a solution to (2) - I used\n\"fetch_if_missing\", as discussed in your previous question and my answer\nto that. A solution to (2) is usually also a solution to (1), so the\ncheck wouldn't help much here.\n"},{"id":"356032","messageId":"CACsJy8D3nC+fXFYMKWh6AStbMVOVfmcAvAT9tBqtG9K6r043eQ@mail.gmail.com","threadId":"47411","inReplyTo":"20171208155851.855-10-git@jeffhostetler.com","subject":"Re: [PATCH v7 09/16] fetch: support filters","fromName":"Duy Nguyen","fromEmail":"pclouds@gmail.com","sentAt":"2018-08-19T11:24:41Z","receivedAt":"2018-08-19T11:25:10Z","isPatch":true,"sender":{"key":"pclouds@gmail.com","avatar":"https://avatars.githubusercontent.com/u/720?v=4"},"body":"On Fri, Dec 8, 2017 at 5:00 PM Jeff Hostetler <git@jeffhostetler.com> wrote:\n>\n> From: Jeff Hostetler <jeffhost@microsoft.com>\n>\n> Teach fetch to support filters. This is only allowed for the remote\n> configured in extensions.partialcloneremote.\n>\n> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n> ---\n>  builtin/fetch.c       | 23 +++++++++++++++++++++--\n>  connected.c           |  2 ++\n>  remote-curl.c         |  6 ++++++\n>  t/t5500-fetch-pack.sh | 36 ++++++++++++++++++++++++++++++++++++\n>  4 files changed, 65 insertions(+), 2 deletions(-)\n>\n> diff --git a/builtin/fetch.c b/builtin/fetch.c\n> index 1b1f039..14aab71 100644\n> --- a/builtin/fetch.c\n> +++ b/builtin/fetch.c\n> @@ -18,6 +18,7 @@\n>  #include \"argv-array.h\"\n>  #include \"utf8.h\"\n>  #include \"packfile.h\"\n> +#include \"list-objects-filter-options.h\"\n>\n>  static const char * const builtin_fetch_usage[] = {\n>         N_(\"git fetch [<options>] [<repository> [<refspec>...]]\"),\n> @@ -55,6 +56,7 @@ static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;\n>  static int shown_url = 0;\n>  static int refmap_alloc, refmap_nr;\n>  static const char **refmap_array;\n> +static struct list_objects_filter_options filter_options;\n>\n>  static int git_fetch_config(const char *k, const char *v, void *cb)\n>  {\n> @@ -160,6 +162,7 @@ static struct option builtin_fetch_options[] = {\n>                         TRANSPORT_FAMILY_IPV4),\n>         OPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n>                         TRANSPORT_FAMILY_IPV6),\n> +       OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n\nDocumentation is missing. Please add something to git-fetch.txt (or\nfetch-options.txt) about this option. I would make a patch but I don't\nknow enough about this to write and I'm in the middle of something\nelse.\n-- \nDuy\n"},{"id":"356131","messageId":"409954a1-6aff-f187-ec3e-c0bf044d279f@jeffhostetler.com","threadId":"47411","inReplyTo":"CACsJy8D3nC+fXFYMKWh6AStbMVOVfmcAvAT9tBqtG9K6r043eQ@mail.gmail.com","subject":"Re: [PATCH v7 09/16] fetch: support filters","fromName":"Jeff Hostetler","fromEmail":"git@jeffhostetler.com","sentAt":"2018-08-20T19:42:33Z","receivedAt":"2018-08-20T19:42:36Z","isPatch":true,"sender":{"key":"git@jeffhostetler.com","avatar":null},"body":"\n\nOn 8/19/2018 7:24 AM, Duy Nguyen wrote:\n> On Fri, Dec 8, 2017 at 5:00 PM Jeff Hostetler <git@jeffhostetler.com> wrote:\n>>\n>> From: Jeff Hostetler <jeffhost@microsoft.com>\n>>\n>> Teach fetch to support filters. This is only allowed for the remote\n>> configured in extensions.partialcloneremote.\n>>\n>> Signed-off-by: Jonathan Tan <jonathantanmy@google.com>\n>> ---\n>>   builtin/fetch.c       | 23 +++++++++++++++++++++--\n>>   connected.c           |  2 ++\n>>   remote-curl.c         |  6 ++++++\n>>   t/t5500-fetch-pack.sh | 36 ++++++++++++++++++++++++++++++++++++\n>>   4 files changed, 65 insertions(+), 2 deletions(-)\n>>\n>> diff --git a/builtin/fetch.c b/builtin/fetch.c\n>> index 1b1f039..14aab71 100644\n>> --- a/builtin/fetch.c\n>> +++ b/builtin/fetch.c\n>> @@ -18,6 +18,7 @@\n>>   #include \"argv-array.h\"\n>>   #include \"utf8.h\"\n>>   #include \"packfile.h\"\n>> +#include \"list-objects-filter-options.h\"\n>>\n>>   static const char * const builtin_fetch_usage[] = {\n>>          N_(\"git fetch [<options>] [<repository> [<refspec>...]]\"),\n>> @@ -55,6 +56,7 @@ static int recurse_submodules_default = RECURSE_SUBMODULES_ON_DEMAND;\n>>   static int shown_url = 0;\n>>   static int refmap_alloc, refmap_nr;\n>>   static const char **refmap_array;\n>> +static struct list_objects_filter_options filter_options;\n>>\n>>   static int git_fetch_config(const char *k, const char *v, void *cb)\n>>   {\n>> @@ -160,6 +162,7 @@ static struct option builtin_fetch_options[] = {\n>>                          TRANSPORT_FAMILY_IPV4),\n>>          OPT_SET_INT('6', \"ipv6\", &family, N_(\"use IPv6 addresses only\"),\n>>                          TRANSPORT_FAMILY_IPV6),\n>> +       OPT_PARSE_LIST_OBJECTS_FILTER(&filter_options),\n> \n> Documentation is missing. Please add something to git-fetch.txt (or\n> fetch-options.txt) about this option. I would make a patch but I don't\n> know enough about this to write and I'm in the middle of something\n> else.\n> \n\nDocumentation for --filter=<fs> (and --no-filter) were added to\nrev-list-options.txt.  The opening paragraph talks about --objects\nwhich would need to be omitted for fetch and clone, but the rest\nof that text is relevant.\n\nI'll push up a patch shortly.\n\nThanks\nJeff\n"}]}