{"thread":{"id":"45293","subject":"[RFC 0/4] Shallow clones with on-demand fetch","startedAt":"2017-03-04T19:36:05Z","lastAt":"2017-03-07T12:32:03Z","messageCount":9,"participants":["Mark Thomas","Junio C Hamano","Jonathan Tan","Stefan Beller","Jeff King"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"313277","messageId":"20170304191901.9622-1-markbt@efaref.net","threadId":"45293","inReplyTo":null,"subject":"[RFC 0/4] Shallow clones with on-demand fetch","fromName":"Mark Thomas","fromEmail":"markbt@efaref.net","sentAt":"2017-03-04T19:18:57Z","receivedAt":"2017-03-04T19:36:05Z","isPatch":false,"sender":{"key":"markbt@efaref.net","avatar":"https://gravatar.com/avatar/1860e0c182858cf7ffa8a642978e8c1b53a1aa4fbd13ef7bbf97007abb61eb45?d=mp&s=160"},"body":"Hello everyone,\n\nThis is an RFC for an enhancement to shallow repositories to make them\nbehave more like full clones.\n\nI was inspired a bit by Microsoft's announcement of their Git VFS.  I\nsaw that people have talked in the past about making git fetch objects\nfrom remotes as they are needed, and decided to give it a try.\n\nThe patch series adds a \"--on-demand\" option to git clone, which, when\nused in conjunction with the existing shallow clone operations, clones\nthe full history of the repository's commits, but only the files that\nwould be included in the shallow clone.\n\nWhen a file that is missing is required, git requests the file on-demand\nfrom the remote, via a new 'upload-file' service.\n\nPublic git servers are unlikely to want to enable this, due to the\naddition load it may cause, but within an organization's own network, it\nwill allow full access to the repository history without needing a full\ninitial clone.\n\nThe patch set is in four parts:\n  1:\n    Adds the \"upload-file\" command, which starts a new protocol\n    conversation with the client allowing it to request file info and\n    file contents.  The connection is kept open so that the client\n    can make as many requests as it likes.  The client terminates the\n    connection by sending a packet containing \"end\".\n  2:\n    Adds the ability for file info and content to be requested from\n    the remote if the file cannot be found in any pack, or loose in\n    the repository.  Currently this only looks at the default remote,\n    but the intention is this would be configurable.\n  3:\n    Adds the \"on-demand\" capability to \"upload-pack\".  When a client\n    requests this capability, \"upload-pack\" includes in the pack\n    all commits, even those that would normally be dropped by the\n    shallow clone.\n  4:\n    Adds the \"--on-demand\" option to clone, to request a shallow\n    clone.\n\nThis is a proof-of-concept, so it is in no way complete.  It contains a\nfew hacks to make it work, but these can be ironed out with a bit more\nwork.  What I have so far is sufficient to try out the idea.  I'd like\nto get people's opinions on it before I spend any more time working on\nit, plus also I'm not very familiar with the git codebase, so some help\nwould be appreciated.\n\nAs an example, the Linux repository currently stands at 2.0GB of packed\ndata.  A \"git clone --shallow-since=2016-01-01 --on-demand\" is only\n561MB, and yet remains fully functional.  A git blame on the Makefile,\nfor example, shows all changes to the file, right back to Linus's\noriginal commit in 2005.\n\nStill to do:\n\n - Fix up the hacks and make everything work correctly.\n - Make fetching of further updates work correctly.\n - Store the retrieved files in an LRU cache, possibly with the option\n   of storing them in the main repo data, too.\n - Add a gc/enshallow operation to make the repo shallower by forgetting\n   old files, or moving them to the LRU cache.\n - Add configurable remote to fetch from.\n - Documentation.\n - Much more.\n\nPlease let me know what you think, and if an experienced git developer\nwould like to help out with finishing this, that would be even better.\n\nMark Thomas (4):\n  upload-file: Add upload-file command\n  on-demand: Fetch missing files from remote\n  upload-pack: Send all commits if client requests on-demand\n  clone: Request on-demand shallow clones\n\n .gitignore             |   1 +\n Makefile               |   3 +\n builtin/clone.c        |   7 +-\n builtin/pack-objects.c |  26 ++++++-\n cache-tree.c           |   2 +-\n cache.h                |   3 +-\n daemon.c               |   6 ++\n fetch-pack.c           |   3 +\n fetch-pack.h           |   1 +\n list-objects.c         |  12 ++--\n list-objects.h         |  13 +++-\n object.h               |   1 +\n on_demand.c            | 183 +++++++++++++++++++++++++++++++++++++++++++++++++\n on_demand.h            |  12 ++++\n sha1_file.c            |   8 ++-\n shallow.c              |   2 +-\n transport.c            |   3 +\n transport.h            |   4 ++\n upload-file.c          |  87 +++++++++++++++++++++++\n upload-pack.c          |   8 ++-\n 20 files changed, 370 insertions(+), 15 deletions(-)\n create mode 100644 on_demand.c\n create mode 100644 on_demand.h\n create mode 100644 upload-file.c\n\n-- \n2.7.4\n\n"},{"id":"313278","messageId":"20170304191901.9622-4-markbt@efaref.net","threadId":"45293","inReplyTo":"20170304191901.9622-1-markbt@efaref.net","subject":"[RFC 3/4] upload-pack: Send all commits if client requests on-demand","fromName":"Mark Thomas","fromEmail":"markbt@efaref.net","sentAt":"2017-03-04T19:19:00Z","receivedAt":"2017-03-04T19:36:07Z","isPatch":false,"sender":{"key":"markbt@efaref.net","avatar":"https://gravatar.com/avatar/1860e0c182858cf7ffa8a642978e8c1b53a1aa4fbd13ef7bbf97007abb61eb45?d=mp&s=160"},"body":"Signed-off-by: Mark Thomas <markbt@efaref.net>\n---\n builtin/pack-objects.c | 26 ++++++++++++++++++++++++--\n list-objects.c         | 12 +++++++-----\n list-objects.h         | 13 ++++++++++++-\n object.h               |  1 +\n on_demand.c            | 26 ++++++++++++++++++++++++++\n on_demand.h            |  4 ++++\n upload-pack.c          |  8 +++++++-\n 7 files changed, 81 insertions(+), 9 deletions(-)\n\ndiff --git a/builtin/pack-objects.c b/builtin/pack-objects.c\nindex f294dcf..c8b2503 100644\n--- a/builtin/pack-objects.c\n+++ b/builtin/pack-objects.c\n@@ -24,6 +24,7 @@\n #include \"sha1-array.h\"\n #include \"argv-array.h\"\n #include \"mru.h\"\n+#include \"on_demand.h\"\n \n static const char *pack_usage[] = {\n \tN_(\"git pack-objects --stdout [<options>...] [< <ref-list> | < <object-list>]\"),\n@@ -77,6 +78,8 @@ static unsigned long cache_max_small_delta_size = 1000;\n \n static unsigned long window_memory_limit = 0;\n \n+static int send_all_commits;\n+\n /*\n  * stats\n  */\n@@ -2750,12 +2753,15 @@ static void record_recent_commit(struct commit *commit, void *data)\n static void get_object_list(int ac, const char **av)\n {\n \tstruct rev_info revs;\n+\tstruct rev_info revs2;\n \tchar line[1000];\n \tint flags = 0;\n \n \tinit_revisions(&revs, NULL);\n+\tinit_revisions(&revs2, NULL);\n \tsave_commit_buffer = 0;\n \tsetup_revisions(ac, av, &revs, NULL);\n+\tsetup_revisions(ac, av, &revs2, NULL);\n \n \t/* make sure shallows are read */\n \tis_repository_shallow();\n@@ -2776,7 +2782,10 @@ static void get_object_list(int ac, const char **av)\n \t\t\t\tunsigned char sha1[20];\n \t\t\t\tif (get_sha1_hex(line + 10, sha1))\n \t\t\t\t\tdie(\"not an SHA-1 '%s'\", line + 10);\n-\t\t\t\tregister_shallow(sha1);\n+\t\t\t\tif (send_all_commits)\n+\t\t\t\t\tregister_on_demand_cutoff(sha1);\n+\t\t\t\telse\n+\t\t\t\t\tregister_shallow(sha1);\n \t\t\t\tuse_bitmap_index = 0;\n \t\t\t\tcontinue;\n \t\t\t}\n@@ -2784,6 +2793,8 @@ static void get_object_list(int ac, const char **av)\n \t\t}\n \t\tif (handle_revision_arg(line, &revs, flags, REVARG_CANNOT_BE_FILENAME))\n \t\t\tdie(\"bad revision '%s'\", line);\n+\t\tif (handle_revision_arg(line, &revs2, flags, REVARG_CANNOT_BE_FILENAME))\n+\t\t\tdie(\"bad revision '%s'\", line);\n \t}\n \n \tif (use_bitmap_index && !get_object_list_from_bitmap(&revs))\n@@ -2792,7 +2803,16 @@ static void get_object_list(int ac, const char **av)\n \tif (prepare_revision_walk(&revs))\n \t\tdie(\"revision walk setup failed\");\n \tmark_edges_uninteresting(&revs, show_edge);\n-\ttraverse_commit_list(&revs, show_commit, show_object, NULL);\n+\n+\tif (send_all_commits) {\n+\t\trevs2.include_check = on_demand_include_check;\n+\t\ttraverse_commit_list(&revs2, on_demand_show_commit_tree, NULL,\n+\t\t\t\t     NULL);\n+\t\treset_revision_walk();\n+\t}\n+\n+\ttraverse_commit_list_extended(&revs, show_commit, show_object,\n+\t\t\t\t      on_demand_show_tree_check, NULL);\n \n \tif (unpack_unreachable_expiration) {\n \t\trevs.ignore_missing_links = 1;\n@@ -2928,6 +2948,8 @@ int cmd_pack_objects(int argc, const char **argv, const char *prefix)\n \t\t\t N_(\"use a bitmap index if available to speed up counting objects\")),\n \t\tOPT_BOOL(0, \"write-bitmap-index\", &write_bitmap_index,\n \t\t\t N_(\"write a bitmap index together with the pack index\")),\n+\t\tOPT_BOOL(0, \"send-all-commits\", &send_all_commits,\n+\t\t\t N_(\"send all commits for on-demand shallow fetches\")),\n \t\tOPT_END(),\n \t};\n \ndiff --git a/list-objects.c b/list-objects.c\nindex f3ca6aa..2607549 100644\n--- a/list-objects.c\n+++ b/list-objects.c\n@@ -183,10 +183,11 @@ static void add_pending_tree(struct rev_info *revs, struct tree *tree)\n \tadd_pending_object(revs, &tree->object, \"\");\n }\n \n-void traverse_commit_list(struct rev_info *revs,\n-\t\t\t  show_commit_fn show_commit,\n-\t\t\t  show_object_fn show_object,\n-\t\t\t  void *data)\n+void traverse_commit_list_extended(struct rev_info *revs,\n+\t\t\t\t   show_commit_fn show_commit,\n+\t\t\t\t   show_object_fn show_object,\n+\t\t\t\t   show_tree_check_fn show_tree_check,\n+\t\t\t\t   void *data)\n {\n \tint i;\n \tstruct commit *commit;\n@@ -198,7 +199,8 @@ void traverse_commit_list(struct rev_info *revs,\n \t\t * an uninteresting boundary commit may not have its tree\n \t\t * parsed yet, but we are not going to show them anyway\n \t\t */\n-\t\tif (commit->tree)\n+\t\tif (show_object && commit->tree &&\n+\t\t    (!show_tree_check || show_tree_check(commit, data)))\n \t\t\tadd_pending_tree(revs, commit->tree);\n \t\tshow_commit(commit, data);\n \t}\ndiff --git a/list-objects.h b/list-objects.h\nindex 0cebf85..e80dc8c 100644\n--- a/list-objects.h\n+++ b/list-objects.h\n@@ -3,7 +3,18 @@\n \n typedef void (*show_commit_fn)(struct commit *, void *);\n typedef void (*show_object_fn)(struct object *, const char *, void *);\n-void traverse_commit_list(struct rev_info *, show_commit_fn, show_object_fn, void *);\n+typedef int (*show_tree_check_fn)(struct commit *, void *);\n+void traverse_commit_list_extended(struct rev_info *, show_commit_fn,\n+\t\t\t\t   show_object_fn, show_tree_check_fn, void *);\n+\n+inline void traverse_commit_list(struct rev_info *revs,\n+\t\t\t\t show_commit_fn show_commit,\n+\t\t\t\t show_object_fn show_object,\n+\t\t\t\t void *data)\n+{\n+\ttraverse_commit_list_extended(revs, show_commit, show_object, NULL,\n+\t\t\t\t      data);\n+}\n \n typedef void (*show_edge_fn)(struct commit *);\n void mark_edges_uninteresting(struct rev_info *, show_edge_fn);\ndiff --git a/object.h b/object.h\nindex f52957d..25177fd 100644\n--- a/object.h\n+++ b/object.h\n@@ -38,6 +38,7 @@ struct object_array {\n  * http-push.c:                            16-----19\n  * commit.c:                               16-----19\n  * sha1_name.c:                                     20\n+ * on_demand.c:                                       21-22\n  */\n #define FLAG_BITS  27\n \ndiff --git a/on_demand.c b/on_demand.c\nindex a0aaf18..c72d7a5 100644\n--- a/on_demand.c\n+++ b/on_demand.c\n@@ -155,3 +155,29 @@ int object_info_on_demand(const unsigned char *sha1, struct object_info *oi)\n \tdie(\"git on-demand: protocol error, \"\n \t    \"unexpected response: '%s'\", line);\n }\n+\n+#define ON_DEMAND_CUTOFF\t(1u << 21)\n+#define ON_DEMAND_SHOW_TREE\t(1u << 22)\n+\n+void register_on_demand_cutoff(const unsigned char *sha1)\n+{\n+\tstruct commit *commit = lookup_commit(sha1);\n+\tif (commit)\n+\t\tcommit->object.flags |= ON_DEMAND_CUTOFF;\n+}\n+\n+int on_demand_include_check(struct commit *commit, void *data)\n+{\n+\treturn !(commit->object.flags & ON_DEMAND_CUTOFF);\n+}\n+\n+void on_demand_show_commit_tree(struct commit *commit, void *data)\n+{\n+\tcommit->object.flags |= ON_DEMAND_SHOW_TREE;\n+}\n+\n+int on_demand_show_tree_check(struct commit *commit, void *data)\n+{\n+\treturn !!(commit->object.flags &\n+\t\t  (ON_DEMAND_SHOW_TREE|ON_DEMAND_CUTOFF));\n+}\ndiff --git a/on_demand.h b/on_demand.h\nindex 09a8072..7bbb523 100644\n--- a/on_demand.h\n+++ b/on_demand.h\n@@ -4,5 +4,9 @@\n void *read_remote_on_demand(const unsigned char *sha1, enum object_type *type,\n \t\t\t    unsigned long *size);\n int object_info_on_demand(const unsigned char *sha1, struct object_info *oi);\n+void register_on_demand_cutoff(const unsigned char *sha1);\n+int on_demand_include_check(struct commit *commit, void *data);\n+void on_demand_show_commit_tree(struct commit *commit, void *data);\n+int on_demand_show_tree_check(struct commit *commit, void *data);\n \n #endif\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 7597ba3..1b552b4 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -42,6 +42,7 @@ static int multi_ack;\n static int no_done;\n static int use_thin_pack, use_ofs_delta, use_include_tag;\n static int no_progress, daemon_mode;\n+static int send_all_commits;\n /* Allow specifying sha1 if it is a ref tip. */\n #define ALLOW_TIP_SHA1\t01\n /* Allow request of a sha1 if it is reachable from a ref (possibly hidden ref). */\n@@ -130,6 +131,8 @@ static void create_pack_file(void)\n \t\targv_array_push(&pack_objects.args, \"--delta-base-offset\");\n \tif (use_include_tag)\n \t\targv_array_push(&pack_objects.args, \"--include-tag\");\n+\tif (send_all_commits)\n+\t\targv_array_push(&pack_objects.args, \"--send-all-commits\");\n \n \tpack_objects.in = -1;\n \tpack_objects.out = -1;\n@@ -820,6 +823,8 @@ static void receive_needs(void)\n \t\t\tno_progress = 1;\n \t\tif (parse_feature_request(features, \"include-tag\"))\n \t\t\tuse_include_tag = 1;\n+\t\tif (parse_feature_request(features, \"on-demand\"))\n+\t\t\tsend_all_commits = 1;\n \n \t\to = parse_object(sha1_buf);\n \t\tif (!o)\n@@ -924,7 +929,8 @@ static int send_ref(const char *refname, const struct object_id *oid,\n {\n \tstatic const char *capabilities = \"multi_ack thin-pack side-band\"\n \t\t\" side-band-64k ofs-delta shallow deepen-since deepen-not\"\n-\t\t\" deepen-relative no-progress include-tag multi_ack_detailed\";\n+\t\t\" deepen-relative no-progress include-tag multi_ack_detailed\"\n+\t\t\" on-demand\";\n \tconst char *refname_nons = strip_namespace(refname);\n \tstruct object_id peeled;\n \n-- \n2.7.4\n\n"},{"id":"313279","messageId":"20170304191901.9622-5-markbt@efaref.net","threadId":"45293","inReplyTo":"20170304191901.9622-1-markbt@efaref.net","subject":"[RFC 4/4] clone: Request on-demand shallow clones","fromName":"Mark Thomas","fromEmail":"markbt@efaref.net","sentAt":"2017-03-04T19:19:01Z","receivedAt":"2017-03-04T19:36:10Z","isPatch":false,"sender":{"key":"markbt@efaref.net","avatar":"https://gravatar.com/avatar/1860e0c182858cf7ffa8a642978e8c1b53a1aa4fbd13ef7bbf97007abb61eb45?d=mp&s=160"},"body":"Add the --on-demand option to git-clone, which, when used in combination\nwith the existing shallow clone options, requests an on-demand shallow\nclone.\n\nAn on-demand shallow clone contains all commits from all history, but\nthe commits that would normally be omitted in the shallow clone do not\nhave their trees or blobs in the repository.  Instead, they will be\nfetched on-demand from the remote.\n\nSigned-off-by: Mark Thomas <markbt@efaref.net>\n---\n builtin/clone.c | 7 ++++++-\n cache-tree.c    | 2 +-\n fetch-pack.c    | 3 +++\n fetch-pack.h    | 1 +\n shallow.c       | 2 +-\n transport.c     | 3 +++\n transport.h     | 4 ++++\n 7 files changed, 19 insertions(+), 3 deletions(-)\n\ndiff --git a/builtin/clone.c b/builtin/clone.c\nindex 3f63edb..7541016 100644\n--- a/builtin/clone.c\n+++ b/builtin/clone.c\n@@ -40,6 +40,7 @@ static const char * const builtin_clone_usage[] = {\n \n static int option_no_checkout, option_bare, option_mirror, option_single_branch = -1;\n static int option_local = -1, option_no_hardlinks, option_shared, option_recursive;\n+static int option_on_demand;\n static int option_shallow_submodules;\n static int deepen;\n static char *option_template, *option_depth, *option_since;\n@@ -100,6 +101,8 @@ static struct option builtin_clone_options[] = {\n \t\t    N_(\"create a shallow clone since a specific time\")),\n \tOPT_STRING_LIST(0, \"shallow-exclude\", &option_not, N_(\"revision\"),\n \t\t\tN_(\"deepen history of shallow clone, excluding rev\")),\n+\tOPT_BOOL(0, \"on-demand\", &option_on_demand,\n+\t\t N_(\"Make shallow clone an on-demand clone\")),\n \tOPT_BOOL(0, \"single-branch\", &option_single_branch,\n \t\t    N_(\"clone only one branch, HEAD or --branch\")),\n \tOPT_BOOL(0, \"shallow-submodules\", &option_shallow_submodules,\n@@ -1045,6 +1048,8 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \tif (option_not.nr)\n \t\ttransport_set_option(transport, TRANS_OPT_DEEPEN_NOT,\n \t\t\t\t     (const char *)&option_not);\n+\tif (option_on_demand)\n+\t\ttransport_set_option(transport, TRANS_OPT_ON_DEMAND, \"1\");\n \tif (option_single_branch)\n \t\ttransport_set_option(transport, TRANS_OPT_FOLLOWTAGS, \"1\");\n \n@@ -1118,7 +1123,7 @@ int cmd_clone(int argc, const char **argv, const char *prefix)\n \t\ttransport_fetch_refs(transport, mapped_refs);\n \n \tupdate_remote_refs(refs, mapped_refs, remote_head_points_at,\n-\t\t\t   branch_top.buf, reflog_msg.buf, transport, !is_local);\n+\t\t\t   branch_top.buf, reflog_msg.buf, transport, !is_local && !option_on_demand);\n \n \tupdate_head(our_head_points_at, remote_head, reflog_msg.buf);\n \ndiff --git a/cache-tree.c b/cache-tree.c\nindex 345ea35..10b14fe 100644\n--- a/cache-tree.c\n+++ b/cache-tree.c\n@@ -356,7 +356,7 @@ static int update_one(struct cache_tree *it,\n \t\t}\n \t\tif (mode != S_IFGITLINK && !missing_ok && !has_sha1_file(sha1)) {\n \t\t\tstrbuf_release(&buffer);\n-\t\t\tif (expected_missing)\n+\t\t\tif (expected_missing || 1 /*** FIXME: markbt temp hack, to allow missing files ***/)\n \t\t\t\treturn -1;\n \t\t\treturn error(\"invalid object %06o %s for '%.*s'\",\n \t\t\t\tmode, sha1_to_hex(sha1), entlen+baselen, path);\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex e0f5d5c..1dd4823 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -372,6 +372,7 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tif (prefer_ofs_delta)   strbuf_addstr(&c, \" ofs-delta\");\n \t\t\tif (deepen_since_ok)    strbuf_addstr(&c, \" deepen-since\");\n \t\t\tif (deepen_not_ok)      strbuf_addstr(&c, \" deepen-not\");\n+\t\t\tif (args->on_demand)    strbuf_addf(&c, \" on-demand\");\n \t\t\tif (agent_supported)    strbuf_addf(&c, \" agent=%s\",\n \t\t\t\t\t\t\t    git_user_agent_sanitized());\n \t\t\tpacket_buf_write(&req_buf, \"want %s%s\\n\", remote_hex, c.buf);\n@@ -936,6 +937,8 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\tdie(_(\"Server does not support --shallow-exclude\"));\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n+\tif (!server_supports(\"on-demand\") && args->on_demand)\n+\t\tdie(_(\"Server does not support --on-demand\"));\n \n \tif (everything_local(args, &ref, sought, nr_sought)) {\n \t\tpacket_flush(fd[1]);\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex c912e3d..16ab8bd 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -29,6 +29,7 @@ struct fetch_pack_args {\n \tunsigned cloning:1;\n \tunsigned update_shallow:1;\n \tunsigned deepen:1;\n+\tunsigned on_demand:1;\n };\n \n /*\ndiff --git a/shallow.c b/shallow.c\nindex 11f7dde..a24292b 100644\n--- a/shallow.c\n+++ b/shallow.c\n@@ -45,7 +45,7 @@ int is_repository_shallow(void)\n \tFILE *fp;\n \tchar buf[1024];\n \tconst char *path = alternate_shallow_file;\n-\n+\tis_shallow = 0;  /*** FIXME: markbt temp hack to allow shallow repos with on-demand files ***/\n \tif (is_shallow >= 0)\n \t\treturn is_shallow;\n \ndiff --git a/transport.c b/transport.c\nindex 5828e06..69f8d72 100644\n--- a/transport.c\n+++ b/transport.c\n@@ -160,6 +160,8 @@ static int set_git_option(struct git_transport_options *opts,\n \t} else if (!strcmp(name, TRANS_OPT_DEEPEN_RELATIVE)) {\n \t\topts->deepen_relative = !!value;\n \t\treturn 0;\n+\t} else if (!strcmp(name, TRANS_OPT_ON_DEMAND)) {\n+\t\topts->on_demand = !!value;\n \t}\n \treturn 1;\n }\n@@ -223,6 +225,7 @@ static int fetch_refs_via_pack(struct transport *transport,\n \targs.deepen_since = data->options.deepen_since;\n \targs.deepen_not = data->options.deepen_not;\n \targs.deepen_relative = data->options.deepen_relative;\n+\targs.on_demand = data->options.on_demand;\n \targs.check_self_contained_and_connected =\n \t\tdata->options.check_self_contained_and_connected;\n \targs.cloning = transport->cloning;\ndiff --git a/transport.h b/transport.h\nindex bc55715..d4f848b 100644\n--- a/transport.h\n+++ b/transport.h\n@@ -15,6 +15,7 @@ struct git_transport_options {\n \tunsigned self_contained_and_connected : 1;\n \tunsigned update_shallow : 1;\n \tunsigned deepen_relative : 1;\n+\tunsigned on_demand : 1;\n \tint depth;\n \tconst char *deepen_since;\n \tconst struct string_list *deepen_not;\n@@ -210,6 +211,9 @@ void transport_check_allowed(const char *type);\n /* Send push certificates */\n #define TRANS_OPT_PUSH_CERT \"pushcert\"\n \n+/* On-demand clone */\n+#define TRANS_OPT_ON_DEMAND \"on-demand\"\n+\n /**\n  * Returns 0 if the option was used, non-zero otherwise. Prints a\n  * message to stderr if the option is not used.\n-- \n2.7.4\n\n"},{"id":"313280","messageId":"20170304191901.9622-2-markbt@efaref.net","threadId":"45293","inReplyTo":"20170304191901.9622-1-markbt@efaref.net","subject":"[RFC 1/4] upload-file: Add upload-file command","fromName":"Mark Thomas","fromEmail":"markbt@efaref.net","sentAt":"2017-03-04T19:18:58Z","receivedAt":"2017-03-04T19:36:12Z","isPatch":false,"sender":{"key":"markbt@efaref.net","avatar":"https://gravatar.com/avatar/1860e0c182858cf7ffa8a642978e8c1b53a1aa4fbd13ef7bbf97007abb61eb45?d=mp&s=160"},"body":"The upload-file command allows a remote to request specific files by sha1.\n\nSigned-off-by: Mark Thomas <markbt@efaref.net>\n---\n .gitignore    |  1 +\n Makefile      |  2 ++\n daemon.c      |  6 +++++\n upload-file.c | 87 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 96 insertions(+)\n create mode 100644 upload-file.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 833ef3b..c2db9c2 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -165,6 +165,7 @@\n /git-update-ref\n /git-update-server-info\n /git-upload-archive\n+/git-upload-file\n /git-upload-pack\n /git-var\n /git-verify-commit\ndiff --git a/Makefile b/Makefile\nindex 9ec6065..1b84322 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -597,6 +597,7 @@ PROGRAM_OBJS += sh-i18n--envsubst.o\n PROGRAM_OBJS += shell.o\n PROGRAM_OBJS += show-index.o\n PROGRAM_OBJS += upload-pack.o\n+PROGRAM_OBJS += upload-file.o\n PROGRAM_OBJS += remote-testsvn.o\n \n # Binary suffix, set to .exe for Windows builds\n@@ -668,6 +669,7 @@ BINDIR_PROGRAMS_NEED_X += git-upload-pack\n BINDIR_PROGRAMS_NEED_X += git-receive-pack\n BINDIR_PROGRAMS_NEED_X += git-upload-archive\n BINDIR_PROGRAMS_NEED_X += git-shell\n+BINDIR_PROGRAMS_NEED_X += git-upload-file\n \n BINDIR_PROGRAMS_NO_X += git-cvsserver\n \ndiff --git a/daemon.c b/daemon.c\nindex 473e6b6..8b5b026 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -479,6 +479,11 @@ static int upload_pack(void)\n \treturn run_service_command(argv);\n }\n \n+static int upload_file(void)\n+{\n+\tconst char *argv[] = { \"upload-file\", \".\", NULL };\n+\treturn run_service_command(argv);\n+}\n static int upload_archive(void)\n {\n \tstatic const char *argv[] = { \"upload-archive\", \".\", NULL };\n@@ -494,6 +499,7 @@ static int receive_pack(void)\n static struct daemon_service daemon_service[] = {\n \t{ \"upload-archive\", \"uploadarch\", upload_archive, 0, 1 },\n \t{ \"upload-pack\", \"uploadpack\", upload_pack, 1, 1 },\n+\t{ \"upload-file\", \"uploadfile\", upload_file, 1, 1 },\n \t{ \"receive-pack\", \"receivepack\", receive_pack, 0, 1 },\n };\n \ndiff --git a/upload-file.c b/upload-file.c\nnew file mode 100644\nindex 0000000..cb2bfe8\n--- /dev/null\n+++ b/upload-file.c\n@@ -0,0 +1,87 @@\n+\n+#include \"cache.h\"\n+#include \"exec_cmd.h\"\n+#include \"parse-options.h\"\n+#include \"pkt-line.h\"\n+\n+static const char * const upload_file_usage[] = {\n+\tN_(\"git upload-file [<options>] <dir>\"),\n+\tNULL\n+};\n+\n+\n+static void upload_file(void)\n+{\n+\tfor (;;) {\n+\t\tchar *line = packet_read_line(0, NULL);\n+\t\tconst char *arg;\n+\t\tif (!line)\n+\t\t\tbreak;\n+\n+\t\tif (skip_prefix(line, \"info \", &arg)) {\n+\t\t\tunsigned char sha1[20];\n+\t\t\tvoid *buffer;\n+\t\t\tenum object_type type;\n+\t\t\tunsigned long size;\n+\n+\t\t\tif (get_sha1_hex(arg, sha1))\n+\t\t\t\tdie(\"invalid sha: %s\", arg);\n+\n+\t\t\tbuffer = read_sha1_file(sha1, &type, &size);\n+\t\t\tif (buffer) {\n+\t\t\t\tpacket_write_fmt(1, \"found %s %d %ld\\n\", sha1_to_hex(sha1), type, size);\n+\t\t\t\tfree(buffer);\n+\t\t\t} else {\n+\t\t\t\tpacket_write_fmt(1, \"missing %s\\n\", sha1_to_hex(sha1));\n+\t\t\t}\n+\t\t}\n+\n+\t\tif (skip_prefix(line, \"get \", &arg)) {\n+\t\t\tunsigned char sha1[20];\n+\t\t\tvoid *buffer;\n+\t\t\tenum object_type type;\n+\t\t\tunsigned long size;\n+\n+\t\t\tif (get_sha1_hex(arg, sha1))\n+\t\t\t\tdie(\"invalid sha: %s\", arg);\n+\n+\t\t\tbuffer = read_sha1_file(sha1, &type, &size);\n+\t\t\tif (buffer) {\n+\t\t\t\tpacket_write_fmt(1, \"found %s %d %ld\\n\", sha1_to_hex(sha1), type, size);\n+\t\t\t\twrite_or_die(1, buffer, size);\n+\t\t\t\tfree(buffer);\n+\t\t\t} else {\n+\t\t\t\tpacket_write_fmt(1, \"missing %s\\n\", sha1_to_hex(sha1));\n+\t\t\t}\n+\t\t\t\n+\t\t}\n+\n+\t\tif (!strcmp(line, \"end\"))\n+\t\t\tbreak;\n+\t}\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tconst char *dir;\n+\tstruct option options[] = {\n+\t\tOPT_END()\n+\t};\n+\n+\tpacket_trace_identity(\"upload-file\");\n+\n+\targc = parse_options(argc, argv, NULL, options, upload_file_usage, 0);\n+\n+\tif (argc != 1)\n+\t\tusage_with_options(upload_file_usage, options);\n+\n+\tsetup_path();\n+\n+\tdir = argv[0];\n+\n+\tif (!enter_repo(dir, 0))\n+\t\tdie(\"'%s' does not appear to be a git repository\", dir);\n+\n+\tupload_file();\n+\treturn 0;\n+}\n-- \n2.7.4\n\n"},{"id":"313281","messageId":"20170304191901.9622-3-markbt@efaref.net","threadId":"45293","inReplyTo":"20170304191901.9622-1-markbt@efaref.net","subject":"[RFC 2/4] on-demand: Fetch missing files from remote","fromName":"Mark Thomas","fromEmail":"markbt@efaref.net","sentAt":"2017-03-04T19:18:59Z","receivedAt":"2017-03-04T19:36:20Z","isPatch":false,"sender":{"key":"markbt@efaref.net","avatar":"https://gravatar.com/avatar/1860e0c182858cf7ffa8a642978e8c1b53a1aa4fbd13ef7bbf97007abb61eb45?d=mp&s=160"},"body":"If an object (tree, blob, ...) is not found either in the\npacks or loose, check if it is available on-demand from\nthe remote.\n\nSigned-off-by: Mark Thomas <markbt@efaref.net>\n---\n Makefile    |   1 +\n cache.h     |   3 +-\n on_demand.c | 157 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++\n on_demand.h |   8 ++++\n sha1_file.c |   8 +++-\n 5 files changed, 174 insertions(+), 3 deletions(-)\n create mode 100644 on_demand.c\n create mode 100644 on_demand.h\n\ndiff --git a/Makefile b/Makefile\nindex 1b84322..fb8ca6c 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -784,6 +784,7 @@ LIB_OBJS += notes-merge.o\n LIB_OBJS += notes-utils.o\n LIB_OBJS += object.o\n LIB_OBJS += oidset.o\n+LIB_OBJS += on_demand.o\n LIB_OBJS += pack-bitmap.o\n LIB_OBJS += pack-bitmap-write.o\n LIB_OBJS += pack-check.o\ndiff --git a/cache.h b/cache.h\nindex 80b6372..d34af06 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -1730,7 +1730,8 @@ struct object_info {\n \t\tOI_CACHED,\n \t\tOI_LOOSE,\n \t\tOI_PACKED,\n-\t\tOI_DBCACHED\n+\t\tOI_DBCACHED,\n+\t\tOI_ONDEMAND\n \t} whence;\n \tunion {\n \t\t/*\ndiff --git a/on_demand.c b/on_demand.c\nnew file mode 100644\nindex 0000000..a0aaf18\n--- /dev/null\n+++ b/on_demand.c\n@@ -0,0 +1,157 @@\n+#include \"transport.h\"\n+#include \"pkt-line.h\"\n+#include \"remote.h\"\n+#include \"commit.h\"\n+\n+static struct remote *remote = NULL;\n+static struct transport *transport = NULL;\n+static int fd[2];\n+static int connected;\n+\n+struct trace_key trace_on_demand = TRACE_KEY_INIT(ON_DEMAND);\n+\n+static void on_demand_cleanup(void)\n+{\n+\tif (connected) {\n+\t\tpacket_write_fmt(fd[1], \"end\\n\");\n+\t\ttransport_disconnect(transport);\n+\t\tconnected = 0;\n+\t}\n+}\n+\n+static int on_demand_connect(void)\n+{\n+\tif (!connected) {\n+\t\tif (!remote)\n+\t\t\tremote = remote_get(NULL);\n+\t\tif (remote && !transport)\n+\t\t\ttransport = transport_get(remote, NULL);\n+\t\tif (!remote || !transport)\n+\t\t\treturn 0;\n+\t\tif (transport_connect(transport, transport->url, \"git-upload-file\", fd))\n+\t\t\treturn 0;\n+\t\tconnected = 1;\n+\t\tatexit(on_demand_cleanup);\n+\t}\n+\treturn 1;\n+}\n+\n+void *read_remote_on_demand(const unsigned char *sha1, enum object_type *type,\n+\t\t\t    unsigned long *size)\n+{\n+\tconst char *line;\n+\tconst char *arg;\n+\tint line_size;\n+\n+\tif (!on_demand_connect())\n+\t\treturn NULL;\n+\n+\tpacket_write_fmt(fd[1], \"get %s\\n\", sha1_to_hex(sha1));\n+\n+\tline = packet_read_line(fd[0], &line_size);\n+\n+\tif (line_size == 0)\n+\t\treturn NULL;\n+\n+\tif (skip_prefix(line, \"missing \", &arg))\n+\t\treturn NULL;\n+\n+\tif (skip_prefix(line, \"found \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\tvoid *buffer;\n+\t\tunsigned char file_sha1[GIT_SHA1_RAWSZ];\n+\t\tenum object_type file_type;\n+\t\tunsigned long file_size;\n+\t\tssize_t size_read;\n+\n+\t\tif (get_sha1_hex(arg, file_sha1))\n+\t\t\tdie(\"git on-demand: protocol error, \"\n+\t\t\t    \"expected to get sha in '%s'\", line);\n+\t\targ += GIT_SHA1_HEXSZ;\n+\n+\t\tfile_type = strtol(arg, &end, 0);\n+\t\tif (!end || file_type < 0 || file_type >= OBJ_MAX)\n+\t\t\tdie(\"git on-demand: protocol error, \"\n+\t\t\t    \"invalid object type in '%s'\", line);\n+\t\targ = end;\n+\n+\t\tfile_size = strtoul(arg, &end, 0);\n+\t\tif (!end || *end || file_size > LONG_MAX)\n+\t\t\tdie(\"git on-demand: protocol error, \"\n+\t\t\t    \"invalid file size in '%s'\", line);\n+\n+\t\tbuffer = xmalloc(file_size);\n+\t\tif (!buffer)\n+\t\t\tdie(\"git on-demand: failed to allocate \"\n+\t\t\t    \"buffer for %ld bytes\", file_size);\n+\n+\t\tsize_read = read_in_full(fd[0], buffer, file_size);\n+\t\tif (size_read != (ssize_t)file_size)\n+\t\t\tdie(\"git on-demand: protocol error, \"\n+\t\t\t    \"failed to read file data\");\n+\n+\t\ttrace_printf_key(&trace_on_demand, \"on-demand: fetched %s\\n\",\n+\t\t\t\t sha1_to_hex(sha1));\n+\t\t*type = file_type;\n+\t\t*size = file_size;\n+\t\treturn buffer;\n+\t}\n+\n+\tdie(\"git on-demand: protocol error, \"\n+\t    \"unexpected response: '%s'\", line);\n+}\n+\n+int object_info_on_demand(const unsigned char *sha1, struct object_info *oi)\n+{\n+\tconst char *line;\n+\tconst char *arg;\n+\tint line_size;\n+\n+\tif (!on_demand_connect())\n+\t\treturn -1;\n+\n+\tpacket_write_fmt(fd[1], \"info %s\\n\", sha1_to_hex(sha1));\n+\n+\tline = packet_read_line(fd[0], &line_size);\n+\n+\tif (line_size == 0)\n+\t\treturn -1;\n+\n+\tif (skip_prefix(line, \"missing \", &arg))\n+\t\treturn -1;\n+\n+\tif (skip_prefix(line, \"found \", &arg)) {\n+\t\tchar *end = NULL;\n+\t\tunsigned char sha1[GIT_SHA1_RAWSZ];\n+\t\tenum object_type file_type;\n+\t\tunsigned long file_size;\n+\n+\t\tif (get_sha1_hex(arg, sha1))\n+\t\t\tdie(\"git on-demand: protocol error, \"\n+\t\t\t    \"expected to get sha in '%s'\", line);\n+\t\targ += GIT_SHA1_HEXSZ;\n+\n+\t\tfile_type = strtol(arg, &end, 0);\n+\t\tif (!end || file_type < 0 || file_type >= OBJ_MAX)\n+\t\t\tdie(\"git on-demand: protocol error, \"\n+\t\t\t    \"invalid object type in '%s'\", line);\n+\t\targ = end;\n+\n+\t\tfile_size = strtoul(arg, &end, 0);\n+\t\tif (!end || *end || file_size > LONG_MAX)\n+\t\t\tdie(\"git on-demand: protocol error, \"\n+\t\t\t    \"invalid file size in '%s'\", line);\n+\n+\t\tif (oi->typep)\n+\t\t\t*oi->typep = file_type;\n+\t\tif (oi->sizep)\n+\t\t\t*oi->typep = file_size;\n+\t\tif (oi->disk_sizep)\n+\t\t\t*oi->disk_sizep = 0;\n+\t\toi->whence = OI_ONDEMAND;\n+\t\treturn 0;\n+\t}\n+\n+\tdie(\"git on-demand: protocol error, \"\n+\t    \"unexpected response: '%s'\", line);\n+}\ndiff --git a/on_demand.h b/on_demand.h\nnew file mode 100644\nindex 0000000..09a8072\n--- /dev/null\n+++ b/on_demand.h\n@@ -0,0 +1,8 @@\n+#ifndef ON_DEMAND_H\n+#define ON_DEMAND_H\n+\n+void *read_remote_on_demand(const unsigned char *sha1, enum object_type *type,\n+\t\t\t    unsigned long *size);\n+int object_info_on_demand(const unsigned char *sha1, struct object_info *oi);\n+\n+#endif\ndiff --git a/sha1_file.c b/sha1_file.c\nindex 6628f06..510da41 100644\n--- a/sha1_file.c\n+++ b/sha1_file.c\n@@ -27,6 +27,7 @@\n #include \"list.h\"\n #include \"mergesort.h\"\n #include \"quote.h\"\n+#include \"on_demand.h\"\n \n #define SZ_FMT PRIuMAX\n static inline uintmax_t sz_fmt(size_t s) { return s; }\n@@ -2979,7 +2980,7 @@ int sha1_object_info_extended(const unsigned char *sha1, struct object_info *oi,\n \t\t/* Not a loose object; someone else may have just packed it. */\n \t\treprepare_packed_git();\n \t\tif (!find_pack_entry(real, &e))\n-\t\t\treturn -1;\n+\t\t\treturn object_info_on_demand(real, oi);\n \t}\n \n \t/*\n@@ -3091,7 +3092,10 @@ static void *read_object(const unsigned char *sha1, enum object_type *type,\n \t\treturn buf;\n \t}\n \treprepare_packed_git();\n-\treturn read_packed_sha1(sha1, type, size);\n+\tbuf = read_packed_sha1(sha1, type, size);\n+\tif (buf)\n+\t\treturn buf;\n+\treturn read_remote_on_demand(sha1, type, size);\n }\n \n /*\n-- \n2.7.4\n\n"},{"id":"313351","messageId":"xmqqr32anri1.fsf@junio-linux.mtv.corp.google.com","threadId":"45293","inReplyTo":"20170304191901.9622-1-markbt@efaref.net","subject":"Re: [RFC 0/4] Shallow clones with on-demand fetch","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-03-06T19:18:30Z","receivedAt":"2017-03-06T19:25:38Z","isPatch":false,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Mark Thomas <markbt@efaref.net> writes:\n\n> This is a proof-of-concept, so it is in no way complete.  It contains a\n> few hacks to make it work, but these can be ironed out with a bit more\n> work.  What I have so far is sufficient to try out the idea.\n\nTwo things that immediately come to mind (which may or may not be\nreal issues) are \n\n (1) What (if any) security model you have in mind.\n\n     From object-confidentiality's point of view, this needs to be\n     enabled only on a host that allows\n     uploadpack.allowAnySHA1InWant but even riskier.\n\n     From DoS point of view, you can make a short 40-byte request to\n     cause the other side emit megabytes of stuff.  I do not think\n     it is a new problem (anybody can repeatedly request a clone of\n     large stuff), but there may be new ramifications.\n\n (2) If the interface to ask just one object kills the whole idea\n     due to roundtrip latency.\n\n     You may want to be able to say \"I want all objects reachable\n     from this tree; please give me a packfile of needed objects\n     assuming that I have all objects reachable from this other tree\n     (or these other trees)\".\n\n"},{"id":"313352","messageId":"c8788f30-263b-c96a-239d-940743b96b53@google.com","threadId":"45293","inReplyTo":"20170304191901.9622-1-markbt@efaref.net","subject":"Re: [RFC 0/4] Shallow clones with on-demand fetch","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-03-06T19:16:23Z","receivedAt":"2017-03-06T19:25:40Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On 03/04/2017 11:18 AM, Mark Thomas wrote:\n> I was inspired a bit by Microsoft's announcement of their Git VFS.  I\n> saw that people have talked in the past about making git fetch objects\n> from remotes as they are needed, and decided to give it a try.\n\nFor reference, one such conversation is [1]. (cc-ing Ben Peart also)\n\n> The patch series adds a \"--on-demand\" option to git clone, which, when\n> used in conjunction with the existing shallow clone operations, clones\n> the full history of the repository's commits, but only the files that\n> would be included in the shallow clone.\n>\n> When a file that is missing is required, git requests the file on-demand\n> from the remote, via a new 'upload-file' service.\n\nA reachability check (of the blob) might be a good idea. The current Git \nimplementation already supports fetching a blob (perhaps a bug) but has \nproblems with reachability calculations that I tried to fix in [2], but \nfound some bugs that weren't easily fixable.\n\nAs I said in [2], I think that proper fetching of blobs on demand is a \nprerequisite to any sort of missing object tolerance (like your \non-demand clones), so I haven't thought much about the topics in the \nrest of your patch set.\n\n[1] <20170113155253.1644-1-benpeart@microsoft.com> (you can search for \nemails by Message ID in online archives like \nhttps://public-inbox.org/git if you don't already have them)\n[2] <cover.1487984670.git.jonathantanmy@google.com>\n"},{"id":"313355","messageId":"CAGZ79kZ1K-4PJC45nSZLX3m+FrYrsz8yy__dpwmMd8niRZX_dQ@mail.gmail.com","threadId":"45293","inReplyTo":"c8788f30-263b-c96a-239d-940743b96b53@google.com","subject":"Re: [RFC 0/4] Shallow clones with on-demand fetch","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-03-06T20:01:35Z","receivedAt":"2017-03-06T20:10:42Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Mar 6, 2017 at 11:16 AM, Jonathan Tan <jonathantanmy@google.com> wrote:\n\n> [1] <20170113155253.1644-1-benpeart@microsoft.com> (you can search for\n> emails by Message ID in online archives like https://public-inbox.org/git if\n> you don't already have them)\n\nNot just search, but the immediate lookup is\n https://public-inbox.org/git/<message-id>\nso\n https://public-inbox.org/git/20170113155253.1644-1-benpeart@microsoft.com\n\n> [2] <cover.1487984670.git.jonathantanmy@google.com>\n\nand\n  https://public-inbox.org/git/cover.1487984670.git.jonathantanmy@google.com\n"},{"id":"313409","messageId":"20170307094247.atdtqpttchk5r6qe@sigill.intra.peff.net","threadId":"45293","inReplyTo":"xmqqr32anri1.fsf@junio-linux.mtv.corp.google.com","subject":"Re: [RFC 0/4] Shallow clones with on-demand fetch","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-03-07T09:42:47Z","receivedAt":"2017-03-07T12:32:03Z","isPatch":false,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Mon, Mar 06, 2017 at 11:18:30AM -0800, Junio C Hamano wrote:\n\n> Mark Thomas <markbt@efaref.net> writes:\n> \n> > This is a proof-of-concept, so it is in no way complete.  It contains a\n> > few hacks to make it work, but these can be ironed out with a bit more\n> > work.  What I have so far is sufficient to try out the idea.\n> \n> Two things that immediately come to mind (which may or may not be\n> real issues) are \n> \n>  (1) What (if any) security model you have in mind.\n> \n>      From object-confidentiality's point of view, this needs to be\n>      enabled only on a host that allows\n>      uploadpack.allowAnySHA1InWant but even riskier.\n> \n>      From DoS point of view, you can make a short 40-byte request to\n>      cause the other side emit megabytes of stuff.  I do not think\n>      it is a new problem (anybody can repeatedly request a clone of\n>      large stuff), but there may be new ramifications.\n> \n>  (2) If the interface to ask just one object kills the whole idea\n>      due to roundtrip latency.\n> \n>      You may want to be able to say \"I want all objects reachable\n>      from this tree; please give me a packfile of needed objects\n>      assuming that I have all objects reachable from this other tree\n>      (or these other trees)\".\n\nNot just latency, but you also lose all of the benefits of delta\ncompression. So if I asked for:\n\n  git log -p -- foo.c\n\nand git is going to fault in all of the various versions of foo.c over\ntime, it's _much_ more efficient to batch them into a single request, so\nthat the server can reuse on-disk deltas between the various versions.\nThat makes the transmission smaller, and it also makes it more likely\nfor the server to be able to transmit the bits straight off the disk\n(rather than assembling each delta itself then zlib-compressing the\nresult).\n\nSimilarly, there's a latency tension in just finding out whether an\nobject exists. When we call has_sha1_file() as part of a fetch, for\nexample, we really want to be able to answer it quickly. So you'd\nprobably want some mechanism to say \"tell me the sha1, type, and size\"\nof each object I _could_ get via upload-file. The size of that data is\nfar from trivial for a large repository, but you're probably better off\ngetting it once than paying the latency cost to fetch it piecemeal.\n\n-Peff\n"}]}