{"thread":{"id":"45652","subject":"[RFC 0/4] Implementation of fetch-blobs and fetch-refs","startedAt":"2017-04-10T20:46:28Z","lastAt":"2017-06-30T22:41:42Z","messageCount":6,"participants":["Jonathan Tan","Stefan Beller"],"isPatch":false,"patchVersion":null,"patchTotal":null},"messages":[{"id":"316511","messageId":"cover.1491851452.git.jonathantanmy@google.com","threadId":"45652","inReplyTo":null,"subject":"[RFC 0/4] Implementation of fetch-blobs and fetch-refs","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-04-10T20:46:06Z","receivedAt":"2017-04-10T20:46:28Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"My previous proposal about a possible new server endpoint that serves\nblobs and refs [1] didn't seem to garner much interest, so I thought I'd\nprovide a possible implementation of that proposal as a work in\nprogress.\n\nIn particular, patch 1 demonstrates that a new server endpoint that\nserves refs without the initial ref advertisement can be done in 228\nlines of code (according to \"git diff --stat\") while accounting for the\nvarious special cases that upload-pack imposes (stateless RPC, inclusion\nof an additional response when depth is requested, handling of \"done\" in\nrequest, sending a packfile directly after a response containing \"ACK\nready\" without waiting for \"done\", and so on).\n\nI'm hoping that these features would be of more interest to more people.\nTo that end, I'm sending these patches in the hope of showing that these\nfeatures are useful (omitting ref advertisements help greatly when\nserving large repos, as described in the commit message of patch 1, and\nserving blobs is useful for any fetch-blob-on-demand repo scheme) and\nthat my proposed way of implementing them can be done in a relatively\nuncomplicated manner (as seen in these patches).\n\nPatch 1-3 show what serving refs without the advertisement would look\nlike from the server's and fetch-pack's points of view. Patch 4 is\nsimilar to some of the other blob-serving patches except that this\ncontains reachability checks and that this bundles the resulting objects\nin a packfile.\n\n[1] <ffd92ad9-39fe-c76b-178d-6e3d6a425037@google.com>\n\nJonathan Tan (4):\n  server-endpoint: serve refs without advertisement\n  fetch-pack: refactor \"want\" pkt-line generation\n  fetch-pack: support new server endpoint\n  server-endpoint: serve blobs by hash\n\n .gitignore                 |   1 +\n Makefile                   |   3 +\n builtin/fetch-pack.c       |  10 +-\n fetch-pack.c               | 129 +++++++++++------\n fetch-pack.h               |   1 +\n server-endpoint.c          | 347 +++++++++++++++++++++++++++++++++++++++++++++\n t/helper/.gitignore        |   1 +\n t/helper/test-un-pkt.c     |  40 ++++++\n t/t5573-server-endpoint.sh |  60 ++++++++\n t/t9999-mytests.sh         | 242 +++++++++++++++++++++++++++++++\n 10 files changed, 790 insertions(+), 44 deletions(-)\n create mode 100644 server-endpoint.c\n create mode 100644 t/helper/test-un-pkt.c\n create mode 100644 t/t5573-server-endpoint.sh\n create mode 100644 t/t9999-mytests.sh\n\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316512","messageId":"247c6b74c9d375f3d34996c1e4465761ee9d49a1.1491851452.git.jonathantanmy@google.com","threadId":"45652","inReplyTo":"cover.1491851452.git.jonathantanmy@google.com","subject":"[RFC 1/4] server-endpoint: serve refs without advertisement","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-04-10T20:46:07Z","receivedAt":"2017-04-10T20:46:31Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Introduce a new binary that can act as an endpoint to serve refs without\nfirst sending the ref advertisement (a list of all ref names and\nassociated hashes that the server contains). For very large\nrepositories, including an internal Android repository with more than\n700000 refs, this would save tens of megabytes of network bandwidth\nduring each fetch.\n\nThis endpoint handles ref namespaces and \"uploadpack.hiderefs\" by\nitself, and handles other functionality by invoking upload-pack and\nacting as an intermediary (therefore having to know the relatively\nminute details of the fetch-pack/upload-pack protocol).\n\nNote: There is still an issue with the handling of \"deepen\" lines. The\ndocumentation for the pack protocol states that \"deepen 0\" is the same\nas not specifying any depth, but upload-pack seems to not accept \"deepen\n0\". I'm not sure if it's better to change the documentation or change\nthe code - I generally prefer to change the code in such cases, but\ntreating \"deepen 0\" (and similar things like \"deepen 000\") differently\nfrom other \"deepen\"s requires multiple components to know about this\nspecial case (upload-pack, fetch-pack, and now server-endpoint) so I'm\ninclined to just forbid it (like in the current code).\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n .gitignore        |   1 +\n Makefile          |   2 +\n server-endpoint.c | 228 ++++++++++++++++++++++++++++++++++++++++++++++++++++++\n 3 files changed, 231 insertions(+)\n create mode 100644 server-endpoint.c\n\ndiff --git a/.gitignore b/.gitignore\nindex 833ef3b0b..761e06d2c 100644\n--- a/.gitignore\n+++ b/.gitignore\n@@ -140,6 +140,7 @@\n /git-rm\n /git-send-email\n /git-send-pack\n+/git-server-endpoint\n /git-sh-i18n\n /git-sh-i18n--envsubst\n /git-sh-setup\ndiff --git a/Makefile b/Makefile\nindex c80fec292..0d3813772 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -603,6 +603,7 @@ PROGRAM_OBJS += shell.o\n PROGRAM_OBJS += show-index.o\n PROGRAM_OBJS += upload-pack.o\n PROGRAM_OBJS += remote-testsvn.o\n+PROGRAM_OBJS += server-endpoint.o\n \n # Binary suffix, set to .exe for Windows builds\n X =\n@@ -673,6 +674,7 @@ BINDIR_PROGRAMS_NEED_X += git-upload-pack\n BINDIR_PROGRAMS_NEED_X += git-receive-pack\n BINDIR_PROGRAMS_NEED_X += git-upload-archive\n BINDIR_PROGRAMS_NEED_X += git-shell\n+BINDIR_PROGRAMS_NEED_X += git-server-endpoint\n \n BINDIR_PROGRAMS_NO_X += git-cvsserver\n \ndiff --git a/server-endpoint.c b/server-endpoint.c\nnew file mode 100644\nindex 000000000..a9c0c7c94\n--- /dev/null\n+++ b/server-endpoint.c\n@@ -0,0 +1,228 @@\n+#include \"cache.h\"\n+#include \"pkt-line.h\"\n+#include \"refs.h\"\n+#include \"revision.h\"\n+#include \"run-command.h\"\n+\n+static const char * const server_endpoint_usage[] = {\n+\tN_(\"git server-endpoint [<options>] <dir>\"),\n+\tNULL\n+};\n+\n+static const char *capabilities = \"multi_ack_detailed side-band-64k shallow\";\n+\n+struct handle_want_data {\n+\tint upload_pack_in_fd;\n+\tint capabilities_sent;\n+\tstruct string_list sent_namespaced_names;\n+};\n+\n+static int send_want(const char *namespaced_name, const struct object_id *oid,\n+\t\t     int flags, void *handle_want_data)\n+{\n+\tstruct handle_want_data *data = handle_want_data;\n+\n+\tif (ref_is_hidden(strip_namespace(namespaced_name), namespaced_name))\n+\t\treturn 0;\n+\tif (string_list_lookup(&data->sent_namespaced_names, namespaced_name))\n+\t\treturn 0;\n+\n+\tstring_list_insert(&data->sent_namespaced_names, namespaced_name);\n+\n+\tif (data->capabilities_sent) {\n+\t\tpacket_write_fmt(data->upload_pack_in_fd, \"want %s\\n\",\n+\t\t\t\t oid_to_hex(oid));\n+\t} else {\n+\t\tpacket_write_fmt(data->upload_pack_in_fd, \"want %s%s\\n\",\n+\t\t\t\t oid_to_hex(oid), capabilities);\n+\t\tdata->capabilities_sent = 1;\n+\t}\n+\n+\treturn 0;\n+}\n+\n+static void handle_want(const char *arg, struct handle_want_data *data) {\n+\tchar *namespaced_name = xstrfmt(\"%s%s\", get_git_namespace(), arg);\n+\tif (has_glob_specials(arg)) {\n+\t\tfor_each_glob_ref(send_want, namespaced_name, data);\n+\t} else {\n+\t\tstruct object_id oid;\n+\t\tif (!read_ref(namespaced_name, oid.hash))\n+\t\t\tsend_want(namespaced_name, &oid, 0, data);\n+\t}\n+\tfree(namespaced_name);\n+}\n+\n+static int fetch_ref(int stateless_rpc)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstatic const char *argv[] = {\n+\t\t\"upload-pack\", \".\", NULL, NULL\n+\t};\n+\tstruct handle_want_data handle_want_data = {0, 0, STRING_LIST_INIT_DUP};\n+\n+\tchar *line;\n+\tint size;\n+\n+\tint upload_pack_will_respond = 0;\n+\tint wanted_refs_sent = 0;\n+\n+\tif (stateless_rpc)\n+\t\targv[2] = \"--stateless-rpc\";\n+\tcmd.argv = argv;\n+\tcmd.git_cmd = 1;\n+\tcmd.in = -1;\n+\tcmd.out = -1;\n+\n+\tif (start_command(&cmd))\n+\t\tgoto error;\n+\n+\thandle_want_data.upload_pack_in_fd = cmd.in;\n+\n+\tif (!stateless_rpc) {\n+\t\t/* Drain the initial ref advertisement (until flush-pkt). */\n+\t\twhile (packet_read_line(cmd.out, NULL))\n+\t\t\t;\n+\t}\n+\n+\t/* Send the wants. Upload-pack will not respond to this unless a depth\n+\t * request is made. */\n+\twhile ((line = packet_read_line(0, NULL))) {\n+\t\tconst char *arg;\n+\t\tif (skip_prefix(line, \"want \", &arg)) {\n+\t\t\thandle_want(arg, &handle_want_data);\n+\t\t} else if (starts_with(line, \"shallow \")) {\n+\t\t\tpacket_write_fmt(cmd.in, \"%s\", line);\n+\t\t} else if (starts_with(line, \"deepen \") ||\n+\t\t\t   starts_with(line, \"deepen-since \") ||\n+\t\t\t   starts_with(line, \"deepen-not \")) {\n+\t\t\tpacket_write_fmt(cmd.in, \"%s\", line);\n+\t\t\tupload_pack_will_respond = 1;\n+\t\t}\n+\t}\n+\tpacket_flush(cmd.in);\n+\n+\tif (upload_pack_will_respond) {\n+\t\twhile ((line = packet_read_line(cmd.out, NULL))) {\n+\t\t\tpacket_write_fmt(1, \"%s\", line);\n+\t\t}\n+\t\tpacket_flush(1);\n+\t}\n+\n+\t/* Continue to copy the conversation. */\n+\tdo {\n+\t\tchar buffer[LARGE_PACKET_DATA_MAX];\n+\t\tchar size_buffer[5]; /* 4 bytes + NUL */\n+\t\tint done_received = 0;\n+\t\tint ready_received = 0;\n+\t\tint options = PACKET_READ_CHOMP_NEWLINE;\n+\n+\t\twhile ((line = packet_read_line(0, NULL))) {\n+\t\t\tpacket_write_fmt(cmd.in, \"%s\", line);\n+\t\t\tif (!strcmp(line, \"done\")) {\n+\t\t\t\tdone_received = 1;\n+\t\t\t\t/* \"done\" also marks the end of the request. */\n+\t\t\t\tgoto after_flush;\n+\t\t\t}\n+\t\t}\n+\t\tpacket_flush(cmd.in);\n+after_flush:\n+\t\twhile ((size = packet_read(cmd.out, NULL, NULL, buffer,\n+\t\t\t\t\t   sizeof(buffer), options))) {\n+\t\t\tint send_wanted_refs = 0;\n+\t\t\tif (!wanted_refs_sent) {\n+\t\t\t\tif ((done_received || ready_received) &&\n+\t\t\t\t    size == strlen(\"ACK \") + GIT_SHA1_HEXSZ &&\n+\t\t\t\t    starts_with(buffer, \"ACK \"))\n+\t\t\t\t\tsend_wanted_refs = 1;\n+\t\t\t\telse if (done_received && !strcmp(buffer, \"NAK\"))\n+\t\t\t\t\tsend_wanted_refs = 1;\n+\t\t\t\telse if (size == strlen(\"ACK  ready\") + GIT_SHA1_HEXSZ &&\n+\t\t\t\t\t starts_with(buffer, \"ACK \") &&\n+\t\t\t\t\t !strcmp(buffer + strlen(\"ACK  \") + GIT_SHA1_HEXSZ, \"ready\"))\n+\t\t\t\t\tready_received = 1;\n+\t\t\t}\n+\t\t\tif (send_wanted_refs) {\n+\t\t\t\tstruct string_list_item *item;\n+\t\t\t\tfor_each_string_list_item(item,\n+\t\t\t\t\t\t\t  &handle_want_data.sent_namespaced_names) {\n+\t\t\t\t\tstruct object_id oid;\n+\t\t\t\t\tif (read_ref(item->string, oid.hash))\n+\t\t\t\t\t\tdie(\"something happened\");\n+\t\t\t\t\tpacket_write_fmt(1, \"wanted %s %s\",\n+\t\t\t\t\t\t\t oid_to_hex(&oid),\n+\t\t\t\t\t\t\t strip_namespace(item->string));\n+\t\t\t\t}\n+\t\t\t\twanted_refs_sent = 1;\n+\t\t\t\t/* Do not chomp any more characters because\n+\t\t\t\t * binary data (packfile) is about to be sent.\n+\t\t\t\t */\n+\t\t\t\toptions = 0;\n+\t\t\t}\n+\t\t\tsprintf(size_buffer, \"%04x\", size + 4);\n+\t\t\twrite_or_die(1, size_buffer, 4);\n+\t\t\twrite_or_die(1, buffer, size);\n+\t\t\tif (!wanted_refs_sent && !strcmp(buffer, \"NAK\")) {\n+\t\t\t\t/* NAK before we send wanted refs marks the end\n+\t\t\t\t * of the response. */\n+\t\t\t\tgoto after_flush_2;\n+\t\t\t}\n+\t\t}\n+\t\tpacket_flush(1);\n+after_flush_2:\n+\t\t;\n+\t} while (!stateless_rpc && !wanted_refs_sent);\n+\n+\tclose(cmd.in);\n+\tcmd.in = -1;\n+\tclose(cmd.out);\n+\tcmd.out = -1;\n+\n+\tif (finish_command(&cmd))\n+\t\treturn -1;\n+\n+\treturn 0;\n+\n+error:\n+\n+\tif (cmd.in >= 0)\n+\t\tclose(cmd.in);\n+\tif (cmd.out >= 0)\n+\t\tclose(cmd.out);\n+\treturn -1;\n+}\n+\n+static int server_endpoint_config(const char *var, const char *value, void *unused)\n+{\n+\treturn parse_hide_refs_config(var, value, \"uploadpack\");\n+}\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tint stateless_rpc = 0;\n+\n+\tstruct option options[] = {\n+\t\tOPT_BOOL(0, \"stateless-rpc\", &stateless_rpc,\n+\t\t\t N_(\"quit after a single request/response exchange\")),\n+\t\tOPT_END()\n+\t};\n+\n+\tchar *line;\n+\n+\tpacket_trace_identity(\"server-endpoint\");\n+\tcheck_replace_refs = 0;\n+\n+\targc = parse_options(argc, argv, NULL, options, server_endpoint_usage, 0);\n+\n+\tif (argc != 1)\n+\t\tdie(\"must have 1 arg\");\n+\n+\tif (!enter_repo(argv[0], 0))\n+\t\tdie(\"does not appear to be a git repository\");\n+\tgit_config(server_endpoint_config, NULL);\n+\n+\tline = packet_read_line(0, NULL);\n+\tif (!strcmp(line, \"fetch-refs\"))\n+\t\treturn fetch_ref(stateless_rpc);\n+\tdie(\"only fetch-refs is supported\");\n+}\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316513","messageId":"dca51f5f394df376162a4cd716688ad14802e3fe.1491851452.git.jonathantanmy@google.com","threadId":"45652","inReplyTo":"cover.1491851452.git.jonathantanmy@google.com","subject":"[RFC 2/4] fetch-pack: refactor \"want\" pkt-line generation","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-04-10T20:46:08Z","receivedAt":"2017-04-10T20:46:33Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"In fetch-pack, refactor the generation of the initial request\n(containing the \"want\" lines) into its own function. This cleans up the\ncode slightly in that the scopes of certain variables are reduced, but\nthis commit mainly is in preparation for a subsequent one.\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n fetch-pack.c | 58 +++++++++++++++++++++++++++++++++-------------------------\n 1 file changed, 33 insertions(+), 25 deletions(-)\n\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex d07d85ce3..74771a283 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -314,29 +314,14 @@ static int next_flush(struct fetch_pack_args *args, int count)\n \treturn count;\n }\n \n-static int find_common(struct fetch_pack_args *args,\n-\t\t       int fd[2], unsigned char *result_sha1,\n-\t\t       struct ref *refs)\n+/*\n+ * Generate the \"want\" lines to be sent. If there are no \"want\" lines to be\n+ * sent, return NULL.\n+ */\n+static char *get_wants(const struct fetch_pack_args *args, struct ref *refs)\n {\n-\tint fetching;\n-\tint count = 0, flushes = 0, flush_at = INITIAL_FLUSH, retval;\n-\tconst unsigned char *sha1;\n-\tunsigned in_vain = 0;\n-\tint got_continue = 0;\n-\tint got_ready = 0;\n \tstruct strbuf req_buf = STRBUF_INIT;\n-\tsize_t state_len = 0;\n \n-\tif (args->stateless_rpc && multi_ack == 1)\n-\t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n-\tif (marked)\n-\t\tfor_each_ref(clear_marks, NULL);\n-\tmarked = 1;\n-\n-\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n-\tfor_each_cached_alternate(insert_one_alternate_object);\n-\n-\tfetching = 0;\n \tfor ( ; refs ; refs = refs->next) {\n \t\tunsigned char *remote = refs->old_oid.hash;\n \t\tconst char *remote_hex;\n@@ -358,7 +343,7 @@ static int find_common(struct fetch_pack_args *args,\n \t\t}\n \n \t\tremote_hex = sha1_to_hex(remote);\n-\t\tif (!fetching) {\n+\t\tif (!req_buf.len) {\n \t\t\tstruct strbuf c = STRBUF_INIT;\n \t\t\tif (multi_ack == 2)     strbuf_addstr(&c, \" multi_ack_detailed\");\n \t\t\tif (multi_ack == 1)     strbuf_addstr(&c, \" multi_ack\");\n@@ -378,14 +363,37 @@ static int find_common(struct fetch_pack_args *args,\n \t\t\tstrbuf_release(&c);\n \t\t} else\n \t\t\tpacket_buf_write(&req_buf, \"want %s\\n\", remote_hex);\n-\t\tfetching++;\n \t}\n \n-\tif (!fetching) {\n-\t\tstrbuf_release(&req_buf);\n+\treturn req_buf.len ? strbuf_detach(&req_buf, NULL) : NULL;\n+}\n+\n+static int find_common(struct fetch_pack_args *args,\n+\t\t       int fd[2], unsigned char *result_sha1,\n+\t\t       char *wants)\n+{\n+\tint count = 0, flushes = 0, flush_at = INITIAL_FLUSH, retval;\n+\tconst unsigned char *sha1;\n+\tunsigned in_vain = 0;\n+\tint got_continue = 0;\n+\tint got_ready = 0;\n+\tstruct strbuf req_buf = STRBUF_INIT;\n+\tsize_t state_len = 0;\n+\n+\tif (args->stateless_rpc && multi_ack == 1)\n+\t\tdie(_(\"--stateless-rpc requires multi_ack_detailed\"));\n+\tif (marked)\n+\t\tfor_each_ref(clear_marks, NULL);\n+\tmarked = 1;\n+\n+\tfor_each_ref(rev_list_insert_ref_oid, NULL);\n+\tfor_each_cached_alternate(insert_one_alternate_object);\n+\n+\tif (!wants) {\n \t\tpacket_flush(fd[1]);\n \t\treturn 1;\n \t}\n+\tstrbuf_attach(&req_buf, wants, strlen(wants), strlen(wants));\n \n \tif (is_repository_shallow())\n \t\twrite_shallow_commits(&req_buf, 1, NULL);\n@@ -943,7 +951,7 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\tpacket_flush(fd[1]);\n \t\tgoto all_done;\n \t}\n-\tif (find_common(args, fd, sha1, ref) < 0)\n+\tif (find_common(args, fd, sha1, get_wants(args, ref)) < 0)\n \t\tif (!args->keep_pack)\n \t\t\t/* When cloning, it is not unusual to have\n \t\t\t * no common commit.\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316514","messageId":"40f36d5eeb984adc220a4038fc77ed6ad1398fef.1491851452.git.jonathantanmy@google.com","threadId":"45652","inReplyTo":"cover.1491851452.git.jonathantanmy@google.com","subject":"[RFC 3/4] fetch-pack: support new server endpoint","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-04-10T20:46:09Z","receivedAt":"2017-04-10T20:46:42Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"This is a patch to demonstrate that fetch-pack only requires a small\namount of changes to support the new server endpoint, and that things\ngenerally work in various situations, including both stateless RPC and\nnon-stateless RPC (as can be seen from the tests).\n\nSome minor issues remain:\n - Names of refs should be treated as strings (when interfacing with the\n   new endpoint is requested) instead of being parsed into struct ref\n   unconditionally.\n - Capability management could probably be done better instead of\n   checking for \"--new-way\" everywhere.\n - I'm not sure how to test the upload-pack code path that sends \"ACK\n   ready\".\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n Makefile               |   1 +\n builtin/fetch-pack.c   |  10 +-\n fetch-pack.c           |  75 +++++++++++----\n fetch-pack.h           |   1 +\n t/helper/.gitignore    |   1 +\n t/helper/test-un-pkt.c |  40 ++++++++\n t/t9999-mytests.sh     | 242 +++++++++++++++++++++++++++++++++++++++++++++++++\n 7 files changed, 349 insertions(+), 21 deletions(-)\n create mode 100644 t/helper/test-un-pkt.c\n create mode 100644 t/t9999-mytests.sh\n\ndiff --git a/Makefile b/Makefile\nindex 0d3813772..0b4510d3f 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -641,6 +641,7 @@ TEST_PROGRAMS_NEED_X += test-string-list\n TEST_PROGRAMS_NEED_X += test-submodule-config\n TEST_PROGRAMS_NEED_X += test-subprocess\n TEST_PROGRAMS_NEED_X += test-svn-fe\n+TEST_PROGRAMS_NEED_X += test-un-pkt\n TEST_PROGRAMS_NEED_X += test-urlmatch-normalization\n TEST_PROGRAMS_NEED_X += test-wildmatch\n \ndiff --git a/builtin/fetch-pack.c b/builtin/fetch-pack.c\nindex 2a1c1c213..4bd83c4ff 100644\n--- a/builtin/fetch-pack.c\n+++ b/builtin/fetch-pack.c\n@@ -143,6 +143,10 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\t\targs.update_shallow = 1;\n \t\t\tcontinue;\n \t\t}\n+\t\tif (!strcmp(\"--new-way\", arg)) {\n+\t\t\targs.new_way = 1;\n+\t\t\tcontinue;\n+\t\t}\n \t\tusage(fetch_pack_usage);\n \t}\n \tif (deepen_not.nr)\n@@ -193,7 +197,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t\tif (!conn)\n \t\t\treturn args.diag_url ? 0 : 1;\n \t}\n-\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n+\tif (!args.new_way)\n+\t\tget_remote_heads(fd[0], NULL, 0, &ref, 0, NULL, &shallow);\n \n \tref = fetch_pack(&args, fd, conn, ref, dest, sought, nr_sought,\n \t\t\t &shallow, pack_lockfile_ptr);\n@@ -219,7 +224,8 @@ int cmd_fetch_pack(int argc, const char **argv, const char *prefix)\n \t * remote no-such-ref' would silently succeed without issuing\n \t * an error.\n \t */\n-\tret |= report_unmatched_refs(sought, nr_sought);\n+\tif (!args.new_way)\n+\t\tret |= report_unmatched_refs(sought, nr_sought);\n \n \twhile (ref) {\n \t\tprintf(\"%s %s\\n\",\ndiff --git a/fetch-pack.c b/fetch-pack.c\nindex 74771a283..4cbdada7b 100644\n--- a/fetch-pack.c\n+++ b/fetch-pack.c\n@@ -251,7 +251,7 @@ static void consume_shallow_list(struct fetch_pack_args *args, int fd)\n \t}\n }\n \n-static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n+static enum ack_type get_ack(int fd, unsigned char *result_sha1, struct ref **ref, struct sha1_array *shallow)\n {\n \tint len;\n \tchar *line = packet_read_line(fd, &len);\n@@ -261,6 +261,23 @@ static enum ack_type get_ack(int fd, unsigned char *result_sha1)\n \t\tdie(_(\"git fetch-pack: expected ACK/NAK, got EOF\"));\n \tif (!strcmp(line, \"NAK\"))\n \t\treturn NAK;\n+\tif (ref && skip_prefix(line, \"wanted \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tif (!get_sha1_hex(arg, oid.hash) && arg[40] == ' ' && arg[41]) {\n+\t\t\tstruct ref *new_ref = alloc_ref(arg + 41);\n+\t\t\toidcpy(&new_ref->old_oid, &oid);\n+\t\t\tnew_ref->next = *ref;\n+\t\t\t*ref = new_ref;\n+\t\t\treturn get_ack(fd, result_sha1, ref, shallow);\n+\t\t}\n+\t}\n+\tif (shallow && skip_prefix(line, \"shallow \", &arg)) {\n+\t\tstruct object_id oid;\n+\t\tif (!get_sha1_hex(arg, oid.hash)) {\n+\t\t\tsha1_array_append(shallow, oid.hash);\n+\t\t\treturn get_ack(fd, result_sha1, ref, shallow);\n+\t\t}\n+\t}\n \tif (skip_prefix(line, \"ACK \", &arg)) {\n \t\tif (!get_sha1_hex(arg, result_sha1)) {\n \t\t\targ += 40;\n@@ -370,7 +387,7 @@ static char *get_wants(const struct fetch_pack_args *args, struct ref *refs)\n \n static int find_common(struct fetch_pack_args *args,\n \t\t       int fd[2], unsigned char *result_sha1,\n-\t\t       char *wants)\n+\t\t       char *wants, struct ref **ref, struct sha1_array *shallow)\n {\n \tint count = 0, flushes = 0, flush_at = INITIAL_FLUSH, retval;\n \tconst unsigned char *sha1;\n@@ -475,7 +492,7 @@ static int find_common(struct fetch_pack_args *args,\n \n \t\t\tconsume_shallow_list(args, fd[0]);\n \t\t\tdo {\n-\t\t\t\tack = get_ack(fd[0], result_sha1);\n+\t\t\t\tack = get_ack(fd[0], result_sha1, NULL, NULL);\n \t\t\t\tif (ack)\n \t\t\t\t\tprint_verbose(args, _(\"got %s %d %s\"), \"ack\",\n \t\t\t\t\t\t      ack, sha1_to_hex(result_sha1));\n@@ -544,7 +561,7 @@ static int find_common(struct fetch_pack_args *args,\n \tif (!got_ready || !no_done)\n \t\tconsume_shallow_list(args, fd[0]);\n \twhile (flushes || multi_ack) {\n-\t\tint ack = get_ack(fd[0], result_sha1);\n+\t\tint ack = get_ack(fd[0], result_sha1, ref, shallow);\n \t\tif (ack) {\n \t\t\tprint_verbose(args, _(\"got %s (%d) %s\"), \"ack\",\n \t\t\t\t      ack, sha1_to_hex(result_sha1));\n@@ -878,22 +895,22 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\t\t struct shallow_info *si,\n \t\t\t\t char **pack_lockfile)\n {\n-\tstruct ref *ref = copy_ref_list(orig_ref);\n+\tstruct ref *ref;\n \tunsigned char sha1[20];\n \tconst char *agent_feature;\n \tint agent_len;\n+\tint find_common_result;\n \n-\tsort_ref_list(&ref, ref_compare_name);\n-\tQSORT(sought, nr_sought, cmp_ref_by_name);\n-\n-\tif ((args->depth > 0 || is_repository_shallow()) && !server_supports(\"shallow\"))\n-\t\tdie(_(\"Server does not support shallow clients\"));\n+\tif (!args->new_way) {\n+\t\tif ((args->depth > 0 || is_repository_shallow()) && !server_supports(\"shallow\"))\n+\t\t\tdie(_(\"Server does not support shallow clients\"));\n+\t}\n \tif (args->depth > 0 || args->deepen_since || args->deepen_not)\n \t\targs->deepen = 1;\n-\tif (server_supports(\"multi_ack_detailed\")) {\n+\tif (server_supports(\"multi_ack_detailed\") || args->new_way) {\n \t\tprint_verbose(args, _(\"Server supports multi_ack_detailed\"));\n \t\tmulti_ack = 2;\n-\t\tif (server_supports(\"no-done\")) {\n+\t\tif (server_supports(\"no-done\") || args->new_way) {\n \t\t\tprint_verbose(args, _(\"Server supports no-done\"));\n \t\t\tif (args->stateless_rpc)\n \t\t\t\tno_done = 1;\n@@ -936,22 +953,42 @@ static struct ref *do_fetch_pack(struct fetch_pack_args *args,\n \t\t\tprint_verbose(args, _(\"Server version is %.*s\"),\n \t\t\t\t      agent_len, agent_feature);\n \t}\n-\tif (server_supports(\"deepen-since\"))\n+\tif (server_supports(\"deepen-since\") || args->new_way)\n \t\tdeepen_since_ok = 1;\n \telse if (args->deepen_since)\n \t\tdie(_(\"Server does not support --shallow-since\"));\n-\tif (server_supports(\"deepen-not\"))\n+\tif (server_supports(\"deepen-not\") || args->new_way)\n \t\tdeepen_not_ok = 1;\n \telse if (args->deepen_not)\n \t\tdie(_(\"Server does not support --shallow-exclude\"));\n \tif (!server_supports(\"deepen-relative\") && args->deepen_relative)\n \t\tdie(_(\"Server does not support --deepen\"));\n \n-\tif (everything_local(args, &ref, sought, nr_sought)) {\n-\t\tpacket_flush(fd[1]);\n-\t\tgoto all_done;\n+\tif (args->new_way) {\n+\t\tstruct strbuf wants = STRBUF_INIT;\n+\t\tint i;\n+\t\tstruct sha1_array shallow = SHA1_ARRAY_INIT;\n+\n+\t\tref = NULL;\n+\t\tmulti_ack = 2;\n+\t\tuse_sideband = 2;\n+\n+\t\tpacket_buf_write(&wants, \"fetch-refs\\n\");\n+\t\tfor (i = 0; i < nr_sought; i++)\n+\t\t\tpacket_buf_write(&wants, \"want %s\\n\", sought[i]->name);\n+\t\tfind_common_result = find_common(args, fd, sha1, strbuf_detach(&wants, NULL), &ref, &shallow);\n+\t\tprepare_shallow_info(si, &shallow);\n+\t} else {\n+\t\tref = copy_ref_list(orig_ref);\n+\t\tsort_ref_list(&ref, ref_compare_name);\n+\t\tQSORT(sought, nr_sought, cmp_ref_by_name);\n+\t\tif (everything_local(args, &ref, sought, nr_sought)) {\n+\t\t\tpacket_flush(fd[1]);\n+\t\t\tgoto all_done;\n+\t\t}\n+\t\tfind_common_result = find_common(args, fd, sha1, get_wants(args, ref), NULL, NULL);\n \t}\n-\tif (find_common(args, fd, sha1, get_wants(args, ref)) < 0)\n+\tif (find_common_result < 0)\n \t\tif (!args->keep_pack)\n \t\t\t/* When cloning, it is not unusual to have\n \t\t\t * no common commit.\n@@ -1128,7 +1165,7 @@ struct ref *fetch_pack(struct fetch_pack_args *args,\n \tif (nr_sought)\n \t\tnr_sought = remove_duplicates_in_refs(sought, nr_sought);\n \n-\tif (!ref) {\n+\tif (!args->new_way && !ref) {\n \t\tpacket_flush(fd[1]);\n \t\tdie(_(\"no matching remote head\"));\n \t}\ndiff --git a/fetch-pack.h b/fetch-pack.h\nindex a2d46e6e7..ab7a80696 100644\n--- a/fetch-pack.h\n+++ b/fetch-pack.h\n@@ -29,6 +29,7 @@ struct fetch_pack_args {\n \tunsigned cloning:1;\n \tunsigned update_shallow:1;\n \tunsigned deepen:1;\n+\tunsigned new_way:1;\n };\n \n /*\ndiff --git a/t/helper/.gitignore b/t/helper/.gitignore\nindex d6e8b3679..cce5069cb 100644\n--- a/t/helper/.gitignore\n+++ b/t/helper/.gitignore\n@@ -29,5 +29,6 @@\n /test-submodule-config\n /test-subprocess\n /test-svn-fe\n+/test-un-pkt\n /test-urlmatch-normalization\n /test-wildmatch\ndiff --git a/t/helper/test-un-pkt.c b/t/helper/test-un-pkt.c\nnew file mode 100644\nindex 000000000..6dcf87980\n--- /dev/null\n+++ b/t/helper/test-un-pkt.c\n@@ -0,0 +1,40 @@\n+/*\n+ * This program takes payloads in pkt-line format (one or more pkt-lines\n+ * followed by a flush pkt) and runs the given command once per payload.\n+ */\n+#include \"cache.h\"\n+#include \"pkt-line.h\"\n+#include \"run-command.h\"\n+\n+int cmd_main(int argc, const char **argv)\n+{\n+\tstruct child_process *cmd = NULL;\n+\tchar buffer[LARGE_PACKET_MAX];\n+\tint size;\n+\n+\twhile ((size = packet_read(0, NULL, NULL, buffer, sizeof(buffer),\n+\t\t\t\t   PACKET_READ_GENTLE_ON_EOF)) >= 0) {\n+\t\tif (size > 0) {\n+\t\t\tif (!cmd) {\n+\t\t\t\tcmd = xmalloc(sizeof(*cmd));\n+\t\t\t\tchild_process_init(cmd);\n+\t\t\t\tcmd->argv = argv + 1;\n+\t\t\t\tcmd->git_cmd = 1;\n+\t\t\t\tcmd->in = -1;\n+\t\t\t\tcmd->out = 0;\n+\t\t\t\tif (start_command(cmd))\n+\t\t\t\t\tdie(\"could not start command\");\n+\t\t\t}\n+\t\t\twrite_in_full(cmd->in, buffer, size);\n+\t\t} else if (cmd) {\n+\t\t\tclose(cmd->in);\n+\t\t\tif (finish_command(cmd))\n+\t\t\t\tdie(\"could not finish command\");\n+\t\t\tfree(cmd);\n+\t\t\tcmd = NULL;\n+\t\t\tif (size < 0)\n+\t\t\t\tbreak;\n+\t\t}\n+\t}\n+\treturn 0;\n+}\ndiff --git a/t/t9999-mytests.sh b/t/t9999-mytests.sh\nnew file mode 100644\nindex 000000000..9bb4e85e3\n--- /dev/null\n+++ b/t/t9999-mytests.sh\n@@ -0,0 +1,242 @@\n+#!/bin/sh\n+\n+test_description='my tests'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'fetch-pack --new-way basic' '\n+\trm -rf server &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0 &&\n+\n+\t\tgit checkout -b one\n+\t\ttest_commit 1 &&\n+\t\tgit checkout master &&\n+\n+\t\tgit checkout -b two\n+\t\ttest_commit 2 &&\n+\t\tgit checkout master &&\n+\n+\t\tgit checkout -b dont_fetch_this\n+\t\ttest_commit 3 &&\n+\t\tgit checkout master\n+\t) &&\n+\tgit fetch-pack --new-way --exec=git-server-endpoint server refs/heads/one refs/heads/two\\* >actual &&\n+\n+\tgrep \"$(printf \"%s refs/heads/one\" $(git -C server rev-parse --verify one))\" actual &&\n+\tgrep \"$(printf \"%s refs/heads/two\" $(git -C server rev-parse --verify two))\" actual &&\n+\t! grep dont_fetch_this actual\n+'\n+\n+test_expect_success 'fetch-pack --new-way hideRefs' '\n+\trm -rf server &&\n+\tgit init server &&\n+\ttest_config -C server transfer.hideRefs refs/heads/b2 &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0 &&\n+\n+\t\tgit checkout -b b1\n+\t\ttest_commit 1 &&\n+\t\tgit checkout master &&\n+\n+\t\tgit checkout -b b2\n+\t\ttest_commit 2 &&\n+\t\tgit checkout master\n+\t) &&\n+\tgit fetch-pack --new-way --exec=git-server-endpoint server refs/heads/b\\* >actual &&\n+\n+\tgrep \"$(printf \"%s refs/heads/b1\" $(git -C server rev-parse --verify 1))\" actual &&\n+\t! grep refs/heads/b2 actual\n+'\n+\n+test_expect_success 'fetch-pack --new-way long negotiation' '\n+\trm -rf server client &&\n+\tgit init server &&\n+\ttest_commit -C server 0 &&\n+\n+\tgit clone server client &&\n+\n+\ttest_commit -C server 1 &&\n+\n+\tgit -C client checkout -b sidebranch &&\n+\tfor i in $(seq 2 32)\n+\tdo\n+\t\ttest_commit -C client $i\n+\tdone &&\n+\n+\tgit -C client fetch-pack --new-way --exec=git-server-endpoint ../server refs/heads/master >actual &&\n+\n+\tgrep \"$(printf \"%s refs/heads/master\" $(git -C server rev-parse --verify 1))\" actual\n+'\n+\n+test_expect_success 'fetch-pack --new-way with shallow client' '\n+\trm -rf server &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0 &&\n+\t\ttest_commit 1\n+\t) &&\n+\trm -rf client &&\n+\tgit clone --depth=1 \"file://$(pwd)/server\" client &&\n+\t(\n+\t\tcd server &&\n+\t\tgit checkout -b two 0 &&\n+\t\ttest_commit 2\n+\t) &&\n+\n+\t# check that the shallow clone does not include this parent commit\n+\ttest_must_fail git -C client cat-file -e $(git -C server rev-parse 0) &&\n+\n+\tgit -C client fetch-pack --new-way --exec=git-server-endpoint ../server refs/heads/two >actual &&\n+\t# 0 is the parent of 2, so it must be included now\n+\tgit -C client cat-file -e $(git -C server rev-parse 0)\n+'\n+\n+test_expect_success 'fetch-pack --new-way --depth' '\n+\trm -rf server &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0\n+\t) &&\n+\trm -rf client &&\n+\tgit clone server client &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 1 &&\n+\t\ttest_commit 2\n+\t) &&\n+\n+\tgit -C client fetch-pack --new-way --exec=git-server-endpoint --depth=1 ../server refs/heads/master >actual &&\n+\tgit -C client cat-file -e $(git -C server rev-parse 2) &&\n+\ttest_must_fail git -C client cat-file -e $(git -C server rev-parse 1)\n+'\n+\n+test_expect_success 'fetch-pack --new-way --shallow-since' '\n+\trm -rf server &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0\n+\t) &&\n+\trm -rf client &&\n+\tgit clone server client &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 1 &&\n+\t\ttest_commit 2\n+\t) &&\n+\tDATE=$(git -C server log --format=\"format:%at\" --no-walk 2) &&\n+\n+\tgit -C client fetch-pack --new-way --exec=git-server-endpoint --shallow-since=$DATE ../server refs/heads/master >actual &&\n+\tgit -C client cat-file -e $(git -C server rev-parse 2) &&\n+\ttest_must_fail git -C client cat-file -e $(git -C server rev-parse 1)\n+'\n+\n+test_expect_success 'fetch-pack --new-way --shallow-exclude' '\n+\trm -rf server &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0\n+\t) &&\n+\trm -rf client &&\n+\tgit clone server client &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 1 &&\n+\t\ttest_commit 2\n+\t) &&\n+\n+\tgit -C client fetch-pack --new-way --exec=git-server-endpoint --shallow-exclude=1 ../server refs/heads/master >actual &&\n+\tgit -C client cat-file -e $(git -C server rev-parse 2) &&\n+\ttest_must_fail git -C client cat-file -e $(git -C server rev-parse 1)\n+'\n+\n+test_expect_success PIPE 'fetch-pack --new-way --stateless-rpc' '\n+\trm -rf server &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0 &&\n+\n+\t\tgit checkout -b one\n+\t\ttest_commit 1 &&\n+\t\tgit checkout master &&\n+\n+\t\tgit checkout -b two\n+\t\ttest_commit 2 &&\n+\t\tgit checkout master &&\n+\n+\t\tgit checkout -b dont_fetch_this\n+\t\ttest_commit 3 &&\n+\t\tgit checkout master\n+\t) &&\n+\trm -rf client &&\n+\tgit init client &&\n+\n+\tmkfifo se-out &&\n+\n+\tgit -C client fetch-pack --new-way --stateless-rpc ../server refs/heads/one refs/heads/two\\* <se-out | test-un-pkt server-endpoint server --stateless-rpc >se-out &&\n+\n+\tgit -C client cat-file -e $(git -C server rev-parse 1) &&\n+\tgit -C client cat-file -e $(git -C server rev-parse 2) &&\n+\ttest_must_fail git -C client cat-file -e $(git -C server rev-parse 3)\n+'\n+\n+test_expect_success 'fetch-pack --new-way --stateless-rpc long negotiation' '\n+\trm -rf server client &&\n+\tgit init server &&\n+\ttest_commit -C server 0 &&\n+\n+\tgit clone server client &&\n+\n+\ttest_commit -C server 1 &&\n+\n+\tgit -C client checkout -b sidebranch &&\n+\tfor i in $(seq 2 32)\n+\tdo\n+\t\ttest_commit -C client $i\n+\tdone &&\n+\n+\trm -f se-out &&\n+\tmkfifo se-out &&\n+\n+\tgit -C client fetch-pack --new-way --stateless-rpc ../server refs/heads/master <se-out | test-un-pkt server-endpoint server --stateless-rpc >se-out &&\n+\n+\tgit -C client cat-file -e $(git -C server rev-parse 1)\n+'\n+\n+test_expect_success 'fetch-pack --new-way --stateless-rpc namespaces' '\n+\trm -rf server &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0 &&\n+\n+\t\tgit checkout -b mybranch\n+\t\ttest_commit 1 &&\n+\t\tgit checkout master &&\n+\n+\t\tgit checkout -b mybranch_ns\n+\t\ttest_commit 2 &&\n+\t\tgit checkout master &&\n+\t\tgit update-ref refs/namespaces/ns/refs/heads/mybranch mybranch_ns\n+\t) &&\n+\trm -rf client &&\n+\tgit init client &&\n+\n+\trm -f se-out &&\n+\tmkfifo se-out &&\n+\n+\tgit -C client fetch-pack --new-way --stateless-rpc ../server refs/heads/mybranch <se-out | GIT_NAMESPACE=ns test-un-pkt server-endpoint server --stateless-rpc >se-out &&\n+\n+\tgit -C client cat-file -e $(git -C server rev-parse 2) &&\n+\ttest_must_fail git -C client cat-file -e $(git -C server rev-parse 1)\n+'\n+\n+test_done\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"316515","messageId":"2cef84c6781af4b36c17968537ed4e492c4435b2.1491851452.git.jonathantanmy@google.com","threadId":"45652","inReplyTo":"cover.1491851452.git.jonathantanmy@google.com","subject":"[RFC 4/4] server-endpoint: serve blobs by hash","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-04-10T20:46:10Z","receivedAt":"2017-04-10T20:46:44Z","isPatch":false,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Upgrade server-endpoint to also serve blobs in a packfile given their\nhashes.  Reachability checks are performed before the packfile is sent -\nboth an absent blob and an unreachable blob are reported to the user in\nthe same way (\"not our blob\").\n\nDue to a bug in \"rev-list\" in the absence of bitmaps (discussed here\n[1]), the server repositories in tests all have bitmaps.\n\n[1] <20170309003547.6930-1-jonathantanmy@google.com>\n\nSigned-off-by: Jonathan Tan <jonathantanmy@google.com>\n---\n server-endpoint.c          | 121 ++++++++++++++++++++++++++++++++++++++++++++-\n t/t5573-server-endpoint.sh |  60 ++++++++++++++++++++++\n 2 files changed, 180 insertions(+), 1 deletion(-)\n create mode 100644 t/t5573-server-endpoint.sh\n\ndiff --git a/server-endpoint.c b/server-endpoint.c\nindex a9c0c7c94..870b853a6 100644\n--- a/server-endpoint.c\n+++ b/server-endpoint.c\n@@ -192,6 +192,123 @@ static int fetch_ref(int stateless_rpc)\n \treturn -1;\n }\n \n+/*\n+ * Returns 1 if all blobs are reachable. If not, returns 0 and stores the hash\n+ * of one of the unreachable blobs in unreachable.\n+ */\n+static int are_all_reachable(const struct object_array *blobs, struct object_id *unreachable)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstatic const char *argv[] = {\n+\t\t\"rev-list\", \"--objects\", \"--use-bitmap-index\", \"--stdin\", \"--not\", \"--all\", \"--not\", NULL,\n+\t};\n+\tint i;\n+\tchar buf[41] = {0};\n+\n+\tcmd.argv = argv;\n+\tcmd.git_cmd = 1;\n+\tcmd.in = -1;\n+\tcmd.out = -1;\n+\n+\tif (start_command(&cmd))\n+\t\tgoto error;\n+\t\n+\tfor (i = 0; i < blobs->nr; i++) {\n+\t\twrite_in_full(cmd.in, sha1_to_hex(blobs->objects[i].item->oid.hash), 40);\n+\t\twrite_in_full(cmd.in, \"\\n\", 1);\n+\t}\n+\tclose(cmd.in);\n+\tcmd.in = -1;\n+\n+\ti = read_in_full(cmd.out, buf, 40);\n+\tclose(cmd.out);\n+\tcmd.out = -1;\n+\n+\tif (finish_command(&cmd))\n+\t\tgoto error;\n+\n+\tif (i) {\n+\t\tif (get_oid_hex(buf, unreachable))\n+\t\t\tgoto error;\n+\t\treturn 0;\n+\t}\n+\n+\treturn 1;\n+\n+error:\n+\tif (cmd.out >= 0)\n+\t\tclose(cmd.out);\n+\tdie(\"problem with running rev-list\");\n+}\n+\n+static void send_blobs(const struct object_array *blobs)\n+{\n+\tstruct child_process cmd = CHILD_PROCESS_INIT;\n+\tstatic const char *argv[] = {\n+\t\t\"pack-objects\", \"--stdout\", NULL\n+\t};\n+\tint i;\n+\n+\tcmd.argv = argv;\n+\tcmd.git_cmd = 1;\n+\tcmd.in = -1;\n+\tcmd.out = 0;\n+\n+\tif (start_command(&cmd))\n+\t\tgoto error;\n+\t\n+\tfor (i = 0; i < blobs->nr; i++) {\n+\t\twrite_in_full(cmd.in, sha1_to_hex(blobs->objects[i].item->oid.hash), 40);\n+\t\twrite_in_full(cmd.in, \"\\n\", 1);\n+\t}\n+\tclose(cmd.in);\n+\tcmd.in = -1;\n+\n+\tif (finish_command(&cmd))\n+\t\tgoto error;\n+\n+\treturn;\n+\n+error:\n+\tdie(\"problem with running pack-objects\");\n+}\n+\n+static int fetch_blob(void)\n+{\n+\tchar *line;\n+\n+\tstruct object_array wanted_blobs = OBJECT_ARRAY_INIT;\n+\tstruct object_id unreachable;\n+\n+\twhile ((line = packet_read_line(0, NULL))) {\n+\t\tconst char *arg;\n+\t\tif (skip_prefix(line, \"want \", &arg)) {\n+\t\t\tstruct object_id oid;\n+\t\t\tstruct object *obj;\n+\t\t\tif (get_oid_hex(arg, &oid)) {\n+\t\t\t\tpacket_write_fmt(1, \"ERR invalid object ID <%s>\", arg);\n+\t\t\t\treturn 0;\n+\t\t\t}\n+\t\t\tobj = parse_object(oid.hash);\n+\t\t\tif (!obj || obj->type != OBJ_BLOB) {\n+\t\t\t\tpacket_write_fmt(1, \"ERR not our blob <%s>\", arg);\n+\t\t\t\treturn 0;\n+\t\t\t}\n+\t\t\tadd_object_array(obj, NULL, &wanted_blobs);\n+\t\t}\n+\t}\n+\n+\tif (!are_all_reachable(&wanted_blobs, &unreachable)) {\n+\t\tpacket_write_fmt(1, \"ERR not our blob <%s>\", oid_to_hex(&unreachable));\n+\t\treturn 0;\n+\t}\n+\n+\tpacket_write_fmt(1, \"ACK\\n\");\n+\tsend_blobs(&wanted_blobs);\n+\n+\treturn 0;\n+}\n+\n static int server_endpoint_config(const char *var, const char *value, void *unused)\n {\n \treturn parse_hide_refs_config(var, value, \"uploadpack\");\n@@ -224,5 +341,7 @@ int cmd_main(int argc, const char **argv)\n \tline = packet_read_line(0, NULL);\n \tif (!strcmp(line, \"fetch-refs\"))\n \t\treturn fetch_ref(stateless_rpc);\n-\tdie(\"only fetch-refs is supported\");\n+\tif (!strcmp(line, \"fetch-blobs\"))\n+\t\treturn fetch_blob();\n+\tdie(\"only fetch-refs and fetch-blobs are supported\");\n }\ndiff --git a/t/t5573-server-endpoint.sh b/t/t5573-server-endpoint.sh\nnew file mode 100644\nindex 000000000..48f052851\n--- /dev/null\n+++ b/t/t5573-server-endpoint.sh\n@@ -0,0 +1,60 @@\n+#!/bin/sh\n+\n+test_description='server-endpoint'\n+\n+. ./test-lib.sh\n+\n+test_expect_success 'fetch-blobs basic' '\n+\trm -rf server client &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0 &&\n+\t\ttest_commit 1 &&\n+\t\tgit repack -a -d --write-bitmap-index\n+\t) &&\n+\tBLOB0=$(git hash-object server/0.t) &&\n+\tBLOB1=$(git hash-object server/1.t) &&\n+\tprintf \"000ffetch-blobs0031want %s0031want %s0000\" \"$BLOB0\" \"$BLOB1\" | git server-endpoint server >out &&\n+\n+\ttest \"$(head -1 out)\" = \"0008ACK\" &&\n+\n+\tgit init client &&\n+\tsed 1d out | git -C client unpack-objects &&\n+\tgit -C client cat-file -e \"$BLOB0\" &&\n+\tgit -C client cat-file -e \"$BLOB1\"\n+'\n+\n+test_expect_success 'fetch-blobs no such object' '\n+\trm -rf server client &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0 &&\n+\t\tgit repack -a -d --write-bitmap-index\n+\t) &&\n+\tBLOB0=$(git hash-object server/0.t) &&\n+\techo myblob >myblob &&\n+\tMYBLOB=$(git hash-object myblob) &&\n+\tprintf \"000ffetch-blobs0031want %s0031want %s0000\" \"$BLOB0\" \"$MYBLOB\" | git server-endpoint server >out &&\n+\n+\ttest_i18ngrep \"$(printf \"ERR not our blob.*%s\" \"$MYBLOB\")\" out\n+'\n+\n+test_expect_success 'fetch-blobs unreachable' '\n+\trm -rf server client &&\n+\tgit init server &&\n+\t(\n+\t\tcd server &&\n+\t\ttest_commit 0 &&\n+\t\tgit repack -a -d --write-bitmap-index\n+\t) &&\n+\tBLOB0=$(git hash-object server/0.t) &&\n+\techo myblob >myblob &&\n+\tMYBLOB=$(git -C server hash-object -w ../myblob) &&\n+\tprintf \"000ffetch-blobs0031want %s0031want %s0000\" \"$BLOB0\" \"$MYBLOB\" | git server-endpoint server >out &&\n+\n+\ttest_i18ngrep \"$(printf \"ERR not our blob.*%s\" \"$MYBLOB\")\" out\n+'\n+\n+test_done\n-- \n2.12.2.715.g7642488e1d-goog\n\n"},{"id":"323652","messageId":"CAGZ79kYdydHLxuzDcfyiahbfszoAGoyVsb4VjTAiC8ATDyLMxA@mail.gmail.com","threadId":"45652","inReplyTo":"cover.1491851452.git.jonathantanmy@google.com","subject":"Re: [RFC 0/4] Implementation of fetch-blobs and fetch-refs","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-06-30T22:41:34Z","receivedAt":"2017-06-30T22:41:42Z","isPatch":false,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Sorry for not paying attention to this patch set earlier. I missed\n(and might still\nmiss) the big picture(tm) here. This is because\nhttp://public-inbox.org/git/ffd92ad9-39fe-c76b-178d-6e3d6a425037@google.com/\nseemed to be specific for big binary files, but ...\n\nOn Mon, Apr 10, 2017 at 1:46 PM, Jonathan Tan <jonathantanmy@google.com> wrote:\n\n> In particular, patch 1 demonstrates that a new server endpoint that\n> serves refs without the initial ref advertisement can be done in 228\n> lines of code (according to \"git diff --stat\") while accounting for the\n> various special cases that upload-pack imposes (stateless RPC, inclusion\n> of an additional response when depth is requested, handling of \"done\" in\n> request, sending a packfile directly after a response containing \"ACK\n> ready\" without waiting for \"done\", and so on).\n\n... from looking at the patches, this actually implements the idea\nthat was thrown around on the mailing list as \"client speaks first\",\nas this essentially omits the refs advertisement, and then continues the\nconversation by running upload-pack as normal.\n\nThat seems very exciting to me!\n\n> To that end, I'm sending these patches in the hope of showing that these\n> features are useful (omitting ref advertisements help greatly when\n> serving large repos, as described in the commit message of patch 1, and\n> serving blobs is useful for any fetch-blob-on-demand repo scheme) and\n> that my proposed way of implementing them can be done in a relatively\n> uncomplicated manner (as seen in these patches).\n\nYes, if we encapsulate v1 protocol on (both?) sides, we seem to need\nvery little additional code, but get the benefit of using lots of well\ntested code.\n\nThanks,\nStefan\n"}]}