{"thread":{"id":"46746","subject":"[PATCH 0/8] protocol transition","startedAt":"2017-09-13T21:55:07Z","lastAt":"2017-11-17T01:12:21Z","messageCount":161,"participants":["Brandon Williams","Stefan Beller","Jeff King","Jonathan Nieder","Jonathan Tan","Junio C Hamano","Simon Ruderich","Martin Ågren","Kevin Daudt","Johannes Schindelin"],"isPatch":true,"patchVersion":1,"patchTotal":8},"messages":[{"id":"328002","messageId":"20170913215448.84674-1-bmwill@google.com","threadId":"46746","inReplyTo":null,"subject":"[PATCH 0/8] protocol transition","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-13T21:54:40Z","receivedAt":"2017-09-13T21:55:07Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Here is the non-RFC version of of my proposed protocol transition plan which\ncan be found here:\nhttps://public-inbox.org/git/20170824225328.8174-1-bmwill@google.com/\n\nThe main take away from the comments on the RFC were that the first transition\nshouldn't be drastic, so this patch set introduces protocol v1 which is simply\nprotocol v0 (which is what I'm calling the current git wire protocol) with a\nsingle pkt-line containing a version string before the ref advertisement.\n\nI have included tests for protocol version 1 to verify that it works with the\nfollowing transports: git://, file://, ssh://, and http://. I have also\nincluded an interop test to ensure that sending the version request out of band\ndoesn't cause issues with older servers.\n\nAny and all comments and feedback are welcome, thanks!\n\nBrandon Williams (8):\n pkt-line: add packet_write function\n protocol: introduce protocol extention mechanisms\n daemon: recognize hidden request arguments\n upload-pack, receive-pack: introduce protocol version 1\n connect: teach client to recognize v1 server response\n connect: tell server that the client understands v1\n http: tell server that the client understands v1\n i5700: add interop test for protocol transition\n\n Documentation/config.txt | 16 ++\n Documentation/git.txt | 5 +\n Makefile | 1 +\n builtin/receive-pack.c | 14 ++\n cache.h | 9 +\n connect.c | 59 ++++++-\n daemon.c | 71 ++++++--\n http.c | 18 ++\n pkt-line.c | 6 +\n pkt-line.h | 1 +\n protocol.c | 72 ++++++++\n protocol.h | 15 ++\n t/interop/i5700-protocol-transition.sh | 68 ++++++++\n t/lib-httpd/apache.conf | 7 +\n t/t5700-protocol-v1.sh | 292 +++++++++++++++++++++++++++++++++\n upload-pack.c | 17 +-\n 16 files changed, 655 insertions(+), 16 deletions(-)\n create mode 100644 protocol.c\n create mode 100644 protocol.h\n create mode 100755 t/interop/i5700-protocol-transition.sh\n create mode 100755 t/t5700-protocol-v1.sh\n\n-- \n2.14.1.690.gbb1197296e-goog\n\n"},{"id":"328003","messageId":"20170913215448.84674-2-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH 1/8] pkt-line: add packet_write function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-13T21:54:41Z","receivedAt":"2017-09-13T21:55:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a function which can be used to write the contents of an arbitrary\nbuffer. This makes it easy to build up data in a buffer before writing\nthe packet instead of formatting the entire contents of the packet using\n'packet_write_fmt()'.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 6 ++++++\n pkt-line.h | 1 +\n 2 files changed, 7 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 7db911957..cf98f371b 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -188,6 +188,12 @@ static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n \treturn error(\"packet write failed\");\n }\n \n+void packet_write(const int fd_out, const char *buf, size_t size)\n+{\n+\tif (packet_write_gently(fd_out, buf, size))\n+\t\tdie_errno(\"packet write failed\");\n+}\n+\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n {\n \tva_list args;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 66ef610fc..d9e9783b1 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -22,6 +22,7 @@\n void packet_flush(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n+void packet_write(const int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n-- \n2.14.1.690.gbb1197296e-goog\n\n"},{"id":"328004","messageId":"20170913215448.84674-3-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH 2/8] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-13T21:54:42Z","receivedAt":"2017-09-13T21:55:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Create protocol.{c,h} and provide functions which future servers and\nclients can use to determine which protocol to use or is being used.\n\nAlso introduce the 'GIT_PROTOCOL' environment variable which will be\nused to communicate a colon separated list of keys with optional values\nto a server. Unknown keys and values must be tolerated. This mechanism\nis used to communicate which version of the wire protocol a client would\nlike to use with a server.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/config.txt | 16 +++++++++++\n Documentation/git.txt | 5 ++++\n Makefile | 1 +\n cache.h | 7 +++++\n protocol.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++\n protocol.h | 15 ++++++++++\n 6 files changed, 116 insertions(+)\n create mode 100644 protocol.c\n create mode 100644 protocol.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dc4e3f58a..d5b28a32c 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2517,6 +2517,22 @@ The protocol names currently used by git are:\n `hg` to allow the `git-remote-hg` helper)\n --\n \n+protocol.version::\n+\tIf set, clients will attempt to communicate with a server using\n+\tthe specified protocol version. If unset, no attempt will be\n+\tmade by the client to communicate using a particular protocol\n+\tversion, this results in protocol version 0 being used.\n+\tSupported versions:\n++\n+--\n+\n+* `0` - the original wire protocol.\n+\n+* `1` - the original wire protocol with the addition of a version string\n+ in the initial respose from the server.\n+\n+--\n+\n pull.ff::\n \tBy default, Git does not create an extra merge commit when merging\n \ta commit that is a descendant of the current commit. Instead, the\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 6e3a6767e..299f75c7b 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -697,6 +697,11 @@ of clones and fetches.\n \twhich feed potentially-untrusted URLS to git commands. See\n \tlinkgit:git-config[1] for more details.\n \n+`GIT_PROTOCOL`::\n+\tFor internal use only. Used in handshaking the wire protocol.\n+\tContains a colon ':' separated list of keys with optional values\n+\t'key[=value]'. Presence of unknown keys must be tolerated.\n+\n Discussion[[Discussion]]\n ------------------------\n \ndiff --git a/Makefile b/Makefile\nindex f2bb7f2f6..1f300bd6b 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -837,6 +837,7 @@ LIB_OBJS += pretty.o\n LIB_OBJS += prio-queue.o\n LIB_OBJS += progress.o\n LIB_OBJS += prompt.o\n+LIB_OBJS += protocol.o\n LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\ndiff --git a/cache.h b/cache.h\nindex a916bc79e..8839b1ed4 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -444,6 +444,13 @@ static inline enum object_type object_type(unsigned int mode)\n #define GIT_ICASE_PATHSPECS_ENVIRONMENT \"GIT_ICASE_PATHSPECS\"\n #define GIT_QUARANTINE_ENVIRONMENT \"GIT_QUARANTINE_PATH\"\n \n+/*\n+ * Environment variable used in handshaking the wire protocol.\n+ * Contains a colon ':' separated list of keys with optional values\n+ * 'key[=value]'. Presence of unknown keys must be tolerated.\n+ */\n+#define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n+\n /*\n * This environment variable is expected to contain a boolean indicating\n * whether we should or should not treat:\ndiff --git a/protocol.c b/protocol.c\nnew file mode 100644\nindex 000000000..1b16c7b9a\n--- /dev/null\n+++ b/protocol.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"protocol.h\"\n+\n+enum protocol_version parse_protocol_version(const char *value)\n+{\n+\tif (!strcmp(value, \"0\"))\n+\t\treturn protocol_v0;\n+\telse if (!strcmp(value, \"1\"))\n+\t\treturn protocol_v1;\n+\telse\n+\t\treturn protocol_unknown_version;\n+}\n+\n+enum protocol_version get_protocol_version_config(void)\n+{\n+\tconst char *value;\n+\tif (!git_config_get_string_const(\"protocol.version\", &value)) {\n+\t\tenum protocol_version version = parse_protocol_version(value);\n+\n+\t\tif (version == protocol_unknown_version)\n+\t\t\tdie(\"unknown value for config 'protocol.version': %s\",\n+\t\t\t value);\n+\n+\t\treturn version;\n+\t}\n+\n+\treturn protocol_v0;\n+}\n+\n+enum protocol_version determine_protocol_version_server(void)\n+{\n+\tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n+\tenum protocol_version version = protocol_v0;\n+\n+\tif (git_protocol) {\n+\t\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\t\tconst struct string_list_item *item;\n+\t\tstring_list_split(&list, git_protocol, ':', -1);\n+\n+\t\tfor_each_string_list_item(item, &list) {\n+\t\t\tconst char *value;\n+\t\t\tenum protocol_version v;\n+\n+\t\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n+\t\t\t\tv = parse_protocol_version(value);\n+\t\t\t\tif (v > version)\n+\t\t\t\t\tversion = v;\n+\t\t\t}\n+\t\t}\n+\n+\t\tstring_list_clear(&list, 0);\n+\t}\n+\n+\treturn version;\n+}\n+\n+enum protocol_version determine_protocol_version_client(const char *server_response)\n+{\n+\tenum protocol_version version = protocol_v0;\n+\n+\tif (skip_prefix(server_response, \"version \", &server_response)) {\n+\t\tversion = parse_protocol_version(server_response);\n+\n+\t\tif (version == protocol_unknown_version)\n+\t\t\tdie(\"server is speaking an unknown protocol\");\n+\t\tif (version == protocol_v0)\n+\t\t\tdie(\"protocol error: server explicitly said version 0\");\n+\t}\n+\n+\treturn version;\n+}\ndiff --git a/protocol.h b/protocol.h\nnew file mode 100644\nindex 000000000..2fa6486d0\n--- /dev/null\n+++ b/protocol.h\n@@ -0,0 +1,15 @@\n+#ifndef PROTOCOL_H\n+#define PROTOCOL_H\n+\n+enum protocol_version {\n+\tprotocol_unknown_version = -1,\n+\tprotocol_v0 = 0,\n+\tprotocol_v1 = 1,\n+};\n+\n+extern enum protocol_version parse_protocol_version(const char *value);\n+extern enum protocol_version get_protocol_version_config(void);\n+extern enum protocol_version determine_protocol_version_server(void);\n+extern enum protocol_version determine_protocol_version_client(const char *server_response);\n+\n+#endif /* PROTOCOL_H */\n-- \n2.14.1.690.gbb1197296e-goog\n\n"},{"id":"328005","messageId":"20170913215448.84674-4-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH 3/8] daemon: recognize hidden request arguments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-13T21:54:43Z","receivedAt":"2017-09-13T21:55:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"A normal request to git-daemon is structured as\n\"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\ngit-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\ncommand, 2009-06-04) we aren't able to place any extra args (separated\nby NULs) besides the host.\n\nIn order to get around this limitation teach git-daemon to recognize\nadditional request arguments hidden behind a second NUL byte. Requests\ncan then be structured like:\n\"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\ncan then parse out the extra arguments and set 'GIT_PROTOCOL'\naccordingly.\n\nSigned-off-by: Brandon Williams \n---\n daemon.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 61 insertions(+), 10 deletions(-)\n\ndiff --git a/daemon.c b/daemon.c\nindex 30747075f..250dbf82c 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -282,7 +282,7 @@ static const char *path_ok(const char *directory, struct hostinfo *hi)\n \treturn NULL;\t\t/* Fallthrough. Deny by default */\n }\n \n-typedef int (*daemon_service_fn)(void);\n+typedef int (*daemon_service_fn)(const struct argv_array *env);\n struct daemon_service {\n \tconst char *name;\n \tconst char *config_name;\n@@ -363,7 +363,7 @@ static int run_access_hook(struct daemon_service *service, const char *dir,\n }\n \n static int run_service(const char *dir, struct daemon_service *service,\n-\t\t struct hostinfo *hi)\n+\t\t struct hostinfo *hi, const struct argv_array *env)\n {\n \tconst char *path;\n \tint enabled = service->enabled;\n@@ -422,7 +422,7 @@ static int run_service(const char *dir, struct daemon_service *service,\n \t */\n \tsignal(SIGTERM, SIG_IGN);\n \n-\treturn service->fn();\n+\treturn service->fn(env);\n }\n \n static void copy_to_log(int fd)\n@@ -462,25 +462,34 @@ static int run_service_command(struct child_process *cld)\n \treturn finish_command(cld);\n }\n \n-static int upload_pack(void)\n+static int upload_pack(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_pushl(&cld.args, \"upload-pack\", \"--strict\", NULL);\n \targv_array_pushf(&cld.args, \"--timeout=%u\", timeout);\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n-static int upload_archive(void)\n+static int upload_archive(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_push(&cld.args, \"upload-archive\");\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n-static int receive_pack(void)\n+static int receive_pack(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_push(&cld.args, \"receive-pack\");\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n@@ -574,7 +583,7 @@ static void canonicalize_client(struct strbuf *out, const char *in)\n /*\n * Read the host as supplied by the client connection.\n */\n-static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n+static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n {\n \tchar *val;\n \tint vallen;\n@@ -602,6 +611,39 @@ static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n \t\tif (extra_args < end && *extra_args)\n \t\t\tdie(\"Invalid request\");\n \t}\n+\n+\treturn extra_args;\n+}\n+\n+static void parse_extra_args(struct argv_array *env, const char *extra_args,\n+\t\t\t int buflen)\n+{\n+\tconst char *end = extra_args + buflen;\n+\tstruct strbuf git_protocol = STRBUF_INIT;\n+\n+\tfor (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n+\t\tconst char *arg = extra_args;\n+\n+\t\t/*\n+\t\t * Parse the extra arguments, adding most to 'git_protocol'\n+\t\t * which will be used to set the 'GIT_PROTOCOL' envvar in the\n+\t\t * service that will be run.\n+\t\t *\n+\t\t * If there ends up being a particular arg in the future that\n+\t\t * git-daemon needs to parse specificly (like the 'host' arg)\n+\t\t * then it can be parsed here and not added to 'git_protocol'.\n+\t\t */\n+\t\tif (*arg) {\n+\t\t\tif (git_protocol.len > 0)\n+\t\t\t\tstrbuf_addch(&git_protocol, ':');\n+\t\t\tstrbuf_addstr(&git_protocol, arg);\n+\t\t}\n+\t}\n+\n+\tif (git_protocol.len > 0)\n+\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n+\t\t\t\t git_protocol.buf);\n+\tstrbuf_release(&git_protocol);\n }\n \n /*\n@@ -695,6 +737,7 @@ static int execute(void)\n \tint pktlen, len, i;\n \tchar *addr = getenv(\"REMOTE_ADDR\"), *port = getenv(\"REMOTE_PORT\");\n \tstruct hostinfo hi;\n+\tstruct argv_array env = ARGV_ARRAY_INIT;\n \n \thostinfo_init(&hi);\n \n@@ -716,8 +759,14 @@ static int execute(void)\n \t\tpktlen--;\n \t}\n \n-\tif (len != pktlen)\n-\t\tparse_host_arg(&hi, line + len + 1, pktlen - len - 1);\n+\tif (len != pktlen) {\n+\t\tconst char *extra_args;\n+\t\t/* retrieve host */\n+\t\textra_args = parse_host_arg(&hi, line + len + 1, pktlen - len - 1);\n+\n+\t\t/* parse additional args hidden behind a second NUL byte */\n+\t\tparse_extra_args(&env, extra_args + 1, pktlen - (extra_args - line) - 1);\n+\t}\n \n \tfor (i = 0; i < ARRAY_SIZE(daemon_service); i++) {\n \t\tstruct daemon_service *s = &(daemon_service[i]);\n@@ -730,13 +779,15 @@ static int execute(void)\n \t\t\t * Note: The directory here is probably context sensitive,\n \t\t\t * and might depend on the actual service being performed.\n \t\t\t */\n-\t\t\tint rc = run_service(arg, s, &hi);\n+\t\t\tint rc = run_service(arg, s, &hi, &env);\n \t\t\thostinfo_clear(&hi);\n+\t\t\targv_array_clear(&env);\n \t\t\treturn rc;\n \t\t}\n \t}\n \n \thostinfo_clear(&hi);\n+\targv_array_clear(&env);\n \tlogerror(\"Protocol error: '%s'\", line);\n \treturn -1;\n }\n-- \n2.14.1.690.gbb1197296e-goog\n\n"},{"id":"328006","messageId":"20170913215448.84674-6-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH 5/8] connect: teach client to recognize v1 server response","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-13T21:54:45Z","receivedAt":"2017-09-13T21:55:19Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach a client to recognize that a server understands protocol v1 by\nlooking at the first pkt-line the server sends in response. This is\ndone by looking for the response \"version 1\" send by upload-pack or\nreceive-pack.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 22 ++++++++++++++++++++++\n 1 file changed, 22 insertions(+)\n\ndiff --git a/connect.c b/connect.c\nindex 49b28b83b..2702e1f2e 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -11,6 +11,7 @@\n #include \"string-list.h\"\n #include \"sha1-array.h\"\n #include \"transport.h\"\n+#include \"protocol.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -142,6 +143,27 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\tif (len < 0)\n \t\t\tdie_initial_contact(saw_response);\n \n+\t\t/* Only check for version information on first response */\n+\t\tif (!saw_response) {\n+\t\t\tswitch (determine_protocol_version_client(buffer)) {\n+\t\t\tcase protocol_v1:\n+\t\t\t\t/*\n+\t\t\t\t * First pkt-line contained the version string.\n+\t\t\t\t * Continue on to process the ref advertisement.\n+\t\t\t\t */\n+\t\t\t\tcontinue;\n+\t\t\tcase protocol_v0:\n+\t\t\t\t/*\n+\t\t\t\t * Server is speaking protocol v0 and sent a\n+\t\t\t\t * ref so we need to process it.\n+\t\t\t\t */\n+\t\t\t\tbreak;\n+\t\t\tdefault:\n+\t\t\t\tdie(\"server is speaking an unknown protocol\");\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t}\n+\n \t\tif (!len)\n \t\t\tbreak;\n \n-- \n2.14.1.690.gbb1197296e-goog\n\n"},{"id":"328007","messageId":"20170913215448.84674-5-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH 4/8] upload-pack, receive-pack: introduce protocol version 1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-13T21:54:44Z","receivedAt":"2017-09-13T21:55:20Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach upload-pack and receive-pack to understand and respond using\nprotocol version 1, if requested.\n\nProtocol version 1 is simply the original and current protocol (what I'm\ncalling version 0) with the addition of a single packet line, which\nprecedes the ref advertisement, indicating the protocol version being\nspoken.\n\nSigned-off-by: Brandon Williams \n---\n builtin/receive-pack.c | 14 ++++++++++++++\n upload-pack.c | 17 ++++++++++++++++-\n 2 files changed, 30 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 52c63ebfd..aebe77cc3 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -24,6 +24,7 @@\n #include \"tmp-objdir.h\"\n #include \"oidset.h\"\n #include \"packfile.h\"\n+#include \"protocol.h\"\n \n static const char * const receive_pack_usage[] = {\n \tN_(\"git receive-pack \"),\n@@ -1963,6 +1964,19 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \telse if (0 <= receive_unpack_limit)\n \t\tunpack_limit = receive_unpack_limit;\n \n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\tcase protocol_v0:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\n \tif (advertise_refs || !stateless_rpc) {\n \t\twrite_head_info();\n \t}\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 7efff2fbf..5cab39819 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -18,6 +18,7 @@\n #include \"parse-options.h\"\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n+#include \"protocol.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -1067,6 +1068,20 @@ int cmd_main(int argc, const char **argv)\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n \tgit_config(upload_pack_config, NULL);\n-\tupload_pack();\n+\n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\tcase protocol_v0:\n+\tdefault:\n+\t\tupload_pack();\n+\t\tbreak;\n+\t}\n+\n \treturn 0;\n }\n-- \n2.14.1.690.gbb1197296e-goog\n\n"},{"id":"328008","messageId":"20170913215448.84674-7-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH 6/8] connect: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-13T21:54:46Z","receivedAt":"2017-09-13T21:55:23Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach the connection logic to tell a serve that it understands protocol\nv1. This is done in 2 different ways for the built in protocols.\n\n1. git://\n A normal request is structured as \"command path/to/repo\\0host=..\\0\"\n and due to a bug in an old version of git-daemon 73bb33a94 (daemon:\n Strictly parse the \"extra arg\" part of the command, 2009-06-04) we\n aren't able to place any extra args (separated by NULs) besides the\n host. In order to get around this limitation put protocol version\n information after a second NUL byte so the request is structured\n like: \"command path/to/repo\\0host=..\\0\\0version=1\\0\". git-daemon can\n then parse out the version number and set GIT_PROTOCOL.\n\n2. ssh://, file://\n Set GIT_PROTOCOL envvar with the desired protocol version. The\n envvar can be sent across ssh by using '-o SendEnv=GIT_PROTOCOL' and\n having the server whitelist this envvar.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 37 ++++++--\n t/t5700-protocol-v1.sh | 223 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 255 insertions(+), 5 deletions(-)\n create mode 100755 t/t5700-protocol-v1.sh\n\ndiff --git a/connect.c b/connect.c\nindex 2702e1f2e..40b388acd 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -815,6 +815,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tprintf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n \t\tconn = NULL;\n \t} else if (protocol == PROTO_GIT) {\n+\t\tstruct strbuf request = STRBUF_INIT;\n \t\t/*\n \t\t * Set up virtual host information based on where we will\n \t\t * connect, unless the user has overridden us in\n@@ -842,12 +843,24 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t * Note: Do not add any other headers here! Doing so\n \t\t * will cause older git-daemon servers to crash.\n \t\t */\n-\t\tpacket_write_fmt(fd[1],\n-\t\t\t \"%s %s%chost=%s%c\",\n-\t\t\t prog, path, 0,\n-\t\t\t target_host, 0);\n+\t\tstrbuf_addf(&request,\n+\t\t\t \"%s %s%chost=%s%c\",\n+\t\t\t prog, path, 0,\n+\t\t\t target_host, 0);\n+\n+\t\t/* If using a new version put that stuff here after a second null byte */\n+\t\tif (get_protocol_version_config() > 0) {\n+\t\t\tstrbuf_addch(&request, '\\0');\n+\t\t\tstrbuf_addf(&request, \"version=%d%c\",\n+\t\t\t\t get_protocol_version_config(), '\\0');\n+\t\t}\n+\n+\t\tpacket_write(fd[1], request.buf, request.len);\n+\n \t\tfree(target_host);\n+\t\tstrbuf_release(&request);\n \t} else {\n+\t\tconst char *const *var;\n \t\tconn = xmalloc(sizeof(*conn));\n \t\tchild_process_init(conn);\n \n@@ -859,7 +872,9 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tsq_quote_buf(&cmd, path);\n \n \t\t/* remove repo-local variables from the environment */\n-\t\tconn->env = local_repo_env;\n+\t\tfor (var = local_repo_env; *var; var++)\n+\t\t\targv_array_push(&conn->env_array, *var);\n+\n \t\tconn->use_shell = 1;\n \t\tconn->in = conn->out = -1;\n \t\tif (protocol == PROTO_SSH) {\n@@ -912,6 +927,14 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t}\n \n \t\t\targv_array_push(&conn->args, ssh);\n+\n+\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\t\targv_array_push(&conn->args, \"-o\");\n+\t\t\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n+\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t}\n+\n \t\t\tif (flags & CONNECT_IPV4)\n \t\t\t\targv_array_push(&conn->args, \"-4\");\n \t\t\telse if (flags & CONNECT_IPV6)\n@@ -926,6 +949,10 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\targv_array_push(&conn->args, ssh_host);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\n+\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t}\n \t\t}\n \t\targv_array_push(&conn->args, cmd.buf);\n \ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nnew file mode 100755\nindex 000000000..1988bbce6\n--- /dev/null\n+++ b/t/t5700-protocol-v1.sh\n@@ -0,0 +1,223 @@\n+#!/bin/sh\n+\n+test_description='test git wire-protocol transition'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+. ./test-lib.sh\n+\n+# Test protocol v1 with 'git://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\ttest_commit -C \"$daemon_parent\" one\n+'\n+\n+test_expect_success 'clone with git:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"$GIT_DAEMON_URL/parent\" daemon_child 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with git:// using protocol v1' '\n+\ttest_commit -C \"$daemon_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with git:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with git:// using protocol v1' '\n+\ttest_commit -C daemon_child three &&\n+\n+\t# Since the repository being served isnt bare we need to push to\n+\t# another branch explicitly to avoid mangling the master branch\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+stop_git_daemon\n+\n+# Test protocol v1 with 'file://' transport\n+#\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init file_parent &&\n+\ttest_commit -C file_parent one\n+'\n+\n+test_expect_success 'clone with file:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"file://$(pwd)/file_parent\" file_child 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with file:// using protocol v1' '\n+\ttest_commit -C file_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with file:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with file:// using protocol v1' '\n+\ttest_commit -C file_child three &&\n+\n+\t# Since the repository being served isnt bare we need to push to\n+\t# another branch explicitly to avoid mangling the master branch\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+# Test protocol v1 with 'ssh://' transport\n+#\n+test_expect_success 'setup ssh wrapper' '\n+\tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh$X\" &&\n+\texport GIT_SSH &&\n+\texport TRASH_DIRECTORY &&\n+\t>\"$TRASH_DIRECTORY\"/ssh-output\n+'\n+\n+expect_ssh () {\n+\ttest_when_finished '(cd \"$TRASH_DIRECTORY\" && rm -f ssh-expect && >ssh-output)' &&\n+\techo \"ssh: -o SendEnv=GIT_PROTOCOL myhost $1 '$PWD/ssh_parent'\" >\"$TRASH_DIRECTORY/ssh-expect\" &&\n+\t(cd \"$TRASH_DIRECTORY\" && test_cmp ssh-expect ssh-output)\n+}\n+\n+test_expect_success 'create repo to be served by ssh:// transport' '\n+\tgit init ssh_parent &&\n+\ttest_commit -C ssh_parent one\n+'\n+\n+test_expect_success 'clone with ssh:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"ssh://myhost:$(pwd)/ssh_parent\" ssh_child 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with ssh:// using protocol v1' '\n+\ttest_commit -C ssh_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with ssh:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with ssh:// using protocol v1' '\n+\ttest_commit -C ssh_child three &&\n+\n+\t# Since the repository being served isnt bare we need to push to\n+\t# another branch explicitly to avoid mangling the master branch\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\texpect_ssh git-receive-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+test_done\n-- \n2.14.1.690.gbb1197296e-goog\n\n"},{"id":"328009","messageId":"20170913215448.84674-9-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH 8/8] i5700: add interop test for protocol transition","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-13T21:54:48Z","receivedAt":"2017-09-13T21:55:28Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams \n---\n t/interop/i5700-protocol-transition.sh | 68 ++++++++++++++++++++++++++++++++++\n 1 file changed, 68 insertions(+)\n create mode 100755 t/interop/i5700-protocol-transition.sh\n\ndiff --git a/t/interop/i5700-protocol-transition.sh b/t/interop/i5700-protocol-transition.sh\nnew file mode 100755\nindex 000000000..9e83428a8\n--- /dev/null\n+++ b/t/interop/i5700-protocol-transition.sh\n@@ -0,0 +1,68 @@\n+#!/bin/sh\n+\n+VERSION_A=.\n+VERSION_B=v2.0.0\n+\n+: ${LIB_GIT_DAEMON_PORT:=5600}\n+LIB_GIT_DAEMON_COMMAND='git.b daemon'\n+\n+test_description='clone and fetch by client who is trying to use a new protocol'\n+. ./interop-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\n+start_git_daemon --export-all\n+\n+repo=$GIT_DAEMON_DOCUMENT_ROOT_PATH/repo\n+\n+test_expect_success \"create repo served by $VERSION_B\" '\n+\tgit.b init \"$repo\" &&\n+\tgit.b -C \"$repo\" commit --allow-empty -m one\n+'\n+\n+test_expect_success \"git:// clone with $VERSION_A and protocol v1\" '\n+\tGIT_TRACE_PACKET=1 git.a -c protocol.version=1 clone \"$GIT_DAEMON_URL/repo\" child 2>log &&\n+\tgit.a -C child log -1 --format=%s >actual &&\n+\tgit.b -C \"$repo\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\tgrep \"version=1\" log\n+'\n+\n+test_expect_success \"git:// fetch with $VERSION_A and protocol v1\" '\n+\tgit.b -C \"$repo\" commit --allow-empty -m two &&\n+\tgit.b -C \"$repo\" log -1 --format=%s >expect &&\n+\n+\tGIT_TRACE_PACKET=1 git.a -C child -c protocol.version=1 fetch 2>log &&\n+\tgit.a -C child log -1 --format=%s FETCH_HEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"version=1\" log &&\n+\t! grep \"version 1\" log\n+'\n+\n+stop_git_daemon\n+\n+test_expect_success \"create repo served by $VERSION_B\" '\n+\tgit.b init parent &&\n+\tgit.b -C parent commit --allow-empty -m one\n+'\n+\n+test_expect_success \"file:// clone with $VERSION_A and protocol v1\" '\n+\tGIT_TRACE_PACKET=1 git.a -c protocol.version=1 clone --upload-pack=\"git.b upload-pack\" parent child2 2>log &&\n+\tgit.a -C child2 log -1 --format=%s >actual &&\n+\tgit.b -C parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\t! grep \"version 1\" log\n+'\n+\n+test_expect_success \"file:// fetch with $VERSION_A and protocol v1\" '\n+\tgit.b -C parent commit --allow-empty -m two &&\n+\tgit.b -C parent log -1 --format=%s >expect &&\n+\n+\tGIT_TRACE_PACKET=1 git.a -C child2 -c protocol.version=1 fetch --upload-pack=\"git.b upload-pack\" 2>log &&\n+\tgit.a -C child2 log -1 --format=%s FETCH_HEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\t! grep \"version 1\" log\n+'\n+\n+test_done\n-- \n2.14.1.690.gbb1197296e-goog\n\n"},{"id":"328010","messageId":"20170913215448.84674-8-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH 7/8] http: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-13T21:54:47Z","receivedAt":"2017-09-13T21:55:30Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Tell a server that protocol v1 can be used by sending the http header\n'Git-Protocol' indicating this.\n\nAlso teach the apache http server to pass through the 'Git-Protocol'\nheader as an environment variable 'GIT_PROTOCOL'.\n\nSigned-off-by: Brandon Williams \n---\n cache.h | 2 ++\n http.c | 18 +++++++++++++\n t/lib-httpd/apache.conf | 7 +++++\n t/t5700-protocol-v1.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 96 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 8839b1ed4..82a643968 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -450,6 +450,8 @@ static inline enum object_type object_type(unsigned int mode)\n * 'key[=value]'. Presence of unknown keys must be tolerated.\n */\n #define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n+/* HTTP header used to handshake the wire protocol */\n+#define GIT_PROTOCOL_HEADER \"Git-Protocol\"\n \n /*\n * This environment variable is expected to contain a boolean indicating\ndiff --git a/http.c b/http.c\nindex 9e40a465f..ffb719216 100644\n--- a/http.c\n+++ b/http.c\n@@ -12,6 +12,7 @@\n #include \"gettext.h\"\n #include \"transport.h\"\n #include \"packfile.h\"\n+#include \"protocol.h\"\n \n static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);\n #if LIBCURL_VERSION_NUM >= 0x070a08\n@@ -897,6 +898,21 @@ static void set_from_env(const char **var, const char *envname)\n \t\t*var = val;\n }\n \n+static void protocol_http_header(void)\n+{\n+\tif (get_protocol_version_config() > 0) {\n+\t\tstruct strbuf protocol_header = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&protocol_header, GIT_PROTOCOL_HEADER \": version=%d\",\n+\t\t\t get_protocol_version_config());\n+\n+\n+\t\textra_http_headers = curl_slist_append(extra_http_headers,\n+\t\t\t\t\t\t protocol_header.buf);\n+\t\tstrbuf_release(&protocol_header);\n+\t}\n+}\n+\n void http_init(struct remote *remote, const char *url, int proactive_auth)\n {\n \tchar *low_speed_limit;\n@@ -927,6 +943,8 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tif (remote)\n \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n \n+\tprotocol_http_header();\n+\n \tpragma_header = curl_slist_append(http_copy_default_headers(),\n \t\t\"Pragma: no-cache\");\n \tno_pragma_header = curl_slist_append(http_copy_default_headers(),\ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 0642ae7e6..df1943631 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -67,6 +67,9 @@ LockFile accept.lock\n \n \tLoadModule unixd_module modules/mod_unixd.so\n \n+\n+\tLoadModule setenvif_module modules/mod_setenvif.so\n+\n \n \n PassEnv GIT_VALGRIND\n@@ -76,6 +79,10 @@ PassEnv ASAN_OPTIONS\n PassEnv GIT_TRACE\n PassEnv GIT_CONFIG_NOSYSTEM\n \n+= 2.4>\n+\tSetEnvIf Git-Protocol \".*\" GIT_PROTOCOL=$0\n+\n+\n Alias /dumb/ www/\n Alias /auth/dumb/ www/auth/dumb/\n \ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nindex 1988bbce6..222265127 100755\n--- a/t/t5700-protocol-v1.sh\n+++ b/t/t5700-protocol-v1.sh\n@@ -220,4 +220,73 @@ test_expect_success 'push with ssh:// using protocol v1' '\n \tgrep \"push< version 1\" log\n '\n \n+# Test protocol v1 with 'http://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one\n+'\n+\n+test_expect_success 'clone with http:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 GIT_TRACE_CURL=1 git -c protocol.version=1 \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"Git-Protocol: version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'fetch with http:// using protocol v1' '\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'pull with http:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'push with http:// using protocol v1' '\n+\ttest_commit -C http_child three &&\n+\n+\t# Since the repository being served isnt bare we need to push to\n+\t# another branch explicitly to avoid mangling the master branch\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch && #2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.14.1.690.gbb1197296e-goog\n\n"},{"id":"328015","messageId":"CAGZ79kaeWsWdhoC7VDy-YkL1wVWqaQ9=TQUti9HfwjbT2poQdw@mail.gmail.com","threadId":"46746","inReplyTo":"20170913215448.84674-3-bmwill@google.com","subject":"Re: [PATCH 2/8] protocol: introduce protocol extention mechanisms","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-09-13T22:27:11Z","receivedAt":"2017-09-13T22:27:19Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Sep 13, 2017 at 2:54 PM, Brandon Williams wrote:\n> Create protocol.{c,h} and provide functions which future servers and\n> clients can use to determine which protocol to use or is being used.\n>\n> Also introduce the 'GIT_PROTOCOL' environment variable which will be\n> used to communicate a colon separated list of keys with optional values\n> to a server. Unknown keys and values must be tolerated. This mechanism\n> is used to communicate which version of the wire protocol a client would\n> like to use with a server.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> Documentation/config.txt | 16 +++++++++++\n> Documentation/git.txt | 5 ++++\n> Makefile | 1 +\n> cache.h | 7 +++++\n> protocol.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++\n> protocol.h | 15 ++++++++++\n> 6 files changed, 116 insertions(+)\n> create mode 100644 protocol.c\n> create mode 100644 protocol.h\n>\n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index dc4e3f58a..d5b28a32c 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -2517,6 +2517,22 @@ The protocol names currently used by git are:\n> `hg` to allow the `git-remote-hg` helper)\n> --\n>\n> +protocol.version::\n\nIt would be cool to set a set of versions that are good. (I am not sure\nif that can be deferred to a later patch.)\n\n Consider we'd have versions 0,1,2,3,4 in the future:\n In an ideal world the client and server would talk using v4\n as it is the most advanced protocol, right?\n Maybe a security/performance issue is found on the server side\n with say protocol v3. Still no big deal as we are speaking v4.\n But then consider an issue is found on the client side with v4.\n Then the client would happily talk 0..3 while the server would\n love to talk using 0,1,2,4.\n\nThe way I think about protocol version negotiation is that\nboth parties involved have a set of versions that they tolerate\nto talk (they might understand more than the tolerated set, but the\nuser forbade some), and the goal of the negotiation is to find\nthe highest version number that is part of both the server set\nand client set. So quite naturally with this line of thinking the\nconfiguration is to configure a set of versions, which is what\nI propose here. Maybe even in the wire format, separated\nwith colons?\n\n> + If set, clients will attempt to communicate with a server using\n> + the specified protocol version. If unset, no attempt will be\n> + made by the client to communicate using a particular protocol\n> + version, this results in protocol version 0 being used.\n\nThis sounds as if we're going to be really shy at first and only\nusers that care will try out new versions at their own risk.\nFrom a users POV this may be frustrating as I would imagine that\npeople want to run\n\n git config --global protocol.version 2\n\nto try it out and then realize that some of their hosts do not speak\n2, so they have to actually configure it per repo/remote.\n\n> + Supported versions:\n\n> +* `0` - the original wire protocol.\n\nIn the future this may be misleading as it doesn't specify the date of\nwhen it was original. e.g. are capabilities already supported in \"original\"?\n\nMaybe phrase it as \"wire protocol as of v2.14\" ? (Though this sounds\nas if new capabilities added in the future are not allowed)\n\n\n> +\n> +extern enum protocol_version parse_protocol_version(const char *value);\n> +extern enum protocol_version get_protocol_version_config(void);\n> +extern enum protocol_version determine_protocol_version_server(void);\n> +extern enum protocol_version determine_protocol_version_client(const char *server_response);\n\nHere is a good place to have some comments.\n"},{"id":"328017","messageId":"CAGZ79kZkGfM=2Nek66gvZnbMAu2HLUkbd4D4S5Rij1D_t5DYKg@mail.gmail.com","threadId":"46746","inReplyTo":"20170913215448.84674-4-bmwill@google.com","subject":"Re: [PATCH 3/8] daemon: recognize hidden request arguments","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-09-13T22:31:54Z","receivedAt":"2017-09-13T22:39:30Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Sep 13, 2017 at 2:54 PM, Brandon Williams wrote:\n> A normal request to git-daemon is structured as\n> \"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\n> git-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\n> command, 2009-06-04) we aren't able to place any extra args (separated\n> by NULs) besides the host.\n>\n> In order to get around this limitation teach git-daemon to recognize\n> additional request arguments hidden behind a second NUL byte. Requests\n> can then be structured like:\n> \"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\n> can then parse out the extra arguments and set 'GIT_PROTOCOL'\n> accordingly.\n>\n> Signed-off-by: Brandon Williams \n> ---\n> daemon.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++---------\n> 1 file changed, 61 insertions(+), 10 deletions(-)\n>\n> diff --git a/daemon.c b/daemon.c\n> index 30747075f..250dbf82c 100644\n> --- a/daemon.c\n> +++ b/daemon.c\n> @@ -282,7 +282,7 @@ static const char *path_ok(const char *directory, struct hostinfo *hi)\n> return NULL; /* Fallthrough. Deny by default */\n> }\n>\n> -typedef int (*daemon_service_fn)(void);\n> +typedef int (*daemon_service_fn)(const struct argv_array *env);\n> struct daemon_service {\n> const char *name;\n> const char *config_name;\n> @@ -363,7 +363,7 @@ static int run_access_hook(struct daemon_service *service, const char *dir,\n> }\n>\n> static int run_service(const char *dir, struct daemon_service *service,\n> - struct hostinfo *hi)\n> + struct hostinfo *hi, const struct argv_array *env)\n> {\n> const char *path;\n> int enabled = service->enabled;\n> @@ -422,7 +422,7 @@ static int run_service(const char *dir, struct daemon_service *service,\n> */\n> signal(SIGTERM, SIG_IGN);\n>\n> - return service->fn();\n> + return service->fn(env);\n> }\n>\n> static void copy_to_log(int fd)\n> @@ -462,25 +462,34 @@ static int run_service_command(struct child_process *cld)\n> return finish_command(cld);\n> }\n>\n> -static int upload_pack(void)\n> +static int upload_pack(const struct argv_array *env)\n> {\n> struct child_process cld = CHILD_PROCESS_INIT;\n> argv_array_pushl(&cld.args, \"upload-pack\", \"--strict\", NULL);\n> argv_array_pushf(&cld.args, \"--timeout=%u\", timeout);\n> +\n> + argv_array_pushv(&cld.env_array, env->argv);\n> +\n> return run_service_command(&cld);\n> }\n>\n> -static int upload_archive(void)\n> +static int upload_archive(const struct argv_array *env)\n> {\n> struct child_process cld = CHILD_PROCESS_INIT;\n> argv_array_push(&cld.args, \"upload-archive\");\n> +\n> + argv_array_pushv(&cld.env_array, env->argv);\n> +\n> return run_service_command(&cld);\n> }\n>\n> -static int receive_pack(void)\n> +static int receive_pack(const struct argv_array *env)\n> {\n> struct child_process cld = CHILD_PROCESS_INIT;\n> argv_array_push(&cld.args, \"receive-pack\");\n> +\n> + argv_array_pushv(&cld.env_array, env->argv);\n> +\n> return run_service_command(&cld);\n> }\n>\n> @@ -574,7 +583,7 @@ static void canonicalize_client(struct strbuf *out, const char *in)\n> /*\n> * Read the host as supplied by the client connection.\n> */\n> -static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> +static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> {\n> char *val;\n> int vallen;\n> @@ -602,6 +611,39 @@ static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> if (extra_args < end && *extra_args)\n> die(\"Invalid request\");\n> }\n> +\n> + return extra_args;\n> +}\n> +\n> +static void parse_extra_args(struct argv_array *env, const char *extra_args,\n> + int buflen)\n> +{\n> + const char *end = extra_args + buflen;\n> + struct strbuf git_protocol = STRBUF_INIT;\n> +\n> + for (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n> + const char *arg = extra_args;\n> +\n> + /*\n> + * Parse the extra arguments, adding most to 'git_protocol'\n> + * which will be used to set the 'GIT_PROTOCOL' envvar in the\n> + * service that will be run.\n> + *\n> + * If there ends up being a particular arg in the future that\n> + * git-daemon needs to parse specificly (like the 'host' arg)\n> + * then it can be parsed here and not added to 'git_protocol'.\n> + */\n> + if (*arg) {\n> + if (git_protocol.len > 0)\n> + strbuf_addch(&git_protocol, ':');\n> + strbuf_addstr(&git_protocol, arg);\n> + }\n> + }\n> +\n> + if (git_protocol.len > 0)\n> + argv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n> + git_protocol.buf);\n> + strbuf_release(&git_protocol);\n> }\n\nI wonder if this could be written as\n\n begin = extra_args;\n p = extra_args;\n end = extra_args + buflen;\n\n while (p < end) {\n if (!*p)\n *p = ':';\n p++;\n }\n argv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\", begin);\n\nto ease the load on the server side, as then we do not\nhave to copy the partial strings into strbufs and then\ncount the length individually? (maybe performance is no big deal here?)\n\n\n>\n> /*\n> @@ -695,6 +737,7 @@ static int execute(void)\n> int pktlen, len, i;\n> char *addr = getenv(\"REMOTE_ADDR\"), *port = getenv(\"REMOTE_PORT\");\n> struct hostinfo hi;\n> + struct argv_array env = ARGV_ARRAY_INIT;\n>\n> hostinfo_init(&hi);\n>\n> @@ -716,8 +759,14 @@ static int execute(void)\n> pktlen--;\n> }\n>\n> - if (len != pktlen)\n> - parse_host_arg(&hi, line + len + 1, pktlen - len - 1);\n> + if (len != pktlen) {\n> + const char *extra_args;\n> + /* retrieve host */\n> + extra_args = parse_host_arg(&hi, line + len + 1, pktlen - len - 1);\n> +\n> + /* parse additional args hidden behind a second NUL byte */\n> + parse_extra_args(&env, extra_args + 1, pktlen - (extra_args - line) - 1);\n> + }\n>\n> for (i = 0; i < ARRAY_SIZE(daemon_service); i++) {\n> struct daemon_service *s = &(daemon_service[i]);\n> @@ -730,13 +779,15 @@ static int execute(void)\n> * Note: The directory here is probably context sensitive,\n> * and might depend on the actual service being performed.\n> */\n> - int rc = run_service(arg, s, &hi);\n> + int rc = run_service(arg, s, &hi, &env);\n> hostinfo_clear(&hi);\n> + argv_array_clear(&env);\n> return rc;\n> }\n> }\n>\n> hostinfo_clear(&hi);\n> + argv_array_clear(&env);\n> logerror(\"Protocol error: '%s'\", line);\n> return -1;\n> }\n> --\n> 2.14.1.690.gbb1197296e-goog\n>\n"},{"id":"328307","messageId":"20170918165615.GE144331@google.com","threadId":"46746","inReplyTo":"CAGZ79kZkGfM=2Nek66gvZnbMAu2HLUkbd4D4S5Rij1D_t5DYKg@mail.gmail.com","subject":"Re: [PATCH 3/8] daemon: recognize hidden request arguments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-18T16:56:15Z","receivedAt":"2017-09-18T16:56:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/13, Stefan Beller wrote:\n> On Wed, Sep 13, 2017 at 2:54 PM, Brandon Williams wrote:\n> > A normal request to git-daemon is structured as\n> > \"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\n> > git-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\n> > command, 2009-06-04) we aren't able to place any extra args (separated\n> > by NULs) besides the host.\n> >\n> > In order to get around this limitation teach git-daemon to recognize\n> > additional request arguments hidden behind a second NUL byte. Requests\n> > can then be structured like:\n> > \"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\n> > can then parse out the extra arguments and set 'GIT_PROTOCOL'\n> > accordingly.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > daemon.c | 71 +++++++++++++++++++++++++++++++++++++++++++++++++++++++---------\n> > 1 file changed, 61 insertions(+), 10 deletions(-)\n> >\n> > diff --git a/daemon.c b/daemon.c\n> > index 30747075f..250dbf82c 100644\n> > --- a/daemon.c\n> > +++ b/daemon.c\n> > @@ -282,7 +282,7 @@ static const char *path_ok(const char *directory, struct hostinfo *hi)\n> > return NULL; /* Fallthrough. Deny by default */\n> > }\n> >\n> > -typedef int (*daemon_service_fn)(void);\n> > +typedef int (*daemon_service_fn)(const struct argv_array *env);\n> > struct daemon_service {\n> > const char *name;\n> > const char *config_name;\n> > @@ -363,7 +363,7 @@ static int run_access_hook(struct daemon_service *service, const char *dir,\n> > }\n> >\n> > static int run_service(const char *dir, struct daemon_service *service,\n> > - struct hostinfo *hi)\n> > + struct hostinfo *hi, const struct argv_array *env)\n> > {\n> > const char *path;\n> > int enabled = service->enabled;\n> > @@ -422,7 +422,7 @@ static int run_service(const char *dir, struct daemon_service *service,\n> > */\n> > signal(SIGTERM, SIG_IGN);\n> >\n> > - return service->fn();\n> > + return service->fn(env);\n> > }\n> >\n> > static void copy_to_log(int fd)\n> > @@ -462,25 +462,34 @@ static int run_service_command(struct child_process *cld)\n> > return finish_command(cld);\n> > }\n> >\n> > -static int upload_pack(void)\n> > +static int upload_pack(const struct argv_array *env)\n> > {\n> > struct child_process cld = CHILD_PROCESS_INIT;\n> > argv_array_pushl(&cld.args, \"upload-pack\", \"--strict\", NULL);\n> > argv_array_pushf(&cld.args, \"--timeout=%u\", timeout);\n> > +\n> > + argv_array_pushv(&cld.env_array, env->argv);\n> > +\n> > return run_service_command(&cld);\n> > }\n> >\n> > -static int upload_archive(void)\n> > +static int upload_archive(const struct argv_array *env)\n> > {\n> > struct child_process cld = CHILD_PROCESS_INIT;\n> > argv_array_push(&cld.args, \"upload-archive\");\n> > +\n> > + argv_array_pushv(&cld.env_array, env->argv);\n> > +\n> > return run_service_command(&cld);\n> > }\n> >\n> > -static int receive_pack(void)\n> > +static int receive_pack(const struct argv_array *env)\n> > {\n> > struct child_process cld = CHILD_PROCESS_INIT;\n> > argv_array_push(&cld.args, \"receive-pack\");\n> > +\n> > + argv_array_pushv(&cld.env_array, env->argv);\n> > +\n> > return run_service_command(&cld);\n> > }\n> >\n> > @@ -574,7 +583,7 @@ static void canonicalize_client(struct strbuf *out, const char *in)\n> > /*\n> > * Read the host as supplied by the client connection.\n> > */\n> > -static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> > +static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> > {\n> > char *val;\n> > int vallen;\n> > @@ -602,6 +611,39 @@ static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> > if (extra_args < end && *extra_args)\n> > die(\"Invalid request\");\n> > }\n> > +\n> > + return extra_args;\n> > +}\n> > +\n> > +static void parse_extra_args(struct argv_array *env, const char *extra_args,\n> > + int buflen)\n> > +{\n> > + const char *end = extra_args + buflen;\n> > + struct strbuf git_protocol = STRBUF_INIT;\n> > +\n> > + for (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n> > + const char *arg = extra_args;\n> > +\n> > + /*\n> > + * Parse the extra arguments, adding most to 'git_protocol'\n> > + * which will be used to set the 'GIT_PROTOCOL' envvar in the\n> > + * service that will be run.\n> > + *\n> > + * If there ends up being a particular arg in the future that\n> > + * git-daemon needs to parse specificly (like the 'host' arg)\n> > + * then it can be parsed here and not added to 'git_protocol'.\n> > + */\n> > + if (*arg) {\n> > + if (git_protocol.len > 0)\n> > + strbuf_addch(&git_protocol, ':');\n> > + strbuf_addstr(&git_protocol, arg);\n> > + }\n> > + }\n> > +\n> > + if (git_protocol.len > 0)\n> > + argv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n> > + git_protocol.buf);\n> > + strbuf_release(&git_protocol);\n> > }\n> \n> I wonder if this could be written as\n> \n> begin = extra_args;\n> p = extra_args;\n> end = extra_args + buflen;\n> \n> while (p < end) {\n> if (!*p)\n> *p = ':';\n> p++;\n> }\n> argv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\", begin);\n> \n> to ease the load on the server side, as then we do not\n> have to copy the partial strings into strbufs and then\n> count the length individually? (maybe performance is no big deal here?)\n\nI'm sure something like that could work, and I don't know how\nperformance sensitive this bit is. That and depending on if we need the\nunmodified string for anything at a later point maybe its best to not\nmodify it in place? I don't know :)\n\n> \n> \n> >\n> > /*\n> > @@ -695,6 +737,7 @@ static int execute(void)\n> > int pktlen, len, i;\n> > char *addr = getenv(\"REMOTE_ADDR\"), *port = getenv(\"REMOTE_PORT\");\n> > struct hostinfo hi;\n> > + struct argv_array env = ARGV_ARRAY_INIT;\n> >\n> > hostinfo_init(&hi);\n> >\n> > @@ -716,8 +759,14 @@ static int execute(void)\n> > pktlen--;\n> > }\n> >\n> > - if (len != pktlen)\n> > - parse_host_arg(&hi, line + len + 1, pktlen - len - 1);\n> > + if (len != pktlen) {\n> > + const char *extra_args;\n> > + /* retrieve host */\n> > + extra_args = parse_host_arg(&hi, line + len + 1, pktlen - len - 1);\n> > +\n> > + /* parse additional args hidden behind a second NUL byte */\n> > + parse_extra_args(&env, extra_args + 1, pktlen - (extra_args - line) - 1);\n> > + }\n> >\n> > for (i = 0; i < ARRAY_SIZE(daemon_service); i++) {\n> > struct daemon_service *s = &(daemon_service[i]);\n> > @@ -730,13 +779,15 @@ static int execute(void)\n> > * Note: The directory here is probably context sensitive,\n> > * and might depend on the actual service being performed.\n> > */\n> > - int rc = run_service(arg, s, &hi);\n> > + int rc = run_service(arg, s, &hi, &env);\n> > hostinfo_clear(&hi);\n> > + argv_array_clear(&env);\n> > return rc;\n> > }\n> > }\n> >\n> > hostinfo_clear(&hi);\n> > + argv_array_clear(&env);\n> > logerror(\"Protocol error: '%s'\", line);\n> > return -1;\n> > }\n> > --\n> > 2.14.1.690.gbb1197296e-goog\n> >\n\n-- \nBrandon Williams\n"},{"id":"328308","messageId":"20170918170207.GF144331@google.com","threadId":"46746","inReplyTo":"CAGZ79kaeWsWdhoC7VDy-YkL1wVWqaQ9=TQUti9HfwjbT2poQdw@mail.gmail.com","subject":"Re: [PATCH 2/8] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-18T17:02:07Z","receivedAt":"2017-09-18T17:02:15Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/13, Stefan Beller wrote:\n> On Wed, Sep 13, 2017 at 2:54 PM, Brandon Williams wrote:\n> > Create protocol.{c,h} and provide functions which future servers and\n> > clients can use to determine which protocol to use or is being used.\n> >\n> > Also introduce the 'GIT_PROTOCOL' environment variable which will be\n> > used to communicate a colon separated list of keys with optional values\n> > to a server. Unknown keys and values must be tolerated. This mechanism\n> > is used to communicate which version of the wire protocol a client would\n> > like to use with a server.\n> >\n> > Signed-off-by: Brandon Williams \n> > ---\n> > Documentation/config.txt | 16 +++++++++++\n> > Documentation/git.txt | 5 ++++\n> > Makefile | 1 +\n> > cache.h | 7 +++++\n> > protocol.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++\n> > protocol.h | 15 ++++++++++\n> > 6 files changed, 116 insertions(+)\n> > create mode 100644 protocol.c\n> > create mode 100644 protocol.h\n> >\n> > diff --git a/Documentation/config.txt b/Documentation/config.txt\n> > index dc4e3f58a..d5b28a32c 100644\n> > --- a/Documentation/config.txt\n> > +++ b/Documentation/config.txt\n> > @@ -2517,6 +2517,22 @@ The protocol names currently used by git are:\n> > `hg` to allow the `git-remote-hg` helper)\n> > --\n> >\n> > +protocol.version::\n> \n> It would be cool to set a set of versions that are good. (I am not sure\n> if that can be deferred to a later patch.)\n> \n> Consider we'd have versions 0,1,2,3,4 in the future:\n> In an ideal world the client and server would talk using v4\n> as it is the most advanced protocol, right?\n> Maybe a security/performance issue is found on the server side\n> with say protocol v3. Still no big deal as we are speaking v4.\n> But then consider an issue is found on the client side with v4.\n> Then the client would happily talk 0..3 while the server would\n> love to talk using 0,1,2,4.\n> \n> The way I think about protocol version negotiation is that\n> both parties involved have a set of versions that they tolerate\n> to talk (they might understand more than the tolerated set, but the\n> user forbade some), and the goal of the negotiation is to find\n> the highest version number that is part of both the server set\n> and client set. So quite naturally with this line of thinking the\n> configuration is to configure a set of versions, which is what\n> I propose here. Maybe even in the wire format, separated\n> with colons?\n\nI'm sure it wouldn't take too much to change this to be a multi-valued\nconfig. Because after this series there is just v0 and v1 I didn't\nthink through this case too much. If others agree then I can go ahead\nand make it so in a reroll.\n\n> \n> > + If set, clients will attempt to communicate with a server using\n> > + the specified protocol version. If unset, no attempt will be\n> > + made by the client to communicate using a particular protocol\n> > + version, this results in protocol version 0 being used.\n> \n> This sounds as if we're going to be really shy at first and only\n> users that care will try out new versions at their own risk.\n> From a users POV this may be frustrating as I would imagine that\n> people want to run\n> \n> git config --global protocol.version 2\n> \n> to try it out and then realize that some of their hosts do not speak\n> 2, so they have to actually configure it per repo/remote.\n\nThe point would be to be able to set this globally, not per-repo. Even\nif a repo doesn't speak v2 then it should be able to gracefully degrade\nto v1 without the user having to do anything. The reason why there is\nthis escape hatch is if doing the protocol negotiation out of band\ncausing issues with communicating with a server that it can be shut off.\n\n\n> > + Supported versions:\n> \n> > +* `0` - the original wire protocol.\n> \n> In the future this may be misleading as it doesn't specify the date of\n> when it was original. e.g. are capabilities already supported in \"original\"?\n> \n> Maybe phrase it as \"wire protocol as of v2.14\" ? (Though this sounds\n> as if new capabilities added in the future are not allowed)\n\nYeah I can see how this could be misleading, though I'm not sure how\nbest to word it to avoid that.\n\n> \n> \n> > +\n> > +extern enum protocol_version parse_protocol_version(const char *value);\n> > +extern enum protocol_version get_protocol_version_config(void);\n> > +extern enum protocol_version determine_protocol_version_server(void);\n> > +extern enum protocol_version determine_protocol_version_client(const char *server_response);\n> \n> Here is a good place to have some comments.\n\n-- \nBrandon Williams\n"},{"id":"328316","messageId":"CAGZ79kYQGguodnaiNfT_vWWDQdDdXwZmTi7JnqOmz+Po+FL1+Q@mail.gmail.com","threadId":"46746","inReplyTo":"20170918170207.GF144331@google.com","subject":"Re: [PATCH 2/8] protocol: introduce protocol extention mechanisms","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-09-18T18:34:35Z","receivedAt":"2017-09-18T18:34:42Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">> From a users POV this may be frustrating as I would imagine that\n>> people want to run\n>>\n>> git config --global protocol.version 2\n>>\n>> to try it out and then realize that some of their hosts do not speak\n>> 2, so they have to actually configure it per repo/remote.\n>\n> The point would be to be able to set this globally, not per-repo. Even\n> if a repo doesn't speak v2 then it should be able to gracefully degrade\n> to v1 without the user having to do anything. The reason why there is\n> this escape hatch is if doing the protocol negotiation out of band\n> causing issues with communicating with a server that it can be shut off.\n\nIn the current situation it is easy to assume that if v1 (and not v0)\nis configured, that the users intent is \"to try out v1 and fallback\ngracefully to v0\".\n\nBut this will change over time in the future!\n\nEventually people will have the desire to say:\n\"Use version N+1, but never version N\", because N has\nperformance or security issues; the user might not want\nto bother to try N or even actively want to be affirmed that\nGit will never use version N.\n\nIn this future we need a mechanism, that either contains a\nwhite list or black list of protocols. To keep it simple (I assume\nthere won't be many protocol versions), a single white list will do.\n\nHowever transitioning from the currently proposed \"try the new\nconfigured thing and fallback to whatever\" to \"this is the exact list\nof options that Git will try for you\" will be hard, as we may break people\nif we do not unconditionally fall back to v0.\n\nThat is why I propose to start with an explicit white list as then we\ndo not have to have a transition plan or otherwise work around the\nissue. Also it doesn't hurt now to use\n\n git config --global protocol.version v1,v0\n\ninstead compared to the proposed configuration above.\n(Even better yet, then people could play around with \"v1 only\"\nand see how it falls apart on old servers)\n"},{"id":"328317","messageId":"20170918195840.GG144331@google.com","threadId":"46746","inReplyTo":"CAGZ79kYQGguodnaiNfT_vWWDQdDdXwZmTi7JnqOmz+Po+FL1+Q@mail.gmail.com","subject":"Re: [PATCH 2/8] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-18T19:58:40Z","receivedAt":"2017-09-18T19:58:48Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/18, Stefan Beller wrote:\n> >> From a users POV this may be frustrating as I would imagine that\n> >> people want to run\n> >>\n> >> git config --global protocol.version 2\n> >>\n> >> to try it out and then realize that some of their hosts do not speak\n> >> 2, so they have to actually configure it per repo/remote.\n> >\n> > The point would be to be able to set this globally, not per-repo. Even\n> > if a repo doesn't speak v2 then it should be able to gracefully degrade\n> > to v1 without the user having to do anything. The reason why there is\n> > this escape hatch is if doing the protocol negotiation out of band\n> > causing issues with communicating with a server that it can be shut off.\n> \n> In the current situation it is easy to assume that if v1 (and not v0)\n> is configured, that the users intent is \"to try out v1 and fallback\n> gracefully to v0\".\n> \n> But this will change over time in the future!\n> \n> Eventually people will have the desire to say:\n> \"Use version N+1, but never version N\", because N has\n> performance or security issues; the user might not want\n> to bother to try N or even actively want to be affirmed that\n> Git will never use version N.\n> \n> In this future we need a mechanism, that either contains a\n> white list or black list of protocols. To keep it simple (I assume\n> there won't be many protocol versions), a single white list will do.\n> \n> However transitioning from the currently proposed \"try the new\n> configured thing and fallback to whatever\" to \"this is the exact list\n> of options that Git will try for you\" will be hard, as we may break people\n> if we do not unconditionally fall back to v0.\n> \n> That is why I propose to start with an explicit white list as then we\n> do not have to have a transition plan or otherwise work around the\n> issue. Also it doesn't hurt now to use\n> \n> git config --global protocol.version v1,v0\n> \n> instead compared to the proposed configuration above.\n> (Even better yet, then people could play around with \"v1 only\"\n> and see how it falls apart on old servers)\n\nExcept we can't start with an explicit whitelist because we must\nfallback to v0 if v1 isn't supported otherwise we would break people.\n\nThat is unless we have the semantics of: If not configured v0 will be\nused, otherwise only use the configured protocol versions.\n\n-- \nBrandon Williams\n"},{"id":"328319","messageId":"CAGZ79kZ1odSp5O+uF_-ZXZfD4M2r=pZD1LADQ9Thfgki1b7nYw@mail.gmail.com","threadId":"46746","inReplyTo":"20170918195840.GG144331@google.com","subject":"Re: [PATCH 2/8] protocol: introduce protocol extention mechanisms","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-09-18T20:06:59Z","receivedAt":"2017-09-18T20:07:06Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">> instead compared to the proposed configuration above.\n>> (Even better yet, then people could play around with \"v1 only\"\n>> and see how it falls apart on old servers)\n>\n> Except we can't start with an explicit whitelist because we must\n> fallback to v0 if v1 isn't supported otherwise we would break people.\n>\n> That is unless we have the semantics of: If not configured v0 will be\n> used, otherwise only use the configured protocol versions.\n>\n\nGood point.\n\nThinking about this, how about:\n\n If not configured, we do as we want. (i.e. Git has full control over\n it's decision making process, which for now is \"favor v0 over v1 as\n we are experimenting with v1\". This strategy may change in the future\n to \"prefer highest version number that both client and server can speak\".)\n\n If it is configured, \"use highest configured number from the given set\".\n\n?\n"},{"id":"328482","messageId":"20170920184832.146564-1-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH 1.5/8] connect: die when a capability line comes after a ref","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-20T18:48:32Z","receivedAt":"2017-09-20T18:48:45Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Commit eb398797c (connect: advertized capability is not a ref,\n2016-09-09) taught 'get_remote_heads()' to recognize that the\n'capabilities^{}' line isn't a ref but required that the\n'capabilities^{}' line came during the first response from the server.\nA future patch will introduce a version string sent by the server during\nits first response which can then cause a client to unnecessarily die if\na 'capabilities^{}' line sent as the first ref.\n\nTeach 'get_remote_heads()' to instead die if a 'capabilities^{}' line is\nsent after a ref.\n\nReported-by: Miguel Alcon \nSigned-off-by: Brandon Williams \n---\nThis is a fix to the bug we found when internally deploying this series. It\njust makes it so that a capability line wont cause a client to error out if its\nnot the first response, because it won't be the first response come protocol\nv1.\n\n connect.c | 8 ++++++--\n 1 file changed, 6 insertions(+), 2 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex df56c0cbf..af5096ec6 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -124,10 +124,11 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t * response does not necessarily mean an ACL problem, though.\n \t */\n \tint saw_response;\n+\tint seen_ref;\n \tint got_dummy_ref_with_capabilities_declaration = 0;\n\n \t*list = NULL;\n-\tfor (saw_response = 0; ; saw_response = 1) {\n+\tfor (saw_response = 0, seen_ref = 0; ; saw_response = 1) {\n \t\tstruct ref *ref;\n \t\tstruct object_id old_oid;\n \t\tchar *name;\n@@ -165,6 +166,8 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n\n \t\tname_len = strlen(name);\n \t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n+\t\t\tif (seen_ref)\n+\t\t\t\t; /* NEEDSWORK: Error out for multiple capabilities lines? */\n \t\t\tfree(server_capabilities);\n \t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n \t\t}\n@@ -175,7 +178,7 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\t}\n\n \t\tif (!strcmp(name, \"capabilities^{}\")) {\n-\t\t\tif (saw_response)\n+\t\t\tif (seen_ref)\n \t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n \t\t\tif (got_dummy_ref_with_capabilities_declaration)\n \t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n@@ -193,6 +196,7 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t\toidcpy(&ref->old_oid, &old_oid);\n \t\t*list = ref;\n \t\tlist = &ref->next;\n+\t\tseen_ref = 1;\n \t}\n\n \tannotate_refs_with_symref_info(*orig_list);\n--\n2.14.1.821.g8fa685d3b7-goog\n\n"},{"id":"328486","messageId":"20170920191431.emornzhse6f7s5mg@sigill.intra.peff.net","threadId":"46746","inReplyTo":"20170920184832.146564-1-bmwill@google.com","subject":"Re: [PATCH 1.5/8] connect: die when a capability line comes after a ref","fromName":"Jeff King","fromEmail":"peff@peff.net","sentAt":"2017-09-20T19:14:31Z","receivedAt":"2017-09-20T19:14:38Z","isPatch":true,"sender":{"key":"peff@peff.net","avatar":"https://avatars.githubusercontent.com/u/45925?v=4"},"body":"On Wed, Sep 20, 2017 at 11:48:32AM -0700, Brandon Williams wrote:\n\n> Commit eb398797c (connect: advertized capability is not a ref,\n> 2016-09-09) taught 'get_remote_heads()' to recognize that the\n> 'capabilities^{}' line isn't a ref but required that the\n> 'capabilities^{}' line came during the first response from the server.\n> A future patch will introduce a version string sent by the server during\n> its first response which can then cause a client to unnecessarily die if\n> a 'capabilities^{}' line sent as the first ref.\n> \n> Teach 'get_remote_heads()' to instead die if a 'capabilities^{}' line is\n> sent after a ref.\n\nHmm. I think I understand why you'd want this loosening. But why are we\nsending a version line to a client that we don't know is speaking v2?\nIOW, shouldn't we be reporting the version to the client in the normal\ncapabilities when we don't know for sure that they can handle the new\nfield? Otherwise we're breaking existing clients.\n\nOr is this only for v2 clients, and we've changed the protocol but\nget_remote_heads() just needs to be updated, too?\n\n> diff --git a/connect.c b/connect.c\n> index df56c0cbf..af5096ec6 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -124,10 +124,11 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t * response does not necessarily mean an ACL problem, though.\n> \t */\n> \tint saw_response;\n> +\tint seen_ref;\n> \tint got_dummy_ref_with_capabilities_declaration = 0;\n> \n> \t*list = NULL;\n> -\tfor (saw_response = 0; ; saw_response = 1) {\n> +\tfor (saw_response = 0, seen_ref = 0; ; saw_response = 1) {\n\nIf we're not going to update it in the right-hand side of the for-loop,\nshould we perhaps not be initializing it in the left-hand side? I.e.,\ncan we just do:\n\n seen_ref = 0;\n\nabove the loop, like we initialize \"list\"?\n\n(For that matter, could we just be checking whether *list is NULL?)\n\n> @@ -165,6 +166,8 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \n> \t\tname_len = strlen(name);\n> \t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n> +\t\t\tif (seen_ref)\n> +\t\t\t\t; /* NEEDSWORK: Error out for multiple capabilities lines? */\n> \t\t\tfree(server_capabilities);\n> \t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n> \t\t}\n\nInteresting question. Probably it would be fine to. Coincidentally I ran\nacross a similar case. It seems that upload-pack will read multiple\ncapabilities lines back from the client. I.e., if it gets:\n\n want 1234abcd... foo\n want 5678abcd... bar\n\nthen it will turn on both the \"foo\" and \"bar\" capabilities. I'm pretty\nsure this is unintended, and is somewhat counter to the way that clients\nhandle multiple lines (which is to forget the old line and respect only\nthe new one, as shown in the quoted hunk).\n\nI wonder if we should be outlawing extra capabilities in both\ndirections. I don't _think_ we've ever relied on that working, and I\ndon't have much sympathy for any 3rd-party implementation that does\n(though I doubt that any exists).\n\nThat tangent aside, I do this hunk is kind of orthogonal to the point of\nyour patch. We're talking about potential _tightening_ here, whereas the\npoint of your patch is loosening. And it's not clear to me what we want\nto tighten:\n\n - should capabilities come as part of the first response, even if we\n have no refs? In which case we really want \"if (saw_response)\" here.\n\n - should they came as part of the first ref (or pseudo-ref), in which\n case \"if (seen_ref)\" is the right thing.\n\n - should we loosen it to complaining when there are multiple\n capabilities sent. In which case \"if (server_capabilities)\" is the\n right thing.\n\nI'm not sure which we'd want, but it really seems like a separate topic\nthat should be explored on top.\n\n-Peff\n"},{"id":"328495","messageId":"20170920200627.GA82704@google.com","threadId":"46746","inReplyTo":"20170920191431.emornzhse6f7s5mg@sigill.intra.peff.net","subject":"Re: [PATCH 1.5/8] connect: die when a capability line comes after a ref","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-20T20:06:27Z","receivedAt":"2017-09-20T20:06:35Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/20, Jeff King wrote:\n> On Wed, Sep 20, 2017 at 11:48:32AM -0700, Brandon Williams wrote:\n> \n> > Commit eb398797c (connect: advertized capability is not a ref,\n> > 2016-09-09) taught 'get_remote_heads()' to recognize that the\n> > 'capabilities^{}' line isn't a ref but required that the\n> > 'capabilities^{}' line came during the first response from the server.\n> > A future patch will introduce a version string sent by the server during\n> > its first response which can then cause a client to unnecessarily die if\n> > a 'capabilities^{}' line sent as the first ref.\n> > \n> > Teach 'get_remote_heads()' to instead die if a 'capabilities^{}' line is\n> > sent after a ref.\n> \n> Hmm. I think I understand why you'd want this loosening. But why are we\n> sending a version line to a client that we don't know is speaking v2?\n> IOW, shouldn't we be reporting the version to the client in the normal\n> capabilities when we don't know for sure that they can handle the new\n> field? Otherwise we're breaking existing clients.\n\nThe client requested the version, this is the servers response. So\nolder clients shouldn't be broken because they wouldn't be requesting\nthe newer versions.\n\n> \n> Or is this only for v2 clients, and we've changed the protocol but\n> get_remote_heads() just needs to be updated, too?\n\nA client which didn't request protocol v1 (I'm calling the current\nprotocol v0, and v1 is just v0 with the initial response from the server\ncontaining a version string) should not receive a version string in the\ninitial response. The problem is that when introducing the version\nstring to protocol version 1, I didn't want to have to do a huge\nrefactoring of ALL of the current transport code so I stuck the version\ncheck in get_remote_heads() since v1 is exactly the same as v0, except\nfor the first line from the server.\n\nWhen we introduce v2, I'm sure we'll have to do more refactoring to\nseparate out the logic for the different versions.\n> \n> > diff --git a/connect.c b/connect.c\n> > index df56c0cbf..af5096ec6 100644\n> > --- a/connect.c\n> > +++ b/connect.c\n> > @@ -124,10 +124,11 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > \t * response does not necessarily mean an ACL problem, though.\n> > \t */\n> > \tint saw_response;\n> > +\tint seen_ref;\n> > \tint got_dummy_ref_with_capabilities_declaration = 0;\n> > \n> > \t*list = NULL;\n> > -\tfor (saw_response = 0; ; saw_response = 1) {\n> > +\tfor (saw_response = 0, seen_ref = 0; ; saw_response = 1) {\n> \n> If we're not going to update it in the right-hand side of the for-loop,\n> should we perhaps not be initializing it in the left-hand side? I.e.,\n> can we just do:\n> \n> seen_ref = 0;\n> \n> above the loop, like we initialize \"list\"?\n> \n> (For that matter, could we just be checking whether *list is NULL?)\n\nTrue, that would probably be the better way to do this.\n\n> \n> > @@ -165,6 +166,8 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > \n> > \t\tname_len = strlen(name);\n> > \t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n> > +\t\t\tif (seen_ref)\n> > +\t\t\t\t; /* NEEDSWORK: Error out for multiple capabilities lines? */\n> > \t\t\tfree(server_capabilities);\n> > \t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n> > \t\t}\n> \n> Interesting question. Probably it would be fine to. Coincidentally I ran\n> across a similar case. It seems that upload-pack will read multiple\n> capabilities lines back from the client. I.e., if it gets:\n> \n> want 1234abcd... foo\n> want 5678abcd... bar\n> \n> then it will turn on both the \"foo\" and \"bar\" capabilities. I'm pretty\n> sure this is unintended, and is somewhat counter to the way that clients\n> handle multiple lines (which is to forget the old line and respect only\n> the new one, as shown in the quoted hunk).\n> \n> I wonder if we should be outlawing extra capabilities in both\n> directions. I don't _think_ we've ever relied on that working, and I\n> don't have much sympathy for any 3rd-party implementation that does\n> (though I doubt that any exists).\n> \n> That tangent aside, I do this hunk is kind of orthogonal to the point of\n> your patch. We're talking about potential _tightening_ here, whereas the\n> point of your patch is loosening. And it's not clear to me what we want\n> to tighten:\n> \n> - should capabilities come as part of the first response, even if we\n> have no refs? In which case we really want \"if (saw_response)\" here.\n> \n> - should they came as part of the first ref (or pseudo-ref), in which\n> case \"if (seen_ref)\" is the right thing.\n> \n> - should we loosen it to complaining when there are multiple\n> capabilities sent. In which case \"if (server_capabilities)\" is the\n> right thing.\n> \n> I'm not sure which we'd want, but it really seems like a separate topic\n> that should be explored on top.\n\nI wasn't sure either, which is why I added the comment to prod\ndiscussion. I agree that is is orthogonal to this series so I'll most\nlikely drop it, as it doesn't help with the protocol transition\ndiscussion.\n\n-- \nBrandon Williams\n"},{"id":"328501","messageId":"20170920204818.GE27425@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170920200627.GA82704@google.com","subject":"Re: [PATCH 1.5/8] connect: die when a capability line comes after a ref","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-09-20T20:48:18Z","receivedAt":"2017-09-20T20:48:28Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n> On 09/20, Jeff King wrote:\n\n>> (For that matter, could we just be checking whether *list is NULL?)\n>\n> True, that would probably be the better way to do this.\n\nNice idea, thank you.\n\nThat doesn't capture a few other cases of pkts that aren't supposed to\ncome before the capabilities^{} line:\n\n * shallow\n * .have\n * capabilities^{}\n * invalid refnames\n\nPerhaps it should check all of those:\n\n\tif ((shallow_points && shallow_points->nr) ||\n\t (extra_have && extra_have->nr) ||\n\t got_dummy_ref_with_capabilities_declaration ||\n\t got_invalid_ref ||\n\t *list)\n\nWhat happens when another type of pkt gets introduced? This feels\npretty error-prone. The underlying problem is that we are emulating a\nstate machine that is not a simple for loop using a simple for loop,\nby piling up variables that keep track of the current state. That\nsuggests one of the following approaches:\n\n A. Replace saw_response with an enum describing the state.\n Immediately after reading the first packet, update the state to\n EXPECTING_FIRST_REF. Immediately after reading the first ref,\n update the state to EXPECTING_SHALLOW.\n\n B. Use instruction flow to encode the state machine. Have separate\n loops for processing refs and shallow lines.\n\nBy the way, there are some other ways the current code is less strict\nthan described in pack-protocol.txt:\n\n - allowing an empty list-of-refs. (This is deliberate ---\n pack-protocol.txt's lack of documentation of this case is a bug.)\n\n - allowing multiple capability-lists\n\n - allowing capabilities^{} combined with other refs\n\n - allowing refs, shallow, and .have to be interleaved\n\nTightening those would likely be good for the ecosystem (so that\nbuggy servers get noticed quickly), but that's a separate topic from\nthis change.\n\n[...]\n> I wasn't sure either, which is why I added the comment to prod\n> discussion. I agree that is is orthogonal to this series so I'll most\n> likely drop it, as it doesn't help with the protocol transition\n> discussion.\n\nI'd be happy to write a separate patch adding the NEEDSWORK comment\n(or even a patch doing what the NEEDSWORK comment suggests) to avoid\nderailing this one. :)\n\nThanks,\nJonathan\n"},{"id":"328512","messageId":"20170920172443.6b42c9ba@twelve2.svl.corp.google.com","threadId":"46746","inReplyTo":"20170913215448.84674-4-bmwill@google.com","subject":"Re: [PATCH 3/8] daemon: recognize hidden request arguments","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-21T00:24:43Z","receivedAt":"2017-09-21T00:24:55Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 13 Sep 2017 14:54:43 -0700\nBrandon Williams wrote:\n\n> A normal request to git-daemon is structured as\n> \"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\n> git-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\n> command, 2009-06-04) we aren't able to place any extra args (separated\n> by NULs) besides the host.\n> \n> In order to get around this limitation teach git-daemon to recognize\n> additional request arguments hidden behind a second NUL byte. Requests\n> can then be structured like:\n> \"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\n> can then parse out the extra arguments and set 'GIT_PROTOCOL'\n> accordingly.\n\nA test in this patch (if possible) would be nice, but it is probably\nclearer to test this when one of the commands (e.g. upload-pack) is\ndone. Could a test be added to the next patch to verify (using\nGIT_TRACE_PACKET, maybe) that the expected strings are sent? Then\nmention in this commit message that this will be tested in the next\npatch too.\n\n> @@ -574,7 +583,7 @@ static void canonicalize_client(struct strbuf *out, const char *in)\n> /*\n> * Read the host as supplied by the client connection.\n> */\n> -static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> +static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> {\n> \tchar *val;\n> \tint vallen;\n> @@ -602,6 +611,39 @@ static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> \t\tif (extra_args < end && *extra_args)\n> \t\t\tdie(\"Invalid request\");\n> \t}\n> +\n> +\treturn extra_args;\n> +}\n> +\n> +static void parse_extra_args(struct argv_array *env, const char *extra_args,\n> +\t\t\t int buflen)\n> +{\n> +\tconst char *end = extra_args + buflen;\n> +\tstruct strbuf git_protocol = STRBUF_INIT;\n> +\n> +\tfor (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n> +\t\tconst char *arg = extra_args;\n> +\n> +\t\t/*\n> +\t\t * Parse the extra arguments, adding most to 'git_protocol'\n> +\t\t * which will be used to set the 'GIT_PROTOCOL' envvar in the\n> +\t\t * service that will be run.\n> +\t\t *\n> +\t\t * If there ends up being a particular arg in the future that\n> +\t\t * git-daemon needs to parse specificly (like the 'host' arg)\n> +\t\t * then it can be parsed here and not added to 'git_protocol'.\n> +\t\t */\n> +\t\tif (*arg) {\n> +\t\t\tif (git_protocol.len > 0)\n> +\t\t\t\tstrbuf_addch(&git_protocol, ':');\n> +\t\t\tstrbuf_addstr(&git_protocol, arg);\n> +\t\t}\n> +\t}\n> +\n> +\tif (git_protocol.len > 0)\n> +\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n> +\t\t\t\t git_protocol.buf);\n> +\tstrbuf_release(&git_protocol);\n> }\n\nI would rewrite this with parse_extra_args() calling parse_host_arg()\ninstead (right now, you have 2 functions with 2 different meanings of\n\"extra_args\"). If you want to keep this arrangement, though, add a\ndocumentation comment about the meaning of the return value of\nparse_host_arg().\n"},{"id":"328513","messageId":"20170920173120.2731e761@twelve2.svl.corp.google.com","threadId":"46746","inReplyTo":"20170920172443.6b42c9ba@twelve2.svl.corp.google.com","subject":"Re: [PATCH 3/8] daemon: recognize hidden request arguments","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-21T00:31:20Z","receivedAt":"2017-09-21T00:31:30Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Wed, 20 Sep 2017 17:24:43 -0700\nJonathan Tan wrote:\n\n> On Wed, 13 Sep 2017 14:54:43 -0700\n> Brandon Williams wrote:\n> \n> > A normal request to git-daemon is structured as\n> > \"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\n> > git-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\n> > command, 2009-06-04) we aren't able to place any extra args (separated\n> > by NULs) besides the host.\n> > \n> > In order to get around this limitation teach git-daemon to recognize\n> > additional request arguments hidden behind a second NUL byte. Requests\n> > can then be structured like:\n> > \"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\n> > can then parse out the extra arguments and set 'GIT_PROTOCOL'\n> > accordingly.\n> \n> A test in this patch (if possible) would be nice, but it is probably\n> clearer to test this when one of the commands (e.g. upload-pack) is\n> done. Could a test be added to the next patch to verify (using\n> GIT_TRACE_PACKET, maybe) that the expected strings are sent? Then\n> mention in this commit message that this will be tested in the next\n> patch too.\n\nAh, I see that it is tested in 6/8. You can ignore this comment.\n"},{"id":"328528","messageId":"xmqqmv5olpon.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170920200627.GA82704@google.com","subject":"Re: [PATCH 1.5/8] connect: die when a capability line comes after a ref","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-21T03:02:16Z","receivedAt":"2017-09-21T03:02:23Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n>> Or is this only for v2 clients, and we've changed the protocol but\n>> get_remote_heads() just needs to be updated, too?\n>\n> A client which didn't request protocol v1 (I'm calling the current\n> protocol v0, and v1 is just v0 with the initial response from the server\n> containing a version string) should not receive a version string in the\n> initial response. The problem is that when introducing the version\n> string to protocol version 1, I didn't want to have to do a huge\n> refactoring of ALL of the current transport code so I stuck the version\n> check in get_remote_heads() since v1 is exactly the same as v0, except\n> for the first line from the server.\n\nIt is still unclear from your response what other things the server\nis now allowed to say before \"version\". I have a slight suspicion\nthat this change makes the input language overly loose. Before\neb398797 (\"connect: advertized capability is not a ref\", 2016-09-09)\nmade the \"dummy ref must come before any ref and no refs should be\nsent if there is a dummy ref sent\", the code before it used to allow\na \".have\" or a \"shallow\" to appear at the beginning, but with the\n\"anything from the other end whatsoever is not allowed before the\ndummy one\" check the commit introduced, it made it a protocol error\nto send these before dummy ref advertisement. But with this patch,\nyou are again allowing them to come before the dummy ref, together\nwith the \"version\" line you recently added. I do not know if it is\na problem in practice or not offhand, though.\n\n"},{"id":"328590","messageId":"20170921204501.32198-1-jonathantanmy@google.com","threadId":"46746","inReplyTo":"20170920200627.GA82704@google.com","subject":"[PATCH] connect: in ref advertisement, shallows are last","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-21T20:45:01Z","receivedAt":"2017-09-21T20:45:11Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, get_remote_heads() parses the ref advertisement in one loop,\nallowing refs and shallow lines to intersperse, despite this not being\nallowed by the specification. Refactor get_remote_heads() to use two\nloops instead, enforcing that refs come first, and then shallows.\n\nThis also makes it easier to teach get_remote_heads() to interpret other\nlines in the ref advertisement, which will be done in a subsequent\npatch.\n\nSigned-off-by: Jonathan Tan \n---\nIt seems that some people are concerned about looseness in interpreting\nthe ref advertisement, so here is a patch to tighten it instead. This is\na replacement for Brandon's PATCH 1.5.\n\nI think this is what Jonathan Nieder meant by his instruction flow idea.\n\nI've checked that Brandon's other patches apply cleanly on this patch,\nexcept for \"connect: teach client to recognize v1 server response\" which\nhas to be modified to the following:\n\n @@ -149,6 +150,26 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t*list = NULL;\n \n \tlen = read_remote_ref(in, &src_buf, &src_len, &responded);\n +\n +\tswitch (determine_protocol_version_client(packet_buffer)) {\n +\tcase protocol_v1:\n +\t\t/*\n +\t\t * First pkt-line contained the version string.\n +\t\t * Continue on to process the ref advertisement.\n +\t\t */\n +\t\tlen = read_remote_ref(in, &src_buf, &src_len, &responded);\n +\t\tbreak;\n +\tcase protocol_v0:\n +\t\t/*\n +\t\t * Server is speaking protocol v0 and sent a\n +\t\t * ref so we need to process it.\n +\t\t */\n +\t\tbreak;\n +\tdefault:\n +\t\tdie(\"server is speaking an unknown protocol\");\n +\t\tbreak;\n +\t}\n +\n\n connect.c | 112 ++++++++++++++++++++++++++++++++++----------------------------\n 1 file changed, 61 insertions(+), 51 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 49b28b83b..9bf97adf6 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -107,6 +107,26 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n+/*\n+ * Read one line of a server's ref advertisement into packet_buffer.\n+ */\n+int read_remote_ref(int in, char **src_buf, size_t *src_len, int *responded)\n+{\n+\tint len = packet_read(in, src_buf, src_len,\n+\t\t\t packet_buffer, sizeof(packet_buffer),\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\tconst char *arg;\n+\tif (len < 0)\n+\t\tdie_initial_contact(*responded);\n+\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n+\t\tdie(\"remote error: %s\", arg);\n+\n+\t*responded = 1;\n+\n+\treturn len;\n+}\n+\n /*\n * Read all the refs from the other end\n */\n@@ -123,46 +143,23 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t * willing to talk to us. A hang-up before seeing any\n \t * response does not necessarily mean an ACL problem, though.\n \t */\n-\tint saw_response;\n-\tint got_dummy_ref_with_capabilities_declaration = 0;\n+\tint responded = 0;\n+\tint len;\n \n \t*list = NULL;\n-\tfor (saw_response = 0; ; saw_response = 1) {\n-\t\tstruct ref *ref;\n+\n+\tlen = read_remote_ref(in, &src_buf, &src_len, &responded);\n+\tdo {\n \t\tstruct object_id old_oid;\n \t\tchar *name;\n-\t\tint len, name_len;\n-\t\tchar *buffer = packet_buffer;\n-\t\tconst char *arg;\n+\t\tint name_len;\n \n-\t\tlen = packet_read(in, &src_buf, &src_len,\n-\t\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\t\tif (len < 0)\n-\t\t\tdie_initial_contact(saw_response);\n-\n-\t\tif (!len)\n+\t\tif (len < GIT_SHA1_HEXSZ + 2 ||\n+\t\t get_oid_hex(packet_buffer, &old_oid) ||\n+\t\t packet_buffer[GIT_SHA1_HEXSZ] != ' ')\n \t\t\tbreak;\n \n-\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n-\t\t\tdie(\"remote error: %s\", arg);\n-\n-\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n-\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &old_oid))\n-\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n-\t\t\tif (!shallow_points)\n-\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n-\t\t\toid_array_append(shallow_points, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n-\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n-\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n-\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n-\n+\t\tname = packet_buffer + GIT_SHA1_HEXSZ + 1;\n \t\tname_len = strlen(name);\n \t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n \t\t\tfree(server_capabilities);\n@@ -171,29 +168,42 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \n \t\tif (extra_have && !strcmp(name, \".have\")) {\n \t\t\toid_array_append(extra_have, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (!strcmp(name, \"capabilities^{}\")) {\n-\t\t\tif (saw_response)\n+\t\t} else if (!strcmp(name, \"capabilities^{}\")) {\n+\t\t\tif (*list)\n+\t\t\t\t/* cannot coexist with other refs */\n \t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n-\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n-\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n-\t\t\tcontinue;\n+\t\t\t/*\n+\t\t\t * There should be no more refs; read the next line and\n+\t\t\t * go to next block.\n+\t\t\t */\n+\t\t\tlen = read_remote_ref(in, &src_buf, &src_len,\n+\t\t\t\t\t &responded);\n+\t\t\tbreak;\n+\t\t} else if (check_ref(name, flags)) {\n+\t\t\tstruct ref *ref = alloc_ref(name);\n+\t\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\t\t*list = ref;\n+\t\t\tlist = &ref->next;\n \t\t}\n+\t} while ((len = read_remote_ref(in, &src_buf, &src_len, &responded)));\n \n-\t\tif (!check_ref(name, flags))\n-\t\t\tcontinue;\n+\tdo {\n+\t\tconst char *arg;\n+\t\tstruct object_id old_oid;\n \n-\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n+\t\tif (skip_prefix(packet_buffer, \"shallow \", &arg)) {\n+\t\t\tif (get_oid_hex(arg, &old_oid))\n+\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n+\t\t\tif (!shallow_points)\n+\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n+\t\t\toid_array_append(shallow_points, &old_oid);\n+\t\t} else {\n+\t\t\tbreak;\n+\t\t}\n+\t} while ((len = read_remote_ref(in, &src_buf, &src_len, &responded)));\n \n-\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n-\t\t*list = ref;\n-\t\tlist = &ref->next;\n-\t}\n+\tif (len)\n+\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n \n \tannotate_refs_with_symref_info(*orig_list);\n \n-- \n2.14.1.728.g20a5b67d5.dirty\n\n"},{"id":"328594","messageId":"20170921215555.GA77641@google.com","threadId":"46746","inReplyTo":"20170920173120.2731e761@twelve2.svl.corp.google.com","subject":"Re: [PATCH 3/8] daemon: recognize hidden request arguments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-21T21:55:55Z","receivedAt":"2017-09-21T21:56:03Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/20, Jonathan Tan wrote:\n> On Wed, 20 Sep 2017 17:24:43 -0700\n> Jonathan Tan wrote:\n> \n> > On Wed, 13 Sep 2017 14:54:43 -0700\n> > Brandon Williams wrote:\n> > \n> > > A normal request to git-daemon is structured as\n> > > \"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\n> > > git-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\n> > > command, 2009-06-04) we aren't able to place any extra args (separated\n> > > by NULs) besides the host.\n> > > \n> > > In order to get around this limitation teach git-daemon to recognize\n> > > additional request arguments hidden behind a second NUL byte. Requests\n> > > can then be structured like:\n> > > \"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\n> > > can then parse out the extra arguments and set 'GIT_PROTOCOL'\n> > > accordingly.\n> > \n> > A test in this patch (if possible) would be nice, but it is probably\n> > clearer to test this when one of the commands (e.g. upload-pack) is\n> > done. Could a test be added to the next patch to verify (using\n> > GIT_TRACE_PACKET, maybe) that the expected strings are sent? Then\n> > mention in this commit message that this will be tested in the next\n> > patch too.\n> \n> Ah, I see that it is tested in 6/8. You can ignore this comment.\n\nYeah I felt it would have been difficult to test any earlier without\nboth the client and server sides done.\n\n-- \nBrandon Williams\n"},{"id":"328599","messageId":"20170921234534.18964-1-jonathantanmy@google.com","threadId":"46746","inReplyTo":"20170921204501.32198-1-jonathantanmy@google.com","subject":"[PATCH v2] connect: in ref advertisement, shallows are last","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-21T23:45:34Z","receivedAt":"2017-09-21T23:45:47Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, get_remote_heads() parses the ref advertisement in one loop,\nallowing refs and shallow lines to intersperse, despite this not being\nallowed by the specification. Refactor get_remote_heads() to use two\nloops instead, enforcing that refs come first, and then shallows.\n\nThis also makes it easier to teach get_remote_heads() to interpret other\nlines in the ref advertisement, which will be done in a subsequent\npatch.\n\nSigned-off-by: Jonathan Tan \n---\nIn some in-office discussion, I was informed that my original patch\nrelaxed the ordering of \".keep\" lines. Here is an update.\n\nI'm also using a switch statement now, which avoids having multiple\nlines of read_remote_ref().\n\n connect.c | 159 ++++++++++++++++++++++++++++++++++++--------------------------\n 1 file changed, 93 insertions(+), 66 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 49b28b83b..e0821dbff 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -107,6 +107,84 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n+/*\n+ * Read one line of a server's ref advertisement into packet_buffer.\n+ */\n+static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n+\t\t\t int *responded)\n+{\n+\tint len = packet_read(in, src_buf, src_len,\n+\t\t\t packet_buffer, sizeof(packet_buffer),\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\tconst char *arg;\n+\tif (len < 0)\n+\t\tdie_initial_contact(*responded);\n+\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n+\t\tdie(\"remote error: %s\", arg);\n+\n+\t*responded = 1;\n+\n+\treturn len;\n+}\n+\n+#define EXPECTING_REF 0\n+#define EXPECTING_SHALLOW 1\n+\n+static int process_ref(int *state, int len, struct ref ***list,\n+\t\t unsigned int flags, struct oid_array *extra_have)\n+{\n+\tstruct object_id old_oid;\n+\tchar *name;\n+\tint name_len;\n+\n+\tif (len < GIT_SHA1_HEXSZ + 2 ||\n+\t get_oid_hex(packet_buffer, &old_oid) ||\n+\t packet_buffer[GIT_SHA1_HEXSZ] != ' ') {\n+\t\t(*state)++;\n+\t\treturn 0;\n+\t}\n+\n+\tname = packet_buffer + GIT_SHA1_HEXSZ + 1;\n+\tname_len = strlen(name);\n+\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n+\t\tfree(server_capabilities);\n+\t\tserver_capabilities = xstrdup(name + name_len + 1);\n+\t}\n+\n+\tif (extra_have && !strcmp(name, \".have\")) {\n+\t\toid_array_append(extra_have, &old_oid);\n+\t} else if (!strcmp(name, \"capabilities^{}\")) {\n+\t\tif (**list)\n+\t\t\t/* cannot coexist with other refs */\n+\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n+\t\t/* There should be no more refs; proceed to the next state. */\n+\t\t(*state)++;\n+\t} else if (check_ref(name, flags)) {\n+\t\tstruct ref *ref = alloc_ref(name);\n+\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\t**list = ref;\n+\t\t*list = &ref->next;\n+\t}\n+\treturn 1;\n+}\n+\n+static int process_shallow(int *state, struct oid_array *shallow_points)\n+{\n+\tconst char *arg;\n+\tstruct object_id old_oid;\n+\n+\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\t\treturn 0;\n+\n+\tif (get_oid_hex(arg, &old_oid))\n+\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n+\tif (!shallow_points)\n+\t\tdie(\"repository on the other end cannot be shallow\");\n+\toid_array_append(shallow_points, &old_oid);\n+\treturn 1;\n+}\n+\n /*\n * Read all the refs from the other end\n */\n@@ -123,76 +201,25 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t * willing to talk to us. A hang-up before seeing any\n \t * response does not necessarily mean an ACL problem, though.\n \t */\n-\tint saw_response;\n-\tint got_dummy_ref_with_capabilities_declaration = 0;\n+\tint responded = 0;\n+\tint len;\n+\tint state = EXPECTING_REF;\n \n \t*list = NULL;\n-\tfor (saw_response = 0; ; saw_response = 1) {\n-\t\tstruct ref *ref;\n-\t\tstruct object_id old_oid;\n-\t\tchar *name;\n-\t\tint len, name_len;\n-\t\tchar *buffer = packet_buffer;\n-\t\tconst char *arg;\n-\n-\t\tlen = packet_read(in, &src_buf, &src_len,\n-\t\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\t\tif (len < 0)\n-\t\t\tdie_initial_contact(saw_response);\n-\n-\t\tif (!len)\n-\t\t\tbreak;\n-\n-\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n-\t\t\tdie(\"remote error: %s\", arg);\n-\n-\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n-\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &old_oid))\n-\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n-\t\t\tif (!shallow_points)\n-\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n-\t\t\toid_array_append(shallow_points, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n-\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n-\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n-\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n-\n-\t\tname_len = strlen(name);\n-\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n-\t\t\tfree(server_capabilities);\n-\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n-\t\t}\n \n-\t\tif (extra_have && !strcmp(name, \".have\")) {\n-\t\t\toid_array_append(extra_have, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (!strcmp(name, \"capabilities^{}\")) {\n-\t\t\tif (saw_response)\n-\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n-\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n-\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n-\t\t\tcontinue;\n+\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\t\tswitch (state) {\n+\t\tcase EXPECTING_REF:\n+\t\t\tif (process_ref(&state, len, &list, flags, extra_have))\n+\t\t\t\tbreak;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_SHALLOW:\n+\t\t\tif (process_shallow(&state, shallow_points))\n+\t\t\t\tbreak;\n+\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n+\t\tdefault:\n+\t\t\tdie(\"unexpected state %d\", state);\n \t\t}\n-\n-\t\tif (!check_ref(name, flags))\n-\t\t\tcontinue;\n-\n-\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n-\n-\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n-\t\t*list = ref;\n-\t\tlist = &ref->next;\n \t}\n \n \tannotate_refs_with_symref_info(*orig_list);\n-- \n2.14.1.728.g20a5b67d5.dirty\n\n"},{"id":"328600","messageId":"20170922000004.GB77641@google.com","threadId":"46746","inReplyTo":"20170921234534.18964-1-jonathantanmy@google.com","subject":"Re: [PATCH v2] connect: in ref advertisement, shallows are last","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-22T00:00:04Z","receivedAt":"2017-09-22T00:00:14Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/21, Jonathan Tan wrote:\n> Currently, get_remote_heads() parses the ref advertisement in one loop,\n> allowing refs and shallow lines to intersperse, despite this not being\n> allowed by the specification. Refactor get_remote_heads() to use two\n> loops instead, enforcing that refs come first, and then shallows.\n> \n> This also makes it easier to teach get_remote_heads() to interpret other\n> lines in the ref advertisement, which will be done in a subsequent\n> patch.\n> \n> Signed-off-by: Jonathan Tan \n> ---\n> In some in-office discussion, I was informed that my original patch\n> relaxed the ordering of \".keep\" lines. Here is an update.\n> \n> I'm also using a switch statement now, which avoids having multiple\n> lines of read_remote_ref().\n\n\nLooks cleaner than the last patch.\n\n> \n> connect.c | 159 ++++++++++++++++++++++++++++++++++++--------------------------\n> 1 file changed, 93 insertions(+), 66 deletions(-)\n> \n> diff --git a/connect.c b/connect.c\n> index 49b28b83b..e0821dbff 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -107,6 +107,84 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n> \tstring_list_clear(&symref, 0);\n> }\n> \n> +/*\n> + * Read one line of a server's ref advertisement into packet_buffer.\n> + */\n> +static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n> +\t\t\t int *responded)\n> +{\n> +\tint len = packet_read(in, src_buf, src_len,\n> +\t\t\t packet_buffer, sizeof(packet_buffer),\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF |\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE);\n> +\tconst char *arg;\n> +\tif (len < 0)\n> +\t\tdie_initial_contact(*responded);\n> +\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n> +\t\tdie(\"remote error: %s\", arg);\n> +\n> +\t*responded = 1;\n> +\n> +\treturn len;\n> +}\n> +\n> +#define EXPECTING_REF 0\n> +#define EXPECTING_SHALLOW 1\n> +\n> +static int process_ref(int *state, int len, struct ref ***list,\n> +\t\t unsigned int flags, struct oid_array *extra_have)\n> +{\n> +\tstruct object_id old_oid;\n> +\tchar *name;\n> +\tint name_len;\n> +\n> +\tif (len < GIT_SHA1_HEXSZ + 2 ||\n> +\t get_oid_hex(packet_buffer, &old_oid) ||\n> +\t packet_buffer[GIT_SHA1_HEXSZ] != ' ') {\n> +\t\t(*state)++;\n\nI think it may be cleaner if the state variable is updated outside of\nthis function based on a return value from this function.\n\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tname = packet_buffer + GIT_SHA1_HEXSZ + 1;\n> +\tname_len = strlen(name);\n> +\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n> +\t\tfree(server_capabilities);\n> +\t\tserver_capabilities = xstrdup(name + name_len + 1);\n> +\t}\n> +\n> +\tif (extra_have && !strcmp(name, \".have\")) {\n> +\t\toid_array_append(extra_have, &old_oid);\n> +\t} else if (!strcmp(name, \"capabilities^{}\")) {\n> +\t\tif (**list)\n> +\t\t\t/* cannot coexist with other refs */\n> +\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n> +\t\t/* There should be no more refs; proceed to the next state. */\n> +\t\t(*state)++;\n> +\t} else if (check_ref(name, flags)) {\n> +\t\tstruct ref *ref = alloc_ref(name);\n> +\t\toidcpy(&ref->old_oid, &old_oid);\n> +\t\t**list = ref;\n> +\t\t*list = &ref->next;\n> +\t}\n> +\treturn 1;\n> +}\n> +\n> +static int process_shallow(int *state, struct oid_array *shallow_points)\n\nstate isn't needed here and could be dropped from the parameter list.\n\n> +{\n> +\tconst char *arg;\n> +\tstruct object_id old_oid;\n> +\n> +\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n> +\t\treturn 0;\n> +\n> +\tif (get_oid_hex(arg, &old_oid))\n> +\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n> +\tif (!shallow_points)\n> +\t\tdie(\"repository on the other end cannot be shallow\");\n> +\toid_array_append(shallow_points, &old_oid);\n> +\treturn 1;\n> +}\n> +\n> /*\n> * Read all the refs from the other end\n> */\n> @@ -123,76 +201,25 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t * willing to talk to us. A hang-up before seeing any\n> \t * response does not necessarily mean an ACL problem, though.\n> \t */\n> -\tint saw_response;\n> -\tint got_dummy_ref_with_capabilities_declaration = 0;\n> +\tint responded = 0;\n> +\tint len;\n> +\tint state = EXPECTING_REF;\n> \n> \t*list = NULL;\n> -\tfor (saw_response = 0; ; saw_response = 1) {\n> -\t\tstruct ref *ref;\n> -\t\tstruct object_id old_oid;\n> -\t\tchar *name;\n> -\t\tint len, name_len;\n> -\t\tchar *buffer = packet_buffer;\n> -\t\tconst char *arg;\n> -\n> -\t\tlen = packet_read(in, &src_buf, &src_len,\n> -\t\t\t\t packet_buffer, sizeof(packet_buffer),\n> -\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n> -\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n> -\t\tif (len < 0)\n> -\t\t\tdie_initial_contact(saw_response);\n> -\n> -\t\tif (!len)\n> -\t\t\tbreak;\n> -\n> -\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n> -\t\t\tdie(\"remote error: %s\", arg);\n> -\n> -\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n> -\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n> -\t\t\tif (get_oid_hex(arg, &old_oid))\n> -\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n> -\t\t\tif (!shallow_points)\n> -\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n> -\t\t\toid_array_append(shallow_points, &old_oid);\n> -\t\t\tcontinue;\n> -\t\t}\n> -\n> -\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n> -\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n> -\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n> -\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n> -\n> -\t\tname_len = strlen(name);\n> -\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n> -\t\t\tfree(server_capabilities);\n> -\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n> -\t\t}\n> \n> -\t\tif (extra_have && !strcmp(name, \".have\")) {\n> -\t\t\toid_array_append(extra_have, &old_oid);\n> -\t\t\tcontinue;\n> -\t\t}\n> -\n> -\t\tif (!strcmp(name, \"capabilities^{}\")) {\n> -\t\t\tif (saw_response)\n> -\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n> -\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n> -\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n> -\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n> -\t\t\tcontinue;\n> +\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> +\t\tswitch (state) {\n> +\t\tcase EXPECTING_REF:\n> +\t\t\tif (process_ref(&state, len, &list, flags, extra_have))\n> +\t\t\t\tbreak;\n> +\t\t\t/* fallthrough */\n> +\t\tcase EXPECTING_SHALLOW:\n> +\t\t\tif (process_shallow(&state, shallow_points))\n> +\t\t\t\tbreak;\n> +\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n> +\t\tdefault:\n> +\t\t\tdie(\"unexpected state %d\", state);\n> \t\t}\n> -\n> -\t\tif (!check_ref(name, flags))\n> -\t\t\tcontinue;\n> -\n> -\t\tif (got_dummy_ref_with_capabilities_declaration)\n> -\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n> -\n> -\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n> -\t\toidcpy(&ref->old_oid, &old_oid);\n> -\t\t*list = ref;\n> -\t\tlist = &ref->next;\n> \t}\n> \n> \tannotate_refs_with_symref_info(*orig_list);\n> -- \n> 2.14.1.728.g20a5b67d5.dirty\n> \n\n-- \nBrandon Williams\n"},{"id":"328601","messageId":"20170922000801.22560-1-jonathantanmy@google.com","threadId":"46746","inReplyTo":"20170922000004.GB77641@google.com","subject":"[PATCH v3] connect: in ref advertisement, shallows are last","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-22T00:08:01Z","receivedAt":"2017-09-22T00:08:18Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, get_remote_heads() parses the ref advertisement in one loop,\nallowing refs and shallow lines to intersperse, despite this not being\nallowed by the specification. Refactor get_remote_heads() to use two\nloops instead, enforcing that refs come first, and then shallows.\n\nThis also makes it easier to teach get_remote_heads() to interpret other\nlines in the ref advertisement, which will be done in a subsequent\npatch.\n\nSigned-off-by: Jonathan Tan \n---\nI sent the wrong version of this patch :-(\n\nThis should be the correct one. A bit less clean because I introduced a\n3rd state, however.\n\n connect.c | 167 +++++++++++++++++++++++++++++++++++++-------------------------\n 1 file changed, 101 insertions(+), 66 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 49b28b83b..ef6358cfc 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -107,6 +107,91 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n+/*\n+ * Read one line of a server's ref advertisement into packet_buffer.\n+ */\n+static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n+\t\t\t int *responded)\n+{\n+\tint len = packet_read(in, src_buf, src_len,\n+\t\t\t packet_buffer, sizeof(packet_buffer),\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\tconst char *arg;\n+\tif (len < 0)\n+\t\tdie_initial_contact(*responded);\n+\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n+\t\tdie(\"remote error: %s\", arg);\n+\n+\t*responded = 1;\n+\n+\treturn len;\n+}\n+\n+#define EXPECTING_REF_WITH_CAPABILITIES 0\n+#define EXPECTING_REF 1\n+#define EXPECTING_SHALLOW 2\n+\n+static int process_ref(int *state, int len, struct ref ***list,\n+\t\t unsigned int flags, struct oid_array *extra_have)\n+{\n+\tstruct object_id old_oid;\n+\tchar *name;\n+\tint name_len;\n+\n+\tif (len < GIT_SHA1_HEXSZ + 2 ||\n+\t get_oid_hex(packet_buffer, &old_oid) ||\n+\t packet_buffer[GIT_SHA1_HEXSZ] != ' ') {\n+\t\t*state = EXPECTING_SHALLOW;\n+\t\treturn 0;\n+\t}\n+\n+\tname = packet_buffer + GIT_SHA1_HEXSZ + 1;\n+\tname_len = strlen(name);\n+\tif (*state == EXPECTING_REF_WITH_CAPABILITIES &&\n+\t len != name_len + GIT_SHA1_HEXSZ + 1) {\n+\t\tfree(server_capabilities);\n+\t\tserver_capabilities = xstrdup(name + name_len + 1);\n+\t} else if (*state == EXPECTING_REF) {\n+\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1)\n+\t\t\tdie(\"unexpected capabilities after ref name\");\n+\t}\n+\n+\tif (extra_have && !strcmp(name, \".have\")) {\n+\t\toid_array_append(extra_have, &old_oid);\n+\t} else if (!strcmp(name, \"capabilities^{}\")) {\n+\t\tif (**list)\n+\t\t\t/* cannot coexist with other refs */\n+\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n+\t\t/* There should be no more refs; proceed to the next state. */\n+\t\t*state = EXPECTING_SHALLOW;\n+\t\treturn 1;\n+\t} else if (check_ref(name, flags)) {\n+\t\tstruct ref *ref = alloc_ref(name);\n+\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\t**list = ref;\n+\t\t*list = &ref->next;\n+\t}\n+\t*state = EXPECTING_REF;\n+\treturn 1;\n+}\n+\n+static int process_shallow(struct oid_array *shallow_points)\n+{\n+\tconst char *arg;\n+\tstruct object_id old_oid;\n+\n+\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\t\treturn 0;\n+\n+\tif (get_oid_hex(arg, &old_oid))\n+\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n+\tif (!shallow_points)\n+\t\tdie(\"repository on the other end cannot be shallow\");\n+\toid_array_append(shallow_points, &old_oid);\n+\treturn 1;\n+}\n+\n /*\n * Read all the refs from the other end\n */\n@@ -123,76 +208,26 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t * willing to talk to us. A hang-up before seeing any\n \t * response does not necessarily mean an ACL problem, though.\n \t */\n-\tint saw_response;\n-\tint got_dummy_ref_with_capabilities_declaration = 0;\n+\tint responded = 0;\n+\tint len;\n+\tint state = EXPECTING_REF_WITH_CAPABILITIES;\n \n \t*list = NULL;\n-\tfor (saw_response = 0; ; saw_response = 1) {\n-\t\tstruct ref *ref;\n-\t\tstruct object_id old_oid;\n-\t\tchar *name;\n-\t\tint len, name_len;\n-\t\tchar *buffer = packet_buffer;\n-\t\tconst char *arg;\n-\n-\t\tlen = packet_read(in, &src_buf, &src_len,\n-\t\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\t\tif (len < 0)\n-\t\t\tdie_initial_contact(saw_response);\n-\n-\t\tif (!len)\n-\t\t\tbreak;\n-\n-\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n-\t\t\tdie(\"remote error: %s\", arg);\n-\n-\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n-\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &old_oid))\n-\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n-\t\t\tif (!shallow_points)\n-\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n-\t\t\toid_array_append(shallow_points, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n-\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n-\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n-\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n-\n-\t\tname_len = strlen(name);\n-\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n-\t\t\tfree(server_capabilities);\n-\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n-\t\t}\n-\n-\t\tif (extra_have && !strcmp(name, \".have\")) {\n-\t\t\toid_array_append(extra_have, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n \n-\t\tif (!strcmp(name, \"capabilities^{}\")) {\n-\t\t\tif (saw_response)\n-\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n-\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n-\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n-\t\t\tcontinue;\n+\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\t\tswitch (state) {\n+\t\tcase EXPECTING_REF_WITH_CAPABILITIES:\n+\t\tcase EXPECTING_REF:\n+\t\t\tif (process_ref(&state, len, &list, flags, extra_have))\n+\t\t\t\tbreak;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_SHALLOW:\n+\t\t\tif (process_shallow(shallow_points))\n+\t\t\t\tbreak;\n+\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n+\t\tdefault:\n+\t\t\tdie(\"unexpected state %d\", state);\n \t\t}\n-\n-\t\tif (!check_ref(name, flags))\n-\t\t\tcontinue;\n-\n-\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n-\n-\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n-\t\t*list = ref;\n-\t\tlist = &ref->next;\n \t}\n \n \tannotate_refs_with_symref_info(*orig_list);\n-- \n2.14.1.728.g20a5b67d5.dirty\n\n"},{"id":"328606","messageId":"xmqqd16jmtjb.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170922000801.22560-1-jonathantanmy@google.com","subject":"Re: [PATCH v3] connect: in ref advertisement, shallows are last","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-22T01:06:00Z","receivedAt":"2017-09-22T01:06:07Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Currently, get_remote_heads() parses the ref advertisement in one loop,\n> allowing refs and shallow lines to intersperse, despite this not being\n> allowed by the specification. Refactor get_remote_heads() to use two\n> loops instead, enforcing that refs come first, and then shallows.\n>\n> This also makes it easier to teach get_remote_heads() to interpret other\n> lines in the ref advertisement, which will be done in a subsequent\n> patch.\n\nSounds sensible. This still replaces the earlier 1.5?\n\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> I sent the wrong version of this patch :-(\n>\n> This should be the correct one. A bit less clean because I introduced a\n> 3rd state, however.\n>\n> connect.c | 167 +++++++++++++++++++++++++++++++++++++-------------------------\n> 1 file changed, 101 insertions(+), 66 deletions(-)\n>\n> diff --git a/connect.c b/connect.c\n> index 49b28b83b..ef6358cfc 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -107,6 +107,91 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n> \tstring_list_clear(&symref, 0);\n> }\n> \n> +/*\n> + * Read one line of a server's ref advertisement into packet_buffer.\n> + */\n> +static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n> +\t\t\t int *responded)\n> +{\n> +\tint len = packet_read(in, src_buf, src_len,\n> +\t\t\t packet_buffer, sizeof(packet_buffer),\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF |\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE);\n> +\tconst char *arg;\n> +\tif (len < 0)\n> +\t\tdie_initial_contact(*responded);\n> +\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n> +\t\tdie(\"remote error: %s\", arg);\n> +\n> +\t*responded = 1;\n> +\n> +\treturn len;\n> +}\n> +\n> +#define EXPECTING_REF_WITH_CAPABILITIES 0\n> +#define EXPECTING_REF 1\n> +#define EXPECTING_SHALLOW 2\n> +\n> +static int process_ref(int *state, int len, struct ref ***list,\n> +\t\t unsigned int flags, struct oid_array *extra_have)\n> +{\n> +\tstruct object_id old_oid;\n> +\tchar *name;\n> +\tint name_len;\n> +\n> +\tif (len < GIT_SHA1_HEXSZ + 2 ||\n> +\t get_oid_hex(packet_buffer, &old_oid) ||\n> +\t packet_buffer[GIT_SHA1_HEXSZ] != ' ') {\n> +\t\t*state = EXPECTING_SHALLOW;\n> +\t\treturn 0;\n> +\t}\n> +\n> +\tname = packet_buffer + GIT_SHA1_HEXSZ + 1;\n> +\tname_len = strlen(name);\n> +\tif (*state == EXPECTING_REF_WITH_CAPABILITIES &&\n> +\t len != name_len + GIT_SHA1_HEXSZ + 1) {\n> +\t\tfree(server_capabilities);\n> +\t\tserver_capabilities = xstrdup(name + name_len + 1);\n> +\t} else if (*state == EXPECTING_REF) {\n> +\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1)\n> +\t\t\tdie(\"unexpected capabilities after ref name\");\n> +\t}\n> +\n> +\tif (extra_have && !strcmp(name, \".have\")) {\n> +\t\toid_array_append(extra_have, &old_oid);\n> +\t} else if (!strcmp(name, \"capabilities^{}\")) {\n> +\t\tif (**list)\n> +\t\t\t/* cannot coexist with other refs */\n> +\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n> +\t\t/* There should be no more refs; proceed to the next state. */\n> +\t\t*state = EXPECTING_SHALLOW;\n> +\t\treturn 1;\n> +\t} else if (check_ref(name, flags)) {\n> +\t\tstruct ref *ref = alloc_ref(name);\n> +\t\toidcpy(&ref->old_oid, &old_oid);\n> +\t\t**list = ref;\n> +\t\t*list = &ref->next;\n> +\t}\n> +\t*state = EXPECTING_REF;\n> +\treturn 1;\n> +}\n> +\n> +static int process_shallow(struct oid_array *shallow_points)\n> +{\n> +\tconst char *arg;\n> +\tstruct object_id old_oid;\n> +\n> +\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n> +\t\treturn 0;\n> +\n> +\tif (get_oid_hex(arg, &old_oid))\n> +\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n> +\tif (!shallow_points)\n> +\t\tdie(\"repository on the other end cannot be shallow\");\n> +\toid_array_append(shallow_points, &old_oid);\n> +\treturn 1;\n> +}\n> +\n> /*\n> * Read all the refs from the other end\n> */\n> @@ -123,76 +208,26 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t * willing to talk to us. A hang-up before seeing any\n> \t * response does not necessarily mean an ACL problem, though.\n> \t */\n> -\tint saw_response;\n> -\tint got_dummy_ref_with_capabilities_declaration = 0;\n> +\tint responded = 0;\n> +\tint len;\n> +\tint state = EXPECTING_REF_WITH_CAPABILITIES;\n> \n> \t*list = NULL;\n> -\tfor (saw_response = 0; ; saw_response = 1) {\n> -\t\tstruct ref *ref;\n> -\t\tstruct object_id old_oid;\n> -\t\tchar *name;\n> -\t\tint len, name_len;\n> -\t\tchar *buffer = packet_buffer;\n> -\t\tconst char *arg;\n> -\n> -\t\tlen = packet_read(in, &src_buf, &src_len,\n> -\t\t\t\t packet_buffer, sizeof(packet_buffer),\n> -\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n> -\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n> -\t\tif (len < 0)\n> -\t\t\tdie_initial_contact(saw_response);\n> -\n> -\t\tif (!len)\n> -\t\t\tbreak;\n> -\n> -\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n> -\t\t\tdie(\"remote error: %s\", arg);\n> -\n> -\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n> -\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n> -\t\t\tif (get_oid_hex(arg, &old_oid))\n> -\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n> -\t\t\tif (!shallow_points)\n> -\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n> -\t\t\toid_array_append(shallow_points, &old_oid);\n> -\t\t\tcontinue;\n> -\t\t}\n> -\n> -\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n> -\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n> -\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n> -\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n> -\n> -\t\tname_len = strlen(name);\n> -\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n> -\t\t\tfree(server_capabilities);\n> -\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n> -\t\t}\n> -\n> -\t\tif (extra_have && !strcmp(name, \".have\")) {\n> -\t\t\toid_array_append(extra_have, &old_oid);\n> -\t\t\tcontinue;\n> -\t\t}\n> \n> -\t\tif (!strcmp(name, \"capabilities^{}\")) {\n> -\t\t\tif (saw_response)\n> -\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n> -\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n> -\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n> -\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n> -\t\t\tcontinue;\n> +\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> +\t\tswitch (state) {\n> +\t\tcase EXPECTING_REF_WITH_CAPABILITIES:\n> +\t\tcase EXPECTING_REF:\n> +\t\t\tif (process_ref(&state, len, &list, flags, extra_have))\n> +\t\t\t\tbreak;\n> +\t\t\t/* fallthrough */\n> +\t\tcase EXPECTING_SHALLOW:\n> +\t\t\tif (process_shallow(shallow_points))\n> +\t\t\t\tbreak;\n> +\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n> +\t\tdefault:\n> +\t\t\tdie(\"unexpected state %d\", state);\n> \t\t}\n> -\n> -\t\tif (!check_ref(name, flags))\n> -\t\t\tcontinue;\n> -\n> -\t\tif (got_dummy_ref_with_capabilities_declaration)\n> -\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n> -\n> -\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n> -\t\toidcpy(&ref->old_oid, &old_oid);\n> -\t\t*list = ref;\n> -\t\tlist = &ref->next;\n> \t}\n> \n> \tannotate_refs_with_symref_info(*orig_list);\n"},{"id":"328607","messageId":"xmqq7ewrmrzo.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"xmqqd16jmtjb.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3] connect: in ref advertisement, shallows are last","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-22T01:39:23Z","receivedAt":"2017-09-22T01:39:32Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Jonathan Tan writes:\n>\n>> Currently, get_remote_heads() parses the ref advertisement in one loop,\n>> allowing refs and shallow lines to intersperse, despite this not being\n>> allowed by the specification. Refactor get_remote_heads() to use two\n>> loops instead, enforcing that refs come first, and then shallows.\n>>\n>> This also makes it easier to teach get_remote_heads() to interpret other\n>> lines in the ref advertisement, which will be done in a subsequent\n>> patch.\n>\n> Sounds sensible. This still replaces the earlier 1.5?\n\nWell, it does, but it also invalidates how the new \"pick the version\noffered and used\" feature is integrated to this callchain. I guess\nwe'd need a new \"we are now expecting the version info\" state in a\npatch to replace \"connect: teach client to recognize v1 server\nresponse\".\n\n>> +static int process_ref(int *state, int len, struct ref ***list,\n>> +\t\t unsigned int flags, struct oid_array *extra_have)\n>> +{\n>> +\tstruct object_id old_oid;\n>> +\tchar *name;\n>> +\tint name_len;\n>> +\n>> +\tif (len < GIT_SHA1_HEXSZ + 2 ||\n>> +\t get_oid_hex(packet_buffer, &old_oid) ||\n>> +\t packet_buffer[GIT_SHA1_HEXSZ] != ' ') {\n>> +\t\t*state = EXPECTING_SHALLOW;\n>> +\t\treturn 0;\n>> +\t}\n>> +\n>> +\tname = packet_buffer + GIT_SHA1_HEXSZ + 1;\n>> +\tname_len = strlen(name);\n>> +\tif (*state == EXPECTING_REF_WITH_CAPABILITIES &&\n>> +\t len != name_len + GIT_SHA1_HEXSZ + 1) {\n>> +\t\tfree(server_capabilities);\n\nIs this free() still needed? After hitting this block, you'd set\n*state to EXPECTING_REF before you return, so nobody would set\nserver_capabilities by hitting this block twice, and an attempt to\ndo so will hit the die(\"unexpected cap\") below, no?\n\nOr it may be a signal that this patch tightens it too much and\nbreaks older or third-party implementations of the other side that\ncan emit more than one refs with capability advertisement?\n\n>> +\t\tserver_capabilities = xstrdup(name + name_len + 1);\n>> +\t} else if (*state == EXPECTING_REF) {\n>> +\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1)\n>> +\t\t\tdie(\"unexpected capabilities after ref name\");\n>> +\t}\n>> +\t...\n>> +\t}\n>> +\t*state = EXPECTING_REF;\n>> +\treturn 1;\n>> +}\n\n>> @@ -123,76 +208,26 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n>> \t * willing to talk to us. A hang-up before seeing any\n>> \t * response does not necessarily mean an ACL problem, though.\n>> \t */\n>> -\tint saw_response;\n>> -\tint got_dummy_ref_with_capabilities_declaration = 0;\n>> +\tint responded = 0;\n>> +\tint len;\n>> +\tint state = EXPECTING_REF_WITH_CAPABILITIES;\n>> \n>> \t*list = NULL;\n\n>> +\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n>> +\t\tswitch (state) {\n>> +\t\tcase EXPECTING_REF_WITH_CAPABILITIES:\n>> +\t\tcase EXPECTING_REF:\n>> +\t\t\tif (process_ref(&state, len, &list, flags, extra_have))\n>> +\t\t\t\tbreak;\n>> +\t\t\t/* fallthrough */\n\nOK. This fallthrough is because expecting-ref is really expecting\nref or shallow and once we see a shallow, we no longer expect ref\nand expect only shallow. So from that point of view, an assignment\nto set state to EXPECTING_SHALLOW could happen here, not inside\nprocess_ref. I mention this because in general, passing state\naround and let it be updated in helper functions would make the\nstate transition harder to follow, not easier, even though\nrefactoring the processing needed in different stages into helper\nfunctions like this patch does ought to make it easier to see by\nshrinking the outer loop (i.e. this one) that controls the whole\nprocess.\n\nI think if we split process_ref() further into two, then we no\nlonger need to pass &state to that function? We start this loop\nwith \"expecting the dummy ref (or other)\" state, have a new\nprocess_dummy_ref() function check if we got \"capabilities^{}\" thing\nand do its thing if that is the case (otherwise we fall through to\nthe call to process_ref(), just like the above code falls through to\ncall process_shallow() when it realizes what it got is not a ref),\nand after the first call to process_dummy_ref() we'd be in the\n\"expecting ref (or other)\" state---and the state transition can\nhappen in this caller, not in process_dummy_ref() or process_ref().\n\nInside process_dummy_ref() and process_ref(), there would be a call\nto the same helper that notices and extracts the server capability\nand stores it (or barfs against the second line that advertises the\ncapability, by noticing that server_capabilities is not NULL).\n\nWouldn't that make the presentation of the state machine cleaner?\n\n\n"},{"id":"328661","messageId":"20170922164504.GD77641@google.com","threadId":"46746","inReplyTo":"xmqq7ewrmrzo.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3] connect: in ref advertisement, shallows are last","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-22T16:45:04Z","receivedAt":"2017-09-22T16:45:13Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/22, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > Jonathan Tan writes:\n> >\n> >> Currently, get_remote_heads() parses the ref advertisement in one loop,\n> >> allowing refs and shallow lines to intersperse, despite this not being\n> >> allowed by the specification. Refactor get_remote_heads() to use two\n> >> loops instead, enforcing that refs come first, and then shallows.\n> >>\n> >> This also makes it easier to teach get_remote_heads() to interpret other\n> >> lines in the ref advertisement, which will be done in a subsequent\n> >> patch.\n> >\n> > Sounds sensible. This still replaces the earlier 1.5?\n> \n> Well, it does, but it also invalidates how the new \"pick the version\n> offered and used\" feature is integrated to this callchain. I guess\n> we'd need a new \"we are now expecting the version info\" state in a\n> patch to replace \"connect: teach client to recognize v1 server\n> response\".\n\nYeah given we go with this patch, which is probably a better cleanup\nthan what I attempted, then I would need to change how a client\nrecognizes a v1 server. That would probably be easily done by adding a\nnew state.\n\nI do think that once a v2 protocol rolls around we'll probably have to\ndo even more refactoring because I don't think we'll want to keep all\nthe version checking logic in get_remote_heads() for different protocol\nversions which may not be interested in a servers ref advertisement, but\nthat'll be for another time.\n\n> \n> >> +static int process_ref(int *state, int len, struct ref ***list,\n> >> +\t\t unsigned int flags, struct oid_array *extra_have)\n> >> +{\n> >> +\tstruct object_id old_oid;\n> >> +\tchar *name;\n> >> +\tint name_len;\n> >> +\n> >> +\tif (len < GIT_SHA1_HEXSZ + 2 ||\n> >> +\t get_oid_hex(packet_buffer, &old_oid) ||\n> >> +\t packet_buffer[GIT_SHA1_HEXSZ] != ' ') {\n> >> +\t\t*state = EXPECTING_SHALLOW;\n> >> +\t\treturn 0;\n> >> +\t}\n> >> +\n> >> +\tname = packet_buffer + GIT_SHA1_HEXSZ + 1;\n> >> +\tname_len = strlen(name);\n> >> +\tif (*state == EXPECTING_REF_WITH_CAPABILITIES &&\n> >> +\t len != name_len + GIT_SHA1_HEXSZ + 1) {\n> >> +\t\tfree(server_capabilities);\n> \n> Is this free() still needed? After hitting this block, you'd set\n> *state to EXPECTING_REF before you return, so nobody would set\n> server_capabilities by hitting this block twice, and an attempt to\n> do so will hit the die(\"unexpected cap\") below, no?\n> \n> Or it may be a signal that this patch tightens it too much and\n> breaks older or third-party implementations of the other side that\n> can emit more than one refs with capability advertisement?\n> \n> >> +\t\tserver_capabilities = xstrdup(name + name_len + 1);\n> >> +\t} else if (*state == EXPECTING_REF) {\n> >> +\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1)\n> >> +\t\t\tdie(\"unexpected capabilities after ref name\");\n> >> +\t}\n> >> +\t...\n> >> +\t}\n> >> +\t*state = EXPECTING_REF;\n> >> +\treturn 1;\n> >> +}\n> \n> >> @@ -123,76 +208,26 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> >> \t * willing to talk to us. A hang-up before seeing any\n> >> \t * response does not necessarily mean an ACL problem, though.\n> >> \t */\n> >> -\tint saw_response;\n> >> -\tint got_dummy_ref_with_capabilities_declaration = 0;\n> >> +\tint responded = 0;\n> >> +\tint len;\n> >> +\tint state = EXPECTING_REF_WITH_CAPABILITIES;\n> >> \n> >> \t*list = NULL;\n> \n> >> +\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> >> +\t\tswitch (state) {\n> >> +\t\tcase EXPECTING_REF_WITH_CAPABILITIES:\n> >> +\t\tcase EXPECTING_REF:\n> >> +\t\t\tif (process_ref(&state, len, &list, flags, extra_have))\n> >> +\t\t\t\tbreak;\n> >> +\t\t\t/* fallthrough */\n> \n> OK. This fallthrough is because expecting-ref is really expecting\n> ref or shallow and once we see a shallow, we no longer expect ref\n> and expect only shallow. So from that point of view, an assignment\n> to set state to EXPECTING_SHALLOW could happen here, not inside\n> process_ref. I mention this because in general, passing state\n> around and let it be updated in helper functions would make the\n> state transition harder to follow, not easier, even though\n> refactoring the processing needed in different stages into helper\n> functions like this patch does ought to make it easier to see by\n> shrinking the outer loop (i.e. this one) that controls the whole\n> process.\n> \n> I think if we split process_ref() further into two, then we no\n> longer need to pass &state to that function? We start this loop\n> with \"expecting the dummy ref (or other)\" state, have a new\n> process_dummy_ref() function check if we got \"capabilities^{}\" thing\n> and do its thing if that is the case (otherwise we fall through to\n> the call to process_ref(), just like the above code falls through to\n> call process_shallow() when it realizes what it got is not a ref),\n> and after the first call to process_dummy_ref() we'd be in the\n> \"expecting ref (or other)\" state---and the state transition can\n> happen in this caller, not in process_dummy_ref() or process_ref().\n> \n> Inside process_dummy_ref() and process_ref(), there would be a call\n> to the same helper that notices and extracts the server capability\n> and stores it (or barfs against the second line that advertises the\n> capability, by noticing that server_capabilities is not NULL).\n> \n> Wouldn't that make the presentation of the state machine cleaner?\n\nI mentioned this when looking at v2 of this patch, that it would\nprobably be cleaner to remove passing the state variable around the\nplace and updating it inside a helper function. It would just make the\nlogic simpler to follow if 'state' is updated directly instead of\nindirectly.\n\n-- \nBrandon Williams\n"},{"id":"328668","messageId":"20170922201551.15012-1-jonathantanmy@google.com","threadId":"46746","inReplyTo":"20170922164504.GD77641@google.com","subject":"[PATCH v4] connect: in ref advertisement, shallows are last","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-22T20:15:51Z","receivedAt":"2017-09-22T20:16:07Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, get_remote_heads() parses the ref advertisement in one loop,\nallowing refs and shallow lines to intersperse, despite this not being\nallowed by the specification. Refactor get_remote_heads() to use two\nloops instead, enforcing that refs come first, and then shallows.\n\nThis also makes it easier to teach get_remote_heads() to interpret other\nlines in the ref advertisement, which will be done in a subsequent\npatch.\n\nAs part of this change, this patch interprets capabilities only on the\nfirst line in the ref advertisement, ignoring all others.\n\nSigned-off-by: Jonathan Tan \n---\nI've updated state transitions to occur in get_remote_heads() instead,\nas suggested. I didn't want to do that previously because each step in\nthe state machine needed to communicate if (i) the line is \"consumed\"\nand (ii) the state needed to be advanced, but with Junio's suggestion to\nreorganize the methods, that is no longer true.\n\nAs Junio said, the free(server_capabilities) can be removed.\n\nAs for whether how capabilities on subsequent lines are handled, I think\nit's better to ignore them - they are behind NULs, after all.\n\nYes, \"connect: teach client to recognize v1 server response\" will need\nto be modified.\n\nThis change does have the side effect that if the server sends a ref\nadvertisement with \"shallow\"s only (and no refs), things will still\nwork, and the server can even tuck capabilities on the first \"shallow\"\nline. I think that's fine, and it does make the client code cleaner.\n---\n connect.c | 171 ++++++++++++++++++++++++++++++++++++++------------------------\n 1 file changed, 105 insertions(+), 66 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 49b28b83b..978d01359 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -11,6 +11,7 @@\n #include \"string-list.h\"\n #include \"sha1-array.h\"\n #include \"transport.h\"\n+#include \"strbuf.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -107,6 +108,86 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n+/*\n+ * Read one line of a server's ref advertisement into packet_buffer.\n+ */\n+static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n+\t\t\t int *responded)\n+{\n+\tint len = packet_read(in, src_buf, src_len,\n+\t\t\t packet_buffer, sizeof(packet_buffer),\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\tconst char *arg;\n+\tif (len < 0)\n+\t\tdie_initial_contact(*responded);\n+\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n+\t\tdie(\"remote error: %s\", arg);\n+\n+\t*responded = 1;\n+\n+\treturn len;\n+}\n+\n+#define EXPECTING_FIRST_REF 0\n+#define EXPECTING_REF 1\n+#define EXPECTING_SHALLOW 2\n+\n+static void process_capabilities(int len)\n+{\n+\tint nul_location = strlen(packet_buffer);\n+\tif (nul_location == len)\n+\t\treturn;\n+\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n+}\n+\n+static int process_dummy_ref(void)\n+{\n+\tstatic char *template;\n+\tif (!template)\n+\t\ttemplate = xstrfmt(\"%040d capabilities^{}\", 0);\n+\treturn !strcmp(packet_buffer, template);\n+}\n+\n+static int process_ref(struct ref ***list, unsigned int flags,\n+\t\t struct oid_array *extra_have)\n+{\n+\tstruct object_id old_oid;\n+\tconst char *name;\n+\n+\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n+\t\treturn 0;\n+\tif (*name != ' ')\n+\t\treturn 0;\n+\tname++;\n+\n+\tif (extra_have && !strcmp(name, \".have\")) {\n+\t\toid_array_append(extra_have, &old_oid);\n+\t} else if (check_ref(name, flags)) {\n+\t\tstruct ref *ref = alloc_ref(name);\n+\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\t**list = ref;\n+\t\t*list = &ref->next;\n+\t}\n+\treturn 1;\n+}\n+\n+static int process_shallow(struct oid_array *shallow_points)\n+{\n+\tconst char *arg;\n+\tstruct object_id old_oid;\n+\n+\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\t\treturn 0;\n+\n+\tif (get_oid_hex(arg, &old_oid))\n+\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n+\tif (!shallow_points)\n+\t\tdie(\"repository on the other end cannot be shallow\");\n+\toid_array_append(shallow_points, &old_oid);\n+\treturn 1;\n+}\n+\n /*\n * Read all the refs from the other end\n */\n@@ -123,76 +204,34 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t * willing to talk to us. A hang-up before seeing any\n \t * response does not necessarily mean an ACL problem, though.\n \t */\n-\tint saw_response;\n-\tint got_dummy_ref_with_capabilities_declaration = 0;\n+\tint responded = 0;\n+\tint len;\n+\tint state = EXPECTING_FIRST_REF;\n \n \t*list = NULL;\n-\tfor (saw_response = 0; ; saw_response = 1) {\n-\t\tstruct ref *ref;\n-\t\tstruct object_id old_oid;\n-\t\tchar *name;\n-\t\tint len, name_len;\n-\t\tchar *buffer = packet_buffer;\n-\t\tconst char *arg;\n-\n-\t\tlen = packet_read(in, &src_buf, &src_len,\n-\t\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\t\tif (len < 0)\n-\t\t\tdie_initial_contact(saw_response);\n-\n-\t\tif (!len)\n-\t\t\tbreak;\n \n-\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n-\t\t\tdie(\"remote error: %s\", arg);\n-\n-\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n-\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &old_oid))\n-\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n-\t\t\tif (!shallow_points)\n-\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n-\t\t\toid_array_append(shallow_points, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n-\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n-\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n-\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n-\n-\t\tname_len = strlen(name);\n-\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n-\t\t\tfree(server_capabilities);\n-\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n-\t\t}\n-\n-\t\tif (extra_have && !strcmp(name, \".have\")) {\n-\t\t\toid_array_append(extra_have, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (!strcmp(name, \"capabilities^{}\")) {\n-\t\t\tif (saw_response)\n-\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n-\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n-\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n-\t\t\tcontinue;\n+\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\t\tswitch (state) {\n+\t\tcase EXPECTING_FIRST_REF:\n+\t\t\tprocess_capabilities(len);\n+\t\t\tif (process_dummy_ref()) {\n+\t\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tstate = EXPECTING_REF;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_REF:\n+\t\t\tif (process_ref(&list, flags, extra_have))\n+\t\t\t\tbreak;\n+\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_SHALLOW:\n+\t\t\tif (process_shallow(shallow_points))\n+\t\t\t\tbreak;\n+\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n+\t\tdefault:\n+\t\t\tdie(\"unexpected state %d\", state);\n \t\t}\n-\n-\t\tif (!check_ref(name, flags))\n-\t\t\tcontinue;\n-\n-\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n-\n-\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n-\t\t*list = ref;\n-\t\tlist = &ref->next;\n \t}\n \n \tannotate_refs_with_symref_info(*orig_list);\n-- \n2.14.1.728.g20a5b67d5.dirty\n\n"},{"id":"328684","messageId":"20170922210104.GB24036@google.com","threadId":"46746","inReplyTo":"20170922201551.15012-1-jonathantanmy@google.com","subject":"Re: [PATCH v4] connect: in ref advertisement, shallows are last","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-22T21:01:04Z","receivedAt":"2017-09-22T21:01:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/22, Jonathan Tan wrote:\n> Currently, get_remote_heads() parses the ref advertisement in one loop,\n> allowing refs and shallow lines to intersperse, despite this not being\n> allowed by the specification. Refactor get_remote_heads() to use two\n> loops instead, enforcing that refs come first, and then shallows.\n> \n> This also makes it easier to teach get_remote_heads() to interpret other\n> lines in the ref advertisement, which will be done in a subsequent\n> patch.\n> \n> As part of this change, this patch interprets capabilities only on the\n> first line in the ref advertisement, ignoring all others.\n> \n> Signed-off-by: Jonathan Tan \n> ---\n> I've updated state transitions to occur in get_remote_heads() instead,\n> as suggested. I didn't want to do that previously because each step in\n> the state machine needed to communicate if (i) the line is \"consumed\"\n> and (ii) the state needed to be advanced, but with Junio's suggestion to\n> reorganize the methods, that is no longer true.\n> \n> As Junio said, the free(server_capabilities) can be removed.\n> \n> As for whether how capabilities on subsequent lines are handled, I think\n> it's better to ignore them - they are behind NULs, after all.\n> \n> Yes, \"connect: teach client to recognize v1 server response\" will need\n> to be modified.\n> \n> This change does have the side effect that if the server sends a ref\n> advertisement with \"shallow\"s only (and no refs), things will still\n> work, and the server can even tuck capabilities on the first \"shallow\"\n> line. I think that's fine, and it does make the client code cleaner.\n> ---\n> connect.c | 171 ++++++++++++++++++++++++++++++++++++++------------------------\n> 1 file changed, 105 insertions(+), 66 deletions(-)\n> \n> diff --git a/connect.c b/connect.c\n> index 49b28b83b..978d01359 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -11,6 +11,7 @@\n> #include \"string-list.h\"\n> #include \"sha1-array.h\"\n> #include \"transport.h\"\n> +#include \"strbuf.h\"\n> \n> static char *server_capabilities;\n> static const char *parse_feature_value(const char *, const char *, int *);\n> @@ -107,6 +108,86 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n> \tstring_list_clear(&symref, 0);\n> }\n> \n> +/*\n> + * Read one line of a server's ref advertisement into packet_buffer.\n> + */\n> +static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n> +\t\t\t int *responded)\n> +{\n> +\tint len = packet_read(in, src_buf, src_len,\n> +\t\t\t packet_buffer, sizeof(packet_buffer),\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF |\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE);\n> +\tconst char *arg;\n> +\tif (len < 0)\n> +\t\tdie_initial_contact(*responded);\n> +\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n> +\t\tdie(\"remote error: %s\", arg);\n> +\n> +\t*responded = 1;\n> +\n> +\treturn len;\n> +}\n> +\n> +#define EXPECTING_FIRST_REF 0\n> +#define EXPECTING_REF 1\n> +#define EXPECTING_SHALLOW 2\n> +\n> +static void process_capabilities(int len)\n> +{\n> +\tint nul_location = strlen(packet_buffer);\n\nIt may make more sense to not rely on accessing a global buffer here\ndirectly and instead pass in the buff you're working on, much like your\nare doing with len.\n\n> +\tif (nul_location == len)\n> +\t\treturn;\n> +\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n> +}\n> +\n> +static int process_dummy_ref(void)\n> +{\n> +\tstatic char *template;\n> +\tif (!template)\n> +\t\ttemplate = xstrfmt(\"%040d capabilities^{}\", 0);\n\nI'm not the biggest fan of dynamically allocating this and then using it\nto compare. Maybe we can check to make sure that the oid matches the\nnull_oid and that the name matches the \"capabilities^{}\" string? That\nway you can avoid the allocation?\n\n> +\treturn !strcmp(packet_buffer, template);\n> +}\n> +\n> +static int process_ref(struct ref ***list, unsigned int flags,\n> +\t\t struct oid_array *extra_have)\n\nSo from comparing this to the current code it doesn't look like there is\na check in 'process_ref' that ensures that a 'capabilities^{}' line\ndoesn't show up after a normal ref, or am I missing something?\n\n> +{\n> +\tstruct object_id old_oid;\n> +\tconst char *name;\n> +\n> +\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n> +\t\treturn 0;\n> +\tif (*name != ' ')\n> +\t\treturn 0;\n> +\tname++;\n> +\n> +\tif (extra_have && !strcmp(name, \".have\")) {\n> +\t\toid_array_append(extra_have, &old_oid);\n> +\t} else if (check_ref(name, flags)) {\n> +\t\tstruct ref *ref = alloc_ref(name);\n> +\t\toidcpy(&ref->old_oid, &old_oid);\n> +\t\t**list = ref;\n> +\t\t*list = &ref->next;\n> +\t}\n> +\treturn 1;\n> +}\n> +\n> +static int process_shallow(struct oid_array *shallow_points)\n> +{\n> +\tconst char *arg;\n> +\tstruct object_id old_oid;\n> +\n> +\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n> +\t\treturn 0;\n> +\n> +\tif (get_oid_hex(arg, &old_oid))\n> +\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n> +\tif (!shallow_points)\n> +\t\tdie(\"repository on the other end cannot be shallow\");\n> +\toid_array_append(shallow_points, &old_oid);\n> +\treturn 1;\n> +}\n> +\n> /*\n> * Read all the refs from the other end\n> */\n> @@ -123,76 +204,34 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t * willing to talk to us. A hang-up before seeing any\n> \t * response does not necessarily mean an ACL problem, though.\n> \t */\n> -\tint saw_response;\n> -\tint got_dummy_ref_with_capabilities_declaration = 0;\n> +\tint responded = 0;\n> +\tint len;\n> +\tint state = EXPECTING_FIRST_REF;\n> \n> \t*list = NULL;\n> -\tfor (saw_response = 0; ; saw_response = 1) {\n> -\t\tstruct ref *ref;\n> -\t\tstruct object_id old_oid;\n> -\t\tchar *name;\n> -\t\tint len, name_len;\n> -\t\tchar *buffer = packet_buffer;\n> -\t\tconst char *arg;\n> -\n> -\t\tlen = packet_read(in, &src_buf, &src_len,\n> -\t\t\t\t packet_buffer, sizeof(packet_buffer),\n> -\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n> -\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n> -\t\tif (len < 0)\n> -\t\t\tdie_initial_contact(saw_response);\n> -\n> -\t\tif (!len)\n> -\t\t\tbreak;\n> \n> -\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n> -\t\t\tdie(\"remote error: %s\", arg);\n> -\n> -\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n> -\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n> -\t\t\tif (get_oid_hex(arg, &old_oid))\n> -\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n> -\t\t\tif (!shallow_points)\n> -\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n> -\t\t\toid_array_append(shallow_points, &old_oid);\n> -\t\t\tcontinue;\n> -\t\t}\n> -\n> -\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n> -\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n> -\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n> -\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n> -\n> -\t\tname_len = strlen(name);\n> -\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n> -\t\t\tfree(server_capabilities);\n> -\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n> -\t\t}\n> -\n> -\t\tif (extra_have && !strcmp(name, \".have\")) {\n> -\t\t\toid_array_append(extra_have, &old_oid);\n> -\t\t\tcontinue;\n> -\t\t}\n> -\n> -\t\tif (!strcmp(name, \"capabilities^{}\")) {\n> -\t\t\tif (saw_response)\n> -\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n> -\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n> -\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n> -\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n> -\t\t\tcontinue;\n> +\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> +\t\tswitch (state) {\n> +\t\tcase EXPECTING_FIRST_REF:\n> +\t\t\tprocess_capabilities(len);\n> +\t\t\tif (process_dummy_ref()) {\n> +\t\t\t\tstate = EXPECTING_SHALLOW;\n> +\t\t\t\tbreak;\n> +\t\t\t}\n> +\t\t\tstate = EXPECTING_REF;\n> +\t\t\t/* fallthrough */\n> +\t\tcase EXPECTING_REF:\n> +\t\t\tif (process_ref(&list, flags, extra_have))\n> +\t\t\t\tbreak;\n> +\t\t\tstate = EXPECTING_SHALLOW;\n> +\t\t\t/* fallthrough */\n> +\t\tcase EXPECTING_SHALLOW:\n> +\t\t\tif (process_shallow(shallow_points))\n> +\t\t\t\tbreak;\n> +\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n> +\t\tdefault:\n> +\t\t\tdie(\"unexpected state %d\", state);\n\nLooks much cleaner, thanks!\n\n> \t\t}\n> -\n> -\t\tif (!check_ref(name, flags))\n> -\t\t\tcontinue;\n> -\n> -\t\tif (got_dummy_ref_with_capabilities_declaration)\n> -\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n> -\n> -\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n> -\t\toidcpy(&ref->old_oid, &old_oid);\n> -\t\t*list = ref;\n> -\t\tlist = &ref->next;\n> \t}\n> \n> \tannotate_refs_with_symref_info(*orig_list);\n> -- \n> 2.14.1.728.g20a5b67d5.dirty\n> \n\n-- \nBrandon Williams\n"},{"id":"328693","messageId":"20170922151635.9c058ce5a06c67ed9610fc6e@google.com","threadId":"46746","inReplyTo":"20170922210104.GB24036@google.com","subject":"Re: [PATCH v4] connect: in ref advertisement, shallows are last","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-22T22:16:35Z","receivedAt":"2017-09-22T22:16:41Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Fri, 22 Sep 2017 14:01:04 -0700\nBrandon Williams wrote:\n\n> > +static void process_capabilities(int len)\n> > +{\n> > +\tint nul_location = strlen(packet_buffer);\n> \n> It may make more sense to not rely on accessing a global buffer here\n> directly and instead pass in the buff you're working on, much like your\n> are doing with len.\n\nI wanted to preserve the existing code's behavior of using the global\nbuffer, and it didn't make sense for me to alias it (like the existing\ncode does).\n\nI pass len in because I need to read beyond NUL.\n\n> I'm not the biggest fan of dynamically allocating this and then using it\n> to compare. Maybe we can check to make sure that the oid matches the\n> null_oid and that the name matches the \"capabilities^{}\" string? That\n> way you can avoid the allocation?\n\nThe dynamic allocation happens only once per process, since it is\nstatic. To check the oid matches null_oid, I would have to parse it\nfirst, and that seemed unnecessary.\n\nIdeally I would just check again \"000...000 capabilities^{}\", but\nwriting it in source code would be error-prone, I think.\n\n> > +static int process_ref(struct ref ***list, unsigned int flags,\n> > +\t\t struct oid_array *extra_have)\n> \n> So from comparing this to the current code it doesn't look like there is\n> a check in 'process_ref' that ensures that a 'capabilities^{}' line\n> doesn't show up after a normal ref, or am I missing something?\n\nAh...yes, you're right. I'll fix this by adding a check in\nprocess_ref().\n\nThis is getting more complicated than I thought, so I'll wait a while\nfor other comments before sending out an updated patch.\n"},{"id":"328755","messageId":"xmqqefqwj4ui.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170922201551.15012-1-jonathantanmy@google.com","subject":"Re: [PATCH v4] connect: in ref advertisement, shallows are last","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-24T00:52:05Z","receivedAt":"2017-09-24T00:52:13Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Tan writes:\n\n> Currently, get_remote_heads() parses the ref advertisement in one loop,\n> allowing refs and shallow lines to intersperse, despite this not being\n> allowed by the specification. Refactor get_remote_heads() to use two\n> loops instead, enforcing that refs come first, and then shallows.\n>\n> This also makes it easier to teach get_remote_heads() to interpret other\n> lines in the ref advertisement, which will be done in a subsequent\n> patch.\n>\n> As part of this change, this patch interprets capabilities only on the\n> first line in the ref advertisement, ignoring all others.\n>\n> Signed-off-by: Jonathan Tan \n> ---\n> I've updated state transitions to occur in get_remote_heads() instead,\n> as suggested. I didn't want to do that previously because each step in\n> the state machine needed to communicate if (i) the line is \"consumed\"\n> and (ii) the state needed to be advanced, but with Junio's suggestion to\n> reorganize the methods, that is no longer true.\n>\n> As Junio said, the free(server_capabilities) can be removed.\n>\n> As for whether how capabilities on subsequent lines are handled, I think\n> it's better to ignore them - they are behind NULs, after all.\n\nNot even a diagnosis? When the other side clearly wants to tell us\nsomething and we deliberately ignore, I'd think we want to at least\nknow about it---that may lead us to notify the implementators of the\nother side of a protocol violation, or rethink the design decision\n(i.e. only the first one matters) ourselves.\n\n> This change does have the side effect that if the server sends a ref\n> advertisement with \"shallow\"s only (and no refs), things will still\n> work, and the server can even tuck capabilities on the first \"shallow\"\n> line. I think that's fine, and it does make the client code cleaner.\n\nI am ambivalent on this aspect of the change.\n\nThe change makes the resulting state transition logic quite easy to\nfollow. Very nicely done.\n\n> +\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> +\t\tswitch (state) {\n> +\t\tcase EXPECTING_FIRST_REF:\n> +\t\t\tprocess_capabilities(len);\n> +\t\t\tif (process_dummy_ref()) {\n> +\t\t\t\tstate = EXPECTING_SHALLOW;\n> +\t\t\t\tbreak;\n> +\t\t\t}\n> +\t\t\tstate = EXPECTING_REF;\n> +\t\t\t/* fallthrough */\n> +\t\tcase EXPECTING_REF:\n> +\t\t\tif (process_ref(&list, flags, extra_have))\n> +\t\t\t\tbreak;\n> +\t\t\tstate = EXPECTING_SHALLOW;\n> +\t\t\t/* fallthrough */\n> +\t\tcase EXPECTING_SHALLOW:\n> +\t\t\tif (process_shallow(shallow_points))\n> +\t\t\t\tbreak;\n> +\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n> +\t\tdefault:\n> +\t\t\tdie(\"unexpected state %d\", state);\n> \t\t}\n> \t}\n\n"},{"id":"328954","messageId":"20170926182147.73336-1-jonathantanmy@google.com","threadId":"46746","inReplyTo":"20170921204501.32198-1-jonathantanmy@google.com","subject":"[PATCH v5] connect: in ref advertisement, shallows are last","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-09-26T18:21:47Z","receivedAt":"2017-09-26T18:22:00Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Currently, get_remote_heads() parses the ref advertisement in one loop,\nallowing refs and shallow lines to intersperse, despite this not being\nallowed by the specification. Refactor get_remote_heads() to use two\nloops instead, enforcing that refs come first, and then shallows.\n\nThis also makes it easier to teach get_remote_heads() to interpret other\nlines in the ref advertisement, which will be done in a subsequent\npatch.\n\nAs part of this change, this patch interprets capabilities only on the\nfirst line in the ref advertisement, printing a warning message when\nencountering capabilities on other lines.\n\nSigned-off-by: Jonathan Tan \n---\nChanges in v5:\n - print warning when encountering capabilities on other lines instead\n of ignoring them (also updated commit message)\n - explicitly disallow refs of name \"capabilities^{}\" (except when it is\n the only ref)\n---\n connect.c | 183 +++++++++++++++++++++++++++++++++++++++-----------------------\n 1 file changed, 117 insertions(+), 66 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex df56c0cbf..df65a3fc4 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -11,6 +11,7 @@\n #include \"string-list.h\"\n #include \"sha1-array.h\"\n #include \"transport.h\"\n+#include \"strbuf.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -107,6 +108,98 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n+/*\n+ * Read one line of a server's ref advertisement into packet_buffer.\n+ */\n+static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n+\t\t\t int *responded)\n+{\n+\tint len = packet_read(in, src_buf, src_len,\n+\t\t\t packet_buffer, sizeof(packet_buffer),\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\tconst char *arg;\n+\tif (len < 0)\n+\t\tdie_initial_contact(*responded);\n+\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n+\t\tdie(\"remote error: %s\", arg);\n+\n+\t*responded = 1;\n+\n+\treturn len;\n+}\n+\n+#define EXPECTING_FIRST_REF 0\n+#define EXPECTING_REF 1\n+#define EXPECTING_SHALLOW 2\n+\n+static void process_capabilities(int *len)\n+{\n+\tint nul_location = strlen(packet_buffer);\n+\tif (nul_location == *len)\n+\t\treturn;\n+\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n+\t*len = nul_location;\n+}\n+\n+static int process_dummy_ref(void)\n+{\n+\tstatic char *template;\n+\tif (!template)\n+\t\ttemplate = xstrfmt(\"%040d capabilities^{}\", 0);\n+\treturn !strcmp(packet_buffer, template);\n+}\n+\n+static void check_no_capabilities(int len)\n+{\n+\tif (strlen(packet_buffer) != len)\n+\t\twarning(\"Ignoring capabilities after first line '%s'\",\n+\t\t\tpacket_buffer + strlen(packet_buffer));\n+}\n+\n+static int process_ref(int len, struct ref ***list, unsigned int flags,\n+\t\t struct oid_array *extra_have)\n+{\n+\tstruct object_id old_oid;\n+\tconst char *name;\n+\n+\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n+\t\treturn 0;\n+\tif (*name != ' ')\n+\t\treturn 0;\n+\tname++;\n+\n+\tif (extra_have && !strcmp(name, \".have\")) {\n+\t\toid_array_append(extra_have, &old_oid);\n+\t} else if (!strcmp(name, \"capabilities^{}\")) {\n+\t\tdie(\"protocol error: unexpected capabilities^{}\");\n+\t} else if (check_ref(name, flags)) {\n+\t\tstruct ref *ref = alloc_ref(name);\n+\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\t**list = ref;\n+\t\t*list = &ref->next;\n+\t}\n+\tcheck_no_capabilities(len);\n+\treturn 1;\n+}\n+\n+static int process_shallow(int len, struct oid_array *shallow_points)\n+{\n+\tconst char *arg;\n+\tstruct object_id old_oid;\n+\n+\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\t\treturn 0;\n+\n+\tif (get_oid_hex(arg, &old_oid))\n+\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n+\tif (!shallow_points)\n+\t\tdie(\"repository on the other end cannot be shallow\");\n+\toid_array_append(shallow_points, &old_oid);\n+\tcheck_no_capabilities(len);\n+\treturn 1;\n+}\n+\n /*\n * Read all the refs from the other end\n */\n@@ -123,76 +216,34 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t * willing to talk to us. A hang-up before seeing any\n \t * response does not necessarily mean an ACL problem, though.\n \t */\n-\tint saw_response;\n-\tint got_dummy_ref_with_capabilities_declaration = 0;\n+\tint responded = 0;\n+\tint len;\n+\tint state = EXPECTING_FIRST_REF;\n \n \t*list = NULL;\n-\tfor (saw_response = 0; ; saw_response = 1) {\n-\t\tstruct ref *ref;\n-\t\tstruct object_id old_oid;\n-\t\tchar *name;\n-\t\tint len, name_len;\n-\t\tchar *buffer = packet_buffer;\n-\t\tconst char *arg;\n-\n-\t\tlen = packet_read(in, &src_buf, &src_len,\n-\t\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\t\tif (len < 0)\n-\t\t\tdie_initial_contact(saw_response);\n-\n-\t\tif (!len)\n-\t\t\tbreak;\n-\n-\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n-\t\t\tdie(\"remote error: %s\", arg);\n-\n-\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n-\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &old_oid))\n-\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n-\t\t\tif (!shallow_points)\n-\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n-\t\t\toid_array_append(shallow_points, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n-\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n-\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n-\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n \n-\t\tname_len = strlen(name);\n-\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n-\t\t\tfree(server_capabilities);\n-\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n-\t\t}\n-\n-\t\tif (extra_have && !strcmp(name, \".have\")) {\n-\t\t\toid_array_append(extra_have, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (!strcmp(name, \"capabilities^{}\")) {\n-\t\t\tif (saw_response)\n-\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n-\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n-\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n-\t\t\tcontinue;\n+\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\t\tswitch (state) {\n+\t\tcase EXPECTING_FIRST_REF:\n+\t\t\tprocess_capabilities(&len);\n+\t\t\tif (process_dummy_ref()) {\n+\t\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tstate = EXPECTING_REF;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_REF:\n+\t\t\tif (process_ref(len, &list, flags, extra_have))\n+\t\t\t\tbreak;\n+\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_SHALLOW:\n+\t\t\tif (process_shallow(len, shallow_points))\n+\t\t\t\tbreak;\n+\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n+\t\tdefault:\n+\t\t\tdie(\"unexpected state %d\", state);\n \t\t}\n-\n-\t\tif (!check_ref(name, flags))\n-\t\t\tcontinue;\n-\n-\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n-\n-\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n-\t\t*list = ref;\n-\t\tlist = &ref->next;\n \t}\n \n \tannotate_refs_with_symref_info(*orig_list);\n-- \n2.14.1.821.g8fa685d3b7-goog\n\n"},{"id":"328956","messageId":"20170926183143.GA144745@google.com","threadId":"46746","inReplyTo":"20170926182147.73336-1-jonathantanmy@google.com","subject":"Re: [PATCH v5] connect: in ref advertisement, shallows are last","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T18:31:43Z","receivedAt":"2017-09-26T18:31:51Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/26, Jonathan Tan wrote:\n> Currently, get_remote_heads() parses the ref advertisement in one loop,\n> allowing refs and shallow lines to intersperse, despite this not being\n> allowed by the specification. Refactor get_remote_heads() to use two\n> loops instead, enforcing that refs come first, and then shallows.\n> \n> This also makes it easier to teach get_remote_heads() to interpret other\n> lines in the ref advertisement, which will be done in a subsequent\n> patch.\n> \n> As part of this change, this patch interprets capabilities only on the\n> first line in the ref advertisement, printing a warning message when\n> encountering capabilities on other lines.\n> \n> Signed-off-by: Jonathan Tan \n> ---\n> Changes in v5:\n> - print warning when encountering capabilities on other lines instead\n> of ignoring them (also updated commit message)\n> - explicitly disallow refs of name \"capabilities^{}\" (except when it is\n> the only ref)\n> ---\n> connect.c | 183 +++++++++++++++++++++++++++++++++++++++-----------------------\n> 1 file changed, 117 insertions(+), 66 deletions(-)\n> \n> diff --git a/connect.c b/connect.c\n> index df56c0cbf..df65a3fc4 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -11,6 +11,7 @@\n> #include \"string-list.h\"\n> #include \"sha1-array.h\"\n> #include \"transport.h\"\n> +#include \"strbuf.h\"\n> \n> static char *server_capabilities;\n> static const char *parse_feature_value(const char *, const char *, int *);\n> @@ -107,6 +108,98 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n> \tstring_list_clear(&symref, 0);\n> }\n> \n> +/*\n> + * Read one line of a server's ref advertisement into packet_buffer.\n> + */\n> +static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n> +\t\t\t int *responded)\n> +{\n> +\tint len = packet_read(in, src_buf, src_len,\n> +\t\t\t packet_buffer, sizeof(packet_buffer),\n> +\t\t\t PACKET_READ_GENTLE_ON_EOF |\n> +\t\t\t PACKET_READ_CHOMP_NEWLINE);\n> +\tconst char *arg;\n> +\tif (len < 0)\n> +\t\tdie_initial_contact(*responded);\n> +\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n> +\t\tdie(\"remote error: %s\", arg);\n> +\n> +\t*responded = 1;\n> +\n> +\treturn len;\n> +}\n> +\n> +#define EXPECTING_FIRST_REF 0\n> +#define EXPECTING_REF 1\n> +#define EXPECTING_SHALLOW 2\n> +\n> +static void process_capabilities(int *len)\n> +{\n> +\tint nul_location = strlen(packet_buffer);\n> +\tif (nul_location == *len)\n> +\t\treturn;\n> +\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n> +\t*len = nul_location;\n> +}\n> +\n> +static int process_dummy_ref(void)\n> +{\n> +\tstatic char *template;\n> +\tif (!template)\n> +\t\ttemplate = xstrfmt(\"%040d capabilities^{}\", 0);\n\nMy only complaint is still here, I don't like the notion of hardcoding\nthe 0's. Its much more future proof and less error prone to call\nparse_oid_hex and require that it matches null_oid.\n\n> +\treturn !strcmp(packet_buffer, template);\n> +}\n> +\n> +static void check_no_capabilities(int len)\n> +{\n> +\tif (strlen(packet_buffer) != len)\n> +\t\twarning(\"Ignoring capabilities after first line '%s'\",\n> +\t\t\tpacket_buffer + strlen(packet_buffer));\n> +}\n> +\n> +static int process_ref(int len, struct ref ***list, unsigned int flags,\n> +\t\t struct oid_array *extra_have)\n> +{\n> +\tstruct object_id old_oid;\n> +\tconst char *name;\n> +\n> +\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n> +\t\treturn 0;\n> +\tif (*name != ' ')\n> +\t\treturn 0;\n> +\tname++;\n> +\n> +\tif (extra_have && !strcmp(name, \".have\")) {\n> +\t\toid_array_append(extra_have, &old_oid);\n> +\t} else if (!strcmp(name, \"capabilities^{}\")) {\n> +\t\tdie(\"protocol error: unexpected capabilities^{}\");\n> +\t} else if (check_ref(name, flags)) {\n> +\t\tstruct ref *ref = alloc_ref(name);\n> +\t\toidcpy(&ref->old_oid, &old_oid);\n> +\t\t**list = ref;\n> +\t\t*list = &ref->next;\n> +\t}\n> +\tcheck_no_capabilities(len);\n> +\treturn 1;\n> +}\n> +\n> +static int process_shallow(int len, struct oid_array *shallow_points)\n> +{\n> +\tconst char *arg;\n> +\tstruct object_id old_oid;\n> +\n> +\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n> +\t\treturn 0;\n> +\n> +\tif (get_oid_hex(arg, &old_oid))\n> +\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n> +\tif (!shallow_points)\n> +\t\tdie(\"repository on the other end cannot be shallow\");\n> +\toid_array_append(shallow_points, &old_oid);\n> +\tcheck_no_capabilities(len);\n> +\treturn 1;\n> +}\n> +\n> /*\n> * Read all the refs from the other end\n> */\n> @@ -123,76 +216,34 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t * willing to talk to us. A hang-up before seeing any\n> \t * response does not necessarily mean an ACL problem, though.\n> \t */\n> -\tint saw_response;\n> -\tint got_dummy_ref_with_capabilities_declaration = 0;\n> +\tint responded = 0;\n> +\tint len;\n> +\tint state = EXPECTING_FIRST_REF;\n> \n> \t*list = NULL;\n> -\tfor (saw_response = 0; ; saw_response = 1) {\n> -\t\tstruct ref *ref;\n> -\t\tstruct object_id old_oid;\n> -\t\tchar *name;\n> -\t\tint len, name_len;\n> -\t\tchar *buffer = packet_buffer;\n> -\t\tconst char *arg;\n> -\n> -\t\tlen = packet_read(in, &src_buf, &src_len,\n> -\t\t\t\t packet_buffer, sizeof(packet_buffer),\n> -\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n> -\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n> -\t\tif (len < 0)\n> -\t\t\tdie_initial_contact(saw_response);\n> -\n> -\t\tif (!len)\n> -\t\t\tbreak;\n> -\n> -\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n> -\t\t\tdie(\"remote error: %s\", arg);\n> -\n> -\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n> -\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n> -\t\t\tif (get_oid_hex(arg, &old_oid))\n> -\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n> -\t\t\tif (!shallow_points)\n> -\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n> -\t\t\toid_array_append(shallow_points, &old_oid);\n> -\t\t\tcontinue;\n> -\t\t}\n> -\n> -\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n> -\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n> -\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n> -\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n> \n> -\t\tname_len = strlen(name);\n> -\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n> -\t\t\tfree(server_capabilities);\n> -\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n> -\t\t}\n> -\n> -\t\tif (extra_have && !strcmp(name, \".have\")) {\n> -\t\t\toid_array_append(extra_have, &old_oid);\n> -\t\t\tcontinue;\n> -\t\t}\n> -\n> -\t\tif (!strcmp(name, \"capabilities^{}\")) {\n> -\t\t\tif (saw_response)\n> -\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n> -\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n> -\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n> -\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n> -\t\t\tcontinue;\n> +\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> +\t\tswitch (state) {\n> +\t\tcase EXPECTING_FIRST_REF:\n> +\t\t\tprocess_capabilities(&len);\n> +\t\t\tif (process_dummy_ref()) {\n> +\t\t\t\tstate = EXPECTING_SHALLOW;\n> +\t\t\t\tbreak;\n> +\t\t\t}\n> +\t\t\tstate = EXPECTING_REF;\n> +\t\t\t/* fallthrough */\n> +\t\tcase EXPECTING_REF:\n> +\t\t\tif (process_ref(len, &list, flags, extra_have))\n> +\t\t\t\tbreak;\n> +\t\t\tstate = EXPECTING_SHALLOW;\n> +\t\t\t/* fallthrough */\n> +\t\tcase EXPECTING_SHALLOW:\n> +\t\t\tif (process_shallow(len, shallow_points))\n> +\t\t\t\tbreak;\n> +\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n> +\t\tdefault:\n> +\t\t\tdie(\"unexpected state %d\", state);\n> \t\t}\n> -\n> -\t\tif (!check_ref(name, flags))\n> -\t\t\tcontinue;\n> -\n> -\t\tif (got_dummy_ref_with_capabilities_declaration)\n> -\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n> -\n> -\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n> -\t\toidcpy(&ref->old_oid, &old_oid);\n> -\t\t*list = ref;\n> -\t\tlist = &ref->next;\n> \t}\n> \n> \tannotate_refs_with_symref_info(*orig_list);\n> -- \n> 2.14.1.821.g8fa685d3b7-goog\n> \n\n-- \nBrandon Williams\n"},{"id":"328976","messageId":"20170926235627.79606-1-bmwill@google.com","threadId":"46746","inReplyTo":"20170913215448.84674-1-bmwill@google.com","subject":"[PATCH v2 0/9] protocol transition","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:18Z","receivedAt":"2017-09-26T23:56:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"v2 of this series has the following changes:\n\n * Included Jonathan Tan's patch as [1/9] with a small tweak to not rely on\n hardcoding the side of sha1 into the capability line check.\n * Reworked some of the logic in daemon.c\n * Added the word 'Experimental' to protocol.version to indicate that user's\n shouldn't rely on its semantics to hold until it has been thoroughly tested.\n\nBrandon Williams (8):\n pkt-line: add packet_write function\n protocol: introduce protocol extention mechanisms\n daemon: recognize hidden request arguments\n upload-pack, receive-pack: introduce protocol version 1\n connect: teach client to recognize v1 server response\n connect: tell server that the client understands v1\n http: tell server that the client understands v1\n i5700: add interop test for protocol transition\n\nJonathan Tan (1):\n connect: in ref advertisement, shallows are last\n\n Documentation/config.txt | 17 ++\n Documentation/git.txt | 5 +\n Makefile | 1 +\n builtin/receive-pack.c | 14 ++\n cache.h | 9 +\n connect.c | 248 ++++++++++++++++++++--------\n daemon.c | 68 +++++++-\n http.c | 18 ++\n pkt-line.c | 6 +\n pkt-line.h | 1 +\n protocol.c | 72 ++++++++\n protocol.h | 14 ++\n t/interop/i5700-protocol-transition.sh | 68 ++++++++\n t/lib-httpd/apache.conf | 7 +\n t/t5700-protocol-v1.sh | 292 +++++++++++++++++++++++++++++++++\n upload-pack.c | 17 +-\n 16 files changed, 776 insertions(+), 81 deletions(-)\n create mode 100644 protocol.c\n create mode 100644 protocol.h\n create mode 100755 t/interop/i5700-protocol-transition.sh\n create mode 100755 t/t5700-protocol-v1.sh\n\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328977","messageId":"20170926235627.79606-3-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v2 2/9] pkt-line: add packet_write function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:20Z","receivedAt":"2017-09-26T23:56:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a function which can be used to write the contents of an arbitrary\nbuffer. This makes it easy to build up data in a buffer before writing\nthe packet instead of formatting the entire contents of the packet using\n'packet_write_fmt()'.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 6 ++++++\n pkt-line.h | 1 +\n 2 files changed, 7 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 647bbd3bc..c025d0332 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -188,6 +188,12 @@ static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n \treturn 0;\n }\n \n+void packet_write(const int fd_out, const char *buf, size_t size)\n+{\n+\tif (packet_write_gently(fd_out, buf, size))\n+\t\tdie_errno(\"packet write failed\");\n+}\n+\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n {\n \tva_list args;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 66ef610fc..d9e9783b1 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -22,6 +22,7 @@\n void packet_flush(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n+void packet_write(const int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328978","messageId":"20170926235627.79606-4-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v2 3/9] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:21Z","receivedAt":"2017-09-26T23:56:57Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Create protocol.{c,h} and provide functions which future servers and\nclients can use to determine which protocol to use or is being used.\n\nAlso introduce the 'GIT_PROTOCOL' environment variable which will be\nused to communicate a colon separated list of keys with optional values\nto a server. Unknown keys and values must be tolerated. This mechanism\nis used to communicate which version of the wire protocol a client would\nlike to use with a server.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/config.txt | 17 ++++++++++++\n Documentation/git.txt | 5 ++++\n Makefile | 1 +\n cache.h | 7 +++++\n protocol.c | 72 ++++++++++++++++++++++++++++++++++++++++++++++++\n protocol.h | 14 ++++++++++\n 6 files changed, 116 insertions(+)\n create mode 100644 protocol.c\n create mode 100644 protocol.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dc4e3f58a..b78747abc 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2517,6 +2517,23 @@ The protocol names currently used by git are:\n `hg` to allow the `git-remote-hg` helper)\n --\n \n+protocol.version::\n+\tExperimental. If set, clients will attempt to communicate with a\n+\tserver using the specified protocol version. If unset, no\n+\tattempt will be made by the client to communicate using a\n+\tparticular protocol version, this results in protocol version 0\n+\tbeing used.\n+\tSupported versions:\n++\n+--\n+\n+* `0` - the original wire protocol.\n+\n+* `1` - the original wire protocol with the addition of a version string\n+ in the initial response from the server.\n+\n+--\n+\n pull.ff::\n \tBy default, Git does not create an extra merge commit when merging\n \ta commit that is a descendant of the current commit. Instead, the\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 6e3a6767e..299f75c7b 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -697,6 +697,11 @@ of clones and fetches.\n \twhich feed potentially-untrusted URLS to git commands. See\n \tlinkgit:git-config[1] for more details.\n \n+`GIT_PROTOCOL`::\n+\tFor internal use only. Used in handshaking the wire protocol.\n+\tContains a colon ':' separated list of keys with optional values\n+\t'key[=value]'. Presence of unknown keys must be tolerated.\n+\n Discussion[[Discussion]]\n ------------------------\n \ndiff --git a/Makefile b/Makefile\nindex ed4ca438b..9ce68cded 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -842,6 +842,7 @@ LIB_OBJS += pretty.o\n LIB_OBJS += prio-queue.o\n LIB_OBJS += progress.o\n LIB_OBJS += prompt.o\n+LIB_OBJS += protocol.o\n LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\ndiff --git a/cache.h b/cache.h\nindex 49b083ee0..0c792545f 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -445,6 +445,13 @@ static inline enum object_type object_type(unsigned int mode)\n #define GIT_ICASE_PATHSPECS_ENVIRONMENT \"GIT_ICASE_PATHSPECS\"\n #define GIT_QUARANTINE_ENVIRONMENT \"GIT_QUARANTINE_PATH\"\n \n+/*\n+ * Environment variable used in handshaking the wire protocol.\n+ * Contains a colon ':' separated list of keys with optional values\n+ * 'key[=value]'. Presence of unknown keys must be tolerated.\n+ */\n+#define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n+\n /*\n * This environment variable is expected to contain a boolean indicating\n * whether we should or should not treat:\ndiff --git a/protocol.c b/protocol.c\nnew file mode 100644\nindex 000000000..369503065\n--- /dev/null\n+++ b/protocol.c\n@@ -0,0 +1,72 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"protocol.h\"\n+\n+static enum protocol_version parse_protocol_version(const char *value)\n+{\n+\tif (!strcmp(value, \"0\"))\n+\t\treturn protocol_v0;\n+\telse if (!strcmp(value, \"1\"))\n+\t\treturn protocol_v1;\n+\telse\n+\t\treturn protocol_unknown_version;\n+}\n+\n+enum protocol_version get_protocol_version_config(void)\n+{\n+\tconst char *value;\n+\tif (!git_config_get_string_const(\"protocol.version\", &value)) {\n+\t\tenum protocol_version version = parse_protocol_version(value);\n+\n+\t\tif (version == protocol_unknown_version)\n+\t\t\tdie(\"unknown value for config 'protocol.version': %s\",\n+\t\t\t value);\n+\n+\t\treturn version;\n+\t}\n+\n+\treturn protocol_v0;\n+}\n+\n+enum protocol_version determine_protocol_version_server(void)\n+{\n+\tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n+\tenum protocol_version version = protocol_v0;\n+\n+\tif (git_protocol) {\n+\t\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\t\tconst struct string_list_item *item;\n+\t\tstring_list_split(&list, git_protocol, ':', -1);\n+\n+\t\tfor_each_string_list_item(item, &list) {\n+\t\t\tconst char *value;\n+\t\t\tenum protocol_version v;\n+\n+\t\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n+\t\t\t\tv = parse_protocol_version(value);\n+\t\t\t\tif (v > version)\n+\t\t\t\t\tversion = v;\n+\t\t\t}\n+\t\t}\n+\n+\t\tstring_list_clear(&list, 0);\n+\t}\n+\n+\treturn version;\n+}\n+\n+enum protocol_version determine_protocol_version_client(const char *server_response)\n+{\n+\tenum protocol_version version = protocol_v0;\n+\n+\tif (skip_prefix(server_response, \"version \", &server_response)) {\n+\t\tversion = parse_protocol_version(server_response);\n+\n+\t\tif (version == protocol_unknown_version)\n+\t\t\tdie(\"server is speaking an unknown protocol\");\n+\t\tif (version == protocol_v0)\n+\t\t\tdie(\"protocol error: server explicitly said version 0\");\n+\t}\n+\n+\treturn version;\n+}\ndiff --git a/protocol.h b/protocol.h\nnew file mode 100644\nindex 000000000..18f9a5235\n--- /dev/null\n+++ b/protocol.h\n@@ -0,0 +1,14 @@\n+#ifndef PROTOCOL_H\n+#define PROTOCOL_H\n+\n+enum protocol_version {\n+\tprotocol_unknown_version = -1,\n+\tprotocol_v0 = 0,\n+\tprotocol_v1 = 1,\n+};\n+\n+extern enum protocol_version get_protocol_version_config(void);\n+extern enum protocol_version determine_protocol_version_server(void);\n+extern enum protocol_version determine_protocol_version_client(const char *server_response);\n+\n+#endif /* PROTOCOL_H */\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328979","messageId":"20170926235627.79606-2-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v2 1/9] connect: in ref advertisement, shallows are last","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:19Z","receivedAt":"2017-09-26T23:57:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"From: Jonathan Tan \n\nCurrently, get_remote_heads() parses the ref advertisement in one loop,\nallowing refs and shallow lines to intersperse, despite this not being\nallowed by the specification. Refactor get_remote_heads() to use two\nloops instead, enforcing that refs come first, and then shallows.\n\nThis also makes it easier to teach get_remote_heads() to interpret other\nlines in the ref advertisement, which will be done in a subsequent\npatch.\n\nAs part of this change, this patch interprets capabilities only on the\nfirst line in the ref advertisement, printing a warning message when\nencountering capabilities on other lines.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Brandon Williams \n---\n connect.c | 189 ++++++++++++++++++++++++++++++++++++++++----------------------\n 1 file changed, 123 insertions(+), 66 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex df56c0cbf..8e2e276b6 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -11,6 +11,7 @@\n #include \"string-list.h\"\n #include \"sha1-array.h\"\n #include \"transport.h\"\n+#include \"strbuf.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -107,6 +108,104 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n+/*\n+ * Read one line of a server's ref advertisement into packet_buffer.\n+ */\n+static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n+\t\t\t int *responded)\n+{\n+\tint len = packet_read(in, src_buf, src_len,\n+\t\t\t packet_buffer, sizeof(packet_buffer),\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\tconst char *arg;\n+\tif (len < 0)\n+\t\tdie_initial_contact(*responded);\n+\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n+\t\tdie(\"remote error: %s\", arg);\n+\n+\t*responded = 1;\n+\n+\treturn len;\n+}\n+\n+#define EXPECTING_FIRST_REF 0\n+#define EXPECTING_REF 1\n+#define EXPECTING_SHALLOW 2\n+\n+static void process_capabilities(int *len)\n+{\n+\tint nul_location = strlen(packet_buffer);\n+\tif (nul_location == *len)\n+\t\treturn;\n+\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n+\t*len = nul_location;\n+}\n+\n+static int process_dummy_ref(void)\n+{\n+\tstruct object_id oid;\n+\tconst char *name;\n+\n+\tif (parse_oid_hex(packet_buffer, &oid, &name))\n+\t\treturn 0;\n+\tif (*name != ' ')\n+\t\treturn 0;\n+\tname++;\n+\n+\treturn !oidcmp(&null_oid, &oid) && !strcmp(name, \"capabilities^{}\");\n+}\n+\n+static void check_no_capabilities(int len)\n+{\n+\tif (strlen(packet_buffer) != len)\n+\t\twarning(\"Ignoring capabilities after first line '%s'\",\n+\t\t\tpacket_buffer + strlen(packet_buffer));\n+}\n+\n+static int process_ref(int len, struct ref ***list, unsigned int flags,\n+\t\t struct oid_array *extra_have)\n+{\n+\tstruct object_id old_oid;\n+\tconst char *name;\n+\n+\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n+\t\treturn 0;\n+\tif (*name != ' ')\n+\t\treturn 0;\n+\tname++;\n+\n+\tif (extra_have && !strcmp(name, \".have\")) {\n+\t\toid_array_append(extra_have, &old_oid);\n+\t} else if (!strcmp(name, \"capabilities^{}\")) {\n+\t\tdie(\"protocol error: unexpected capabilities^{}\");\n+\t} else if (check_ref(name, flags)) {\n+\t\tstruct ref *ref = alloc_ref(name);\n+\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\t**list = ref;\n+\t\t*list = &ref->next;\n+\t}\n+\tcheck_no_capabilities(len);\n+\treturn 1;\n+}\n+\n+static int process_shallow(int len, struct oid_array *shallow_points)\n+{\n+\tconst char *arg;\n+\tstruct object_id old_oid;\n+\n+\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\t\treturn 0;\n+\n+\tif (get_oid_hex(arg, &old_oid))\n+\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n+\tif (!shallow_points)\n+\t\tdie(\"repository on the other end cannot be shallow\");\n+\toid_array_append(shallow_points, &old_oid);\n+\tcheck_no_capabilities(len);\n+\treturn 1;\n+}\n+\n /*\n * Read all the refs from the other end\n */\n@@ -123,76 +222,34 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t * willing to talk to us. A hang-up before seeing any\n \t * response does not necessarily mean an ACL problem, though.\n \t */\n-\tint saw_response;\n-\tint got_dummy_ref_with_capabilities_declaration = 0;\n+\tint responded = 0;\n+\tint len;\n+\tint state = EXPECTING_FIRST_REF;\n \n \t*list = NULL;\n-\tfor (saw_response = 0; ; saw_response = 1) {\n-\t\tstruct ref *ref;\n-\t\tstruct object_id old_oid;\n-\t\tchar *name;\n-\t\tint len, name_len;\n-\t\tchar *buffer = packet_buffer;\n-\t\tconst char *arg;\n-\n-\t\tlen = packet_read(in, &src_buf, &src_len,\n-\t\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\t\tif (len < 0)\n-\t\t\tdie_initial_contact(saw_response);\n-\n-\t\tif (!len)\n-\t\t\tbreak;\n-\n-\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n-\t\t\tdie(\"remote error: %s\", arg);\n-\n-\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n-\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &old_oid))\n-\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n-\t\t\tif (!shallow_points)\n-\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n-\t\t\toid_array_append(shallow_points, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n \n-\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n-\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n-\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n-\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n-\n-\t\tname_len = strlen(name);\n-\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n-\t\t\tfree(server_capabilities);\n-\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n-\t\t}\n-\n-\t\tif (extra_have && !strcmp(name, \".have\")) {\n-\t\t\toid_array_append(extra_have, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (!strcmp(name, \"capabilities^{}\")) {\n-\t\t\tif (saw_response)\n-\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n-\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n-\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n-\t\t\tcontinue;\n+\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\t\tswitch (state) {\n+\t\tcase EXPECTING_FIRST_REF:\n+\t\t\tprocess_capabilities(&len);\n+\t\t\tif (process_dummy_ref()) {\n+\t\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tstate = EXPECTING_REF;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_REF:\n+\t\t\tif (process_ref(len, &list, flags, extra_have))\n+\t\t\t\tbreak;\n+\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_SHALLOW:\n+\t\t\tif (process_shallow(len, shallow_points))\n+\t\t\t\tbreak;\n+\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n+\t\tdefault:\n+\t\t\tdie(\"unexpected state %d\", state);\n \t\t}\n-\n-\t\tif (!check_ref(name, flags))\n-\t\t\tcontinue;\n-\n-\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n-\n-\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n-\t\t*list = ref;\n-\t\tlist = &ref->next;\n \t}\n \n \tannotate_refs_with_symref_info(*orig_list);\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328980","messageId":"20170926235627.79606-7-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v2 6/9] connect: teach client to recognize v1 server response","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:24Z","receivedAt":"2017-09-26T23:57:04Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach a client to recognize that a server understands protocol v1 by\nlooking at the first pkt-line the server sends in response. This is\ndone by looking for the response \"version 1\" send by upload-pack or\nreceive-pack.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 30 ++++++++++++++++++++++++++----\n 1 file changed, 26 insertions(+), 4 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 8e2e276b6..1805debf3 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -12,6 +12,7 @@\n #include \"sha1-array.h\"\n #include \"transport.h\"\n #include \"strbuf.h\"\n+#include \"protocol.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -129,9 +130,23 @@ static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n \treturn len;\n }\n \n-#define EXPECTING_FIRST_REF 0\n-#define EXPECTING_REF 1\n-#define EXPECTING_SHALLOW 2\n+#define EXPECTING_PROTOCOL_VERSION 0\n+#define EXPECTING_FIRST_REF 1\n+#define EXPECTING_REF 2\n+#define EXPECTING_SHALLOW 3\n+\n+/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n+static int process_protocol_version(void)\n+{\n+\tswitch (determine_protocol_version_client(packet_buffer)) {\n+\t\tcase protocol_v1:\n+\t\t\treturn 1;\n+\t\tcase protocol_v0:\n+\t\t\treturn 0;\n+\t\tdefault:\n+\t\t\tdie(\"server is speaking an unknown protocol\");\n+\t}\n+}\n \n static void process_capabilities(int *len)\n {\n@@ -224,12 +239,19 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t */\n \tint responded = 0;\n \tint len;\n-\tint state = EXPECTING_FIRST_REF;\n+\tint state = EXPECTING_PROTOCOL_VERSION;\n \n \t*list = NULL;\n \n \twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n \t\tswitch (state) {\n+\t\tcase EXPECTING_PROTOCOL_VERSION:\n+\t\t\tif (process_protocol_version()) {\n+\t\t\t\tstate = EXPECTING_FIRST_REF;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tstate = EXPECTING_FIRST_REF;\n+\t\t\t/* fallthrough */\n \t\tcase EXPECTING_FIRST_REF:\n \t\t\tprocess_capabilities(&len);\n \t\t\tif (process_dummy_ref()) {\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328981","messageId":"20170926235627.79606-8-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v2 7/9] connect: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:25Z","receivedAt":"2017-09-26T23:57:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach the connection logic to tell a serve that it understands protocol\nv1. This is done in 2 different ways for the built in protocols.\n\n1. git://\n A normal request is structured as \"command path/to/repo\\0host=..\\0\"\n and due to a bug in an old version of git-daemon 73bb33a94 (daemon:\n Strictly parse the \"extra arg\" part of the command, 2009-06-04) we\n aren't able to place any extra args (separated by NULs) besides the\n host. In order to get around this limitation put protocol version\n information after a second NUL byte so the request is structured\n like: \"command path/to/repo\\0host=..\\0\\0version=1\\0\". git-daemon can\n then parse out the version number and set GIT_PROTOCOL.\n\n2. ssh://, file://\n Set GIT_PROTOCOL envvar with the desired protocol version. The\n envvar can be sent across ssh by using '-o SendEnv=GIT_PROTOCOL' and\n having the server whitelist this envvar.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 37 ++++++--\n t/t5700-protocol-v1.sh | 223 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 255 insertions(+), 5 deletions(-)\n create mode 100755 t/t5700-protocol-v1.sh\n\ndiff --git a/connect.c b/connect.c\nindex 1805debf3..12ebab724 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -871,6 +871,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tprintf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n \t\tconn = NULL;\n \t} else if (protocol == PROTO_GIT) {\n+\t\tstruct strbuf request = STRBUF_INIT;\n \t\t/*\n \t\t * Set up virtual host information based on where we will\n \t\t * connect, unless the user has overridden us in\n@@ -898,13 +899,25 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t * Note: Do not add any other headers here! Doing so\n \t\t * will cause older git-daemon servers to crash.\n \t\t */\n-\t\tpacket_write_fmt(fd[1],\n-\t\t\t \"%s %s%chost=%s%c\",\n-\t\t\t prog, path, 0,\n-\t\t\t target_host, 0);\n+\t\tstrbuf_addf(&request,\n+\t\t\t \"%s %s%chost=%s%c\",\n+\t\t\t prog, path, 0,\n+\t\t\t target_host, 0);\n+\n+\t\t/* If using a new version put that stuff here after a second null byte */\n+\t\tif (get_protocol_version_config() > 0) {\n+\t\t\tstrbuf_addch(&request, '\\0');\n+\t\t\tstrbuf_addf(&request, \"version=%d%c\",\n+\t\t\t\t get_protocol_version_config(), '\\0');\n+\t\t}\n+\n+\t\tpacket_write(fd[1], request.buf, request.len);\n+\n \t\tfree(target_host);\n+\t\tstrbuf_release(&request);\n \t} else {\n \t\tstruct strbuf cmd = STRBUF_INIT;\n+\t\tconst char *const *var;\n \n \t\tconn = xmalloc(sizeof(*conn));\n \t\tchild_process_init(conn);\n@@ -917,7 +930,9 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tsq_quote_buf(&cmd, path);\n \n \t\t/* remove repo-local variables from the environment */\n-\t\tconn->env = local_repo_env;\n+\t\tfor (var = local_repo_env; *var; var++)\n+\t\t\targv_array_push(&conn->env_array, *var);\n+\n \t\tconn->use_shell = 1;\n \t\tconn->in = conn->out = -1;\n \t\tif (protocol == PROTO_SSH) {\n@@ -971,6 +986,14 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t}\n \n \t\t\targv_array_push(&conn->args, ssh);\n+\n+\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\t\targv_array_push(&conn->args, \"-o\");\n+\t\t\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n+\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t}\n+\n \t\t\tif (flags & CONNECT_IPV4)\n \t\t\t\targv_array_push(&conn->args, \"-4\");\n \t\t\telse if (flags & CONNECT_IPV6)\n@@ -985,6 +1008,10 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\targv_array_push(&conn->args, ssh_host);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\n+\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t}\n \t\t}\n \t\targv_array_push(&conn->args, cmd.buf);\n \ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nnew file mode 100755\nindex 000000000..1988bbce6\n--- /dev/null\n+++ b/t/t5700-protocol-v1.sh\n@@ -0,0 +1,223 @@\n+#!/bin/sh\n+\n+test_description='test git wire-protocol transition'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+. ./test-lib.sh\n+\n+# Test protocol v1 with 'git://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\ttest_commit -C \"$daemon_parent\" one\n+'\n+\n+test_expect_success 'clone with git:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"$GIT_DAEMON_URL/parent\" daemon_child 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with git:// using protocol v1' '\n+\ttest_commit -C \"$daemon_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with git:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with git:// using protocol v1' '\n+\ttest_commit -C daemon_child three &&\n+\n+\t# Since the repository being served isnt bare we need to push to\n+\t# another branch explicitly to avoid mangling the master branch\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+stop_git_daemon\n+\n+# Test protocol v1 with 'file://' transport\n+#\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init file_parent &&\n+\ttest_commit -C file_parent one\n+'\n+\n+test_expect_success 'clone with file:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"file://$(pwd)/file_parent\" file_child 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with file:// using protocol v1' '\n+\ttest_commit -C file_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with file:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with file:// using protocol v1' '\n+\ttest_commit -C file_child three &&\n+\n+\t# Since the repository being served isnt bare we need to push to\n+\t# another branch explicitly to avoid mangling the master branch\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+# Test protocol v1 with 'ssh://' transport\n+#\n+test_expect_success 'setup ssh wrapper' '\n+\tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh$X\" &&\n+\texport GIT_SSH &&\n+\texport TRASH_DIRECTORY &&\n+\t>\"$TRASH_DIRECTORY\"/ssh-output\n+'\n+\n+expect_ssh () {\n+\ttest_when_finished '(cd \"$TRASH_DIRECTORY\" && rm -f ssh-expect && >ssh-output)' &&\n+\techo \"ssh: -o SendEnv=GIT_PROTOCOL myhost $1 '$PWD/ssh_parent'\" >\"$TRASH_DIRECTORY/ssh-expect\" &&\n+\t(cd \"$TRASH_DIRECTORY\" && test_cmp ssh-expect ssh-output)\n+}\n+\n+test_expect_success 'create repo to be served by ssh:// transport' '\n+\tgit init ssh_parent &&\n+\ttest_commit -C ssh_parent one\n+'\n+\n+test_expect_success 'clone with ssh:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"ssh://myhost:$(pwd)/ssh_parent\" ssh_child 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with ssh:// using protocol v1' '\n+\ttest_commit -C ssh_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with ssh:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with ssh:// using protocol v1' '\n+\ttest_commit -C ssh_child three &&\n+\n+\t# Since the repository being served isnt bare we need to push to\n+\t# another branch explicitly to avoid mangling the master branch\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\texpect_ssh git-receive-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+test_done\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328982","messageId":"20170926235627.79606-10-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v2 9/9] i5700: add interop test for protocol transition","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:27Z","receivedAt":"2017-09-26T23:57:08Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams \n---\n t/interop/i5700-protocol-transition.sh | 68 ++++++++++++++++++++++++++++++++++\n 1 file changed, 68 insertions(+)\n create mode 100755 t/interop/i5700-protocol-transition.sh\n\ndiff --git a/t/interop/i5700-protocol-transition.sh b/t/interop/i5700-protocol-transition.sh\nnew file mode 100755\nindex 000000000..9e83428a8\n--- /dev/null\n+++ b/t/interop/i5700-protocol-transition.sh\n@@ -0,0 +1,68 @@\n+#!/bin/sh\n+\n+VERSION_A=.\n+VERSION_B=v2.0.0\n+\n+: ${LIB_GIT_DAEMON_PORT:=5600}\n+LIB_GIT_DAEMON_COMMAND='git.b daemon'\n+\n+test_description='clone and fetch by client who is trying to use a new protocol'\n+. ./interop-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\n+start_git_daemon --export-all\n+\n+repo=$GIT_DAEMON_DOCUMENT_ROOT_PATH/repo\n+\n+test_expect_success \"create repo served by $VERSION_B\" '\n+\tgit.b init \"$repo\" &&\n+\tgit.b -C \"$repo\" commit --allow-empty -m one\n+'\n+\n+test_expect_success \"git:// clone with $VERSION_A and protocol v1\" '\n+\tGIT_TRACE_PACKET=1 git.a -c protocol.version=1 clone \"$GIT_DAEMON_URL/repo\" child 2>log &&\n+\tgit.a -C child log -1 --format=%s >actual &&\n+\tgit.b -C \"$repo\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\tgrep \"version=1\" log\n+'\n+\n+test_expect_success \"git:// fetch with $VERSION_A and protocol v1\" '\n+\tgit.b -C \"$repo\" commit --allow-empty -m two &&\n+\tgit.b -C \"$repo\" log -1 --format=%s >expect &&\n+\n+\tGIT_TRACE_PACKET=1 git.a -C child -c protocol.version=1 fetch 2>log &&\n+\tgit.a -C child log -1 --format=%s FETCH_HEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"version=1\" log &&\n+\t! grep \"version 1\" log\n+'\n+\n+stop_git_daemon\n+\n+test_expect_success \"create repo served by $VERSION_B\" '\n+\tgit.b init parent &&\n+\tgit.b -C parent commit --allow-empty -m one\n+'\n+\n+test_expect_success \"file:// clone with $VERSION_A and protocol v1\" '\n+\tGIT_TRACE_PACKET=1 git.a -c protocol.version=1 clone --upload-pack=\"git.b upload-pack\" parent child2 2>log &&\n+\tgit.a -C child2 log -1 --format=%s >actual &&\n+\tgit.b -C parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\t! grep \"version 1\" log\n+'\n+\n+test_expect_success \"file:// fetch with $VERSION_A and protocol v1\" '\n+\tgit.b -C parent commit --allow-empty -m two &&\n+\tgit.b -C parent log -1 --format=%s >expect &&\n+\n+\tGIT_TRACE_PACKET=1 git.a -C child2 -c protocol.version=1 fetch --upload-pack=\"git.b upload-pack\" 2>log &&\n+\tgit.a -C child2 log -1 --format=%s FETCH_HEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\t! grep \"version 1\" log\n+'\n+\n+test_done\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328983","messageId":"20170926235627.79606-9-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v2 8/9] http: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:26Z","receivedAt":"2017-09-26T23:57:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Tell a server that protocol v1 can be used by sending the http header\n'Git-Protocol' indicating this.\n\nAlso teach the apache http server to pass through the 'Git-Protocol'\nheader as an environment variable 'GIT_PROTOCOL'.\n\nSigned-off-by: Brandon Williams \n---\n cache.h | 2 ++\n http.c | 18 +++++++++++++\n t/lib-httpd/apache.conf | 7 +++++\n t/t5700-protocol-v1.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 96 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex 0c792545f..aaadac1f0 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -451,6 +451,8 @@ static inline enum object_type object_type(unsigned int mode)\n * 'key[=value]'. Presence of unknown keys must be tolerated.\n */\n #define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n+/* HTTP header used to handshake the wire protocol */\n+#define GIT_PROTOCOL_HEADER \"Git-Protocol\"\n \n /*\n * This environment variable is expected to contain a boolean indicating\ndiff --git a/http.c b/http.c\nindex 9e40a465f..ffb719216 100644\n--- a/http.c\n+++ b/http.c\n@@ -12,6 +12,7 @@\n #include \"gettext.h\"\n #include \"transport.h\"\n #include \"packfile.h\"\n+#include \"protocol.h\"\n \n static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);\n #if LIBCURL_VERSION_NUM >= 0x070a08\n@@ -897,6 +898,21 @@ static void set_from_env(const char **var, const char *envname)\n \t\t*var = val;\n }\n \n+static void protocol_http_header(void)\n+{\n+\tif (get_protocol_version_config() > 0) {\n+\t\tstruct strbuf protocol_header = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&protocol_header, GIT_PROTOCOL_HEADER \": version=%d\",\n+\t\t\t get_protocol_version_config());\n+\n+\n+\t\textra_http_headers = curl_slist_append(extra_http_headers,\n+\t\t\t\t\t\t protocol_header.buf);\n+\t\tstrbuf_release(&protocol_header);\n+\t}\n+}\n+\n void http_init(struct remote *remote, const char *url, int proactive_auth)\n {\n \tchar *low_speed_limit;\n@@ -927,6 +943,8 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tif (remote)\n \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n \n+\tprotocol_http_header();\n+\n \tpragma_header = curl_slist_append(http_copy_default_headers(),\n \t\t\"Pragma: no-cache\");\n \tno_pragma_header = curl_slist_append(http_copy_default_headers(),\ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 0642ae7e6..df1943631 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -67,6 +67,9 @@ LockFile accept.lock\n \n \tLoadModule unixd_module modules/mod_unixd.so\n \n+\n+\tLoadModule setenvif_module modules/mod_setenvif.so\n+\n \n \n PassEnv GIT_VALGRIND\n@@ -76,6 +79,10 @@ PassEnv ASAN_OPTIONS\n PassEnv GIT_TRACE\n PassEnv GIT_CONFIG_NOSYSTEM\n \n+= 2.4>\n+\tSetEnvIf Git-Protocol \".*\" GIT_PROTOCOL=$0\n+\n+\n Alias /dumb/ www/\n Alias /auth/dumb/ www/auth/dumb/\n \ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nindex 1988bbce6..222265127 100755\n--- a/t/t5700-protocol-v1.sh\n+++ b/t/t5700-protocol-v1.sh\n@@ -220,4 +220,73 @@ test_expect_success 'push with ssh:// using protocol v1' '\n \tgrep \"push< version 1\" log\n '\n \n+# Test protocol v1 with 'http://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one\n+'\n+\n+test_expect_success 'clone with http:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 GIT_TRACE_CURL=1 git -c protocol.version=1 \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"Git-Protocol: version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'fetch with http:// using protocol v1' '\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'pull with http:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'push with http:// using protocol v1' '\n+\ttest_commit -C http_child three &&\n+\n+\t# Since the repository being served isnt bare we need to push to\n+\t# another branch explicitly to avoid mangling the master branch\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch && #2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328984","messageId":"20170926235627.79606-6-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v2 5/9] upload-pack, receive-pack: introduce protocol version 1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:23Z","receivedAt":"2017-09-26T23:57:12Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach upload-pack and receive-pack to understand and respond using\nprotocol version 1, if requested.\n\nProtocol version 1 is simply the original and current protocol (what I'm\ncalling version 0) with the addition of a single packet line, which\nprecedes the ref advertisement, indicating the protocol version being\nspoken.\n\nSigned-off-by: Brandon Williams \n---\n builtin/receive-pack.c | 14 ++++++++++++++\n upload-pack.c | 17 ++++++++++++++++-\n 2 files changed, 30 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex dd06b3fb4..cb179367b 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -24,6 +24,7 @@\n #include \"tmp-objdir.h\"\n #include \"oidset.h\"\n #include \"packfile.h\"\n+#include \"protocol.h\"\n \n static const char * const receive_pack_usage[] = {\n \tN_(\"git receive-pack \"),\n@@ -1963,6 +1964,19 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \telse if (0 <= receive_unpack_limit)\n \t\tunpack_limit = receive_unpack_limit;\n \n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\tcase protocol_v0:\n+\tdefault:\n+\t\tbreak;\n+\t}\n+\n \tif (advertise_refs || !stateless_rpc) {\n \t\twrite_head_info();\n \t}\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 7efff2fbf..5cab39819 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -18,6 +18,7 @@\n #include \"parse-options.h\"\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n+#include \"protocol.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -1067,6 +1068,20 @@ int cmd_main(int argc, const char **argv)\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n \tgit_config(upload_pack_config, NULL);\n-\tupload_pack();\n+\n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\tcase protocol_v0:\n+\tdefault:\n+\t\tupload_pack();\n+\t\tbreak;\n+\t}\n+\n \treturn 0;\n }\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328985","messageId":"20170926235627.79606-5-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v2 4/9] daemon: recognize hidden request arguments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-26T23:56:22Z","receivedAt":"2017-09-26T23:57:14Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"A normal request to git-daemon is structured as\n\"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\ngit-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\ncommand, 2009-06-04) we aren't able to place any extra args (separated\nby NULs) besides the host.\n\nIn order to get around this limitation teach git-daemon to recognize\nadditional request arguments hidden behind a second NUL byte. Requests\ncan then be structured like:\n\"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\ncan then parse out the extra arguments and set 'GIT_PROTOCOL'\naccordingly.\n\nSigned-off-by: Brandon Williams \n---\n daemon.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 59 insertions(+), 9 deletions(-)\n\ndiff --git a/daemon.c b/daemon.c\nindex 30747075f..36cc794c9 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -282,7 +282,7 @@ static const char *path_ok(const char *directory, struct hostinfo *hi)\n \treturn NULL;\t\t/* Fallthrough. Deny by default */\n }\n \n-typedef int (*daemon_service_fn)(void);\n+typedef int (*daemon_service_fn)(const struct argv_array *env);\n struct daemon_service {\n \tconst char *name;\n \tconst char *config_name;\n@@ -363,7 +363,7 @@ static int run_access_hook(struct daemon_service *service, const char *dir,\n }\n \n static int run_service(const char *dir, struct daemon_service *service,\n-\t\t struct hostinfo *hi)\n+\t\t struct hostinfo *hi, const struct argv_array *env)\n {\n \tconst char *path;\n \tint enabled = service->enabled;\n@@ -422,7 +422,7 @@ static int run_service(const char *dir, struct daemon_service *service,\n \t */\n \tsignal(SIGTERM, SIG_IGN);\n \n-\treturn service->fn();\n+\treturn service->fn(env);\n }\n \n static void copy_to_log(int fd)\n@@ -462,25 +462,34 @@ static int run_service_command(struct child_process *cld)\n \treturn finish_command(cld);\n }\n \n-static int upload_pack(void)\n+static int upload_pack(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_pushl(&cld.args, \"upload-pack\", \"--strict\", NULL);\n \targv_array_pushf(&cld.args, \"--timeout=%u\", timeout);\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n-static int upload_archive(void)\n+static int upload_archive(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_push(&cld.args, \"upload-archive\");\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n-static int receive_pack(void)\n+static int receive_pack(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_push(&cld.args, \"receive-pack\");\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n@@ -574,7 +583,7 @@ static void canonicalize_client(struct strbuf *out, const char *in)\n /*\n * Read the host as supplied by the client connection.\n */\n-static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n+static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n {\n \tchar *val;\n \tint vallen;\n@@ -602,6 +611,43 @@ static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n \t\tif (extra_args < end && *extra_args)\n \t\t\tdie(\"Invalid request\");\n \t}\n+\n+\treturn extra_args;\n+}\n+\n+static void parse_extra_args(struct hostinfo *hi, struct argv_array *env,\n+\t\t\t char *extra_args, int buflen)\n+{\n+\tconst char *end = extra_args + buflen;\n+\tstruct strbuf git_protocol = STRBUF_INIT;\n+\n+\t/* First look for the host argument */\n+\textra_args = parse_host_arg(hi, extra_args, buflen);\n+\n+\t/* Look for additional arguments places after a second NUL byte */\n+\tfor (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n+\t\tconst char *arg = extra_args;\n+\n+\t\t/*\n+\t\t * Parse the extra arguments, adding most to 'git_protocol'\n+\t\t * which will be used to set the 'GIT_PROTOCOL' envvar in the\n+\t\t * service that will be run.\n+\t\t *\n+\t\t * If there ends up being a particular arg in the future that\n+\t\t * git-daemon needs to parse specificly (like the 'host' arg)\n+\t\t * then it can be parsed here and not added to 'git_protocol'.\n+\t\t */\n+\t\tif (*arg) {\n+\t\t\tif (git_protocol.len > 0)\n+\t\t\t\tstrbuf_addch(&git_protocol, ':');\n+\t\t\tstrbuf_addstr(&git_protocol, arg);\n+\t\t}\n+\t}\n+\n+\tif (git_protocol.len > 0)\n+\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n+\t\t\t\t git_protocol.buf);\n+\tstrbuf_release(&git_protocol);\n }\n \n /*\n@@ -695,6 +741,7 @@ static int execute(void)\n \tint pktlen, len, i;\n \tchar *addr = getenv(\"REMOTE_ADDR\"), *port = getenv(\"REMOTE_PORT\");\n \tstruct hostinfo hi;\n+\tstruct argv_array env = ARGV_ARRAY_INIT;\n \n \thostinfo_init(&hi);\n \n@@ -716,8 +763,9 @@ static int execute(void)\n \t\tpktlen--;\n \t}\n \n+\t/* parse additional args hidden behind a NUL byte */\n \tif (len != pktlen)\n-\t\tparse_host_arg(&hi, line + len + 1, pktlen - len - 1);\n+\t\tparse_extra_args(&hi, &env, line + len + 1, pktlen - len - 1);\n \n \tfor (i = 0; i < ARRAY_SIZE(daemon_service); i++) {\n \t\tstruct daemon_service *s = &(daemon_service[i]);\n@@ -730,13 +778,15 @@ static int execute(void)\n \t\t\t * Note: The directory here is probably context sensitive,\n \t\t\t * and might depend on the actual service being performed.\n \t\t\t */\n-\t\t\tint rc = run_service(arg, s, &hi);\n+\t\t\tint rc = run_service(arg, s, &hi, &env);\n \t\t\thostinfo_clear(&hi);\n+\t\t\targv_array_clear(&env);\n \t\t\treturn rc;\n \t\t}\n \t}\n \n \thostinfo_clear(&hi);\n+\targv_array_clear(&env);\n \tlogerror(\"Protocol error: '%s'\", line);\n \treturn -1;\n }\n-- \n2.14.1.992.g2c7b836f3a-goog\n\n"},{"id":"328992","messageId":"xmqqh8vp7xv4.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170926235627.79606-7-bmwill@google.com","subject":"Re: [PATCH v2 6/9] connect: teach client to recognize v1 server response","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-27T01:07:27Z","receivedAt":"2017-09-27T01:07:34Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> +/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n> +static int process_protocol_version(void)\n> +{\n> +\tswitch (determine_protocol_version_client(packet_buffer)) {\n> +\t\tcase protocol_v1:\n> +\t\t\treturn 1;\n> +\t\tcase protocol_v0:\n> +\t\t\treturn 0;\n> +\t\tdefault:\n> +\t\t\tdie(\"server is speaking an unknown protocol\");\n> +\t}\n> +}\n\ncheckpatch.pl yells at me:\n\n ERROR: switch and case should be at the same indent\n\nand we would probably want to teach \"make style\" the same, if we\nalready don't.\n"},{"id":"328999","messageId":"xmqq4lro7mab.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170926235627.79606-4-bmwill@google.com","subject":"Re: [PATCH v2 3/9] protocol: introduce protocol extention mechanisms","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-27T05:17:32Z","receivedAt":"2017-09-27T05:17:46Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> +`GIT_PROTOCOL`::\n> +\tFor internal use only. Used in handshaking the wire protocol.\n> +\tContains a colon ':' separated list of keys with optional values\n> +\t'key[=value]'. Presence of unknown keys must be tolerated.\n\nIs this meant to be used only on the \"server\" end? Am I correct to\ninterpret \"handshaking\" to mean the initial connection acceptor\n(e.g. \"git daemon\") uses it to pass what it decided to the programs\nthat implement the service (e.g. \"git receive-pack\")?\n\n> +/*\n> + * Environment variable used in handshaking the wire protocol.\n> + * Contains a colon ':' separated list of keys with optional values\n> + * 'key[=value]'. Presence of unknown keys must be tolerated.\n> + */\n> +#define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n\n\"Must be tolerated\" feels a bit strange. When somebody asks you to\nuse \"version 3\" or \"version 1 variant 2\", when you only know\n\"version 0\" or \"version 1\" and you are not yet even aware of the\nconcept of \"variant\", we simply ignore \"variant=2\" as if it wasn't\nthere, even though \"version=3\" will be rejected (because we know of\n\"version\"; it's just that we don't know \"version=3\").\n\n> +enum protocol_version determine_protocol_version_server(void)\n> +{\n> +\tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n> +\tenum protocol_version version = protocol_v0;\n> +\n> +\tif (git_protocol) {\n> +\t\tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\t\tconst struct string_list_item *item;\n> +\t\tstring_list_split(&list, git_protocol, ':', -1);\n> +\n> +\t\tfor_each_string_list_item(item, &list) {\n> +\t\t\tconst char *value;\n> +\t\t\tenum protocol_version v;\n> +\n> +\t\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n> +\t\t\t\tv = parse_protocol_version(value);\n> +\t\t\t\tif (v > version)\n> +\t\t\t\t\tversion = v;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tstring_list_clear(&list, 0);\n> +\t}\n> +\n> +\treturn version;\n> +}\n\nThis implements \"the largest one wins\", not \"the last one wins\". Is\nthere a particular reason why the former is chosen?\n\n"},{"id":"329001","messageId":"xmqqzi9g67kl.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170926235627.79606-5-bmwill@google.com","subject":"Re: [PATCH v2 4/9] daemon: recognize hidden request arguments","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-27T05:20:42Z","receivedAt":"2017-09-27T05:20:48Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> A normal request to git-daemon is structured as\n> \"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\n> git-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\n> command, 2009-06-04) we aren't able to place any extra args (separated\n> by NULs) besides the host.\n\nIt's a bit unclear if that commit _introduced_ a bug, or just\nnoticed an old bug and documented it in its log message. How does\nthat commit impact the versons of Git that the updated code is\ncapable of interracting with?\n\n> +static void parse_extra_args(struct hostinfo *hi, struct argv_array *env,\n> +\t\t\t char *extra_args, int buflen)\n> +{\n> +\tconst char *end = extra_args + buflen;\n> +\tstruct strbuf git_protocol = STRBUF_INIT;\n> +\n> +\t/* First look for the host argument */\n> +\textra_args = parse_host_arg(hi, extra_args, buflen);\n> +\n> +\t/* Look for additional arguments places after a second NUL byte */\n> +\tfor (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n> +\t\tconst char *arg = extra_args;\n> +\n> +\t\t/*\n> +\t\t * Parse the extra arguments, adding most to 'git_protocol'\n> +\t\t * which will be used to set the 'GIT_PROTOCOL' envvar in the\n> +\t\t * service that will be run.\n> +\t\t *\n> +\t\t * If there ends up being a particular arg in the future that\n> +\t\t * git-daemon needs to parse specificly (like the 'host' arg)\n> +\t\t * then it can be parsed here and not added to 'git_protocol'.\n> +\t\t */\n> +\t\tif (*arg) {\n> +\t\t\tif (git_protocol.len > 0)\n> +\t\t\t\tstrbuf_addch(&git_protocol, ':');\n> +\t\t\tstrbuf_addstr(&git_protocol, arg);\n> +\t\t}\n> +\t}\n> +\n> +\tif (git_protocol.len > 0)\n> +\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n> +\t\t\t\t git_protocol.buf);\n> +\tstrbuf_release(&git_protocol);\n> }\n"},{"id":"329002","messageId":"xmqqvak467fy.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170926235627.79606-6-bmwill@google.com","subject":"Re: [PATCH v2 5/9] upload-pack, receive-pack: introduce protocol version 1","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-27T05:23:29Z","receivedAt":"2017-09-27T05:23:38Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> @@ -1963,6 +1964,19 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> \telse if (0 <= receive_unpack_limit)\n> \t\tunpack_limit = receive_unpack_limit;\n> \n> +\tswitch (determine_protocol_version_server()) {\n> +\tcase protocol_v1:\n> +\t\tif (advertise_refs || !stateless_rpc)\n> +\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n> +\t\t/*\n> +\t\t * v1 is just the original protocol with a version string,\n> +\t\t * so just fall through after writing the version string.\n> +\t\t */\n> +\tcase protocol_v0:\n> +\tdefault:\n> +\t\tbreak;\n\nWhen protocol_v2 is introduced in the other part of the codebase\n(i.e. in protocol.[ch]), until these lines are updated accordingly\nto take care of the new protocol, we'd pretend that client asked\n(and the server accepted) v0, even though the client and the daemon\nagreed to talk v2.\n\nShouldn't the \"default:\" die instead? The same for upload-pack.c\n"},{"id":"329003","messageId":"xmqqpoac676l.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170926235627.79606-7-bmwill@google.com","subject":"Re: [PATCH v2 6/9] connect: teach client to recognize v1 server response","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-27T05:29:06Z","receivedAt":"2017-09-27T05:29:17Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> +/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n> +static int process_protocol_version(void)\n> +{\n> +\tswitch (determine_protocol_version_client(packet_buffer)) {\n> +\t\tcase protocol_v1:\n> +\t\t\treturn 1;\n> +\t\tcase protocol_v0:\n> +\t\t\treturn 0;\n> +\t\tdefault:\n> +\t\t\tdie(\"server is speaking an unknown protocol\");\n> +\t}\n> +}\n\nFor the purpose of \"technology demonstration\" v1 protocol, it is OK\nto discard the result of \"determine_pvc()\" like the above code, but\nin a real application, we would do a bit more than just ignoring an\nextra \"version #\" packet that appears at the beginning, no?\n\nIt would be sensible to design how the result of determien_pvc()\ncall is propagated to the remainder of the program in this patch and\nimplement it. Perhaps add a new global (like server_capabilities\nalready is) and store the value there, or something? Or pass a\npointer to enum protocol_version as a return-location parameter to\nthis helper function so that the process_capabilities() can pass a\npointer to its local variable?\n\n> static void process_capabilities(int *len)\n> {\n> @@ -224,12 +239,19 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> \t */\n> \tint responded = 0;\n> \tint len;\n> -\tint state = EXPECTING_FIRST_REF;\n> +\tint state = EXPECTING_PROTOCOL_VERSION;\n> \n> \t*list = NULL;\n> \n> \twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> \t\tswitch (state) {\n> +\t\tcase EXPECTING_PROTOCOL_VERSION:\n> +\t\t\tif (process_protocol_version()) {\n> +\t\t\t\tstate = EXPECTING_FIRST_REF;\n> +\t\t\t\tbreak;\n> +\t\t\t}\n> +\t\t\tstate = EXPECTING_FIRST_REF;\n> +\t\t\t/* fallthrough */\n> \t\tcase EXPECTING_FIRST_REF:\n> \t\t\tprocess_capabilities(&len);\n> \t\t\tif (process_dummy_ref()) {\n"},{"id":"329016","messageId":"xmqqk20k64r4.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170926235627.79606-8-bmwill@google.com","subject":"Re: [PATCH v2 7/9] connect: tell server that the client understands v1","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-27T06:21:35Z","receivedAt":"2017-09-27T06:21:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> Teach the connection logic to tell a serve that it understands protocol\n> v1. This is done in 2 different ways for the built in protocols.\n>\n> 1. git://\n> A normal request is structured as \"command path/to/repo\\0host=..\\0\"\n> and due to a bug in an old version of git-daemon 73bb33a94 (daemon:\n> Strictly parse the \"extra arg\" part of the command, 2009-06-04) we\n> aren't able to place any extra args (separated by NULs) besides the\n> host. In order to get around this limitation put protocol version\n> information after a second NUL byte so the request is structured\n> like: \"command path/to/repo\\0host=..\\0\\0version=1\\0\". git-daemon can\n> then parse out the version number and set GIT_PROTOCOL.\n\nSame question as a previous step, wrt the cited commit. It reads as\nif we are saying that the commit introduced a bug and left it there,\nthat we cannot use \\0host=..\\0version=..\\0other=..\\0 until that bug\nis fixed, and that in the meantime we use \\0host=..\\0\\0version=.. as\na workaround, but that reading leaves readers wondering if we want\nto eventually drop this double-NUL workaround. I am guessing that\nwe want to declare that the current protocol has a glitch that\nprevents us to use \\0host=..\\0version=..\\0 but we accept that and\nplan to keep it that way, and we'll use the double-NUL for anything\nother than host from now on, as it is compatible with the current\nversion of Git before this patch (the extras are safely ignored),\nbut then it still leaves readers wonder if the mention of the\nold commit from 2009 means that this double-NUL would not even work\nif the other end is running a version of Git before that commit, or\nwe are safe to talk with versions of Git even older than that.\n\nI do not think it is a showstopper if we did not work with v1.6.4,\nbut it still needs to be clarified.\n\n> 2. ssh://, file://\n> Set GIT_PROTOCOL envvar with the desired protocol version. The\n> envvar can be sent across ssh by using '-o SendEnv=GIT_PROTOCOL' and\n> having the server whitelist this envvar.\n\nOpenSSH lets us do this, but I do not know how well this works with\nother implementations of SSH clients. The log message perhaps needs\nto ask for volunteers to check if it is OK with the implementations\nthey use, and offer conditional code (just like we have for putty\nand plink customizations) otherwise.\n\nOther than that, the code changes looked good.\n\n> diff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\n> new file mode 100755\n> index 000000000..1988bbce6\n> --- /dev/null\n> +++ b/t/t5700-protocol-v1.sh\n> @@ -0,0 +1,223 @@\n> +#!/bin/sh\n> +\n> +test_description='test git wire-protocol transition'\n> +\n> +TEST_NO_CREATE_REPO=1\n> +\n> +. ./test-lib.sh\n> +\n> +# Test protocol v1 with 'git://' transport\n> +#\n> +. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n> +start_git_daemon --export-all --enable=receive-pack\n> +daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n> +\n> +test_expect_success 'create repo to be served by git-daemon' '\n> +\tgit init \"$daemon_parent\" &&\n> +\ttest_commit -C \"$daemon_parent\" one\n> +'\n> +\n> +test_expect_success 'clone with git:// using protocol v1' '\n> +\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n> +\t\tclone \"$GIT_DAEMON_URL/parent\" daemon_child 2>log &&\n> +\n> +\tgit -C daemon_child log -1 --format=%s >actual &&\n> +\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n> +\ttest_cmp expect actual &&\n> +\n> +\t# Client requested to use protocol v1\n> +\tgrep \"version=1\" log &&\n> +\t# Server responded using protocol v1\n> +\tgrep \"clone< version 1\" log\n\nThis looked a bit strange to check \"clone< version 1\" for one\ndirection, but did not check \"$something> version 1\" for the other\ndirection. Doesn't \"version=1\" end up producing 2 hits?\n\nNot a complaint, but wondering if we can write it in such a way that\ndoes not have to make readers wonder.\n\n> +'\n> +\n> +test_expect_success 'fetch with git:// using protocol v1' '\n> +\ttest_commit -C \"$daemon_parent\" two &&\n> +\n> +\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n> +\t\tfetch 2>log &&\n> +\n> +\tgit -C daemon_child log -1 --format=%s FETCH_HEAD >actual &&\n> +\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n> +\ttest_cmp expect actual &&\n\nOK. So the origin repository gained one commit on the 'master'\nbranch (and a tag 'two'). By fetching, but not pulling, our\n'master' would not advance, and that is where check on FETCH_HEAD\ncomes from. I suspect that the tag 'two' is also auto-followed with\nthis operation and would be in FETCH_HEAD; is that something we want\nto check? Alternatively, the \"actual\" log may want to see what the\nremote tracking branch for their 'master' has---then we do not have\nto worry about \"FETCH_HEAD has two refs---which one are we checking?\"\n\n> +\n> +\t# Client requested to use protocol v1\n> +\tgrep \"version=1\" log &&\n> +\t# Server responded using protocol v1\n> +\tgrep \"fetch< version 1\" log\n> +'\n\nSame \"version=1\" vs \"fetch< version 1\" strangeness appears here.\n\n> +test_expect_success 'pull with git:// using protocol v1' '\n> +\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n> +\t\tpull 2>log &&\n> +\n> +\tgit -C daemon_child log -1 --format=%s >actual &&\n> +\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n> +\ttest_cmp expect actual &&\n\nHere we can check our 'master', as we pulled their 'master' into it.\nWhat is this testing, though? The fact that protocol.version=1\ngiven via \"git -c var=val\" mechanism is propagated to the underlying\nfetch?\n\n> +\t# Client requested to use protocol v1\n> +\tgrep \"version=1\" log &&\n> +\t# Server responded using protocol v1\n> +\tgrep \"fetch< version 1\" log\n> +'\n> +\n> +test_expect_success 'push with git:// using protocol v1' '\n> +\ttest_commit -C daemon_child three &&\n> +\n> +\t# Since the repository being served isnt bare we need to push to\n> +\t# another branch explicitly to avoid mangling the master branch\n\nThe other end avoids mangling the master just fine without us doing\nanything special ;-). You are pushing to another branch because you\ncannot push into a branch that is currently checked out.\n\n\t# Push to another branch, as the target repository has the\n\t# master branch checked out and we cannot push into it.\n\nperhaps?\n\nThe tests for file:// looked identical, so the same set of comments\napply.\n\n> +# Test protocol v1 with 'ssh://' transport\n> +#\n> +test_expect_success 'setup ssh wrapper' '\n> +\tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh$X\" &&\n> +\texport GIT_SSH &&\n> +\texport TRASH_DIRECTORY &&\n> +\t>\"$TRASH_DIRECTORY\"/ssh-output\n> +'\n> +\n> +expect_ssh () {\n> +\ttest_when_finished '(cd \"$TRASH_DIRECTORY\" && rm -f ssh-expect && >ssh-output)' &&\n> +\techo \"ssh: -o SendEnv=GIT_PROTOCOL myhost $1 '$PWD/ssh_parent'\" >\"$TRASH_DIRECTORY/ssh-expect\" &&\n> +\t(cd \"$TRASH_DIRECTORY\" && test_cmp ssh-expect ssh-output)\n> +}\n> +\n> +test_expect_success 'create repo to be served by ssh:// transport' '\n> +\tgit init ssh_parent &&\n> +\ttest_commit -C ssh_parent one\n> +'\n> +\n> +test_expect_success 'clone with ssh:// using protocol v1' '\n> +\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n> +\t\tclone \"ssh://myhost:$(pwd)/ssh_parent\" ssh_child 2>log &&\n\nHmm, this is a fun one, as we deliberately make $(pwd) to have\nwhitespace in the test setup. I am impressed/kinda surprised that\nthis works ;-)\n\nOther than that, these also look more or less identical to file://\nand git:// tests, so the same set of comments apply.\n\nOverall very nicely done.\n\nThanks.\n"},{"id":"329017","messageId":"xmqqfub864m8.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20170926235627.79606-9-bmwill@google.com","subject":"Re: [PATCH v2 8/9] http: tell server that the client understands v1","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-27T06:24:31Z","receivedAt":"2017-09-27T06:24:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> @@ -897,6 +898,21 @@ static void set_from_env(const char **var, const char *envname)\n> \t\t*var = val;\n> }\n> \n> +static void protocol_http_header(void)\n> +{\n> +\tif (get_protocol_version_config() > 0) {\n> +\t\tstruct strbuf protocol_header = STRBUF_INIT;\n> +\n> +\t\tstrbuf_addf(&protocol_header, GIT_PROTOCOL_HEADER \": version=%d\",\n> +\t\t\t get_protocol_version_config());\n> +\n> +\n> +\t\textra_http_headers = curl_slist_append(extra_http_headers,\n> +\t\t\t\t\t\t protocol_header.buf);\n> +\t\tstrbuf_release(&protocol_header);\n> +\t}\n> +}\n> +\n> void http_init(struct remote *remote, const char *url, int proactive_auth)\n> {\n> \tchar *low_speed_limit;\n> @@ -927,6 +943,8 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n> \tif (remote)\n> \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n> \n> +\tprotocol_http_header();\n> +\n> \tpragma_header = curl_slist_append(http_copy_default_headers(),\n> \t\t\"Pragma: no-cache\");\n> \tno_pragma_header = curl_slist_append(http_copy_default_headers(),\n> diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n> index 0642ae7e6..df1943631 100644\n> --- a/t/lib-httpd/apache.conf\n> +++ b/t/lib-httpd/apache.conf\n> @@ -67,6 +67,9 @@ LockFile accept.lock\n> \n> \tLoadModule unixd_module modules/mod_unixd.so\n> \n> +\n> +\tLoadModule setenvif_module modules/mod_setenvif.so\n> +\n> \n> \n> PassEnv GIT_VALGRIND\n> @@ -76,6 +79,10 @@ PassEnv ASAN_OPTIONS\n> PassEnv GIT_TRACE\n> PassEnv GIT_CONFIG_NOSYSTEM\n> \n> += 2.4>\n> +\tSetEnvIf Git-Protocol \".*\" GIT_PROTOCOL=$0\n> +\n> +\n\nIt is very nice to see that only with a single extra HTTP header and\nthe server configuration, everybody else does not have to care how\nthe version information is plumbed through ;-)\n"},{"id":"329018","messageId":"xmqqbmlw64dr.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"xmqqk20k64r4.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 7/9] connect: tell server that the client understands v1","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-27T06:29:36Z","receivedAt":"2017-09-27T06:29:43Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> +\t# Client requested to use protocol v1\n>> +\tgrep \"version=1\" log &&\n>> +\t# Server responded using protocol v1\n>> +\tgrep \"clone< version 1\" log\n>\n> This looked a bit strange to check \"clone< version 1\" for one\n> direction, but did not check \"$something> version 1\" for the other\n> direction. Doesn't \"version=1\" end up producing 2 hits?\n>\n> Not a complaint, but wondering if we can write it in such a way that\n> does not have to make readers wonder.\n\nAh, the check for \"version=1\" is a short-hand for\n\n\tgrep \"clone> git-upload-pack ...\\\\0\\\\0version=1\\\\0$\" log\n\nand the symmetry I sought is already there. So ignore the above; if\nwe wanted to make the symmetry more explicit, it would not hurt to\nspell the first one as\n\n\tgrep \"clone> .*\\\\0\\\\0version=1\\\\0$\" log\n\nthough.\n\n"},{"id":"329019","messageId":"CAGZ79kZxOr3ug7V40kv77e2_kF9MBviCu99-6rcXy76tg2YVGg@mail.gmail.com","threadId":"46746","inReplyTo":"20170926235627.79606-4-bmwill@google.com","subject":"Re: [PATCH v2 3/9] protocol: introduce protocol extention mechanisms","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-09-27T06:30:22Z","receivedAt":"2017-09-27T06:30:32Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> +extern enum protocol_version get_protocol_version_config(void);\n> +extern enum protocol_version determine_protocol_version_server(void);\n> +extern enum protocol_version determine_protocol_version_client(const char *server_response);\n\nIt would be cool to have some documentation here.\n"},{"id":"329033","messageId":"xmqqvak42xn3.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"xmqq4lro7mab.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 3/9] protocol: introduce protocol extention mechanisms","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-09-27T11:23:28Z","receivedAt":"2017-09-27T11:23:44Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n>> +enum protocol_version determine_protocol_version_server(void)\n>> +{\n>> +\tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n>> +\tenum protocol_version version = protocol_v0;\n>> +\n>> +\tif (git_protocol) {\n>> +\t\tstruct string_list list = STRING_LIST_INIT_DUP;\n>> +\t\tconst struct string_list_item *item;\n>> +\t\tstring_list_split(&list, git_protocol, ':', -1);\n>> +\n>> +\t\tfor_each_string_list_item(item, &list) {\n>> +\t\t\tconst char *value;\n>> +\t\t\tenum protocol_version v;\n>> +\n>> +\t\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n>> +\t\t\t\tv = parse_protocol_version(value);\n>> +\t\t\t\tif (v > version)\n>> +\t\t\t\t\tversion = v;\n>> +\t\t\t}\n>> +\t\t}\n>> +\n>> +\t\tstring_list_clear(&list, 0);\n>> +\t}\n>> +\n>> +\treturn version;\n>> +}\n>\n> This implements \"the largest one wins\", not \"the last one wins\". Is\n> there a particular reason why the former is chosen?\n\nLet me give my version of why the usual \"the last one wins\" would\nnot necessarily a good idea. I would imagine that a client\ncontacting the server may want to say \"I understand v3, v2 (but not\nv1 nor v0)\" and in order to influence the server's choice between\nthe available two, it may want to somehow say it prefers v3 over v2\n(or v2 over v3). \n\nOne way to implement such a behaviour would be \"the first one that\nis understood is used\", i.e. something along this line:\n\n enum protocol_version version = protocol_unknown;\n\n\tfor_each_string_list_item(item, &list) {\n\t\tconst char *value;\n\t\tenum protocol_version v;\n\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n \tif (version == protocol_unknown) {\n \tv = parse_protocol_version(value);\n\t\t\t if (v != protocol_unknown)\n\t\t\t\t\tversion = v;\n\t\t\t}\n\t\t}\n\t}\n\n\tif (version == protocol_unknown)\n\t\tversion = protocol_v0;\n\nand not \"the largest one wins\" nor \"the last one wins\".\n\nI am not saying your code or the choice of \"the largest one wins\" is\nnecessarily wrong. I am just illlustrating the way to explain\n\"because I want to support a usecase like _this_, I define the way\nin which multiple values to the version variable is parsed like so,\nhence this code\". IOW, I think this commit should mention how the\n\"largest one wins\" rule would be useful to the clients and the\nservers when they want to achieve X---and that X is left unexplained.\n\n\n\n"},{"id":"329047","messageId":"20170927173409.GA161070@google.com","threadId":"46746","inReplyTo":"xmqqh8vp7xv4.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 6/9] connect: teach client to recognize v1 server response","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-27T17:34:09Z","receivedAt":"2017-09-27T17:34:17Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > +/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n> > +static int process_protocol_version(void)\n> > +{\n> > +\tswitch (determine_protocol_version_client(packet_buffer)) {\n> > +\t\tcase protocol_v1:\n> > +\t\t\treturn 1;\n> > +\t\tcase protocol_v0:\n> > +\t\t\treturn 0;\n> > +\t\tdefault:\n> > +\t\t\tdie(\"server is speaking an unknown protocol\");\n> > +\t}\n> > +}\n> \n> checkpatch.pl yells at me:\n> \n> ERROR: switch and case should be at the same indent\n> \n> and we would probably want to teach \"make style\" the same, if we\n> already don't.\n\n'make style' actually already understands this, I just forgot it run it\non this change :)\n\n-- \nBrandon Williams\n"},{"id":"329063","messageId":"20170927212258.GA68699@google.com","threadId":"46746","inReplyTo":"xmqqzi9g67kl.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 4/9] daemon: recognize hidden request arguments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-27T21:22:58Z","receivedAt":"2017-09-27T21:23:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > A normal request to git-daemon is structured as\n> > \"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\n> > git-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\n> > command, 2009-06-04) we aren't able to place any extra args (separated\n> > by NULs) besides the host.\n> \n> It's a bit unclear if that commit _introduced_ a bug, or just\n> noticed an old bug and documented it in its log message. How does\n> that commit impact the versons of Git that the updated code is\n> capable of interracting with?\n\nYou're right, after reading it again it isn't clear. I'll change this\nto indicate that the commit is a fix to a bug and that the fix doesn't\nallow us to place any additional args.\n\n> \n> > +static void parse_extra_args(struct hostinfo *hi, struct argv_array *env,\n> > +\t\t\t char *extra_args, int buflen)\n> > +{\n> > +\tconst char *end = extra_args + buflen;\n> > +\tstruct strbuf git_protocol = STRBUF_INIT;\n> > +\n> > +\t/* First look for the host argument */\n> > +\textra_args = parse_host_arg(hi, extra_args, buflen);\n> > +\n> > +\t/* Look for additional arguments places after a second NUL byte */\n> > +\tfor (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n> > +\t\tconst char *arg = extra_args;\n> > +\n> > +\t\t/*\n> > +\t\t * Parse the extra arguments, adding most to 'git_protocol'\n> > +\t\t * which will be used to set the 'GIT_PROTOCOL' envvar in the\n> > +\t\t * service that will be run.\n> > +\t\t *\n> > +\t\t * If there ends up being a particular arg in the future that\n> > +\t\t * git-daemon needs to parse specificly (like the 'host' arg)\n> > +\t\t * then it can be parsed here and not added to 'git_protocol'.\n> > +\t\t */\n> > +\t\tif (*arg) {\n> > +\t\t\tif (git_protocol.len > 0)\n> > +\t\t\t\tstrbuf_addch(&git_protocol, ':');\n> > +\t\t\tstrbuf_addstr(&git_protocol, arg);\n> > +\t\t}\n> > +\t}\n> > +\n> > +\tif (git_protocol.len > 0)\n> > +\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n> > +\t\t\t\t git_protocol.buf);\n> > +\tstrbuf_release(&git_protocol);\n> > }\n\n-- \nBrandon Williams\n"},{"id":"329064","messageId":"20170927212948.GB68699@google.com","threadId":"46746","inReplyTo":"xmqqvak467fy.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 5/9] upload-pack, receive-pack: introduce protocol version 1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-27T21:29:48Z","receivedAt":"2017-09-27T21:30:00Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > @@ -1963,6 +1964,19 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n> > \telse if (0 <= receive_unpack_limit)\n> > \t\tunpack_limit = receive_unpack_limit;\n> > \n> > +\tswitch (determine_protocol_version_server()) {\n> > +\tcase protocol_v1:\n> > +\t\tif (advertise_refs || !stateless_rpc)\n> > +\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n> > +\t\t/*\n> > +\t\t * v1 is just the original protocol with a version string,\n> > +\t\t * so just fall through after writing the version string.\n> > +\t\t */\n> > +\tcase protocol_v0:\n> > +\tdefault:\n> > +\t\tbreak;\n> \n> When protocol_v2 is introduced in the other part of the codebase\n> (i.e. in protocol.[ch]), until these lines are updated accordingly\n> to take care of the new protocol, we'd pretend that client asked\n> (and the server accepted) v0, even though the client and the daemon\n> agreed to talk v2.\n> \n> Shouldn't the \"default:\" die instead? The same for upload-pack.c\n\nGood catch. Yeah you're right, the default should probably die saying\nthat receive pack or upload pack doesn't support the protocol version.\n\n-- \nBrandon Williams\n"},{"id":"329065","messageId":"20170927213616.GC68699@google.com","threadId":"46746","inReplyTo":"xmqqfub864m8.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 8/9] http: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-27T21:36:16Z","receivedAt":"2017-09-27T21:36:24Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > @@ -897,6 +898,21 @@ static void set_from_env(const char **var, const char *envname)\n> > \t\t*var = val;\n> > }\n> > \n> > +static void protocol_http_header(void)\n> > +{\n> > +\tif (get_protocol_version_config() > 0) {\n> > +\t\tstruct strbuf protocol_header = STRBUF_INIT;\n> > +\n> > +\t\tstrbuf_addf(&protocol_header, GIT_PROTOCOL_HEADER \": version=%d\",\n> > +\t\t\t get_protocol_version_config());\n> > +\n> > +\n> > +\t\textra_http_headers = curl_slist_append(extra_http_headers,\n> > +\t\t\t\t\t\t protocol_header.buf);\n> > +\t\tstrbuf_release(&protocol_header);\n> > +\t}\n> > +}\n> > +\n> > void http_init(struct remote *remote, const char *url, int proactive_auth)\n> > {\n> > \tchar *low_speed_limit;\n> > @@ -927,6 +943,8 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n> > \tif (remote)\n> > \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n> > \n> > +\tprotocol_http_header();\n> > +\n> > \tpragma_header = curl_slist_append(http_copy_default_headers(),\n> > \t\t\"Pragma: no-cache\");\n> > \tno_pragma_header = curl_slist_append(http_copy_default_headers(),\n> > diff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\n> > index 0642ae7e6..df1943631 100644\n> > --- a/t/lib-httpd/apache.conf\n> > +++ b/t/lib-httpd/apache.conf\n> > @@ -67,6 +67,9 @@ LockFile accept.lock\n> > \n> > \tLoadModule unixd_module modules/mod_unixd.so\n> > \n> > +\n> > +\tLoadModule setenvif_module modules/mod_setenvif.so\n> > +\n> > \n> > \n> > PassEnv GIT_VALGRIND\n> > @@ -76,6 +79,10 @@ PassEnv ASAN_OPTIONS\n> > PassEnv GIT_TRACE\n> > PassEnv GIT_CONFIG_NOSYSTEM\n> > \n> > += 2.4>\n> > +\tSetEnvIf Git-Protocol \".*\" GIT_PROTOCOL=$0\n> > +\n> > +\n> \n> It is very nice to see that only with a single extra HTTP header and\n> the server configuration, everybody else does not have to care how\n> the version information is plumbed through ;-)\n\nHaving limited experience working with HTTP, it took me a bit to\nfigure out how to get the server configuration right, but once I got it\nworking it seemed to work pretty seamlessly :)\n\n-- \nBrandon Williams\n"},{"id":"329110","messageId":"20170928165751.GA153914@google.com","threadId":"46746","inReplyTo":"20170927212258.GA68699@google.com","subject":"Re: [PATCH v2 4/9] daemon: recognize hidden request arguments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-28T16:57:51Z","receivedAt":"2017-09-28T16:57:59Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Brandon Williams wrote:\n> On 09/27, Junio C Hamano wrote:\n> > Brandon Williams writes:\n> >\n> > > A normal request to git-daemon is structured as\n> > > \"command path/to/repo\\0host=..\\0\" and due to a bug in an old version of\n> > > git-daemon 73bb33a94 (daemon: Strictly parse the \"extra arg\" part of the\n> > > command, 2009-06-04) we aren't able to place any extra args (separated\n> > > by NULs) besides the host.\n> >\n> > It's a bit unclear if that commit _introduced_ a bug, or just\n> > noticed an old bug and documented it in its log message. How does\n> > that commit impact the versons of Git that the updated code is\n> > capable of interracting with?\n>\n> You're right, after reading it again it isn't clear. I'll change this\n> to indicate that the commit is a fix to a bug and that the fix doesn't\n> allow us to place any additional args.\n\nOk how about this wording for the commit msg:\n\n daemon: recognize hidden request arguments\n\n A normal request to git-daemon is structured as \"command\n path/to/repo\\0host=..\\0\" and due to a bug introduced in\n 49ba83fb6 (Add virtualization support to git-daemon, 2006-09-19)\n we aren't able to place any extra arguments (separated by NULs)\n besides the host otherwise the parsing of those arguments would\n enter an infinite loop. This bug was fixed in 73bb33a94\n (daemon: Strictly parse the \"extra arg\" part of the command,\n 2009-06-04) but a check was put in place to disallow extra\n arguments so that new clients wouldn't trigger this bug in older\n servers.\n\n In order to get around this limitation teach git-daemon to\n recognize additional request arguments hidden behind a second\n NUL byte. Requests can then be structured like: \"command\n path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\n can then parse out the extra arguments and set 'GIT_PROTOCOL'\n accordingly.\n\n By placing these extra arguments behind a second NUL byte we can\n skirt around both the infinite loop bug in 49ba83fb6 (Add\n virtualization support to git-daemon, 2006-09-19) as well as the\n explicit disallowing of extra arguments introduced in 73bb33a94\n (daemon: Strictly parse the \"extra arg\" part of the command,\n 2009-06-04) because both of these versions of git-daemon check\n for a single NUL byte after the host argument before terminating\n the argument parsing.\n\nThis way I'm citing when the bug was actually introduced as well as\ndescribing why the 'fix' didn't completely resolve the issue. I also\nexplain a little bit about how this change should work even with very\nold servers which still have the bug. (I tried to get the introp test\nto work on a version of git that old but am having some difficulty even\ngetting the old version to launch git-daemon without hanging for some\nunknown reason)\n\n--\nBrandon Williams\n"},{"id":"329122","messageId":"20170928210451.GA177031@google.com","threadId":"46746","inReplyTo":"CAGZ79kZxOr3ug7V40kv77e2_kF9MBviCu99-6rcXy76tg2YVGg@mail.gmail.com","subject":"Re: [PATCH v2 3/9] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-28T21:04:51Z","receivedAt":"2017-09-28T21:04:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/26, Stefan Beller wrote:\n> > +extern enum protocol_version get_protocol_version_config(void);\n> > +extern enum protocol_version determine_protocol_version_server(void);\n> > +extern enum protocol_version determine_protocol_version_client(const char *server_response);\n> \n> It would be cool to have some documentation here.\n\nThanks for reminding me, I'll get to writing some more documentation :)\n\n-- \nBrandon Williams\n"},{"id":"329126","messageId":"20170928215858.GB177031@google.com","threadId":"46746","inReplyTo":"xmqq4lro7mab.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 3/9] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-28T21:58:58Z","receivedAt":"2017-09-28T21:59:05Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > +`GIT_PROTOCOL`::\n> > +\tFor internal use only. Used in handshaking the wire protocol.\n> > +\tContains a colon ':' separated list of keys with optional values\n> > +\t'key[=value]'. Presence of unknown keys must be tolerated.\n> \n> Is this meant to be used only on the \"server\" end? Am I correct to\n> interpret \"handshaking\" to mean the initial connection acceptor\n> (e.g. \"git daemon\") uses it to pass what it decided to the programs\n> that implement the service (e.g. \"git receive-pack\")?\n\nYes, the idea is that the client will request a protocol version by\nsetting GIT_PROTOCOL (or indirectly set when using git:// or http://).\nupload-pack and receive-pack will use the keys and values set in\nGIT_PROTOCOL to determine which version of the protocol to use. At some\npoint in the future they may even use other keys and values as a means\nof sending more information in an initial request from the client.\n\n> \n> > +/*\n> > + * Environment variable used in handshaking the wire protocol.\n> > + * Contains a colon ':' separated list of keys with optional values\n> > + * 'key[=value]'. Presence of unknown keys must be tolerated.\n> > + */\n> > +#define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n> \n> \"Must be tolerated\" feels a bit strange. When somebody asks you to\n> use \"version 3\" or \"version 1 variant 2\", when you only know\n> \"version 0\" or \"version 1\" and you are not yet even aware of the\n> concept of \"variant\", we simply ignore \"variant=2\" as if it wasn't\n> there, even though \"version=3\" will be rejected (because we know of\n> \"version\"; it's just that we don't know \"version=3\").\n\nBy \"Must be tolerated\" I was trying to get across that if the server\nseeing something it doesn't understand, it shouldn't choke. Maybe a\nbetter wording would be to use the word \"ignored\"?\n\n> \n> > +enum protocol_version determine_protocol_version_server(void)\n> > +{\n> > +\tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n> > +\tenum protocol_version version = protocol_v0;\n> > +\n> > +\tif (git_protocol) {\n> > +\t\tstruct string_list list = STRING_LIST_INIT_DUP;\n> > +\t\tconst struct string_list_item *item;\n> > +\t\tstring_list_split(&list, git_protocol, ':', -1);\n> > +\n> > +\t\tfor_each_string_list_item(item, &list) {\n> > +\t\t\tconst char *value;\n> > +\t\t\tenum protocol_version v;\n> > +\n> > +\t\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n> > +\t\t\t\tv = parse_protocol_version(value);\n> > +\t\t\t\tif (v > version)\n> > +\t\t\t\t\tversion = v;\n> > +\t\t\t}\n> > +\t\t}\n> > +\n> > +\t\tstring_list_clear(&list, 0);\n> > +\t}\n> > +\n> > +\treturn version;\n> > +}\n> \n> This implements \"the largest one wins\", not \"the last one wins\". Is\n> there a particular reason why the former is chosen?\n> \n\nI envision this logic changing for newer servers once more protocol\nversions are added because at some point a server may want to disallow a\nparticular version (because of a security issue or what have you). So I\nfigured the easiest thing to do for now was to implement \"Newest version\nwins\".\n\n-- \nBrandon Williams\n"},{"id":"329128","messageId":"20170928220814.GC177031@google.com","threadId":"46746","inReplyTo":"xmqqpoac676l.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 6/9] connect: teach client to recognize v1 server response","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-28T22:08:14Z","receivedAt":"2017-09-28T22:08:22Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > +/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n> > +static int process_protocol_version(void)\n> > +{\n> > +\tswitch (determine_protocol_version_client(packet_buffer)) {\n> > +\t\tcase protocol_v1:\n> > +\t\t\treturn 1;\n> > +\t\tcase protocol_v0:\n> > +\t\t\treturn 0;\n> > +\t\tdefault:\n> > +\t\t\tdie(\"server is speaking an unknown protocol\");\n> > +\t}\n> > +}\n> \n> For the purpose of \"technology demonstration\" v1 protocol, it is OK\n> to discard the result of \"determine_pvc()\" like the above code, but\n> in a real application, we would do a bit more than just ignoring an\n> extra \"version #\" packet that appears at the beginning, no?\n> \n> It would be sensible to design how the result of determien_pvc()\n> call is propagated to the remainder of the program in this patch and\n> implement it. Perhaps add a new global (like server_capabilities\n> already is) and store the value there, or something? Or pass a\n> pointer to enum protocol_version as a return-location parameter to\n> this helper function so that the process_capabilities() can pass a\n> pointer to its local variable?\n\nYes, once we actually implement a v2 we would need to not throw away the\nresult of 'determine_pvc()' and instead do control flow based on the\nresultant version. I was trying to implement 'v1' as simply as possible\nso that I wouldn't have to do a large amount of refactoring when\nproposing this transition, though it seems Jonathan ended up doing more\nthan I planned, as I figured we didn't really know what the code will\nneed to be refactored to, in order to handle another protocol version.\nI would suspect that we maybe wouldn't want to determine which version a\nserver is speaking in 'get_remote_heads()' but rather at some point\nbefore that so we can branch off to do v2 like things, for example,\ncapability discovery and not ref discovery.\n\nIf you do think we need to do more of that refactoring now, before a v2,\nI can most certainly work on that.\n\n\n> \n> > static void process_capabilities(int *len)\n> > {\n> > @@ -224,12 +239,19 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n> > \t */\n> > \tint responded = 0;\n> > \tint len;\n> > -\tint state = EXPECTING_FIRST_REF;\n> > +\tint state = EXPECTING_PROTOCOL_VERSION;\n> > \n> > \t*list = NULL;\n> > \n> > \twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n> > \t\tswitch (state) {\n> > +\t\tcase EXPECTING_PROTOCOL_VERSION:\n> > +\t\t\tif (process_protocol_version()) {\n> > +\t\t\t\tstate = EXPECTING_FIRST_REF;\n> > +\t\t\t\tbreak;\n> > +\t\t\t}\n> > +\t\t\tstate = EXPECTING_FIRST_REF;\n> > +\t\t\t/* fallthrough */\n> > \t\tcase EXPECTING_FIRST_REF:\n> > \t\t\tprocess_capabilities(&len);\n> > \t\t\tif (process_dummy_ref()) {\n\n-- \nBrandon Williams\n"},{"id":"329130","messageId":"20170928222009.GD177031@google.com","threadId":"46746","inReplyTo":"xmqqk20k64r4.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 7/9] connect: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-28T22:20:09Z","receivedAt":"2017-09-28T22:20:18Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Junio C Hamano wrote:\n> Brandon Williams writes:\n> \n> > Teach the connection logic to tell a serve that it understands protocol\n> > v1. This is done in 2 different ways for the built in protocols.\n> >\n> > 1. git://\n> > A normal request is structured as \"command path/to/repo\\0host=..\\0\"\n> > and due to a bug in an old version of git-daemon 73bb33a94 (daemon:\n> > Strictly parse the \"extra arg\" part of the command, 2009-06-04) we\n> > aren't able to place any extra args (separated by NULs) besides the\n> > host. In order to get around this limitation put protocol version\n> > information after a second NUL byte so the request is structured\n> > like: \"command path/to/repo\\0host=..\\0\\0version=1\\0\". git-daemon can\n> > then parse out the version number and set GIT_PROTOCOL.\n> \n> Same question as a previous step, wrt the cited commit. It reads as\n> if we are saying that the commit introduced a bug and left it there,\n> that we cannot use \\0host=..\\0version=..\\0other=..\\0 until that bug\n> is fixed, and that in the meantime we use \\0host=..\\0\\0version=.. as\n> a workaround, but that reading leaves readers wondering if we want\n> to eventually drop this double-NUL workaround. I am guessing that\n> we want to declare that the current protocol has a glitch that\n> prevents us to use \\0host=..\\0version=..\\0 but we accept that and\n> plan to keep it that way, and we'll use the double-NUL for anything\n> other than host from now on, as it is compatible with the current\n> version of Git before this patch (the extras are safely ignored),\n> but then it still leaves readers wonder if the mention of the\n> old commit from 2009 means that this double-NUL would not even work\n> if the other end is running a version of Git before that commit, or\n> we are safe to talk with versions of Git even older than that.\n> \n> I do not think it is a showstopper if we did not work with v1.6.4,\n> but it still needs to be clarified.\n\nI wrote an updated commit msg for the daemon change, I can make a\nsimilar change here. And this mechanism shouldn't cause any issues with\nboth the pre and post 73bb33a94 git-daemon servers.\n\n> \n> > 2. ssh://, file://\n> > Set GIT_PROTOCOL envvar with the desired protocol version. The\n> > envvar can be sent across ssh by using '-o SendEnv=GIT_PROTOCOL' and\n> > having the server whitelist this envvar.\n> \n> OpenSSH lets us do this, but I do not know how well this works with\n> other implementations of SSH clients. The log message perhaps needs\n> to ask for volunteers to check if it is OK with the implementations\n> they use, and offer conditional code (just like we have for putty\n> and plink customizations) otherwise.\n\nI'll make a comment indicating that\n\n> \n> Other than that, the code changes looked good.\n> \n> > diff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\n> > new file mode 100755\n> > index 000000000..1988bbce6\n> > --- /dev/null\n> > +++ b/t/t5700-protocol-v1.sh\n> > @@ -0,0 +1,223 @@\n> > +#!/bin/sh\n> > +\n> > +test_description='test git wire-protocol transition'\n> > +\n> > +TEST_NO_CREATE_REPO=1\n> > +\n> > +. ./test-lib.sh\n> > +\n> > +# Test protocol v1 with 'git://' transport\n> > +#\n> > +. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n> > +start_git_daemon --export-all --enable=receive-pack\n> > +daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n> > +\n> > +test_expect_success 'create repo to be served by git-daemon' '\n> > +\tgit init \"$daemon_parent\" &&\n> > +\ttest_commit -C \"$daemon_parent\" one\n> > +'\n> > +\n> > +test_expect_success 'clone with git:// using protocol v1' '\n> > +\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n> > +\t\tclone \"$GIT_DAEMON_URL/parent\" daemon_child 2>log &&\n> > +\n> > +\tgit -C daemon_child log -1 --format=%s >actual &&\n> > +\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n> > +\ttest_cmp expect actual &&\n> > +\n> > +\t# Client requested to use protocol v1\n> > +\tgrep \"version=1\" log &&\n> > +\t# Server responded using protocol v1\n> > +\tgrep \"clone< version 1\" log\n> \n> This looked a bit strange to check \"clone< version 1\" for one\n> direction, but did not check \"$something> version 1\" for the other\n> direction. Doesn't \"version=1\" end up producing 2 hits?\n\nI think you discovered this in your next email but the \"version=1\" check\nis to check for the request sent to git-daemon, the \"command\npath/to/repo\\0host=blah\\0\\0version=1\\0\" one. While the \"clone< version\n1\" check is to make sure that the server responded with the correct\nversion.\n\n> \n> Not a complaint, but wondering if we can write it in such a way that\n> does not have to make readers wonder.\n> \n> > +'\n> > +\n> > +test_expect_success 'fetch with git:// using protocol v1' '\n> > +\ttest_commit -C \"$daemon_parent\" two &&\n> > +\n> > +\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n> > +\t\tfetch 2>log &&\n> > +\n> > +\tgit -C daemon_child log -1 --format=%s FETCH_HEAD >actual &&\n> > +\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n> > +\ttest_cmp expect actual &&\n> \n> OK. So the origin repository gained one commit on the 'master'\n> branch (and a tag 'two'). By fetching, but not pulling, our\n> 'master' would not advance, and that is where check on FETCH_HEAD\n> comes from. I suspect that the tag 'two' is also auto-followed with\n> this operation and would be in FETCH_HEAD; is that something we want\n> to check? Alternatively, the \"actual\" log may want to see what the\n> remote tracking branch for their 'master' has---then we do not have\n> to worry about \"FETCH_HEAD has two refs---which one are we checking?\"\n\nYeah I can do that instead if you would prefer.\n\n> \n> > +\n> > +\t# Client requested to use protocol v1\n> > +\tgrep \"version=1\" log &&\n> > +\t# Server responded using protocol v1\n> > +\tgrep \"fetch< version 1\" log\n> > +'\n> \n> Same \"version=1\" vs \"fetch< version 1\" strangeness appears here.\n> \n> > +test_expect_success 'pull with git:// using protocol v1' '\n> > +\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n> > +\t\tpull 2>log &&\n> > +\n> > +\tgit -C daemon_child log -1 --format=%s >actual &&\n> > +\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n> > +\ttest_cmp expect actual &&\n> \n> Here we can check our 'master', as we pulled their 'master' into it.\n> What is this testing, though? The fact that protocol.version=1\n> given via \"git -c var=val\" mechanism is propagated to the underlying\n> fetch?\n\nYeah, i guess we could realistically drop either the fetch or pull test\nas they essentially do the same thing. I was just being overly\ncautious.\n\n> \n> > +\t# Client requested to use protocol v1\n> > +\tgrep \"version=1\" log &&\n> > +\t# Server responded using protocol v1\n> > +\tgrep \"fetch< version 1\" log\n> > +'\n> > +\n> > +test_expect_success 'push with git:// using protocol v1' '\n> > +\ttest_commit -C daemon_child three &&\n> > +\n> > +\t# Since the repository being served isnt bare we need to push to\n> > +\t# another branch explicitly to avoid mangling the master branch\n> \n> The other end avoids mangling the master just fine without us doing\n> anything special ;-). You are pushing to another branch because you\n> cannot push into a branch that is currently checked out.\n> \n> \t# Push to another branch, as the target repository has the\n> \t# master branch checked out and we cannot push into it.\n\nSounds good I'll change that.\n\n> \n> perhaps?\n> \n> The tests for file:// looked identical, so the same set of comments\n> apply.\n> \n> > +# Test protocol v1 with 'ssh://' transport\n> > +#\n> > +test_expect_success 'setup ssh wrapper' '\n> > +\tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh$X\" &&\n> > +\texport GIT_SSH &&\n> > +\texport TRASH_DIRECTORY &&\n> > +\t>\"$TRASH_DIRECTORY\"/ssh-output\n> > +'\n> > +\n> > +expect_ssh () {\n> > +\ttest_when_finished '(cd \"$TRASH_DIRECTORY\" && rm -f ssh-expect && >ssh-output)' &&\n> > +\techo \"ssh: -o SendEnv=GIT_PROTOCOL myhost $1 '$PWD/ssh_parent'\" >\"$TRASH_DIRECTORY/ssh-expect\" &&\n> > +\t(cd \"$TRASH_DIRECTORY\" && test_cmp ssh-expect ssh-output)\n> > +}\n> > +\n> > +test_expect_success 'create repo to be served by ssh:// transport' '\n> > +\tgit init ssh_parent &&\n> > +\ttest_commit -C ssh_parent one\n> > +'\n> > +\n> > +test_expect_success 'clone with ssh:// using protocol v1' '\n> > +\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n> > +\t\tclone \"ssh://myhost:$(pwd)/ssh_parent\" ssh_child 2>log &&\n> \n> Hmm, this is a fun one, as we deliberately make $(pwd) to have\n> whitespace in the test setup. I am impressed/kinda surprised that\n> this works ;-)\n> \n> Other than that, these also look more or less identical to file://\n> and git:// tests, so the same set of comments apply.\n> \n> Overall very nicely done.\n\nThanks! :D\n\n> \n> Thanks.\n\n-- \nBrandon Williams\n"},{"id":"329217","messageId":"20170929212018.GA92120@google.com","threadId":"46746","inReplyTo":"xmqqvak42xn3.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 3/9] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-29T21:20:18Z","receivedAt":"2017-09-29T21:20:28Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> >> +enum protocol_version determine_protocol_version_server(void)\n> >> +{\n> >> +\tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n> >> +\tenum protocol_version version = protocol_v0;\n> >> +\n> >> +\tif (git_protocol) {\n> >> +\t\tstruct string_list list = STRING_LIST_INIT_DUP;\n> >> +\t\tconst struct string_list_item *item;\n> >> +\t\tstring_list_split(&list, git_protocol, ':', -1);\n> >> +\n> >> +\t\tfor_each_string_list_item(item, &list) {\n> >> +\t\t\tconst char *value;\n> >> +\t\t\tenum protocol_version v;\n> >> +\n> >> +\t\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n> >> +\t\t\t\tv = parse_protocol_version(value);\n> >> +\t\t\t\tif (v > version)\n> >> +\t\t\t\t\tversion = v;\n> >> +\t\t\t}\n> >> +\t\t}\n> >> +\n> >> +\t\tstring_list_clear(&list, 0);\n> >> +\t}\n> >> +\n> >> +\treturn version;\n> >> +}\n> >\n> > This implements \"the largest one wins\", not \"the last one wins\". Is\n> > there a particular reason why the former is chosen?\n> \n> Let me give my version of why the usual \"the last one wins\" would\n> not necessarily a good idea. I would imagine that a client\n> contacting the server may want to say \"I understand v3, v2 (but not\n> v1 nor v0)\" and in order to influence the server's choice between\n> the available two, it may want to somehow say it prefers v3 over v2\n> (or v2 over v3). \n> \n> One way to implement such a behaviour would be \"the first one that\n> is understood is used\", i.e. something along this line:\n> \n> enum protocol_version version = protocol_unknown;\n> \n> \tfor_each_string_list_item(item, &list) {\n> \t\tconst char *value;\n> \t\tenum protocol_version v;\n> \t\tif (skip_prefix(item->string, \"version=\", &value)) {\n> \tif (version == protocol_unknown) {\n> \tv = parse_protocol_version(value);\n> \t\t\t if (v != protocol_unknown)\n> \t\t\t\t\tversion = v;\n> \t\t\t}\n> \t\t}\n> \t}\n> \n> \tif (version == protocol_unknown)\n> \t\tversion = protocol_v0;\n> \n> and not \"the largest one wins\" nor \"the last one wins\".\n> \n> I am not saying your code or the choice of \"the largest one wins\" is\n> necessarily wrong. I am just illlustrating the way to explain\n> \"because I want to support a usecase like _this_, I define the way\n> in which multiple values to the version variable is parsed like so,\n> hence this code\". IOW, I think this commit should mention how the\n> \"largest one wins\" rule would be useful to the clients and the\n> servers when they want to achieve X---and that X is left unexplained.\n\nI believe I mentioned this elsewhere but I think that at some point this\nlogic will probably have to be tweaked again at some point so that a\nserver may be able to prefer one version to another.\n\nThat being said I can definitely add a comment indicating how this code\nselects the version and that it can be used to ensure that the latest\nand greatest protocol version is used.\n\n-- \nBrandon Williams\n"},{"id":"329218","messageId":"20170929213238.GB92120@google.com","threadId":"46746","inReplyTo":"xmqqbmlw64dr.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v2 7/9] connect: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-09-29T21:32:38Z","receivedAt":"2017-09-29T21:32:46Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 09/27, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> >> +\t# Client requested to use protocol v1\n> >> +\tgrep \"version=1\" log &&\n> >> +\t# Server responded using protocol v1\n> >> +\tgrep \"clone< version 1\" log\n> >\n> > This looked a bit strange to check \"clone< version 1\" for one\n> > direction, but did not check \"$something> version 1\" for the other\n> > direction. Doesn't \"version=1\" end up producing 2 hits?\n> >\n> > Not a complaint, but wondering if we can write it in such a way that\n> > does not have to make readers wonder.\n> \n> Ah, the check for \"version=1\" is a short-hand for\n> \n> \tgrep \"clone> git-upload-pack ...\\\\0\\\\0version=1\\\\0$\" log\n> \n> and the symmetry I sought is already there. So ignore the above; if\n> we wanted to make the symmetry more explicit, it would not hurt to\n> spell the first one as\n> \n> \tgrep \"clone> .*\\\\0\\\\0version=1\\\\0$\" log\n\nI think you need three '\\' to get an escaped backslash, but I agree,\nI'll spell this out more explicitly in the tests.\n\n> \n> though.\n> \n\n-- \nBrandon Williams\n"},{"id":"329592","messageId":"20171003201507.3589-1-bmwill@google.com","threadId":"46746","inReplyTo":"20170926235627.79606-1-bmwill@google.com","subject":"[PATCH v3 00/10] protocol transition","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:14:57Z","receivedAt":"2017-10-03T20:15:33Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v3:\n * added a new ssh variant 'simple' and update documentation to better reflect\n the command-line parameters passed to the ssh command.\n * updated various commit messages based on feedback.\n * tighten the wording for 'GIT_PROTOCOL' to indicate that both unknown keys\n and values must be ignored.\n * added API comments for functions in protocol.h\n * updated various tests in t5700 based on reviewer feedback\n\nBrandon Williams (9):\n pkt-line: add packet_write function\n protocol: introduce protocol extention mechanisms\n daemon: recognize hidden request arguments\n upload-pack, receive-pack: introduce protocol version 1\n connect: teach client to recognize v1 server response\n connect: tell server that the client understands v1\n http: tell server that the client understands v1\n i5700: add interop test for protocol transition\n ssh: introduce a 'simple' ssh variant\n\nJonathan Tan (1):\n connect: in ref advertisement, shallows are last\n\n Documentation/config.txt | 44 +++-\n Documentation/git.txt | 15 +-\n Makefile | 1 +\n builtin/receive-pack.c | 15 ++\n cache.h | 10 +\n connect.c | 353 ++++++++++++++++++++++-----------\n daemon.c | 68 ++++++-\n http.c | 18 ++\n pkt-line.c | 6 +\n pkt-line.h | 1 +\n protocol.c | 79 ++++++++\n protocol.h | 33 +++\n t/interop/i5700-protocol-transition.sh | 68 +++++++\n t/lib-httpd/apache.conf | 7 +\n t/t5601-clone.sh | 9 +-\n t/t5700-protocol-v1.sh | 294 +++++++++++++++++++++++++++\n upload-pack.c | 18 +-\n 17 files changed, 900 insertions(+), 139 deletions(-)\n create mode 100644 protocol.c\n create mode 100644 protocol.h\n create mode 100755 t/interop/i5700-protocol-transition.sh\n create mode 100755 t/t5700-protocol-v1.sh\n\n--- interdiff with 'origin/bw/protocol-v1'\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex b78747abc..0460af37e 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2084,12 +2084,31 @@ ssh.variant::\n \tDepending on the value of the environment variables `GIT_SSH` or\n \t`GIT_SSH_COMMAND`, or the config setting `core.sshCommand`, Git\n \tauto-detects whether to adjust its command-line parameters for use\n-\twith plink or tortoiseplink, as opposed to the default (OpenSSH).\n+\twith ssh (OpenSSH), plink or tortoiseplink, as opposed to the default\n+\t(simple).\n +\n The config variable `ssh.variant` can be set to override this auto-detection;\n-valid values are `ssh`, `plink`, `putty` or `tortoiseplink`. Any other value\n-will be treated as normal ssh. This setting can be overridden via the\n-environment variable `GIT_SSH_VARIANT`.\n+valid values are `ssh`, `simple`, `plink`, `putty` or `tortoiseplink`. Any\n+other value will be treated as normal ssh. This setting can be overridden via\n+the environment variable `GIT_SSH_VARIANT`.\n++\n+The current command-line parameters used for each variant are as\n+follows:\n++\n+--\n+\n+* `ssh` - [-p port] [-4] [-6] [-o option] [username@]host command\n+\n+* `simple` - [username@]host command\n+\n+* `plink` or `putty` - [-P port] [-4] [-6] [username@]host command\n+\n+* `tortoiseplink` - [-P port] [-4] [-6] -batch [username@]host command\n+\n+--\n++\n+Except for the `simple` variant, command-line parameters are likely to\n+change as git gains new features.\n \n i18n.commitEncoding::\n \tCharacter encoding the commit messages are stored in; Git itself\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 299f75c7b..8bc3f2147 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -518,11 +518,10 @@ other\n \tIf either of these environment variables is set then 'git fetch'\n \tand 'git push' will use the specified command instead of 'ssh'\n \twhen they need to connect to a remote system.\n-\tThe command will be given exactly two or four arguments: the\n-\t'username@host' (or just 'host') from the URL and the shell\n-\tcommand to execute on that remote system, optionally preceded by\n-\t`-p` (literally) and the 'port' from the URL when it specifies\n-\tsomething other than the default SSH port.\n+\tThe command-line parameters passed to the configured command are\n+\tdetermined by the ssh variant. See `ssh.variant` option in\n+\tlinkgit:git-config[1] for details.\n+\n +\n `$GIT_SSH_COMMAND` takes precedence over `$GIT_SSH`, and is interpreted\n by the shell, which allows additional arguments to be included.\n@@ -700,7 +699,8 @@ of clones and fetches.\n `GIT_PROTOCOL`::\n \tFor internal use only. Used in handshaking the wire protocol.\n \tContains a colon ':' separated list of keys with optional values\n-\t'key[=value]'. Presence of unknown keys must be tolerated.\n+\t'key[=value]'. Presence of unknown keys and values must be\n+\tignored.\n \n Discussion[[Discussion]]\n ------------------------\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex cb179367b..94b7d29ea 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1973,8 +1973,9 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \t\t * so just fall through after writing the version string.\n \t\t */\n \tcase protocol_v0:\n-\tdefault:\n \t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown protocol version\");\n \t}\n \n \tif (advertise_refs || !stateless_rpc) {\ndiff --git a/cache.h b/cache.h\nindex aaadac1f0..3a6b869c2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -448,7 +448,8 @@ static inline enum object_type object_type(unsigned int mode)\n /*\n * Environment variable used in handshaking the wire protocol.\n * Contains a colon ':' separated list of keys with optional values\n- * 'key[=value]'. Presence of unknown keys must be tolerated.\n+ * 'key[=value]'. Presence of unknown keys and values must be\n+ * ignored.\n */\n #define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n /* HTTP header used to handshake the wire protocol */\ndiff --git a/connect.c b/connect.c\nindex 12ebab724..65cee49b6 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -139,12 +139,12 @@ static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n static int process_protocol_version(void)\n {\n \tswitch (determine_protocol_version_client(packet_buffer)) {\n-\t\tcase protocol_v1:\n-\t\t\treturn 1;\n-\t\tcase protocol_v0:\n-\t\t\treturn 0;\n-\t\tdefault:\n-\t\t\tdie(\"server is speaking an unknown protocol\");\n+\tcase protocol_v1:\n+\t\treturn 1;\n+\tcase protocol_v0:\n+\t\treturn 0;\n+\tdefault:\n+\t\tdie(\"server is speaking an unknown protocol\");\n \t}\n }\n \n@@ -776,37 +776,44 @@ static const char *get_ssh_command(void)\n \treturn NULL;\n }\n \n-static int override_ssh_variant(int *port_option, int *needs_batch)\n+enum ssh_variant {\n+\tVARIANT_SIMPLE,\n+\tVARIANT_SSH,\n+\tVARIANT_PLINK,\n+\tVARIANT_PUTTY,\n+\tVARIANT_TORTOISEPLINK,\n+};\n+\n+static int override_ssh_variant(enum ssh_variant *ssh_variant)\n {\n-\tchar *variant;\n+\tconst char *variant = getenv(\"GIT_SSH_VARIANT\");\n \n-\tvariant = xstrdup_or_null(getenv(\"GIT_SSH_VARIANT\"));\n-\tif (!variant &&\n-\t git_config_get_string(\"ssh.variant\", &variant))\n+\tif (!variant && git_config_get_string_const(\"ssh.variant\", &variant))\n \t\treturn 0;\n \n-\tif (!strcmp(variant, \"plink\") || !strcmp(variant, \"putty\")) {\n-\t\t*port_option = 'P';\n-\t\t*needs_batch = 0;\n-\t} else if (!strcmp(variant, \"tortoiseplink\")) {\n-\t\t*port_option = 'P';\n-\t\t*needs_batch = 1;\n-\t} else {\n-\t\t*port_option = 'p';\n-\t\t*needs_batch = 0;\n-\t}\n-\tfree(variant);\n+\tif (!strcmp(variant, \"plink\"))\n+\t\t*ssh_variant = VARIANT_PLINK;\n+\telse if (!strcmp(variant, \"putty\"))\n+\t\t*ssh_variant = VARIANT_PUTTY;\n+\telse if (!strcmp(variant, \"tortoiseplink\"))\n+\t\t*ssh_variant = VARIANT_TORTOISEPLINK;\n+\telse if (!strcmp(variant, \"simple\"))\n+\t\t*ssh_variant = VARIANT_SIMPLE;\n+\telse\n+\t\t*ssh_variant = VARIANT_SSH;\n+\n \treturn 1;\n }\n \n-static void handle_ssh_variant(const char *ssh_command, int is_cmdline,\n-\t\t\t int *port_option, int *needs_batch)\n+static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n+\t\t\t\t\t int is_cmdline)\n {\n+\tenum ssh_variant ssh_variant = VARIANT_SIMPLE;\n \tconst char *variant;\n \tchar *p = NULL;\n \n-\tif (override_ssh_variant(port_option, needs_batch))\n-\t\treturn;\n+\tif (override_ssh_variant(&ssh_variant))\n+\t\treturn ssh_variant;\n \n \tif (!is_cmdline) {\n \t\tp = xstrdup(ssh_command);\n@@ -825,19 +832,21 @@ static void handle_ssh_variant(const char *ssh_command, int is_cmdline,\n \t\t\tfree(ssh_argv);\n \t\t} else {\n \t\t\tfree(p);\n-\t\t\treturn;\n+\t\t\treturn ssh_variant;\n \t\t}\n \t}\n \n-\tif (!strcasecmp(variant, \"plink\") ||\n-\t !strcasecmp(variant, \"plink.exe\"))\n-\t\t*port_option = 'P';\n+\tif (!strcasecmp(variant, \"ssh\"))\n+\t\tssh_variant = VARIANT_SSH;\n+\telse if (!strcasecmp(variant, \"plink\") ||\n+\t\t !strcasecmp(variant, \"plink.exe\"))\n+\t\tssh_variant = VARIANT_PLINK;\n \telse if (!strcasecmp(variant, \"tortoiseplink\") ||\n-\t\t !strcasecmp(variant, \"tortoiseplink.exe\")) {\n-\t\t*port_option = 'P';\n-\t\t*needs_batch = 1;\n-\t}\n+\t\t !strcasecmp(variant, \"tortoiseplink.exe\"))\n+\t\tssh_variant = VARIANT_TORTOISEPLINK;\n+\n \tfree(p);\n+\treturn ssh_variant;\n }\n \n /*\n@@ -937,8 +946,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tconn->in = conn->out = -1;\n \t\tif (protocol == PROTO_SSH) {\n \t\t\tconst char *ssh;\n-\t\t\tint needs_batch = 0;\n-\t\t\tint port_option = 'p';\n+\t\t\tenum ssh_variant variant;\n \t\t\tchar *ssh_host = hostandport;\n \t\t\tconst char *port = NULL;\n \t\t\ttransport_check_allowed(\"ssh\");\n@@ -965,10 +973,9 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tdie(\"strange hostname '%s' blocked\", ssh_host);\n \n \t\t\tssh = get_ssh_command();\n-\t\t\tif (ssh)\n-\t\t\t\thandle_ssh_variant(ssh, 1, &port_option,\n-\t\t\t\t\t\t &needs_batch);\n-\t\t\telse {\n+\t\t\tif (ssh) {\n+\t\t\t\tvariant = determine_ssh_variant(ssh, 1);\n+\t\t\t} else {\n \t\t\t\t/*\n \t\t\t\t * GIT_SSH is the no-shell version of\n \t\t\t\t * GIT_SSH_COMMAND (and must remain so for\n@@ -979,32 +986,38 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tssh = getenv(\"GIT_SSH\");\n \t\t\t\tif (!ssh)\n \t\t\t\t\tssh = \"ssh\";\n-\t\t\t\telse\n-\t\t\t\t\thandle_ssh_variant(ssh, 0,\n-\t\t\t\t\t\t\t &port_option,\n-\t\t\t\t\t\t\t &needs_batch);\n+\t\t\t\tvariant = determine_ssh_variant(ssh, 0);\n \t\t\t}\n \n \t\t\targv_array_push(&conn->args, ssh);\n \n-\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\tif (variant == VARIANT_SSH &&\n+\t\t\t get_protocol_version_config() > 0) {\n \t\t\t\targv_array_push(&conn->args, \"-o\");\n \t\t\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n \t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n \t\t\t\t\t\t get_protocol_version_config());\n \t\t\t}\n \n-\t\t\tif (flags & CONNECT_IPV4)\n-\t\t\t\targv_array_push(&conn->args, \"-4\");\n-\t\t\telse if (flags & CONNECT_IPV6)\n-\t\t\t\targv_array_push(&conn->args, \"-6\");\n-\t\t\tif (needs_batch)\n+\t\t\tif (variant != VARIANT_SIMPLE) {\n+\t\t\t\tif (flags & CONNECT_IPV4)\n+\t\t\t\t\targv_array_push(&conn->args, \"-4\");\n+\t\t\t\telse if (flags & CONNECT_IPV6)\n+\t\t\t\t\targv_array_push(&conn->args, \"-6\");\n+\t\t\t}\n+\n+\t\t\tif (variant == VARIANT_TORTOISEPLINK)\n \t\t\t\targv_array_push(&conn->args, \"-batch\");\n-\t\t\tif (port) {\n-\t\t\t\targv_array_pushf(&conn->args,\n-\t\t\t\t\t\t \"-%c\", port_option);\n+\n+\t\t\tif (port && variant != VARIANT_SIMPLE) {\n+\t\t\t\tif (variant == VARIANT_SSH)\n+\t\t\t\t\targv_array_push(&conn->args, \"-p\");\n+\t\t\t\telse\n+\t\t\t\t\targv_array_push(&conn->args, \"-P\");\n+\n \t\t\t\targv_array_push(&conn->args, port);\n \t\t\t}\n+\n \t\t\targv_array_push(&conn->args, ssh_host);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\ndiff --git a/protocol.c b/protocol.c\nindex 369503065..43012b7eb 100644\n--- a/protocol.c\n+++ b/protocol.c\n@@ -33,6 +33,13 @@ enum protocol_version determine_protocol_version_server(void)\n \tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n \tenum protocol_version version = protocol_v0;\n \n+\t/*\n+\t * Determine which protocol version the client has requested. Since\n+\t * multiple 'version' keys can be sent by the client, indicating that\n+\t * the client is okay to speak any of them, select the greatest version\n+\t * that the client has requested. This is due to the assumption that\n+\t * the most recent protocol version will be the most state-of-the-art.\n+\t */\n \tif (git_protocol) {\n \t\tstruct string_list list = STRING_LIST_INIT_DUP;\n \t\tconst struct string_list_item *item;\ndiff --git a/protocol.h b/protocol.h\nindex 18f9a5235..1b2bc94a8 100644\n--- a/protocol.h\n+++ b/protocol.h\n@@ -7,8 +7,27 @@ enum protocol_version {\n \tprotocol_v1 = 1,\n };\n \n+/*\n+ * Used by a client to determine which protocol version to request be used when\n+ * communicating with a server, reflecting the configured value of the\n+ * 'protocol.version' config. If unconfigured, a value of 'protocol_v0' is\n+ * returned.\n+ */\n extern enum protocol_version get_protocol_version_config(void);\n+\n+/*\n+ * Used by a server to determine which protocol version should be used based on\n+ * a client's request, communicated via the 'GIT_PROTOCOL' environment variable\n+ * by setting appropriate values for the key 'version'. If a client doesn't\n+ * request a particular protocol version, a default of 'protocol_v0' will be\n+ * used.\n+ */\n extern enum protocol_version determine_protocol_version_server(void);\n+\n+/*\n+ * Used by a client to determine which protocol version the server is speaking\n+ * based on the server's initial response.\n+ */\n extern enum protocol_version determine_protocol_version_client(const char *server_response);\n \n #endif /* PROTOCOL_H */\ndiff --git a/t/interop/i5700-protocol-transition.sh b/t/interop/i5700-protocol-transition.sh\nindex 9e83428a8..97e8e580e 100755\n--- a/t/interop/i5700-protocol-transition.sh\n+++ b/t/interop/i5700-protocol-transition.sh\n@@ -3,7 +3,7 @@\n VERSION_A=.\n VERSION_B=v2.0.0\n \n-: ${LIB_GIT_DAEMON_PORT:=5600}\n+: ${LIB_GIT_DAEMON_PORT:=5700}\n LIB_GIT_DAEMON_COMMAND='git.b daemon'\n \n test_description='clone and fetch by client who is trying to use a new protocol'\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 9c56f771b..ee1a24c5b 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -312,6 +312,8 @@ setup_ssh_wrapper () {\n \t\t\t\"$TRASH_DIRECTORY/ssh-wrapper$X\" &&\n \t\tGIT_SSH=\"$TRASH_DIRECTORY/ssh-wrapper$X\" &&\n \t\texport GIT_SSH &&\n+\t\tGIT_SSH_VARIANT=ssh &&\n+\t\texport GIT_SSH_VARIANT &&\n \t\texport TRASH_DIRECTORY &&\n \t\t>\"$TRASH_DIRECTORY\"/ssh-output\n \t'\n@@ -320,7 +322,8 @@ setup_ssh_wrapper () {\n copy_ssh_wrapper_as () {\n \tcp \"$TRASH_DIRECTORY/ssh-wrapper$X\" \"${1%$X}$X\" &&\n \tGIT_SSH=\"${1%$X}$X\" &&\n-\texport GIT_SSH\n+\texport GIT_SSH &&\n+\tunset GIT_SSH_VARIANT\n }\n \n expect_ssh () {\n@@ -362,10 +365,10 @@ test_expect_success 'bracketed hostnames are still ssh' '\n \texpect_ssh \"-p 123\" myhost src\n '\n \n-test_expect_success 'uplink is not treated as putty' '\n+test_expect_success 'uplink is treated as simple' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/uplink\" &&\n \tgit clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n-\texpect_ssh \"-p 123\" myhost src\n+\texpect_ssh myhost src\n '\n \n test_expect_success 'plink is treated specially (as putty)' '\ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nindex 222265127..ba86a44eb 100755\n--- a/t/t5700-protocol-v1.sh\n+++ b/t/t5700-protocol-v1.sh\n@@ -26,7 +26,7 @@ test_expect_success 'clone with git:// using protocol v1' '\n \ttest_cmp expect actual &&\n \n \t# Client requested to use protocol v1\n-\tgrep \"version=1\" log &&\n+\tgrep \"clone> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n \t# Server responded using protocol v1\n \tgrep \"clone< version 1\" log\n '\n@@ -37,12 +37,12 @@ test_expect_success 'fetch with git:// using protocol v1' '\n \tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n \t\tfetch 2>log &&\n \n-\tgit -C daemon_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C daemon_child log -1 --format=%s origin/master >actual &&\n \tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n \ttest_cmp expect actual &&\n \n \t# Client requested to use protocol v1\n-\tgrep \"version=1\" log &&\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n \t# Server responded using protocol v1\n \tgrep \"fetch< version 1\" log\n '\n@@ -56,7 +56,7 @@ test_expect_success 'pull with git:// using protocol v1' '\n \ttest_cmp expect actual &&\n \n \t# Client requested to use protocol v1\n-\tgrep \"version=1\" log &&\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n \t# Server responded using protocol v1\n \tgrep \"fetch< version 1\" log\n '\n@@ -64,8 +64,8 @@ test_expect_success 'pull with git:// using protocol v1' '\n test_expect_success 'push with git:// using protocol v1' '\n \ttest_commit -C daemon_child three &&\n \n-\t# Since the repository being served isnt bare we need to push to\n-\t# another branch explicitly to avoid mangling the master branch\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n \tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n \t\tpush origin HEAD:client_branch 2>log &&\n \n@@ -74,7 +74,7 @@ test_expect_success 'push with git:// using protocol v1' '\n \ttest_cmp expect actual &&\n \n \t# Client requested to use protocol v1\n-\tgrep \"version=1\" log &&\n+\tgrep \"push> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n \t# Server responded using protocol v1\n \tgrep \"push< version 1\" log\n '\n@@ -106,7 +106,7 @@ test_expect_success 'fetch with file:// using protocol v1' '\n \tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n \t\tfetch 2>log &&\n \n-\tgit -C file_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n \tgit -C file_parent log -1 --format=%s >expect &&\n \ttest_cmp expect actual &&\n \n@@ -129,8 +129,8 @@ test_expect_success 'pull with file:// using protocol v1' '\n test_expect_success 'push with file:// using protocol v1' '\n \ttest_commit -C file_child three &&\n \n-\t# Since the repository being served isnt bare we need to push to\n-\t# another branch explicitly to avoid mangling the master branch\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n \tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n \t\tpush origin HEAD:client_branch 2>log &&\n \n@@ -145,8 +145,10 @@ test_expect_success 'push with file:// using protocol v1' '\n # Test protocol v1 with 'ssh://' transport\n #\n test_expect_success 'setup ssh wrapper' '\n-\tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh$X\" &&\n+\tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh\" &&\n \texport GIT_SSH &&\n+\tGIT_SSH_VARIANT=ssh &&\n+\texport GIT_SSH_VARIANT &&\n \texport TRASH_DIRECTORY &&\n \t>\"$TRASH_DIRECTORY\"/ssh-output\n '\n@@ -182,7 +184,7 @@ test_expect_success 'fetch with ssh:// using protocol v1' '\n \t\tfetch 2>log &&\n \texpect_ssh git-upload-pack &&\n \n-\tgit -C ssh_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C ssh_child log -1 --format=%s origin/master >actual &&\n \tgit -C ssh_parent log -1 --format=%s >expect &&\n \ttest_cmp expect actual &&\n \n@@ -206,8 +208,8 @@ test_expect_success 'pull with ssh:// using protocol v1' '\n test_expect_success 'push with ssh:// using protocol v1' '\n \ttest_commit -C ssh_child three &&\n \n-\t# Since the repository being served isnt bare we need to push to\n-\t# another branch explicitly to avoid mangling the master branch\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n \tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n \t\tpush origin HEAD:client_branch 2>log &&\n \texpect_ssh git-receive-pack &&\n@@ -251,7 +253,7 @@ test_expect_success 'fetch with http:// using protocol v1' '\n \tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n \t\tfetch 2>log &&\n \n-\tgit -C http_child log -1 --format=%s FETCH_HEAD >actual &&\n+\tgit -C http_child log -1 --format=%s origin/master >actual &&\n \tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n \ttest_cmp expect actual &&\n \n@@ -274,8 +276,8 @@ test_expect_success 'pull with http:// using protocol v1' '\n test_expect_success 'push with http:// using protocol v1' '\n \ttest_commit -C http_child three &&\n \n-\t# Since the repository being served isnt bare we need to push to\n-\t# another branch explicitly to avoid mangling the master branch\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n \tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n \t\tpush origin HEAD:client_branch && #2>log &&\n \ndiff --git a/upload-pack.c b/upload-pack.c\nindex 5cab39819..ef438e9c2 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1078,9 +1078,10 @@ int cmd_main(int argc, const char **argv)\n \t\t * so just fall through after writing the version string.\n \t\t */\n \tcase protocol_v0:\n-\tdefault:\n \t\tupload_pack();\n \t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown protocol version\");\n \t}\n \n \treturn 0;\n\n\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329593","messageId":"20171003201507.3589-5-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 04/10] daemon: recognize hidden request arguments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:15:01Z","receivedAt":"2017-10-03T20:15:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"A normal request to git-daemon is structured as\n\"command path/to/repo\\0host=..\\0\" and due to a bug introduced in\n49ba83fb6 (Add virtualization support to git-daemon, 2006-09-19) we\naren't able to place any extra arguments (separated by NULs) besides the\nhost otherwise the parsing of those arguments would enter an infinite\nloop. This bug was fixed in 73bb33a94 (daemon: Strictly parse the\n\"extra arg\" part of the command, 2009-06-04) but a check was put in\nplace to disallow extra arguments so that new clients wouldn't trigger\nthis bug in older servers.\n\nIn order to get around this limitation teach git-daemon to recognize\nadditional request arguments hidden behind a second NUL byte. Requests\ncan then be structured like:\n\"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\ncan then parse out the extra arguments and set 'GIT_PROTOCOL'\naccordingly.\n\nBy placing these extra arguments behind a second NUL byte we can skirt\naround both the infinite loop bug in 49ba83fb6 (Add virtualization\nsupport to git-daemon, 2006-09-19) as well as the explicit disallowing\nof extra arguments introduced in 73bb33a94 (daemon: Strictly parse the\n\"extra arg\" part of the command, 2009-06-04) because both of these\nversions of git-daemon check for a single NUL byte after the host\nargument before terminating the argument parsing.\n\nSigned-off-by: Brandon Williams \n---\n daemon.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++++++---------\n 1 file changed, 59 insertions(+), 9 deletions(-)\n\ndiff --git a/daemon.c b/daemon.c\nindex 30747075f..36cc794c9 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -282,7 +282,7 @@ static const char *path_ok(const char *directory, struct hostinfo *hi)\n \treturn NULL;\t\t/* Fallthrough. Deny by default */\n }\n \n-typedef int (*daemon_service_fn)(void);\n+typedef int (*daemon_service_fn)(const struct argv_array *env);\n struct daemon_service {\n \tconst char *name;\n \tconst char *config_name;\n@@ -363,7 +363,7 @@ static int run_access_hook(struct daemon_service *service, const char *dir,\n }\n \n static int run_service(const char *dir, struct daemon_service *service,\n-\t\t struct hostinfo *hi)\n+\t\t struct hostinfo *hi, const struct argv_array *env)\n {\n \tconst char *path;\n \tint enabled = service->enabled;\n@@ -422,7 +422,7 @@ static int run_service(const char *dir, struct daemon_service *service,\n \t */\n \tsignal(SIGTERM, SIG_IGN);\n \n-\treturn service->fn();\n+\treturn service->fn(env);\n }\n \n static void copy_to_log(int fd)\n@@ -462,25 +462,34 @@ static int run_service_command(struct child_process *cld)\n \treturn finish_command(cld);\n }\n \n-static int upload_pack(void)\n+static int upload_pack(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_pushl(&cld.args, \"upload-pack\", \"--strict\", NULL);\n \targv_array_pushf(&cld.args, \"--timeout=%u\", timeout);\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n-static int upload_archive(void)\n+static int upload_archive(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_push(&cld.args, \"upload-archive\");\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n-static int receive_pack(void)\n+static int receive_pack(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_push(&cld.args, \"receive-pack\");\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n@@ -574,7 +583,7 @@ static void canonicalize_client(struct strbuf *out, const char *in)\n /*\n * Read the host as supplied by the client connection.\n */\n-static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n+static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n {\n \tchar *val;\n \tint vallen;\n@@ -602,6 +611,43 @@ static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n \t\tif (extra_args < end && *extra_args)\n \t\t\tdie(\"Invalid request\");\n \t}\n+\n+\treturn extra_args;\n+}\n+\n+static void parse_extra_args(struct hostinfo *hi, struct argv_array *env,\n+\t\t\t char *extra_args, int buflen)\n+{\n+\tconst char *end = extra_args + buflen;\n+\tstruct strbuf git_protocol = STRBUF_INIT;\n+\n+\t/* First look for the host argument */\n+\textra_args = parse_host_arg(hi, extra_args, buflen);\n+\n+\t/* Look for additional arguments places after a second NUL byte */\n+\tfor (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n+\t\tconst char *arg = extra_args;\n+\n+\t\t/*\n+\t\t * Parse the extra arguments, adding most to 'git_protocol'\n+\t\t * which will be used to set the 'GIT_PROTOCOL' envvar in the\n+\t\t * service that will be run.\n+\t\t *\n+\t\t * If there ends up being a particular arg in the future that\n+\t\t * git-daemon needs to parse specificly (like the 'host' arg)\n+\t\t * then it can be parsed here and not added to 'git_protocol'.\n+\t\t */\n+\t\tif (*arg) {\n+\t\t\tif (git_protocol.len > 0)\n+\t\t\t\tstrbuf_addch(&git_protocol, ':');\n+\t\t\tstrbuf_addstr(&git_protocol, arg);\n+\t\t}\n+\t}\n+\n+\tif (git_protocol.len > 0)\n+\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n+\t\t\t\t git_protocol.buf);\n+\tstrbuf_release(&git_protocol);\n }\n \n /*\n@@ -695,6 +741,7 @@ static int execute(void)\n \tint pktlen, len, i;\n \tchar *addr = getenv(\"REMOTE_ADDR\"), *port = getenv(\"REMOTE_PORT\");\n \tstruct hostinfo hi;\n+\tstruct argv_array env = ARGV_ARRAY_INIT;\n \n \thostinfo_init(&hi);\n \n@@ -716,8 +763,9 @@ static int execute(void)\n \t\tpktlen--;\n \t}\n \n+\t/* parse additional args hidden behind a NUL byte */\n \tif (len != pktlen)\n-\t\tparse_host_arg(&hi, line + len + 1, pktlen - len - 1);\n+\t\tparse_extra_args(&hi, &env, line + len + 1, pktlen - len - 1);\n \n \tfor (i = 0; i < ARRAY_SIZE(daemon_service); i++) {\n \t\tstruct daemon_service *s = &(daemon_service[i]);\n@@ -730,13 +778,15 @@ static int execute(void)\n \t\t\t * Note: The directory here is probably context sensitive,\n \t\t\t * and might depend on the actual service being performed.\n \t\t\t */\n-\t\t\tint rc = run_service(arg, s, &hi);\n+\t\t\tint rc = run_service(arg, s, &hi, &env);\n \t\t\thostinfo_clear(&hi);\n+\t\t\targv_array_clear(&env);\n \t\t\treturn rc;\n \t\t}\n \t}\n \n \thostinfo_clear(&hi);\n+\targv_array_clear(&env);\n \tlogerror(\"Protocol error: '%s'\", line);\n \treturn -1;\n }\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329594","messageId":"20171003201507.3589-4-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:15:00Z","receivedAt":"2017-10-03T20:15:43Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Create protocol.{c,h} and provide functions which future servers and\nclients can use to determine which protocol to use or is being used.\n\nAlso introduce the 'GIT_PROTOCOL' environment variable which will be\nused to communicate a colon separated list of keys with optional values\nto a server. Unknown keys and values must be tolerated. This mechanism\nis used to communicate which version of the wire protocol a client would\nlike to use with a server.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/config.txt | 17 +++++++++++\n Documentation/git.txt | 6 ++++\n Makefile | 1 +\n cache.h | 8 +++++\n protocol.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++++\n protocol.h | 33 ++++++++++++++++++++\n 6 files changed, 144 insertions(+)\n create mode 100644 protocol.c\n create mode 100644 protocol.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dc4e3f58a..b78747abc 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2517,6 +2517,23 @@ The protocol names currently used by git are:\n `hg` to allow the `git-remote-hg` helper)\n --\n \n+protocol.version::\n+\tExperimental. If set, clients will attempt to communicate with a\n+\tserver using the specified protocol version. If unset, no\n+\tattempt will be made by the client to communicate using a\n+\tparticular protocol version, this results in protocol version 0\n+\tbeing used.\n+\tSupported versions:\n++\n+--\n+\n+* `0` - the original wire protocol.\n+\n+* `1` - the original wire protocol with the addition of a version string\n+ in the initial response from the server.\n+\n+--\n+\n pull.ff::\n \tBy default, Git does not create an extra merge commit when merging\n \ta commit that is a descendant of the current commit. Instead, the\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 6e3a6767e..7518ea3af 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -697,6 +697,12 @@ of clones and fetches.\n \twhich feed potentially-untrusted URLS to git commands. See\n \tlinkgit:git-config[1] for more details.\n \n+`GIT_PROTOCOL`::\n+\tFor internal use only. Used in handshaking the wire protocol.\n+\tContains a colon ':' separated list of keys with optional values\n+\t'key[=value]'. Presence of unknown keys and values must be\n+\tignored.\n+\n Discussion[[Discussion]]\n ------------------------\n \ndiff --git a/Makefile b/Makefile\nindex ed4ca438b..9ce68cded 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -842,6 +842,7 @@ LIB_OBJS += pretty.o\n LIB_OBJS += prio-queue.o\n LIB_OBJS += progress.o\n LIB_OBJS += prompt.o\n+LIB_OBJS += protocol.o\n LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\ndiff --git a/cache.h b/cache.h\nindex 49b083ee0..c74b73671 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -445,6 +445,14 @@ static inline enum object_type object_type(unsigned int mode)\n #define GIT_ICASE_PATHSPECS_ENVIRONMENT \"GIT_ICASE_PATHSPECS\"\n #define GIT_QUARANTINE_ENVIRONMENT \"GIT_QUARANTINE_PATH\"\n \n+/*\n+ * Environment variable used in handshaking the wire protocol.\n+ * Contains a colon ':' separated list of keys with optional values\n+ * 'key[=value]'. Presence of unknown keys and values must be\n+ * ignored.\n+ */\n+#define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n+\n /*\n * This environment variable is expected to contain a boolean indicating\n * whether we should or should not treat:\ndiff --git a/protocol.c b/protocol.c\nnew file mode 100644\nindex 000000000..43012b7eb\n--- /dev/null\n+++ b/protocol.c\n@@ -0,0 +1,79 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"protocol.h\"\n+\n+static enum protocol_version parse_protocol_version(const char *value)\n+{\n+\tif (!strcmp(value, \"0\"))\n+\t\treturn protocol_v0;\n+\telse if (!strcmp(value, \"1\"))\n+\t\treturn protocol_v1;\n+\telse\n+\t\treturn protocol_unknown_version;\n+}\n+\n+enum protocol_version get_protocol_version_config(void)\n+{\n+\tconst char *value;\n+\tif (!git_config_get_string_const(\"protocol.version\", &value)) {\n+\t\tenum protocol_version version = parse_protocol_version(value);\n+\n+\t\tif (version == protocol_unknown_version)\n+\t\t\tdie(\"unknown value for config 'protocol.version': %s\",\n+\t\t\t value);\n+\n+\t\treturn version;\n+\t}\n+\n+\treturn protocol_v0;\n+}\n+\n+enum protocol_version determine_protocol_version_server(void)\n+{\n+\tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n+\tenum protocol_version version = protocol_v0;\n+\n+\t/*\n+\t * Determine which protocol version the client has requested. Since\n+\t * multiple 'version' keys can be sent by the client, indicating that\n+\t * the client is okay to speak any of them, select the greatest version\n+\t * that the client has requested. This is due to the assumption that\n+\t * the most recent protocol version will be the most state-of-the-art.\n+\t */\n+\tif (git_protocol) {\n+\t\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\t\tconst struct string_list_item *item;\n+\t\tstring_list_split(&list, git_protocol, ':', -1);\n+\n+\t\tfor_each_string_list_item(item, &list) {\n+\t\t\tconst char *value;\n+\t\t\tenum protocol_version v;\n+\n+\t\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n+\t\t\t\tv = parse_protocol_version(value);\n+\t\t\t\tif (v > version)\n+\t\t\t\t\tversion = v;\n+\t\t\t}\n+\t\t}\n+\n+\t\tstring_list_clear(&list, 0);\n+\t}\n+\n+\treturn version;\n+}\n+\n+enum protocol_version determine_protocol_version_client(const char *server_response)\n+{\n+\tenum protocol_version version = protocol_v0;\n+\n+\tif (skip_prefix(server_response, \"version \", &server_response)) {\n+\t\tversion = parse_protocol_version(server_response);\n+\n+\t\tif (version == protocol_unknown_version)\n+\t\t\tdie(\"server is speaking an unknown protocol\");\n+\t\tif (version == protocol_v0)\n+\t\t\tdie(\"protocol error: server explicitly said version 0\");\n+\t}\n+\n+\treturn version;\n+}\ndiff --git a/protocol.h b/protocol.h\nnew file mode 100644\nindex 000000000..1b2bc94a8\n--- /dev/null\n+++ b/protocol.h\n@@ -0,0 +1,33 @@\n+#ifndef PROTOCOL_H\n+#define PROTOCOL_H\n+\n+enum protocol_version {\n+\tprotocol_unknown_version = -1,\n+\tprotocol_v0 = 0,\n+\tprotocol_v1 = 1,\n+};\n+\n+/*\n+ * Used by a client to determine which protocol version to request be used when\n+ * communicating with a server, reflecting the configured value of the\n+ * 'protocol.version' config. If unconfigured, a value of 'protocol_v0' is\n+ * returned.\n+ */\n+extern enum protocol_version get_protocol_version_config(void);\n+\n+/*\n+ * Used by a server to determine which protocol version should be used based on\n+ * a client's request, communicated via the 'GIT_PROTOCOL' environment variable\n+ * by setting appropriate values for the key 'version'. If a client doesn't\n+ * request a particular protocol version, a default of 'protocol_v0' will be\n+ * used.\n+ */\n+extern enum protocol_version determine_protocol_version_server(void);\n+\n+/*\n+ * Used by a client to determine which protocol version the server is speaking\n+ * based on the server's initial response.\n+ */\n+extern enum protocol_version determine_protocol_version_client(const char *server_response);\n+\n+#endif /* PROTOCOL_H */\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329595","messageId":"20171003201507.3589-3-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 02/10] pkt-line: add packet_write function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:14:59Z","receivedAt":"2017-10-03T20:15:47Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a function which can be used to write the contents of an arbitrary\nbuffer. This makes it easy to build up data in a buffer before writing\nthe packet instead of formatting the entire contents of the packet using\n'packet_write_fmt()'.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 6 ++++++\n pkt-line.h | 1 +\n 2 files changed, 7 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 647bbd3bc..c025d0332 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -188,6 +188,12 @@ static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n \treturn 0;\n }\n \n+void packet_write(const int fd_out, const char *buf, size_t size)\n+{\n+\tif (packet_write_gently(fd_out, buf, size))\n+\t\tdie_errno(\"packet write failed\");\n+}\n+\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n {\n \tva_list args;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 66ef610fc..d9e9783b1 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -22,6 +22,7 @@\n void packet_flush(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n+void packet_write(const int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329596","messageId":"20171003201507.3589-6-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 05/10] upload-pack, receive-pack: introduce protocol version 1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:15:02Z","receivedAt":"2017-10-03T20:15:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach upload-pack and receive-pack to understand and respond using\nprotocol version 1, if requested.\n\nProtocol version 1 is simply the original and current protocol (what I'm\ncalling version 0) with the addition of a single packet line, which\nprecedes the ref advertisement, indicating the protocol version being\nspoken.\n\nSigned-off-by: Brandon Williams \n---\n builtin/receive-pack.c | 15 +++++++++++++++\n upload-pack.c | 18 +++++++++++++++++-\n 2 files changed, 32 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex dd06b3fb4..94b7d29ea 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -24,6 +24,7 @@\n #include \"tmp-objdir.h\"\n #include \"oidset.h\"\n #include \"packfile.h\"\n+#include \"protocol.h\"\n \n static const char * const receive_pack_usage[] = {\n \tN_(\"git receive-pack \"),\n@@ -1963,6 +1964,20 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \telse if (0 <= receive_unpack_limit)\n \t\tunpack_limit = receive_unpack_limit;\n \n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\tcase protocol_v0:\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \tif (advertise_refs || !stateless_rpc) {\n \t\twrite_head_info();\n \t}\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 7efff2fbf..ef438e9c2 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -18,6 +18,7 @@\n #include \"parse-options.h\"\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n+#include \"protocol.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -1067,6 +1068,21 @@ int cmd_main(int argc, const char **argv)\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n \tgit_config(upload_pack_config, NULL);\n-\tupload_pack();\n+\n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\tcase protocol_v0:\n+\t\tupload_pack();\n+\t\tbreak;\n+\tdefault:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \treturn 0;\n }\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329597","messageId":"20171003201507.3589-2-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 01/10] connect: in ref advertisement, shallows are last","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:14:58Z","receivedAt":"2017-10-03T20:15:58Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"From: Jonathan Tan \n\nCurrently, get_remote_heads() parses the ref advertisement in one loop,\nallowing refs and shallow lines to intersperse, despite this not being\nallowed by the specification. Refactor get_remote_heads() to use two\nloops instead, enforcing that refs come first, and then shallows.\n\nThis also makes it easier to teach get_remote_heads() to interpret other\nlines in the ref advertisement, which will be done in a subsequent\npatch.\n\nAs part of this change, this patch interprets capabilities only on the\nfirst line in the ref advertisement, printing a warning message when\nencountering capabilities on other lines.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Brandon Williams \n---\n connect.c | 189 ++++++++++++++++++++++++++++++++++++++++----------------------\n 1 file changed, 123 insertions(+), 66 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex df56c0cbf..8e2e276b6 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -11,6 +11,7 @@\n #include \"string-list.h\"\n #include \"sha1-array.h\"\n #include \"transport.h\"\n+#include \"strbuf.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -107,6 +108,104 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n+/*\n+ * Read one line of a server's ref advertisement into packet_buffer.\n+ */\n+static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n+\t\t\t int *responded)\n+{\n+\tint len = packet_read(in, src_buf, src_len,\n+\t\t\t packet_buffer, sizeof(packet_buffer),\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\tconst char *arg;\n+\tif (len < 0)\n+\t\tdie_initial_contact(*responded);\n+\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n+\t\tdie(\"remote error: %s\", arg);\n+\n+\t*responded = 1;\n+\n+\treturn len;\n+}\n+\n+#define EXPECTING_FIRST_REF 0\n+#define EXPECTING_REF 1\n+#define EXPECTING_SHALLOW 2\n+\n+static void process_capabilities(int *len)\n+{\n+\tint nul_location = strlen(packet_buffer);\n+\tif (nul_location == *len)\n+\t\treturn;\n+\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n+\t*len = nul_location;\n+}\n+\n+static int process_dummy_ref(void)\n+{\n+\tstruct object_id oid;\n+\tconst char *name;\n+\n+\tif (parse_oid_hex(packet_buffer, &oid, &name))\n+\t\treturn 0;\n+\tif (*name != ' ')\n+\t\treturn 0;\n+\tname++;\n+\n+\treturn !oidcmp(&null_oid, &oid) && !strcmp(name, \"capabilities^{}\");\n+}\n+\n+static void check_no_capabilities(int len)\n+{\n+\tif (strlen(packet_buffer) != len)\n+\t\twarning(\"Ignoring capabilities after first line '%s'\",\n+\t\t\tpacket_buffer + strlen(packet_buffer));\n+}\n+\n+static int process_ref(int len, struct ref ***list, unsigned int flags,\n+\t\t struct oid_array *extra_have)\n+{\n+\tstruct object_id old_oid;\n+\tconst char *name;\n+\n+\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n+\t\treturn 0;\n+\tif (*name != ' ')\n+\t\treturn 0;\n+\tname++;\n+\n+\tif (extra_have && !strcmp(name, \".have\")) {\n+\t\toid_array_append(extra_have, &old_oid);\n+\t} else if (!strcmp(name, \"capabilities^{}\")) {\n+\t\tdie(\"protocol error: unexpected capabilities^{}\");\n+\t} else if (check_ref(name, flags)) {\n+\t\tstruct ref *ref = alloc_ref(name);\n+\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\t**list = ref;\n+\t\t*list = &ref->next;\n+\t}\n+\tcheck_no_capabilities(len);\n+\treturn 1;\n+}\n+\n+static int process_shallow(int len, struct oid_array *shallow_points)\n+{\n+\tconst char *arg;\n+\tstruct object_id old_oid;\n+\n+\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\t\treturn 0;\n+\n+\tif (get_oid_hex(arg, &old_oid))\n+\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n+\tif (!shallow_points)\n+\t\tdie(\"repository on the other end cannot be shallow\");\n+\toid_array_append(shallow_points, &old_oid);\n+\tcheck_no_capabilities(len);\n+\treturn 1;\n+}\n+\n /*\n * Read all the refs from the other end\n */\n@@ -123,76 +222,34 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t * willing to talk to us. A hang-up before seeing any\n \t * response does not necessarily mean an ACL problem, though.\n \t */\n-\tint saw_response;\n-\tint got_dummy_ref_with_capabilities_declaration = 0;\n+\tint responded = 0;\n+\tint len;\n+\tint state = EXPECTING_FIRST_REF;\n \n \t*list = NULL;\n-\tfor (saw_response = 0; ; saw_response = 1) {\n-\t\tstruct ref *ref;\n-\t\tstruct object_id old_oid;\n-\t\tchar *name;\n-\t\tint len, name_len;\n-\t\tchar *buffer = packet_buffer;\n-\t\tconst char *arg;\n-\n-\t\tlen = packet_read(in, &src_buf, &src_len,\n-\t\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\t\tif (len < 0)\n-\t\t\tdie_initial_contact(saw_response);\n-\n-\t\tif (!len)\n-\t\t\tbreak;\n-\n-\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n-\t\t\tdie(\"remote error: %s\", arg);\n-\n-\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n-\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &old_oid))\n-\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n-\t\t\tif (!shallow_points)\n-\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n-\t\t\toid_array_append(shallow_points, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n \n-\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n-\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n-\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n-\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n-\n-\t\tname_len = strlen(name);\n-\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n-\t\t\tfree(server_capabilities);\n-\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n-\t\t}\n-\n-\t\tif (extra_have && !strcmp(name, \".have\")) {\n-\t\t\toid_array_append(extra_have, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (!strcmp(name, \"capabilities^{}\")) {\n-\t\t\tif (saw_response)\n-\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n-\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n-\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n-\t\t\tcontinue;\n+\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\t\tswitch (state) {\n+\t\tcase EXPECTING_FIRST_REF:\n+\t\t\tprocess_capabilities(&len);\n+\t\t\tif (process_dummy_ref()) {\n+\t\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tstate = EXPECTING_REF;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_REF:\n+\t\t\tif (process_ref(len, &list, flags, extra_have))\n+\t\t\t\tbreak;\n+\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_SHALLOW:\n+\t\t\tif (process_shallow(len, shallow_points))\n+\t\t\t\tbreak;\n+\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n+\t\tdefault:\n+\t\t\tdie(\"unexpected state %d\", state);\n \t\t}\n-\n-\t\tif (!check_ref(name, flags))\n-\t\t\tcontinue;\n-\n-\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n-\n-\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n-\t\t*list = ref;\n-\t\tlist = &ref->next;\n \t}\n \n \tannotate_refs_with_symref_info(*orig_list);\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329598","messageId":"20171003201507.3589-10-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 09/10] i5700: add interop test for protocol transition","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:15:06Z","receivedAt":"2017-10-03T20:15:59Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams \n---\n t/interop/i5700-protocol-transition.sh | 68 ++++++++++++++++++++++++++++++++++\n 1 file changed, 68 insertions(+)\n create mode 100755 t/interop/i5700-protocol-transition.sh\n\ndiff --git a/t/interop/i5700-protocol-transition.sh b/t/interop/i5700-protocol-transition.sh\nnew file mode 100755\nindex 000000000..97e8e580e\n--- /dev/null\n+++ b/t/interop/i5700-protocol-transition.sh\n@@ -0,0 +1,68 @@\n+#!/bin/sh\n+\n+VERSION_A=.\n+VERSION_B=v2.0.0\n+\n+: ${LIB_GIT_DAEMON_PORT:=5700}\n+LIB_GIT_DAEMON_COMMAND='git.b daemon'\n+\n+test_description='clone and fetch by client who is trying to use a new protocol'\n+. ./interop-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\n+start_git_daemon --export-all\n+\n+repo=$GIT_DAEMON_DOCUMENT_ROOT_PATH/repo\n+\n+test_expect_success \"create repo served by $VERSION_B\" '\n+\tgit.b init \"$repo\" &&\n+\tgit.b -C \"$repo\" commit --allow-empty -m one\n+'\n+\n+test_expect_success \"git:// clone with $VERSION_A and protocol v1\" '\n+\tGIT_TRACE_PACKET=1 git.a -c protocol.version=1 clone \"$GIT_DAEMON_URL/repo\" child 2>log &&\n+\tgit.a -C child log -1 --format=%s >actual &&\n+\tgit.b -C \"$repo\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\tgrep \"version=1\" log\n+'\n+\n+test_expect_success \"git:// fetch with $VERSION_A and protocol v1\" '\n+\tgit.b -C \"$repo\" commit --allow-empty -m two &&\n+\tgit.b -C \"$repo\" log -1 --format=%s >expect &&\n+\n+\tGIT_TRACE_PACKET=1 git.a -C child -c protocol.version=1 fetch 2>log &&\n+\tgit.a -C child log -1 --format=%s FETCH_HEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"version=1\" log &&\n+\t! grep \"version 1\" log\n+'\n+\n+stop_git_daemon\n+\n+test_expect_success \"create repo served by $VERSION_B\" '\n+\tgit.b init parent &&\n+\tgit.b -C parent commit --allow-empty -m one\n+'\n+\n+test_expect_success \"file:// clone with $VERSION_A and protocol v1\" '\n+\tGIT_TRACE_PACKET=1 git.a -c protocol.version=1 clone --upload-pack=\"git.b upload-pack\" parent child2 2>log &&\n+\tgit.a -C child2 log -1 --format=%s >actual &&\n+\tgit.b -C parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\t! grep \"version 1\" log\n+'\n+\n+test_expect_success \"file:// fetch with $VERSION_A and protocol v1\" '\n+\tgit.b -C parent commit --allow-empty -m two &&\n+\tgit.b -C parent log -1 --format=%s >expect &&\n+\n+\tGIT_TRACE_PACKET=1 git.a -C child2 -c protocol.version=1 fetch --upload-pack=\"git.b upload-pack\" 2>log &&\n+\tgit.a -C child2 log -1 --format=%s FETCH_HEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\t! grep \"version 1\" log\n+'\n+\n+test_done\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329599","messageId":"20171003201507.3589-9-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 08/10] http: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:15:05Z","receivedAt":"2017-10-03T20:16:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Tell a server that protocol v1 can be used by sending the http header\n'Git-Protocol' indicating this.\n\nAlso teach the apache http server to pass through the 'Git-Protocol'\nheader as an environment variable 'GIT_PROTOCOL'.\n\nSigned-off-by: Brandon Williams \n---\n cache.h | 2 ++\n http.c | 18 +++++++++++++\n t/lib-httpd/apache.conf | 7 +++++\n t/t5700-protocol-v1.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 96 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex c74b73671..3a6b869c2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -452,6 +452,8 @@ static inline enum object_type object_type(unsigned int mode)\n * ignored.\n */\n #define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n+/* HTTP header used to handshake the wire protocol */\n+#define GIT_PROTOCOL_HEADER \"Git-Protocol\"\n \n /*\n * This environment variable is expected to contain a boolean indicating\ndiff --git a/http.c b/http.c\nindex 9e40a465f..ffb719216 100644\n--- a/http.c\n+++ b/http.c\n@@ -12,6 +12,7 @@\n #include \"gettext.h\"\n #include \"transport.h\"\n #include \"packfile.h\"\n+#include \"protocol.h\"\n \n static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);\n #if LIBCURL_VERSION_NUM >= 0x070a08\n@@ -897,6 +898,21 @@ static void set_from_env(const char **var, const char *envname)\n \t\t*var = val;\n }\n \n+static void protocol_http_header(void)\n+{\n+\tif (get_protocol_version_config() > 0) {\n+\t\tstruct strbuf protocol_header = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&protocol_header, GIT_PROTOCOL_HEADER \": version=%d\",\n+\t\t\t get_protocol_version_config());\n+\n+\n+\t\textra_http_headers = curl_slist_append(extra_http_headers,\n+\t\t\t\t\t\t protocol_header.buf);\n+\t\tstrbuf_release(&protocol_header);\n+\t}\n+}\n+\n void http_init(struct remote *remote, const char *url, int proactive_auth)\n {\n \tchar *low_speed_limit;\n@@ -927,6 +943,8 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tif (remote)\n \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n \n+\tprotocol_http_header();\n+\n \tpragma_header = curl_slist_append(http_copy_default_headers(),\n \t\t\"Pragma: no-cache\");\n \tno_pragma_header = curl_slist_append(http_copy_default_headers(),\ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 0642ae7e6..df1943631 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -67,6 +67,9 @@ LockFile accept.lock\n \n \tLoadModule unixd_module modules/mod_unixd.so\n \n+\n+\tLoadModule setenvif_module modules/mod_setenvif.so\n+\n \n \n PassEnv GIT_VALGRIND\n@@ -76,6 +79,10 @@ PassEnv ASAN_OPTIONS\n PassEnv GIT_TRACE\n PassEnv GIT_CONFIG_NOSYSTEM\n \n+= 2.4>\n+\tSetEnvIf Git-Protocol \".*\" GIT_PROTOCOL=$0\n+\n+\n Alias /dumb/ www/\n Alias /auth/dumb/ www/auth/dumb/\n \ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nindex 6551932da..b0779d362 100755\n--- a/t/t5700-protocol-v1.sh\n+++ b/t/t5700-protocol-v1.sh\n@@ -220,4 +220,73 @@ test_expect_success 'push with ssh:// using protocol v1' '\n \tgrep \"push< version 1\" log\n '\n \n+# Test protocol v1 with 'http://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one\n+'\n+\n+test_expect_success 'clone with http:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 GIT_TRACE_CURL=1 git -c protocol.version=1 \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"Git-Protocol: version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'fetch with http:// using protocol v1' '\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'pull with http:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'push with http:// using protocol v1' '\n+\ttest_commit -C http_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch && #2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329600","messageId":"20171003201507.3589-11-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 10/10] ssh: introduce a 'simple' ssh variant","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:15:07Z","receivedAt":"2017-10-03T20:16:03Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"When using the 'ssh' transport, the '-o' option is used to specify an\nenvironment variable which should be set on the remote end. This allows\ngit to send additional information when contacting the server,\nrequesting the use of a different protocol version via the\n'GIT_PROTOCOL' environment variable like so: \"-o SendEnv=GIT_PROTOCOL\"\n\nUnfortunately not all ssh variants support the sending of environment\nvariables to the remote end. To account for this, only use the '-o'\noption for ssh variants which are OpenSSH compliant. This is done by\nchecking that the basename of the ssh command is 'ssh' or the ssh\nvariant is overridden to be 'ssh' (via the ssh.variant config).\n\nPreviously if an ssh command's basename wasn't 'plink' or\n'tortoiseplink' git assumed that the command was an OpenSSH variant.\nSince user configured ssh commands may not be OpenSSH compliant, tighten\nthis constraint and assume a variant of 'simple' if the basename of the\ncommand doesn't match the variants known to git. The new ssh variant\n'simple' will only have the host and command to execute ([username@]host\ncommand) passed as parameters to the ssh command.\n\nUpdate the Documentation to better reflect the command-line options sent\nto ssh commands based on their variant.\n\nReported-by: Jeffrey Yasskin \nSigned-off-by: Brandon Williams \n---\n Documentation/config.txt | 27 ++++++++++--\n Documentation/git.txt | 9 ++--\n connect.c | 107 ++++++++++++++++++++++++++---------------------\n t/t5601-clone.sh | 9 ++--\n t/t5700-protocol-v1.sh | 2 +\n 5 files changed, 95 insertions(+), 59 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex b78747abc..0460af37e 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2084,12 +2084,31 @@ ssh.variant::\n \tDepending on the value of the environment variables `GIT_SSH` or\n \t`GIT_SSH_COMMAND`, or the config setting `core.sshCommand`, Git\n \tauto-detects whether to adjust its command-line parameters for use\n-\twith plink or tortoiseplink, as opposed to the default (OpenSSH).\n+\twith ssh (OpenSSH), plink or tortoiseplink, as opposed to the default\n+\t(simple).\n +\n The config variable `ssh.variant` can be set to override this auto-detection;\n-valid values are `ssh`, `plink`, `putty` or `tortoiseplink`. Any other value\n-will be treated as normal ssh. This setting can be overridden via the\n-environment variable `GIT_SSH_VARIANT`.\n+valid values are `ssh`, `simple`, `plink`, `putty` or `tortoiseplink`. Any\n+other value will be treated as normal ssh. This setting can be overridden via\n+the environment variable `GIT_SSH_VARIANT`.\n++\n+The current command-line parameters used for each variant are as\n+follows:\n++\n+--\n+\n+* `ssh` - [-p port] [-4] [-6] [-o option] [username@]host command\n+\n+* `simple` - [username@]host command\n+\n+* `plink` or `putty` - [-P port] [-4] [-6] [username@]host command\n+\n+* `tortoiseplink` - [-P port] [-4] [-6] -batch [username@]host command\n+\n+--\n++\n+Except for the `simple` variant, command-line parameters are likely to\n+change as git gains new features.\n \n i18n.commitEncoding::\n \tCharacter encoding the commit messages are stored in; Git itself\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 7518ea3af..8bc3f2147 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -518,11 +518,10 @@ other\n \tIf either of these environment variables is set then 'git fetch'\n \tand 'git push' will use the specified command instead of 'ssh'\n \twhen they need to connect to a remote system.\n-\tThe command will be given exactly two or four arguments: the\n-\t'username@host' (or just 'host') from the URL and the shell\n-\tcommand to execute on that remote system, optionally preceded by\n-\t`-p` (literally) and the 'port' from the URL when it specifies\n-\tsomething other than the default SSH port.\n+\tThe command-line parameters passed to the configured command are\n+\tdetermined by the ssh variant. See `ssh.variant` option in\n+\tlinkgit:git-config[1] for details.\n+\n +\n `$GIT_SSH_COMMAND` takes precedence over `$GIT_SSH`, and is interpreted\n by the shell, which allows additional arguments to be included.\ndiff --git a/connect.c b/connect.c\nindex b8695a2fa..65cee49b6 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -776,37 +776,44 @@ static const char *get_ssh_command(void)\n \treturn NULL;\n }\n \n-static int override_ssh_variant(int *port_option, int *needs_batch)\n+enum ssh_variant {\n+\tVARIANT_SIMPLE,\n+\tVARIANT_SSH,\n+\tVARIANT_PLINK,\n+\tVARIANT_PUTTY,\n+\tVARIANT_TORTOISEPLINK,\n+};\n+\n+static int override_ssh_variant(enum ssh_variant *ssh_variant)\n {\n-\tchar *variant;\n+\tconst char *variant = getenv(\"GIT_SSH_VARIANT\");\n \n-\tvariant = xstrdup_or_null(getenv(\"GIT_SSH_VARIANT\"));\n-\tif (!variant &&\n-\t git_config_get_string(\"ssh.variant\", &variant))\n+\tif (!variant && git_config_get_string_const(\"ssh.variant\", &variant))\n \t\treturn 0;\n \n-\tif (!strcmp(variant, \"plink\") || !strcmp(variant, \"putty\")) {\n-\t\t*port_option = 'P';\n-\t\t*needs_batch = 0;\n-\t} else if (!strcmp(variant, \"tortoiseplink\")) {\n-\t\t*port_option = 'P';\n-\t\t*needs_batch = 1;\n-\t} else {\n-\t\t*port_option = 'p';\n-\t\t*needs_batch = 0;\n-\t}\n-\tfree(variant);\n+\tif (!strcmp(variant, \"plink\"))\n+\t\t*ssh_variant = VARIANT_PLINK;\n+\telse if (!strcmp(variant, \"putty\"))\n+\t\t*ssh_variant = VARIANT_PUTTY;\n+\telse if (!strcmp(variant, \"tortoiseplink\"))\n+\t\t*ssh_variant = VARIANT_TORTOISEPLINK;\n+\telse if (!strcmp(variant, \"simple\"))\n+\t\t*ssh_variant = VARIANT_SIMPLE;\n+\telse\n+\t\t*ssh_variant = VARIANT_SSH;\n+\n \treturn 1;\n }\n \n-static void handle_ssh_variant(const char *ssh_command, int is_cmdline,\n-\t\t\t int *port_option, int *needs_batch)\n+static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n+\t\t\t\t\t int is_cmdline)\n {\n+\tenum ssh_variant ssh_variant = VARIANT_SIMPLE;\n \tconst char *variant;\n \tchar *p = NULL;\n \n-\tif (override_ssh_variant(port_option, needs_batch))\n-\t\treturn;\n+\tif (override_ssh_variant(&ssh_variant))\n+\t\treturn ssh_variant;\n \n \tif (!is_cmdline) {\n \t\tp = xstrdup(ssh_command);\n@@ -825,19 +832,21 @@ static void handle_ssh_variant(const char *ssh_command, int is_cmdline,\n \t\t\tfree(ssh_argv);\n \t\t} else {\n \t\t\tfree(p);\n-\t\t\treturn;\n+\t\t\treturn ssh_variant;\n \t\t}\n \t}\n \n-\tif (!strcasecmp(variant, \"plink\") ||\n-\t !strcasecmp(variant, \"plink.exe\"))\n-\t\t*port_option = 'P';\n+\tif (!strcasecmp(variant, \"ssh\"))\n+\t\tssh_variant = VARIANT_SSH;\n+\telse if (!strcasecmp(variant, \"plink\") ||\n+\t\t !strcasecmp(variant, \"plink.exe\"))\n+\t\tssh_variant = VARIANT_PLINK;\n \telse if (!strcasecmp(variant, \"tortoiseplink\") ||\n-\t\t !strcasecmp(variant, \"tortoiseplink.exe\")) {\n-\t\t*port_option = 'P';\n-\t\t*needs_batch = 1;\n-\t}\n+\t\t !strcasecmp(variant, \"tortoiseplink.exe\"))\n+\t\tssh_variant = VARIANT_TORTOISEPLINK;\n+\n \tfree(p);\n+\treturn ssh_variant;\n }\n \n /*\n@@ -937,8 +946,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tconn->in = conn->out = -1;\n \t\tif (protocol == PROTO_SSH) {\n \t\t\tconst char *ssh;\n-\t\t\tint needs_batch = 0;\n-\t\t\tint port_option = 'p';\n+\t\t\tenum ssh_variant variant;\n \t\t\tchar *ssh_host = hostandport;\n \t\t\tconst char *port = NULL;\n \t\t\ttransport_check_allowed(\"ssh\");\n@@ -965,10 +973,9 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tdie(\"strange hostname '%s' blocked\", ssh_host);\n \n \t\t\tssh = get_ssh_command();\n-\t\t\tif (ssh)\n-\t\t\t\thandle_ssh_variant(ssh, 1, &port_option,\n-\t\t\t\t\t\t &needs_batch);\n-\t\t\telse {\n+\t\t\tif (ssh) {\n+\t\t\t\tvariant = determine_ssh_variant(ssh, 1);\n+\t\t\t} else {\n \t\t\t\t/*\n \t\t\t\t * GIT_SSH is the no-shell version of\n \t\t\t\t * GIT_SSH_COMMAND (and must remain so for\n@@ -979,32 +986,38 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tssh = getenv(\"GIT_SSH\");\n \t\t\t\tif (!ssh)\n \t\t\t\t\tssh = \"ssh\";\n-\t\t\t\telse\n-\t\t\t\t\thandle_ssh_variant(ssh, 0,\n-\t\t\t\t\t\t\t &port_option,\n-\t\t\t\t\t\t\t &needs_batch);\n+\t\t\t\tvariant = determine_ssh_variant(ssh, 0);\n \t\t\t}\n \n \t\t\targv_array_push(&conn->args, ssh);\n \n-\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\tif (variant == VARIANT_SSH &&\n+\t\t\t get_protocol_version_config() > 0) {\n \t\t\t\targv_array_push(&conn->args, \"-o\");\n \t\t\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n \t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n \t\t\t\t\t\t get_protocol_version_config());\n \t\t\t}\n \n-\t\t\tif (flags & CONNECT_IPV4)\n-\t\t\t\targv_array_push(&conn->args, \"-4\");\n-\t\t\telse if (flags & CONNECT_IPV6)\n-\t\t\t\targv_array_push(&conn->args, \"-6\");\n-\t\t\tif (needs_batch)\n+\t\t\tif (variant != VARIANT_SIMPLE) {\n+\t\t\t\tif (flags & CONNECT_IPV4)\n+\t\t\t\t\targv_array_push(&conn->args, \"-4\");\n+\t\t\t\telse if (flags & CONNECT_IPV6)\n+\t\t\t\t\targv_array_push(&conn->args, \"-6\");\n+\t\t\t}\n+\n+\t\t\tif (variant == VARIANT_TORTOISEPLINK)\n \t\t\t\targv_array_push(&conn->args, \"-batch\");\n-\t\t\tif (port) {\n-\t\t\t\targv_array_pushf(&conn->args,\n-\t\t\t\t\t\t \"-%c\", port_option);\n+\n+\t\t\tif (port && variant != VARIANT_SIMPLE) {\n+\t\t\t\tif (variant == VARIANT_SSH)\n+\t\t\t\t\targv_array_push(&conn->args, \"-p\");\n+\t\t\t\telse\n+\t\t\t\t\targv_array_push(&conn->args, \"-P\");\n+\n \t\t\t\targv_array_push(&conn->args, port);\n \t\t\t}\n+\n \t\t\targv_array_push(&conn->args, ssh_host);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 9c56f771b..ee1a24c5b 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -312,6 +312,8 @@ setup_ssh_wrapper () {\n \t\t\t\"$TRASH_DIRECTORY/ssh-wrapper$X\" &&\n \t\tGIT_SSH=\"$TRASH_DIRECTORY/ssh-wrapper$X\" &&\n \t\texport GIT_SSH &&\n+\t\tGIT_SSH_VARIANT=ssh &&\n+\t\texport GIT_SSH_VARIANT &&\n \t\texport TRASH_DIRECTORY &&\n \t\t>\"$TRASH_DIRECTORY\"/ssh-output\n \t'\n@@ -320,7 +322,8 @@ setup_ssh_wrapper () {\n copy_ssh_wrapper_as () {\n \tcp \"$TRASH_DIRECTORY/ssh-wrapper$X\" \"${1%$X}$X\" &&\n \tGIT_SSH=\"${1%$X}$X\" &&\n-\texport GIT_SSH\n+\texport GIT_SSH &&\n+\tunset GIT_SSH_VARIANT\n }\n \n expect_ssh () {\n@@ -362,10 +365,10 @@ test_expect_success 'bracketed hostnames are still ssh' '\n \texpect_ssh \"-p 123\" myhost src\n '\n \n-test_expect_success 'uplink is not treated as putty' '\n+test_expect_success 'uplink is treated as simple' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/uplink\" &&\n \tgit clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n-\texpect_ssh \"-p 123\" myhost src\n+\texpect_ssh myhost src\n '\n \n test_expect_success 'plink is treated specially (as putty)' '\ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nindex b0779d362..ba86a44eb 100755\n--- a/t/t5700-protocol-v1.sh\n+++ b/t/t5700-protocol-v1.sh\n@@ -147,6 +147,8 @@ test_expect_success 'push with file:// using protocol v1' '\n test_expect_success 'setup ssh wrapper' '\n \tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh\" &&\n \texport GIT_SSH &&\n+\tGIT_SSH_VARIANT=ssh &&\n+\texport GIT_SSH_VARIANT &&\n \texport TRASH_DIRECTORY &&\n \t>\"$TRASH_DIRECTORY\"/ssh-output\n '\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329601","messageId":"20171003201507.3589-8-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 07/10] connect: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:15:04Z","receivedAt":"2017-10-03T20:16:09Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach the connection logic to tell a serve that it understands protocol\nv1. This is done in 2 different ways for the builtin transports.\n\n1. git://\n A normal request to git-daemon is structured as\n \"command path/to/repo\\0host=..\\0\" and due to a bug introduced in\n 49ba83fb6 (Add virtualization support to git-daemon, 2006-09-19) we\n aren't able to place any extra arguments (separated by NULs) besides\n the host otherwise the parsing of those arguments would enter an\n infinite loop. This bug was fixed in 73bb33a94 (daemon: Strictly\n parse the \"extra arg\" part of the command, 2009-06-04) but a check\n was put in place to disallow extra arguments so that new clients\n wouldn't trigger this bug in older servers.\n\n In order to get around this limitation git-daemon was taught to\n recognize additional request arguments hidden behind a second\n NUL byte. Requests can then be structured like:\n \"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\".\n git-daemon can then parse out the extra arguments and set\n 'GIT_PROTOCOL' accordingly.\n\n By placing these extra arguments behind a second NUL byte we can\n skirt around both the infinite loop bug in 49ba83fb6 (Add\n virtualization support to git-daemon, 2006-09-19) as well as the\n explicit disallowing of extra arguments introduced in 73bb33a94\n (daemon: Strictly parse the \"extra arg\" part of the command,\n 2009-06-04) because both of these versions of git-daemon check for a\n single NUL byte after the host argument before terminating the\n argument parsing.\n\n2. ssh://, file://\n Set 'GIT_PROTOCOL' environment variable with the desired protocol\n version. With the file:// transport, 'GIT_PROTOCOL' can be set\n explicitly in the locally running git-upload-pack or git-receive-pack\n processes. With the ssh:// transport and OpenSSH compliant ssh\n programs, 'GIT_PROTOCOL' can be sent across ssh by using '-o\n SendEnv=GIT_PROTOCOL' and having the server whitelist this\n environment variable.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 37 ++++++--\n t/t5700-protocol-v1.sh | 223 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 255 insertions(+), 5 deletions(-)\n create mode 100755 t/t5700-protocol-v1.sh\n\ndiff --git a/connect.c b/connect.c\nindex a5e708a61..b8695a2fa 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -871,6 +871,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tprintf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n \t\tconn = NULL;\n \t} else if (protocol == PROTO_GIT) {\n+\t\tstruct strbuf request = STRBUF_INIT;\n \t\t/*\n \t\t * Set up virtual host information based on where we will\n \t\t * connect, unless the user has overridden us in\n@@ -898,13 +899,25 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t * Note: Do not add any other headers here! Doing so\n \t\t * will cause older git-daemon servers to crash.\n \t\t */\n-\t\tpacket_write_fmt(fd[1],\n-\t\t\t \"%s %s%chost=%s%c\",\n-\t\t\t prog, path, 0,\n-\t\t\t target_host, 0);\n+\t\tstrbuf_addf(&request,\n+\t\t\t \"%s %s%chost=%s%c\",\n+\t\t\t prog, path, 0,\n+\t\t\t target_host, 0);\n+\n+\t\t/* If using a new version put that stuff here after a second null byte */\n+\t\tif (get_protocol_version_config() > 0) {\n+\t\t\tstrbuf_addch(&request, '\\0');\n+\t\t\tstrbuf_addf(&request, \"version=%d%c\",\n+\t\t\t\t get_protocol_version_config(), '\\0');\n+\t\t}\n+\n+\t\tpacket_write(fd[1], request.buf, request.len);\n+\n \t\tfree(target_host);\n+\t\tstrbuf_release(&request);\n \t} else {\n \t\tstruct strbuf cmd = STRBUF_INIT;\n+\t\tconst char *const *var;\n \n \t\tconn = xmalloc(sizeof(*conn));\n \t\tchild_process_init(conn);\n@@ -917,7 +930,9 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tsq_quote_buf(&cmd, path);\n \n \t\t/* remove repo-local variables from the environment */\n-\t\tconn->env = local_repo_env;\n+\t\tfor (var = local_repo_env; *var; var++)\n+\t\t\targv_array_push(&conn->env_array, *var);\n+\n \t\tconn->use_shell = 1;\n \t\tconn->in = conn->out = -1;\n \t\tif (protocol == PROTO_SSH) {\n@@ -971,6 +986,14 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t}\n \n \t\t\targv_array_push(&conn->args, ssh);\n+\n+\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\t\targv_array_push(&conn->args, \"-o\");\n+\t\t\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n+\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t}\n+\n \t\t\tif (flags & CONNECT_IPV4)\n \t\t\t\targv_array_push(&conn->args, \"-4\");\n \t\t\telse if (flags & CONNECT_IPV6)\n@@ -985,6 +1008,10 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\targv_array_push(&conn->args, ssh_host);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\n+\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t}\n \t\t}\n \t\targv_array_push(&conn->args, cmd.buf);\n \ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nnew file mode 100755\nindex 000000000..6551932da\n--- /dev/null\n+++ b/t/t5700-protocol-v1.sh\n@@ -0,0 +1,223 @@\n+#!/bin/sh\n+\n+test_description='test git wire-protocol transition'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+. ./test-lib.sh\n+\n+# Test protocol v1 with 'git://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\ttest_commit -C \"$daemon_parent\" one\n+'\n+\n+test_expect_success 'clone with git:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"$GIT_DAEMON_URL/parent\" daemon_child 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"clone> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with git:// using protocol v1' '\n+\ttest_commit -C \"$daemon_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with git:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with git:// using protocol v1' '\n+\ttest_commit -C daemon_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"push> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+stop_git_daemon\n+\n+# Test protocol v1 with 'file://' transport\n+#\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init file_parent &&\n+\ttest_commit -C file_parent one\n+'\n+\n+test_expect_success 'clone with file:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"file://$(pwd)/file_parent\" file_child 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with file:// using protocol v1' '\n+\ttest_commit -C file_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with file:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with file:// using protocol v1' '\n+\ttest_commit -C file_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+# Test protocol v1 with 'ssh://' transport\n+#\n+test_expect_success 'setup ssh wrapper' '\n+\tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh\" &&\n+\texport GIT_SSH &&\n+\texport TRASH_DIRECTORY &&\n+\t>\"$TRASH_DIRECTORY\"/ssh-output\n+'\n+\n+expect_ssh () {\n+\ttest_when_finished '(cd \"$TRASH_DIRECTORY\" && rm -f ssh-expect && >ssh-output)' &&\n+\techo \"ssh: -o SendEnv=GIT_PROTOCOL myhost $1 '$PWD/ssh_parent'\" >\"$TRASH_DIRECTORY/ssh-expect\" &&\n+\t(cd \"$TRASH_DIRECTORY\" && test_cmp ssh-expect ssh-output)\n+}\n+\n+test_expect_success 'create repo to be served by ssh:// transport' '\n+\tgit init ssh_parent &&\n+\ttest_commit -C ssh_parent one\n+'\n+\n+test_expect_success 'clone with ssh:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"ssh://myhost:$(pwd)/ssh_parent\" ssh_child 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with ssh:// using protocol v1' '\n+\ttest_commit -C ssh_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s origin/master >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with ssh:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with ssh:// using protocol v1' '\n+\ttest_commit -C ssh_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\texpect_ssh git-receive-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+test_done\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329602","messageId":"20171003201507.3589-7-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v3 06/10] connect: teach client to recognize v1 server response","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-03T20:15:03Z","receivedAt":"2017-10-03T20:16:38Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach a client to recognize that a server understands protocol v1 by\nlooking at the first pkt-line the server sends in response. This is\ndone by looking for the response \"version 1\" send by upload-pack or\nreceive-pack.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 30 ++++++++++++++++++++++++++----\n 1 file changed, 26 insertions(+), 4 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 8e2e276b6..a5e708a61 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -12,6 +12,7 @@\n #include \"sha1-array.h\"\n #include \"transport.h\"\n #include \"strbuf.h\"\n+#include \"protocol.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -129,9 +130,23 @@ static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n \treturn len;\n }\n \n-#define EXPECTING_FIRST_REF 0\n-#define EXPECTING_REF 1\n-#define EXPECTING_SHALLOW 2\n+#define EXPECTING_PROTOCOL_VERSION 0\n+#define EXPECTING_FIRST_REF 1\n+#define EXPECTING_REF 2\n+#define EXPECTING_SHALLOW 3\n+\n+/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n+static int process_protocol_version(void)\n+{\n+\tswitch (determine_protocol_version_client(packet_buffer)) {\n+\tcase protocol_v1:\n+\t\treturn 1;\n+\tcase protocol_v0:\n+\t\treturn 0;\n+\tdefault:\n+\t\tdie(\"server is speaking an unknown protocol\");\n+\t}\n+}\n \n static void process_capabilities(int *len)\n {\n@@ -224,12 +239,19 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t */\n \tint responded = 0;\n \tint len;\n-\tint state = EXPECTING_FIRST_REF;\n+\tint state = EXPECTING_PROTOCOL_VERSION;\n \n \t*list = NULL;\n \n \twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n \t\tswitch (state) {\n+\t\tcase EXPECTING_PROTOCOL_VERSION:\n+\t\t\tif (process_protocol_version()) {\n+\t\t\t\tstate = EXPECTING_FIRST_REF;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tstate = EXPECTING_FIRST_REF;\n+\t\t\t/* fallthrough */\n \t\tcase EXPECTING_FIRST_REF:\n \t\t\tprocess_capabilities(&len);\n \t\t\tif (process_dummy_ref()) {\n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"329608","messageId":"20171003214206.GY19555@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171003201507.3589-11-bmwill@google.com","subject":"Re: [PATCH v3 10/10] ssh: introduce a 'simple' ssh variant","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-03T21:42:06Z","receivedAt":"2017-10-03T21:42:14Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> When using the 'ssh' transport, the '-o' option is used to specify an\n> environment variable which should be set on the remote end. This allows\n> git to send additional information when contacting the server,\n> requesting the use of a different protocol version via the\n> 'GIT_PROTOCOL' environment variable like so: \"-o SendEnv=GIT_PROTOCOL\"\n>\n> Unfortunately not all ssh variants support the sending of environment\n> variables to the remote end. To account for this, only use the '-o'\n> option for ssh variants which are OpenSSH compliant. This is done by\n> checking that the basename of the ssh command is 'ssh' or the ssh\n> variant is overridden to be 'ssh' (via the ssh.variant config).\n\nThis also affects -p (port), right?\n\nWhat happens if I specify a ssh://host:port/path URL and the SSH\nimplementation is of 'simple' type?\n\n> Previously if an ssh command's basename wasn't 'plink' or\n\nGit's commit messages use the present tense to describe the current\nstate of the code, so this is \"Currently\". :)\n\n> 'tortoiseplink' git assumed that the command was an OpenSSH variant.\n> Since user configured ssh commands may not be OpenSSH compliant, tighten\n> this constraint and assume a variant of 'simple' if the basename of the\n> command doesn't match the variants known to git. The new ssh variant\n> 'simple' will only have the host and command to execute ([username@]host\n> command) passed as parameters to the ssh command.\n>\n> Update the Documentation to better reflect the command-line options sent\n> to ssh commands based on their variant.\n>\n> Reported-by: Jeffrey Yasskin \n> Signed-off-by: Brandon Williams \n\nThanks for working on this.\n\nFor background, the GIT_SSH implementation that motivated this is\nhttps://github.com/travis-ci/dpl/blob/6c3fddfda1f2a85944c544446b068bac0a77c049/lib/dpl/provider.rb#L215,\nwhich does not support -p or -4/-6, either.\n\n> ---\n> Documentation/config.txt | 27 ++++++++++--\n> Documentation/git.txt | 9 ++--\n> connect.c | 107 ++++++++++++++++++++++++++---------------------\n> t/t5601-clone.sh | 9 ++--\n> t/t5700-protocol-v1.sh | 2 +\n> 5 files changed, 95 insertions(+), 59 deletions(-)\n[...]\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -776,37 +776,44 @@ static const char *get_ssh_command(void)\n[...]\n> +static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n> +\t\t\t\t\t int is_cmdline)\n[...]\n> -\tif (!strcasecmp(variant, \"plink\") ||\n> -\t !strcasecmp(variant, \"plink.exe\"))\n> -\t\t*port_option = 'P';\n> +\tif (!strcasecmp(variant, \"ssh\"))\n> +\t\tssh_variant = VARIANT_SSH;\n\nCould this handle ssh.exe, too?\n\n[...]\n> --- a/t/t5601-clone.sh\n> +++ b/t/t5601-clone.sh\n\nCan this get tests for the new defaulting behavior? E.g.\n\n - default is \"simple\"\n - how \"simple\" treats an ssh://host:port/path URL\n - how \"simple\" treats ipv4/ipv6 switching\n - ssh defaults to \"ssh\"\n - if GIT_SSH=ssh, can set ssh.variant to \"simple\" to force the \"simple\"\n mode\n\nOne other worry: this (intentionally) changes the behavior of a\npreviously-working GIT_SSH=ssh-wrapper that wants to support\nOpenSSH-style options but does not declare ssh.variant=ssh. When\ndiscovering this change, what should the author of such an ssh-wrapper\ndo?\n\nThey could instruct their users to set ssh.variant or GIT_SSH_VARIANT\nto \"ssh\", but then they are at the mercy of future additional options\nsupported by OpenSSH we may want to start using in the future (e.g.,\nmaybe we will start passing \"--\" to separate options from the\nhostname). So this is not a futureproof option for them.\n\nThey could take the new default behavior or instruct their users to\nset ssh.variant or GIT_SSH_VARIANT to \"simple\", but then they lose\nsupport for handling alternate ports, ipv4/ipv6, and specifying -o\nSendEnv to propagate GIT_PROTOCOL or other envvars. They can handle\nGIT_PROTOCOL propagation manually, but losing port support seems like\na heavy cost.\n\nThey could send a patch to define yet another variant that is\nforward-compatible, for example using an interface similar to what\ngit-credential(1) uses. Then they can set GIT_SSH to their\nOpenSSH-style helper and GIT_FANCY_NEW_SSH to their more modern\nhelper, so that old Git versions could use GIT_SSH and new Git\nversions could use GIT_FANCY_NEW_SSH. This might be their best\noption. It feels odd to say that their only good way forward is to\nsend a patch, but on the other hand someone with such an itch is\nlikely to be in the best position to define an appropriate interface.\n\nThey could send a documentation patch to make more promises about the\ncommandline used in OpenSSH mode: e.g. setting a rule in advance about\nwhich options can take an argument so that they can properly parse an\nOpenSSH command line in a future-compatible way.\n\nOr they could send a patch to allow passing the port in \"simple\"\nmode, for example using an environment variable.\n\nAm I missing another option? What advice do we give to this person?\n\nThanks,\nJonathan\n"},{"id":"329657","messageId":"xmqqh8vfcu2l.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"Re: [PATCH v3 00/10] protocol transition","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-04T06:20:50Z","receivedAt":"2017-10-04T06:20:57Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Thanks. All of my review comments from the previous round seem to\nhave been addressed, so this is Reviewed-by: me ;-)\n\n"},{"id":"329863","messageId":"20171006090907.gbsgygulzj3bgqrg@ruderich.org","threadId":"46746","inReplyTo":"20171003201507.3589-4-bmwill@google.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Simon Ruderich","fromEmail":"simon@ruderich.org","sentAt":"2017-10-06T09:09:07Z","receivedAt":"2017-10-06T09:09:15Z","isPatch":true,"sender":{"key":"simon@ruderich.org","avatar":"https://avatars.githubusercontent.com/u/390994?v=4"},"body":"On Tue, Oct 03, 2017 at 01:15:00PM -0700, Brandon Williams wrote:\n> [snip]\n>\n> +protocol.version::\n> +\tExperimental. If set, clients will attempt to communicate with a\n> +\tserver using the specified protocol version. If unset, no\n> +\tattempt will be made by the client to communicate using a\n> +\tparticular protocol version, this results in protocol version 0\n> +\tbeing used.\n> +\tSupported versions:\n\nDid you consider Stefan Beller's suggestion regarding a\n(white)list of allowed versions?\n\nOn Mon, Sep 18, 2017 at 01:06:59PM -0700, Stefan Beller wrote:\n> Thinking about this, how about:\n>\n> If not configured, we do as we want. (i.e. Git has full control over\n> it's decision making process, which for now is \"favor v0 over v1 as\n> we are experimenting with v1\". This strategy may change in the future\n> to \"prefer highest version number that both client and server can speak\".)\n>\n> If it is configured, \"use highest configured number from the given set\".\n>\n> ?\n\nIt would also allow the server operator to configure only a\nspecific set of versions (to handle the \"version x is\ninsecure/slow\"-issue raised by Stefan Beller). The current code\nalways uses the latest protocol supported by the git binary.\n\nMinor nit, s/extention/extension/ in the patch name?\n\nRegards\nSimon\n-- \n+ privacy is necessary\n+ using gnupg http://gnupg.org\n+ public key id: 0x92FEFDB7E44C32F9\n"},{"id":"329864","messageId":"xmqq4lrcwr5c.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171006090907.gbsgygulzj3bgqrg@ruderich.org","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-06T09:40:31Z","receivedAt":"2017-10-06T09:40:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Simon Ruderich writes:\n\n> Did you consider Stefan Beller's suggestion regarding a\n> (white)list of allowed versions?\n>\n> On Mon, Sep 18, 2017 at 01:06:59PM -0700, Stefan Beller wrote:\n>> Thinking about this, how about:\n>>\n>> If not configured, we do as we want. (i.e. Git has full control over\n>> it's decision making process, which for now is \"favor v0 over v1 as\n>> we are experimenting with v1\". This strategy may change in the future\n>> to \"prefer highest version number that both client and server can speak\".)\n>>\n>> If it is configured, \"use highest configured number from the given set\".\n>>\n>> ?\n>\n> It would also allow the server operator to configure only a\n> specific set of versions (to handle the \"version x is\n> insecure/slow\"-issue raised by Stefan Beller). The current code\n> always uses the latest protocol supported by the git binary.\n\nIf we do anything less trivial than \"highest supported by both\" (and\nI suspect we want to in the final production version), I'd prefer\nthe configuration to list versions one side supports in decreasing\norder of preference (e.g. \"v3 v0 v2\"), and take the earliest from\nthis list that both sides know how to talk, so that we can skip\ninsecure versions altogether by omitting, and we can express that we\nwould rather avoid talking expensive versions unless there is no\nother version that is understood by the other side.\n"},{"id":"329873","messageId":"CAN0heSrWekXdzqzhL3Lk37YOGPHUdKCDU85t3hJgqK7PSVBZ_g@mail.gmail.com","threadId":"46746","inReplyTo":"xmqq4lrcwr5c.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2017-10-06T11:11:50Z","receivedAt":"2017-10-06T11:11:58Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 6 October 2017 at 11:40, Junio C Hamano wrote:\n> Simon Ruderich writes:\n>\n>> Did you consider Stefan Beller's suggestion regarding a\n>> (white)list of allowed versions?\n>>\n>> On Mon, Sep 18, 2017 at 01:06:59PM -0700, Stefan Beller wrote:\n>>> Thinking about this, how about:\n>>>\n>>> If not configured, we do as we want. (i.e. Git has full control over\n>>> it's decision making process, which for now is \"favor v0 over v1 as\n>>> we are experimenting with v1\". This strategy may change in the future\n>>> to \"prefer highest version number that both client and server can speak\".)\n>>>\n>>> If it is configured, \"use highest configured number from the given set\".\n>>>\n>>> ?\n>>\n>> It would also allow the server operator to configure only a\n>> specific set of versions (to handle the \"version x is\n>> insecure/slow\"-issue raised by Stefan Beller). The current code\n>> always uses the latest protocol supported by the git binary.\n>\n> If we do anything less trivial than \"highest supported by both\" (and\n> I suspect we want to in the final production version), I'd prefer\n> the configuration to list versions one side supports in decreasing\n> order of preference (e.g. \"v3 v0 v2\"), and take the earliest from\n> this list that both sides know how to talk, so that we can skip\n> insecure versions altogether by omitting, and we can express that we\n> would rather avoid talking expensive versions unless there is no\n> other version that is understood by the other side.\n\nMaybe I'm missing something Git-specific, but isn't the only thing that\nneeds to be done now, to document/specify that 1) the client should send\nits list ordered by preference, 2) how preference is signalled, and 3)\nthat the server gets to choose?\n\nWhy would a server operator with only v0 and v1 at their disposal want\nto choose v0 instead of v1, considering that -- as far as I understand\n-- they are in fact the same?\n\nDifferent server implementations and different server configurations\nwill be able to apply whatever rules they want in order to decide which\nversion to use. (It's not like the client can verify the choice that the\nserver makes.) And Brandon's \"pick the highest number\" will do for now.\n\nThere are many possible rules that the server could apply and they\nshouldn't affect other servers or what the client does. For example, the\nserver can go \"You seem to know lots of versions, including X and Y.\nThose are the two that I really prefer, but between those two I'm not\npicky, so I'll use whichever of X and Y that you seem to prefer.\" Unless\nI've missed something, we'll never need to implement -- nor specify --\nanything like that before we have learned both v2 and v3.\n\nI guess my thinking is, there's a difference between the protocol and\nthe implementation.\n\nMartin\n"},{"id":"329876","messageId":"xmqqr2ugv5om.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"CAN0heSrWekXdzqzhL3Lk37YOGPHUdKCDU85t3hJgqK7PSVBZ_g@mail.gmail.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-06T12:09:29Z","receivedAt":"2017-10-06T12:09:36Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Martin Ågren writes:\n\n> Maybe I'm missing something Git-specific, but isn't the only thing that\n> needs to be done now, to document/specify that 1) the client should send\n> its list ordered by preference, 2) how preference is signalled, and 3)\n> that the server gets to choose?\n\nI think Simon's reminder of Stefan's was about specifying something\ndifferent from (1) above---it was just a list of good ones (as\nopposed to ones to be avoided). I was suggesting to tweak that to\nmatch what you wrote above.\n\n> Why would a server operator with only v0 and v1 at their disposal want\n> to choose v0 instead of v1, considering that -- as far as I understand\n> -- they are in fact the same?\n\nBecause we may later discover some reason we not yet know that makes\nv$n+1 unsuitable after we introduce it, and we need to avoid it by\npreferring v$n instead?\n"},{"id":"329921","messageId":"CAN0heSo28RYmZKxY4UXq9ewVgYGJCXGf-6KSzkOB-enZppv5qg@mail.gmail.com","threadId":"46746","inReplyTo":"xmqqr2ugv5om.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2017-10-06T19:42:11Z","receivedAt":"2017-10-06T19:42:20Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 6 October 2017 at 14:09, Junio C Hamano wrote:\n> Martin Ågren writes:\n>\n>> Maybe I'm missing something Git-specific, but isn't the only thing that\n>> needs to be done now, to document/specify that 1) the client should send\n>> its list ordered by preference, 2) how preference is signalled, and 3)\n>> that the server gets to choose?\n>\n> I think Simon's reminder of Stefan's was about specifying something\n> different from (1) above---it was just a list of good ones (as\n> opposed to ones to be avoided). I was suggesting to tweak that to\n> match what you wrote above.\n\nI replied to your mail, but also to the general notion of \"we need a\ncarefully designed configuration and fall-back strategy before we can\ninclude this series\" which I sensed (and which I hope I didn't just\nmisrepresent). I didn't make it very clear exactly what I was replying\nto, sorry about that.\n\nNote that my 1-3 above are not about \"configuration\", which I interpret\nas \"how does the user tell Git how to select a protocol version?\", but\nabout the protocol, i.e., \"how do the client and server agree on which\nversion to use?\".\n\n>> Why would a server operator with only v0 and v1 at their disposal want\n>> to choose v0 instead of v1, considering that -- as far as I understand\n>> -- they are in fact the same?\n>\n> Because we may later discover some reason we not yet know that makes\n> v$n+1 unsuitable after we introduce it, and we need to avoid it by\n> preferring v$n instead?\n\nFor n in general, I agree completely. For n=0, and in particular for a\nGit which only knows v0 and v1, I'm not so sure. If v1 has a problem\nwhich v0 doesn't, then to the best of my understanding, that problem\nwould be in this series, i.e., in the version-negotiation. And to\nminimize risk in that area, we'd want to make this series more complex?\nThat might be the correct thing to do -- certainly, if we botch this\nseries totally, then we might be in big trouble going forward --, but\nit's not at all obvious to me. Nor is it obvious to me that such an\nescape-hatch needs to be a multi-choice, prioritized configuration.\n\nIf a fall-back mechanism to v0 is wanted on the first Git server with\nv1/v0, the simplest approach might be to make the server respect\nprotocol.version (possibly with a default of 1!?).\n\nI might be naive in thinking that protocol.version could be removed or\nredefined at our discretion just because it's marked as \"experimental\".\n\nMartin\n"},{"id":"329937","messageId":"CAGZ79kYnhxZs0ptpuiSuBhwYBfOm8NiK4a15Zwas6vbjWWZukA@mail.gmail.com","threadId":"46746","inReplyTo":"CAN0heSo28RYmZKxY4UXq9ewVgYGJCXGf-6KSzkOB-enZppv5qg@mail.gmail.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-06T20:27:56Z","receivedAt":"2017-10-06T20:28:03Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"> I might be naive in thinking that protocol.version could be removed or\n> redefined at our discretion just because it's marked as \"experimental\".\n\nWell the redefinition might very well occur, when we now say \"set to v1\nto test v1 and fallback to v0 otherwise\", but long term we want a white\nor black list or some other protocol selection strategy encoded in this\nconfiguration (we would not want to introduce yet another config to work\naround the initial \"failed experiment\", would we?)\n\nAnd hence I would be careful how we define the meaning of\nprotocol.version now.\n\nFor example we could instead now claim \"protocol.version is a whitelist\nof protocol versions, order is not specified. The only guarantee we're willing\nto give is that no protocol is used that is not on the list\".\n\nLater we may want to either add another variable '.versionSelectionStrategy'\nthat helps out there or we'd just say protocol.version tries to select\nthe leftmost (first) protocol that both ends support.\n\nAll I was trying to say initially is that \"we may try (one of) protocol.version,\nbut fall back to whatever (currently v0)\" is too broad. We'd need to redefine\nit shortly in the foreseeable future already.\n"},{"id":"329991","messageId":"CAN0heSpgZij9qO7NqoT6JWjDdGcJHJjReB-4Ps1MmOvs=857Lw@mail.gmail.com","threadId":"46746","inReplyTo":"CAGZ79kYnhxZs0ptpuiSuBhwYBfOm8NiK4a15Zwas6vbjWWZukA@mail.gmail.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2017-10-08T14:24:53Z","receivedAt":"2017-10-08T14:24:59Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 6 October 2017 at 22:27, Stefan Beller wrote:\n>> I might be naive in thinking that protocol.version could be removed or\n>> redefined at our discretion just because it's marked as \"experimental\".\n>\n> Well the redefinition might very well occur, when we now say \"set to v1\n> to test v1 and fallback to v0 otherwise\", but long term we want a white\n> or black list or some other protocol selection strategy encoded in this\n> configuration (we would not want to introduce yet another config to work\n> around the initial \"failed experiment\", would we?)\n>\n> And hence I would be careful how we define the meaning of\n> protocol.version now.\n\nGood points. If we want to go for a more general / future-proof wording\nnow, then we must already now implement the config-parsing as \"does this\nstring contain the word '1'\" instead of \"is this string exactly '1'\". If\nwe claim that \"34 1 5\" is a valid configuration, then the implementation\nshould accept it. (We'd probably also want to verify that there are only\nintegers and spaces in the string.)\n\n> For example we could instead now claim \"protocol.version is a whitelist\n> of protocol versions, order is not specified. The only guarantee we're willing\n> to give is that no protocol is used that is not on the list\".\n\nIf we want to be able to list more than one version, we need to define\nhow to signal preference from the first day, IMHO. (I know you just gave\nan example; I'm simply responding with what I think makes that example\nnon-ideal.)\n\nThe fact that v0 is requested by lack of data and all other protocols\n(whether v1 or v34) have to be requested by presence of data, is a bit\nunfortunate and it is bound to bleed through into the definitions, at\nleast until v0 is simply ripped out of git.git. Ok, this definition\nsuggests that \"1 0\" will be the preferred variant for checking basic\nrobustness, while \"1\" will be how to ensure you have a peer which knows\nv1.\n\n> All I was trying to say initially is that \"we may try (one of) protocol.version,\n> but fall back to whatever (currently v0)\" is too broad. We'd need to redefine\n> it shortly in the foreseeable future already.\n\nYes we would. I'll post a suggestion elsewhere in the thread.\n\nMartin\n"},{"id":"330033","messageId":"CAN0heSo3fQzqv8xqqBs8S5MST0fs2ABh69L0cL1mi+pwUTG8oQ@mail.gmail.com","threadId":"46746","inReplyTo":"xmqq4lrcwr5c.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Martin Ågren","fromEmail":"martin.agren@gmail.com","sentAt":"2017-10-09T04:05:29Z","receivedAt":"2017-10-09T04:05:36Z","isPatch":true,"sender":{"key":"martin.agren@gmail.com","avatar":null},"body":"On 6 October 2017 at 11:40, Junio C Hamano wrote:\n> Simon Ruderich writes:\n>\n>> Did you consider Stefan Beller's suggestion regarding a\n>> (white)list of allowed versions?\n>>\n>> On Mon, Sep 18, 2017 at 01:06:59PM -0700, Stefan Beller wrote:\n>>> Thinking about this, how about:\n>>>\n>>> If not configured, we do as we want. (i.e. Git has full control over\n>>> it's decision making process, which for now is \"favor v0 over v1 as\n>>> we are experimenting with v1\". This strategy may change in the future\n>>> to \"prefer highest version number that both client and server can speak\".)\n>>>\n>>> If it is configured, \"use highest configured number from the given set\".\n>>>\n>>> ?\n>>\n>> It would also allow the server operator to configure only a\n>> specific set of versions (to handle the \"version x is\n>> insecure/slow\"-issue raised by Stefan Beller). The current code\n>> always uses the latest protocol supported by the git binary.\n>\n> If we do anything less trivial than \"highest supported by both\" (and\n> I suspect we want to in the final production version), I'd prefer\n> the configuration to list versions one side supports in decreasing\n> order of preference (e.g. \"v3 v0 v2\"), and take the earliest from\n> this list that both sides know how to talk, so that we can skip\n> insecure versions altogether by omitting, and we can express that we\n> would rather avoid talking expensive versions unless there is no\n> other version that is understood by the other side.\n\nI think I've managed to convince myself that a blacklist would be the\nmost future-proof approach, simply because it cannot be overloaded with\nany other meanings in the future.\n\nIf an ordering needs to be possible, that would have to go into another\nconfig item. An ordering would open up for some interesting issues, but\nat least that shouldn't be any worse because the blacklist-approach has\nbeen taken (as opposed to a whitelist-approach).\n\nTo aid with a slow roll-out, the default blacklist could be used (start\nby blacklisting v1), but after that the default list should be empty. It\nshould not be misused for slowly rolling out any later experimental\nversions.\n\nLetting the blacklist be different server- and client-side seems useful\nfor driving the experiment forwards. Post-experiment, I'm not so sure,\nthat just seems unnecessarily complicated.\n\nSo, here's a suggestion:\n\n* experimental.{client,server}ProtocolV1 is \"0\" (don't experiment) or\n \"1\" (experiment).\n\n* experimental.serverProtocolV1 has default \"0\". Unless early feedback\n is negative, the default is changed to \"1\".\n\n* experimental.clientProtocolV1 has default \"0\". Switch the default to\n \"1\" after some time.\n\n* Big warnings that if someone finds themselves switching to \"0\" they\n should get in touch.\n\nOnce we feel confident, we implement protocol.blacklist and let the\ndefault be \"\". The experimental.* are simply dropped, no \"aliasing\" or\n\"transitioning\". That is, we activate v0 and v1. We don't respect \"0\" in\na blacklist (but don't forbid it either). Once we introduce v2, sure,\nbut until then, some will just be tempted to blacklist v0 \"to get the\nmodern v1\" -- they will have risk, but no benefits.\n\nMartin\n"},{"id":"330125","messageId":"20171010111453.21d858dc38d8bfe6590c3579@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-2-bmwill@google.com","subject":"Re: [PATCH v3 01/10] connect: in ref advertisement, shallows are last","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-10T18:14:53Z","receivedAt":"2017-10-10T18:15:01Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Oct 2017 13:14:58 -0700\nBrandon Williams wrote:\n\n> +static int process_dummy_ref(void)\n> +{\n> +\tstruct object_id oid;\n> +\tconst char *name;\n> +\n> +\tif (parse_oid_hex(packet_buffer, &oid, &name))\n> +\t\treturn 0;\n> +\tif (*name != ' ')\n> +\t\treturn 0;\n> +\tname++;\n> +\n> +\treturn !oidcmp(&null_oid, &oid) && !strcmp(name, \"capabilities^{}\");\n> +}\n\nNit: If you're planning to parse_oid_hex, you can strcmp with\n\" capabilities^{}\" directly (note the space at the start) instead of\nfirst checking for ' ' then \"capabilities^{}\".\n"},{"id":"330126","messageId":"20171010111542.1db0ccdf8899a9f501f4e1ca@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-3-bmwill@google.com","subject":"Re: [PATCH v3 02/10] pkt-line: add packet_write function","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-10T18:15:42Z","receivedAt":"2017-10-10T18:15:49Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Oct 2017 13:14:59 -0700\nBrandon Williams wrote:\n\n> +void packet_write(const int fd_out, const char *buf, size_t size)\n\nNo need for \"const\" in \"const int fd_out\", I think. Same comment for the\nheader file.\n\n> +{\n> +\tif (packet_write_gently(fd_out, buf, size))\n> +\t\tdie_errno(\"packet write failed\");\n> +}\n"},{"id":"330127","messageId":"20171010112416.107a7325cef61cd2425d6204@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-5-bmwill@google.com","subject":"Re: [PATCH v3 04/10] daemon: recognize hidden request arguments","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-10T18:24:16Z","receivedAt":"2017-10-10T18:24:22Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Oct 2017 13:15:01 -0700\nBrandon Williams wrote:\n\n> /*\n> * Read the host as supplied by the client connection.\n\nThe return value is probably worth documenting. Something like \"Returns\na pointer to the character *after* the NUL byte terminating the host\nargument, or extra_args if there is no host argument.\"\n\n> */\n> -static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> +static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n\n[snip]\n\n> +static void parse_extra_args(struct hostinfo *hi, struct argv_array *env,\n> +\t\t\t char *extra_args, int buflen)\n> +{\n> +\tconst char *end = extra_args + buflen;\n> +\tstruct strbuf git_protocol = STRBUF_INIT;\n> +\n> +\t/* First look for the host argument */\n> +\textra_args = parse_host_arg(hi, extra_args, buflen);\n\nThis works, but is a bit loose in what it accepts. I think it's better\nto be tighter - in particular, if there is no host argument, we\nshouldn't be looking for extra args.\n\n> +\n> +\t/* Look for additional arguments places after a second NUL byte */\n> +\tfor (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n\nAssuming that the host argument exists, this for loop should start at\nextra_args + 1 to skip the second NUL byte. This currently works\nbecause this code is lenient towards empty strings.\n\n> +\t\tconst char *arg = extra_args;\n> +\n> +\t\t/*\n> +\t\t * Parse the extra arguments, adding most to 'git_protocol'\n> +\t\t * which will be used to set the 'GIT_PROTOCOL' envvar in the\n> +\t\t * service that will be run.\n> +\t\t *\n> +\t\t * If there ends up being a particular arg in the future that\n> +\t\t * git-daemon needs to parse specificly (like the 'host' arg)\n> +\t\t * then it can be parsed here and not added to 'git_protocol'.\n> +\t\t */\n> +\t\tif (*arg) {\n> +\t\t\tif (git_protocol.len > 0)\n> +\t\t\t\tstrbuf_addch(&git_protocol, ':');\n> +\t\t\tstrbuf_addstr(&git_protocol, arg);\n> +\t\t}\n> +\t}\n\nBut I think we shouldn't be lenient towards empty strings.\n\n> +\n> +\tif (git_protocol.len > 0)\n> +\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n> +\t\t\t\t git_protocol.buf);\n> +\tstrbuf_release(&git_protocol);\n> }\n"},{"id":"330128","messageId":"20171010112818.bd75e8dbe4ae39c90e6512da@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-6-bmwill@google.com","subject":"Re: [PATCH v3 05/10] upload-pack, receive-pack: introduce protocol version 1","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-10T18:28:18Z","receivedAt":"2017-10-10T18:28:25Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Oct 2017 13:15:02 -0700\nBrandon Williams wrote:\n\n> +\tswitch (determine_protocol_version_server()) {\n> +\tcase protocol_v1:\n> +\t\tif (advertise_refs || !stateless_rpc)\n> +\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n> +\t\t/*\n> +\t\t * v1 is just the original protocol with a version string,\n> +\t\t * so just fall through after writing the version string.\n> +\t\t */\n\nPeff sent out at least one patch [1] that reformats fallthrough comments\nto be understood by GCC. It's probably worth doing here too.\n\nIn this case, I would do the 2-comment system that Peff suggested:\n\n\t/*\n\t * v1 is just the original protocol with a version string,\n\t * so just fall through after writing the version string.\n\t */\n\tif (advertise_refs || !stateless_rpc)\n\t\tpacket_write_fmt(1, \"version 1\\n\");\n\t/* fallthrough */\n\n(I put the first comment before the code, so it doesn't look so weird.)\n\n[1] https://public-inbox.org/git/20170921062541.ew67gyvrmb2ot4sf@sigill.intra.peff.net/\n\n> +\tcase protocol_v0:\n> +\t\tbreak;\n> +\tdefault:\n> +\t\tBUG(\"unknown protocol version\");\n> +\t}\n"},{"id":"330129","messageId":"20171010113021.0a23506a5d1ebaf4e8ff1f9a@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-8-bmwill@google.com","subject":"Re: [PATCH v3 07/10] connect: tell server that the client understands v1","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-10T18:30:21Z","receivedAt":"2017-10-10T18:30:29Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Oct 2017 13:15:04 -0700\nBrandon Williams wrote:\n\n> 2. ssh://, file://\n> Set 'GIT_PROTOCOL' environment variable with the desired protocol\n> version. With the file:// transport, 'GIT_PROTOCOL' can be set\n> explicitly in the locally running git-upload-pack or git-receive-pack\n> processes. With the ssh:// transport and OpenSSH compliant ssh\n> programs, 'GIT_PROTOCOL' can be sent across ssh by using '-o\n> SendEnv=GIT_PROTOCOL' and having the server whitelist this\n> environment variable.\n\nIn your commit message, also mention what GIT_PROTOCOL contains\n(version=?). (From this commit message alone, I would have expected a\nlone integer, but that is not the case.)\n\nSame comment for the commit message of PATCH v3 08/10.\n"},{"id":"330134","messageId":"20171010193956.168385-1-jonathantanmy@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH] Documentation: document Extra Parameters","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-10T19:39:56Z","receivedAt":"2017-10-10T19:40:10Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"Document the server support for Extra Parameters, additional information\nthat the client can send in its first message to the server during a\nGit client-server interaction.\n\nSigned-off-by: Jonathan Tan \n---\nI noticed that Documentation/technical was not updated in this patch\nseries. It probably should, and here is a suggestion on how to do it.\n\nAlso, I'm not sure what to call the extra sendable information. I'm open\nto suggestions for a better name than Extra Parameters.\n---\n Documentation/technical/http-protocol.txt | 8 ++++++\n Documentation/technical/pack-protocol.txt | 43 ++++++++++++++++++++++++++-----\n 2 files changed, 44 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt\nindex 1c561bdd9..a0e45f288 100644\n--- a/Documentation/technical/http-protocol.txt\n+++ b/Documentation/technical/http-protocol.txt\n@@ -219,6 +219,10 @@ smart server reply:\n S: 003c2cb58b79488a98d2721cea644875a8dd0026b115 refs/tags/v1.0\\n\n S: 003fa3c2e2402b99163d1d59756e5f207ae21cccba4c refs/tags/v1.0^{}\\n\n \n+The client may send Extra Parameters (see\n+Documentation/technical/pack-protocol.txt) as a colon-separated string\n+in the Git-Protocol HTTP header.\n+\n Dumb Server Response\n ^^^^^^^^^^^^^^^^^^^^\n Dumb servers MUST respond with the dumb server reply format.\n@@ -269,7 +273,11 @@ the C locale ordering. The stream SHOULD include the default ref\n named `HEAD` as the first ref. The stream MUST include capability\n declarations behind a NUL on the first ref.\n \n+The returned response contains \"version 1\" if \"version=1\" was sent as an\n+Extra Parameter.\n+\n smart_reply = PKT-LINE(\"# service=$servicename\" LF)\n+\t\t *1(\"version 1\")\n \t\t ref_list\n \t\t \"0000\"\n ref_list = empty_list / non_empty_list\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex ed1eae8b8..f9ebfb23e 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -39,6 +39,19 @@ communicates with that invoked process over the SSH connection.\n The file:// transport runs the 'upload-pack' or 'receive-pack'\n process locally and communicates with it over a pipe.\n \n+Extra Parameters\n+----------------\n+\n+The protocol provides a mechanism in which clients can send additional\n+information in its first message to the server. These are called \"Extra\n+Parameters\", and are supported by the Git, SSH, and HTTP protocols.\n+\n+Each Extra Parameter takes the form of `=` or ``.\n+\n+Servers that receive any such Extra Parameters MUST ignore all\n+unrecognized keys. Currently, the only Extra Parameter recognized is\n+\"version=1\".\n+\n Git Transport\n -------------\n \n@@ -46,18 +59,25 @@ The Git transport starts off by sending the command and repository\n on the wire using the pkt-line format, followed by a NUL byte and a\n hostname parameter, terminated by a NUL byte.\n \n- 0032git-upload-pack /project.git\\0host=myserver.com\\0\n+ 0033git-upload-pack /project.git\\0host=myserver.com\\0\n+\n+The transport may send Extra Parameters by adding an additional NUL\n+byte, and then adding one or more NUL-terminated strings:\n+\n+ 003egit-upload-pack /project.git\\0host=myserver.com\\0\\0version=1\\0\n \n --\n- git-proto-request = request-command SP pathname NUL [ host-parameter NUL ]\n+ git-proto-request = request-command SP pathname NUL\n+\t\t [ host-parameter NUL [ NUL extra-parameters ] ]\n request-command = \"git-upload-pack\" / \"git-receive-pack\" /\n \t\t \"git-upload-archive\" ; case sensitive\n pathname = *( %x01-ff ) ; exclude NUL\n host-parameter = \"host=\" hostname [ \":\" port ]\n+ extra-parameters = 1*extra-parameter\n+ extra-parameter = 1*( %x01-ff ) NUL\n --\n \n-Only host-parameter is allowed in the git-proto-request. Clients\n-MUST NOT attempt to send additional parameters. It is used for the\n+host-parameter is used for the\n git-daemon name based virtual hosting. See --interpolated-path\n option to git daemon, with the %H/%CH format characters.\n \n@@ -117,6 +137,12 @@ we execute it without the leading '/'.\n \t\t v\n ssh user@example.com \"git-upload-pack '~alice/project.git'\"\n \n+Depending on the value of the `protocol.version` configuration variable,\n+Git may attempt to send Extra Parameters as a colon-separated string in\n+the GIT_PROTOCOL environment variable. This is done only if\n+the `ssh.variant` configuration variable indicates that the ssh command\n+supports passing environment variables as an argument.\n+\n A few things to remember here:\n \n - The \"command name\" is spelled with dash (e.g. git-upload-pack), but\n@@ -137,11 +163,13 @@ Reference Discovery\n -------------------\n \n When the client initially connects the server will immediately respond\n-with a listing of each reference it has (all branches and tags) along\n+with a version number (if \"version=1\" is sent as an Extra Parameter),\n+and a listing of each reference it has (all branches and tags) along\n with the object name that each reference currently points to.\n \n- $ echo -e -n \"0039git-upload-pack /schacon/gitbook.git\\0host=example.com\\0\" |\n+ $ echo -e -n \"0044git-upload-pack /schacon/gitbook.git\\0host=example.com\\0\\0version=1\\0\" |\n nc -v example.com 9418\n+ 000aversion 1\n 00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\\0multi_ack thin-pack\n \t\tside-band side-band-64k ofs-delta shallow no-progress include-tag\n 00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration\n@@ -165,7 +193,8 @@ immediately after the ref itself, if presented. A conforming server\n MUST peel the ref if it's an annotated tag.\n \n ----\n- advertised-refs = (no-refs / list-of-refs)\n+ advertised-refs = *1(\"version 1\")\n+\t\t (no-refs / list-of-refs)\n \t\t *shallow\n \t\t flush-pkt\n \n-- \n2.14.2.920.gcf0c67979c-goog\n\n"},{"id":"330136","messageId":"20171010210049.GA77383@google.com","threadId":"46746","inReplyTo":"CAN0heSrWekXdzqzhL3Lk37YOGPHUdKCDU85t3hJgqK7PSVBZ_g@mail.gmail.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-10T21:00:49Z","receivedAt":"2017-10-10T21:01:01Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/06, Martin Ågren wrote:\n> On 6 October 2017 at 11:40, Junio C Hamano wrote:\n> > Simon Ruderich writes:\n> >\n> >> Did you consider Stefan Beller's suggestion regarding a\n> >> (white)list of allowed versions?\n> >>\n> >> On Mon, Sep 18, 2017 at 01:06:59PM -0700, Stefan Beller wrote:\n> >>> Thinking about this, how about:\n> >>>\n> >>> If not configured, we do as we want. (i.e. Git has full control over\n> >>> it's decision making process, which for now is \"favor v0 over v1 as\n> >>> we are experimenting with v1\". This strategy may change in the future\n> >>> to \"prefer highest version number that both client and server can speak\".)\n> >>>\n> >>> If it is configured, \"use highest configured number from the given set\".\n> >>>\n> >>> ?\n> >>\n> >> It would also allow the server operator to configure only a\n> >> specific set of versions (to handle the \"version x is\n> >> insecure/slow\"-issue raised by Stefan Beller). The current code\n> >> always uses the latest protocol supported by the git binary.\n> >\n> > If we do anything less trivial than \"highest supported by both\" (and\n> > I suspect we want to in the final production version), I'd prefer\n> > the configuration to list versions one side supports in decreasing\n> > order of preference (e.g. \"v3 v0 v2\"), and take the earliest from\n> > this list that both sides know how to talk, so that we can skip\n> > insecure versions altogether by omitting, and we can express that we\n> > would rather avoid talking expensive versions unless there is no\n> > other version that is understood by the other side.\n> \n> Maybe I'm missing something Git-specific, but isn't the only thing that\n> needs to be done now, to document/specify that 1) the client should send\n> its list ordered by preference, 2) how preference is signalled, and 3)\n> that the server gets to choose?\n> \n> Why would a server operator with only v0 and v1 at their disposal want\n> to choose v0 instead of v1, considering that -- as far as I understand\n> -- they are in fact the same?\n> \n> Different server implementations and different server configurations\n> will be able to apply whatever rules they want in order to decide which\n> version to use. (It's not like the client can verify the choice that the\n> server makes.) And Brandon's \"pick the highest number\" will do for now.\n> \n> There are many possible rules that the server could apply and they\n> shouldn't affect other servers or what the client does. For example, the\n> server can go \"You seem to know lots of versions, including X and Y.\n> Those are the two that I really prefer, but between those two I'm not\n> picky, so I'll use whichever of X and Y that you seem to prefer.\" Unless\n> I've missed something, we'll never need to implement -- nor specify --\n> anything like that before we have learned both v2 and v3.\n> \n> I guess my thinking is, there's a difference between the protocol and\n> the implementation.\n\nI've been busy the last week or so and I probably wont get much more\ntime to go into more detail on this until the end of the week, so sorry\nfor not being super active on this thread in the past couple of days.\n\nOne of the key things about this transition is ensuring that we get it\nright, because if we get it wrong and find out years later we'll have\nto live with it. So I'm excited and happy that people are taking a\nclose look at this.\n\nThat being said I don't think we need to worry too much about how one\nversion of the protocol is selected over another. I fully expect this\nto change based on many different factors. Maybe one particular server\nimplementation favors v4 over v5, or another serve has no preference at\nall. We may learn something later on, based on security or other\nreasons, that we want to prefer one version or another. Because of that\n(and because I'm hoping that once we have a v2 built that we don't have\nto move to another protocol version any time soon) I think it would be a\nmistake to hard-code or design in inherent preferences that a client\nexpresses that servers are 'required' to respect.\n\nSo I agree with Martin here that if more complicated use cases arise we\ncan design in a preference system for them at a later time.\n\nGiven some of this discussion though, maybe we want to change the\nsemantics of 'protocol.version' so that both servers and clients respect\nit. As of right now, these patches have the server always allow\nprotocol v0 and v1? Though that doesn't do much right now since v1 is\nthe same as v0.\n\nOne other considerations that I should probably handle is that a client\ndoesn't do any verification right now to ensure that the protocol\nversion the server selects was indeed the protocol that the client\nrequested. Is that something you think we need to check for?\n\n-- \nBrandon Williams\n"},{"id":"330137","messageId":"20171010211755.GY19555@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171010210049.GA77383@google.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-10T21:17:55Z","receivedAt":"2017-10-10T21:18:05Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n\n> Given some of this discussion though, maybe we want to change the\n> semantics of 'protocol.version' so that both servers and clients respect\n> it. As of right now, these patches have the server always allow\n> protocol v0 and v1? Though that doesn't do much right now since v1 is\n> the same as v0.\n\nI strongly prefer not to do that.\n\nIf we want to make the advertised protocol versions on the server side\nconfigurable, I think it should be independent from the configuration\nfor protocol versions to use on the client side. Rationale:\n\n - As a client-side user, I may want to (after reporting the bug, of\n course!) blacklist certain protocol versions to work around server\n bugs. But this should not affect my behavior as a server --- in\n my role as a server, these server-side bugs have no effect on me.\n\n - As a server operator, I may want to (after reporting the bug, of\n course!) blacklist certain protocol versions to work around client\n bugs. But this should not affect my behavior as a client --- in my\n role as a client, these client-side bugs have no effect on me.\n\nMaking the client-side case configurable seems important since Git is\nwidely used in environments where it may not be easy to control the\ndeployed version (so having configuration as an escape hatch is\nimportant).\n\nMaking the server-side case configurable seems less important since\nGit server operators usually have tight control over the deployed Git\nversion and can apply a targetted fix or workaround.\n\n> One other considerations that I should probably handle is that a client\n> doesn't do any verification right now to ensure that the protocol\n> version the server selects was indeed the protocol that the client\n> requested. Is that something you think we need to check for?\n\nDo you mean in tests, or are you referring to something else?\n\nThanks,\nJonathan\n"},{"id":"330138","messageId":"20171010125142.87959e72be8ea5f4abdda0e6@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-4-bmwill@google.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-10T19:51:42Z","receivedAt":"2017-10-10T21:21:03Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Tue, 3 Oct 2017 13:15:00 -0700\nBrandon Williams wrote:\n\n> +enum protocol_version determine_protocol_version_server(void)\n> +{\n> +\tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n> +\tenum protocol_version version = protocol_v0;\n> +\n> +\t/*\n> +\t * Determine which protocol version the client has requested. Since\n> +\t * multiple 'version' keys can be sent by the client, indicating that\n> +\t * the client is okay to speak any of them, select the greatest version\n> +\t * that the client has requested. This is due to the assumption that\n> +\t * the most recent protocol version will be the most state-of-the-art.\n> +\t */\n> +\tif (git_protocol) {\n> +\t\tstruct string_list list = STRING_LIST_INIT_DUP;\n> +\t\tconst struct string_list_item *item;\n> +\t\tstring_list_split(&list, git_protocol, ':', -1);\n> +\n> +\t\tfor_each_string_list_item(item, &list) {\n> +\t\t\tconst char *value;\n> +\t\t\tenum protocol_version v;\n> +\n> +\t\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n\nAfter writing some protocol docs [1], I wonder if this is also too\nlenient. The code should probably die if a lone \"version\" (without the\nequals sign) is given.\n\n[1] https://public-inbox.org/git/20171010193956.168385-1-jonathantanmy@google.com/\n\n> +\t\t\t\tv = parse_protocol_version(value);\n> +\t\t\t\tif (v > version)\n> +\t\t\t\t\tversion = v;\n> +\t\t\t}\n> +\t\t}\n> +\n> +\t\tstring_list_clear(&list, 0);\n> +\t}\n> +\n> +\treturn version;\n> +}\n\nAlso, in your commit title, it is \"extension\", not \"extention\".\n"},{"id":"330139","messageId":"CAGZ79kawV+GZ22D_nCr676iMqFR5-aFE6LNpAVm9yDghudkuhg@mail.gmail.com","threadId":"46746","inReplyTo":"20171010211755.GY19555@aiede.mtv.corp.google.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-10T21:32:29Z","receivedAt":"2017-10-10T21:32:36Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Tue, Oct 10, 2017 at 2:17 PM, Jonathan Nieder wrote:\n> Hi,\n>\n> Brandon Williams wrote:\n>\n>> Given some of this discussion though, maybe we want to change the\n>> semantics of 'protocol.version' so that both servers and clients respect\n>> it.\n\nI have no preference for this, but I agree with Jonathans reasoning.\n\n\n>> As of right now, these patches have the server always allow\n>> protocol v0 and v1? Though that doesn't do much right now since v1 is\n>> the same as v0.\n\nI would strongly prefer if the user configures \"v0 and v1\", being explicit,\nif the client wants to talk either of them.\n\n(v0 is not any more special than any future protocol after all -- from the\nusers perspective, configuring protocol.version)\n\nOn the wire transfer we may want to omit the v0, but for simplicity we could\njust dump the clients config of \"v0 and v1\" onto the wire, and the server\nwould either ignore the \"v0 and\" part (when speaking v1), or ignore it\ncompletely (old server, speaking v0).\n\nGiven this model, we have\n* a strict whitelist clientside,\n* the ordering decision can be deferred until later,\n when we have an actual v2.\n\n>> One other considerations that I should probably handle is that a client\n>> doesn't do any verification right now to ensure that the protocol\n>> version the server selects was indeed the protocol that the client\n>> requested. Is that something you think we need to check for?\n\nYes, we would want to see if the protocol version matches the configured\nwhite list. (\"Once we have a v2, I would no longer want a server talking\nv0,v1 to me, because I consider v0 insecure[1], and I personally want\nto rather have no communication than a v0 protocol. After all I configured\n'protocol.version = v2' only\")\n\n[1] e.g. https://public-inbox.org/git/1477690790.2904.22.camel@mattmccutchen.net/\n\n> Do you mean in tests, or are you referring to something else?\n\nA test would be lovely.\n\nThanks,\nStefan\n"},{"id":"330143","messageId":"xmqqvajmo6vq.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171010211755.GY19555@aiede.mtv.corp.google.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-11T00:39:05Z","receivedAt":"2017-10-11T00:39:20Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> If we want to make the advertised protocol versions on the server side\n> configurable, I think it should be independent from the configuration\n> for protocol versions to use on the client side. Rationale:\n>\n> - As a client-side user, I may want to (after reporting the bug, of\n> course!) blacklist certain protocol versions to work around server\n> bugs. But this should not affect my behavior as a server --- in\n> my role as a server, these server-side bugs have no effect on me.\n>\n> - As a server operator, I may want to (after reporting the bug, of\n> course!) blacklist certain protocol versions to work around client\n> bugs. But this should not affect my behavior as a client --- in my\n> role as a client, these client-side bugs have no effect on me.\n\nGood point. \n\n> Making the client-side case configurable seems important since Git is\n> widely used in environments where it may not be easy to control the\n> deployed version (so having configuration as an escape hatch is\n> important).\n>\n> Making the server-side case configurable seems less important since\n> Git server operators usually have tight control over the deployed Git\n> version and can apply a targetted fix or workaround.\n\nThe above also suggests that the configuration variable that lets\nyou specify the protocol version should hint in its name which\ndirection of the protocol it controls. Even if we decide that we'd\nadd only the variable for the initiator side for now, if it is\npossible that we later may want to add another for the acceptor\nside, we need to do it right from day one.\n\nPerhaps protocol.connectVersion(s) vs protocol.acceptVersion(s) or\nsomething like that?\n"},{"id":"330381","messageId":"20171013220405.GA101395@google.com","threadId":"46746","inReplyTo":"20171010112416.107a7325cef61cd2425d6204@google.com","subject":"Re: [PATCH v3 04/10] daemon: recognize hidden request arguments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-13T22:04:05Z","receivedAt":"2017-10-13T22:04:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/10, Jonathan Tan wrote:\n> On Tue, 3 Oct 2017 13:15:01 -0700\n> Brandon Williams wrote:\n> \n> > /*\n> > * Read the host as supplied by the client connection.\n> \n> The return value is probably worth documenting. Something like \"Returns\n> a pointer to the character *after* the NUL byte terminating the host\n> argument, or extra_args if there is no host argument.\"\n\nI can add that comment.\n\n> \n> > */\n> > -static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> > +static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n> \n> [snip]\n> \n> > +static void parse_extra_args(struct hostinfo *hi, struct argv_array *env,\n> > +\t\t\t char *extra_args, int buflen)\n> > +{\n> > +\tconst char *end = extra_args + buflen;\n> > +\tstruct strbuf git_protocol = STRBUF_INIT;\n> > +\n> > +\t/* First look for the host argument */\n> > +\textra_args = parse_host_arg(hi, extra_args, buflen);\n> \n> This works, but is a bit loose in what it accepts. I think it's better\n> to be tighter - in particular, if there is no host argument, we\n> shouldn't be looking for extra args.\n\nI disagree, you shouldn't be precluded from using protocol v2 if you\ndon't include a host argument.\n\n> \n> > +\n> > +\t/* Look for additional arguments places after a second NUL byte */\n> > +\tfor (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n> \n> Assuming that the host argument exists, this for loop should start at\n> extra_args + 1 to skip the second NUL byte. This currently works\n> because this code is lenient towards empty strings.\n\nBeing lenient towards empty strings is fine, I don't see any reason why\nwe should disallow them. Also, this code already\nrequires that there be the second NUL byte because if there isn't then\nthe code which parses the host argument would fail out.\n\n> \n> > +\t\tconst char *arg = extra_args;\n> > +\n> > +\t\t/*\n> > +\t\t * Parse the extra arguments, adding most to 'git_protocol'\n> > +\t\t * which will be used to set the 'GIT_PROTOCOL' envvar in the\n> > +\t\t * service that will be run.\n> > +\t\t *\n> > +\t\t * If there ends up being a particular arg in the future that\n> > +\t\t * git-daemon needs to parse specificly (like the 'host' arg)\n> > +\t\t * then it can be parsed here and not added to 'git_protocol'.\n> > +\t\t */\n> > +\t\tif (*arg) {\n> > +\t\t\tif (git_protocol.len > 0)\n> > +\t\t\t\tstrbuf_addch(&git_protocol, ':');\n> > +\t\t\tstrbuf_addstr(&git_protocol, arg);\n> > +\t\t}\n> > +\t}\n> \n> But I think we shouldn't be lenient towards empty strings.\n\nWhy not? I see no issue with allowing them. In fact if we error out we\ncould be painting ourselves into a corner much like how we did with the\nhost parsing logic.\n\n> \n> > +\n> > +\tif (git_protocol.len > 0)\n> > +\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n> > +\t\t\t\t git_protocol.buf);\n> > +\tstrbuf_release(&git_protocol);\n> > }\n\n-- \nBrandon Williams\n"},{"id":"330382","messageId":"20171013221846.GB101395@google.com","threadId":"46746","inReplyTo":"20171010112818.bd75e8dbe4ae39c90e6512da@google.com","subject":"Re: [PATCH v3 05/10] upload-pack, receive-pack: introduce protocol version 1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-13T22:18:46Z","receivedAt":"2017-10-13T22:18:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/10, Jonathan Tan wrote:\n> On Tue, 3 Oct 2017 13:15:02 -0700\n> Brandon Williams wrote:\n> \n> > +\tswitch (determine_protocol_version_server()) {\n> > +\tcase protocol_v1:\n> > +\t\tif (advertise_refs || !stateless_rpc)\n> > +\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n> > +\t\t/*\n> > +\t\t * v1 is just the original protocol with a version string,\n> > +\t\t * so just fall through after writing the version string.\n> > +\t\t */\n> \n> Peff sent out at least one patch [1] that reformats fallthrough comments\n> to be understood by GCC. It's probably worth doing here too.\n> \n> In this case, I would do the 2-comment system that Peff suggested:\n> \n> \t/*\n> \t * v1 is just the original protocol with a version string,\n> \t * so just fall through after writing the version string.\n> \t */\n> \tif (advertise_refs || !stateless_rpc)\n> \t\tpacket_write_fmt(1, \"version 1\\n\");\n> \t/* fallthrough */\n> \n> (I put the first comment before the code, so it doesn't look so weird.)\n\nSounds good.\n\n> \n> [1] https://public-inbox.org/git/20170921062541.ew67gyvrmb2ot4sf@sigill.intra.peff.net/\n> \n> > +\tcase protocol_v0:\n> > +\t\tbreak;\n> > +\tdefault:\n\nI'm also going to change this to from default to\n'protocol_unknown_version' that way we get a compiler error instead of a\nrun-time BUG when introducing a new protocol version number.\n\n> > +\t\tBUG(\"unknown protocol version\");\n> > +\t}\n\n-- \nBrandon Williams\n"},{"id":"330383","messageId":"20171013222601.GC101395@google.com","threadId":"46746","inReplyTo":"20171010193956.168385-1-jonathantanmy@google.com","subject":"Re: [PATCH] Documentation: document Extra Parameters","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-13T22:26:01Z","receivedAt":"2017-10-13T22:26:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/10, Jonathan Tan wrote:\n> Document the server support for Extra Parameters, additional information\n> that the client can send in its first message to the server during a\n> Git client-server interaction.\n> \n> Signed-off-by: Jonathan Tan \n> ---\n> I noticed that Documentation/technical was not updated in this patch\n> series. It probably should, and here is a suggestion on how to do it.\n> \n> Also, I'm not sure what to call the extra sendable information. I'm open\n> to suggestions for a better name than Extra Parameters.\n\nThanks for writing this up.\n\n> ---\n> Documentation/technical/http-protocol.txt | 8 ++++++\n> Documentation/technical/pack-protocol.txt | 43 ++++++++++++++++++++++++++-----\n> 2 files changed, 44 insertions(+), 7 deletions(-)\n> \n> diff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt\n> index 1c561bdd9..a0e45f288 100644\n> --- a/Documentation/technical/http-protocol.txt\n> +++ b/Documentation/technical/http-protocol.txt\n> @@ -219,6 +219,10 @@ smart server reply:\n> S: 003c2cb58b79488a98d2721cea644875a8dd0026b115 refs/tags/v1.0\\n\n> S: 003fa3c2e2402b99163d1d59756e5f207ae21cccba4c refs/tags/v1.0^{}\\n\n> \n> +The client may send Extra Parameters (see\n> +Documentation/technical/pack-protocol.txt) as a colon-separated string\n> +in the Git-Protocol HTTP header.\n> +\n> Dumb Server Response\n> ^^^^^^^^^^^^^^^^^^^^\n> Dumb servers MUST respond with the dumb server reply format.\n> @@ -269,7 +273,11 @@ the C locale ordering. The stream SHOULD include the default ref\n> named `HEAD` as the first ref. The stream MUST include capability\n> declarations behind a NUL on the first ref.\n> \n> +The returned response contains \"version 1\" if \"version=1\" was sent as an\n> +Extra Parameter.\n> +\n> smart_reply = PKT-LINE(\"# service=$servicename\" LF)\n> +\t\t *1(\"version 1\")\n> \t\t ref_list\n> \t\t \"0000\"\n> ref_list = empty_list / non_empty_list\n> diff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\n> index ed1eae8b8..f9ebfb23e 100644\n> --- a/Documentation/technical/pack-protocol.txt\n> +++ b/Documentation/technical/pack-protocol.txt\n> @@ -39,6 +39,19 @@ communicates with that invoked process over the SSH connection.\n> The file:// transport runs the 'upload-pack' or 'receive-pack'\n> process locally and communicates with it over a pipe.\n> \n> +Extra Parameters\n> +----------------\n> +\n> +The protocol provides a mechanism in which clients can send additional\n> +information in its first message to the server. These are called \"Extra\n> +Parameters\", and are supported by the Git, SSH, and HTTP protocols.\n> +\n> +Each Extra Parameter takes the form of `=` or ``.\n> +\n> +Servers that receive any such Extra Parameters MUST ignore all\n> +unrecognized keys. Currently, the only Extra Parameter recognized is\n> +\"version=1\".\n> +\n> Git Transport\n> -------------\n> \n> @@ -46,18 +59,25 @@ The Git transport starts off by sending the command and repository\n> on the wire using the pkt-line format, followed by a NUL byte and a\n> hostname parameter, terminated by a NUL byte.\n> \n> - 0032git-upload-pack /project.git\\0host=myserver.com\\0\n> + 0033git-upload-pack /project.git\\0host=myserver.com\\0\n> +\n> +The transport may send Extra Parameters by adding an additional NUL\n> +byte, and then adding one or more NUL-terminated strings:\n> +\n> + 003egit-upload-pack /project.git\\0host=myserver.com\\0\\0version=1\\0\n> \n> --\n> - git-proto-request = request-command SP pathname NUL [ host-parameter NUL ]\n> + git-proto-request = request-command SP pathname NUL\n> +\t\t [ host-parameter NUL [ NUL extra-parameters ] ]\n\nThis should probably be \"[ host-parameter NUL ] [ NUL extra-parameters ]\"\nbecause we don't want to require sending a host parameter in order to\nsend extra parameters.\n\n> request-command = \"git-upload-pack\" / \"git-receive-pack\" /\n> \t\t \"git-upload-archive\" ; case sensitive\n> pathname = *( %x01-ff ) ; exclude NUL\n> host-parameter = \"host=\" hostname [ \":\" port ]\n> + extra-parameters = 1*extra-parameter\n> + extra-parameter = 1*( %x01-ff ) NUL\n> --\n> \n> -Only host-parameter is allowed in the git-proto-request. Clients\n> -MUST NOT attempt to send additional parameters. It is used for the\n> +host-parameter is used for the\n> git-daemon name based virtual hosting. See --interpolated-path\n> option to git daemon, with the %H/%CH format characters.\n> \n> @@ -117,6 +137,12 @@ we execute it without the leading '/'.\n> \t\t v\n> ssh user@example.com \"git-upload-pack '~alice/project.git'\"\n> \n> +Depending on the value of the `protocol.version` configuration variable,\n> +Git may attempt to send Extra Parameters as a colon-separated string in\n> +the GIT_PROTOCOL environment variable. This is done only if\n> +the `ssh.variant` configuration variable indicates that the ssh command\n> +supports passing environment variables as an argument.\n> +\n> A few things to remember here:\n> \n> - The \"command name\" is spelled with dash (e.g. git-upload-pack), but\n> @@ -137,11 +163,13 @@ Reference Discovery\n> -------------------\n> \n> When the client initially connects the server will immediately respond\n> -with a listing of each reference it has (all branches and tags) along\n> +with a version number (if \"version=1\" is sent as an Extra Parameter),\n> +and a listing of each reference it has (all branches and tags) along\n> with the object name that each reference currently points to.\n> \n> - $ echo -e -n \"0039git-upload-pack /schacon/gitbook.git\\0host=example.com\\0\" |\n> + $ echo -e -n \"0044git-upload-pack /schacon/gitbook.git\\0host=example.com\\0\\0version=1\\0\" |\n> nc -v example.com 9418\n> + 000aversion 1\n> 00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\\0multi_ack thin-pack\n> \t\tside-band side-band-64k ofs-delta shallow no-progress include-tag\n> 00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration\n> @@ -165,7 +193,8 @@ immediately after the ref itself, if presented. A conforming server\n> MUST peel the ref if it's an annotated tag.\n> \n> ----\n> - advertised-refs = (no-refs / list-of-refs)\n> + advertised-refs = *1(\"version 1\")\n> +\t\t (no-refs / list-of-refs)\n> \t\t *shallow\n> \t\t flush-pkt\n> \n> -- \n> 2.14.2.920.gcf0c67979c-goog\n> \n\n-- \nBrandon Williams\n"},{"id":"330384","messageId":"20171013224600.GD101395@google.com","threadId":"46746","inReplyTo":"20171010211755.GY19555@aiede.mtv.corp.google.com","subject":"Re: [PATCH v3 03/10] protocol: introduce protocol extention mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-13T22:46:00Z","receivedAt":"2017-10-13T22:46:08Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/10, Jonathan Nieder wrote:\n> Hi,\n> \n> Brandon Williams wrote:\n> \n> > Given some of this discussion though, maybe we want to change the\n> > semantics of 'protocol.version' so that both servers and clients respect\n> > it. As of right now, these patches have the server always allow\n> > protocol v0 and v1? Though that doesn't do much right now since v1 is\n> > the same as v0.\n> \n> I strongly prefer not to do that.\n> \n> If we want to make the advertised protocol versions on the server side\n> configurable, I think it should be independent from the configuration\n> for protocol versions to use on the client side. Rationale:\n> \n> - As a client-side user, I may want to (after reporting the bug, of\n> course!) blacklist certain protocol versions to work around server\n> bugs. But this should not affect my behavior as a server --- in\n> my role as a server, these server-side bugs have no effect on me.\n> \n> - As a server operator, I may want to (after reporting the bug, of\n> course!) blacklist certain protocol versions to work around client\n> bugs. But this should not affect my behavior as a client --- in my\n> role as a client, these client-side bugs have no effect on me.\n> \n> Making the client-side case configurable seems important since Git is\n> widely used in environments where it may not be easy to control the\n> deployed version (so having configuration as an escape hatch is\n> important).\n> \n> Making the server-side case configurable seems less important since\n> Git server operators usually have tight control over the deployed Git\n> version and can apply a targetted fix or workaround.\n\nThis is fine with me. Realistically, as I mentioned, all of this is\nunimportant at the moment as it doesn't prevent us from moving forward\nwith the transition or with implementing a v2. If we do get to a point\nin the future where we need to explicitly care about blacklisting or\nwhitelisting protocol versions from the config then we can take care of\nthat then. The important thing is that servers won't die if they see\nmultiple 'version=?' entries or unknown values of '?' in GIT_PROTOCOL.\n\n\n-- \nBrandon Williams\n"},{"id":"330385","messageId":"20171013225603.GE101395@google.com","threadId":"46746","inReplyTo":"20171010113021.0a23506a5d1ebaf4e8ff1f9a@google.com","subject":"Re: [PATCH v3 07/10] connect: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-13T22:56:03Z","receivedAt":"2017-10-13T22:56:10Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/10, Jonathan Tan wrote:\n> On Tue, 3 Oct 2017 13:15:04 -0700\n> Brandon Williams wrote:\n> \n> > 2. ssh://, file://\n> > Set 'GIT_PROTOCOL' environment variable with the desired protocol\n> > version. With the file:// transport, 'GIT_PROTOCOL' can be set\n> > explicitly in the locally running git-upload-pack or git-receive-pack\n> > processes. With the ssh:// transport and OpenSSH compliant ssh\n> > programs, 'GIT_PROTOCOL' can be sent across ssh by using '-o\n> > SendEnv=GIT_PROTOCOL' and having the server whitelist this\n> > environment variable.\n> \n> In your commit message, also mention what GIT_PROTOCOL contains\n> (version=?). (From this commit message alone, I would have expected a\n> lone integer, but that is not the case.)\n> \n> Same comment for the commit message of PATCH v3 08/10.\n\nI'll update the commit msgs.\n\n-- \nBrandon Williams\n"},{"id":"330461","messageId":"20171016171812.GA4487@google.com","threadId":"46746","inReplyTo":"20171003214206.GY19555@aiede.mtv.corp.google.com","subject":"Re: [PATCH v3 10/10] ssh: introduce a 'simple' ssh variant","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:18:12Z","receivedAt":"2017-10-16T17:18:20Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/03, Jonathan Nieder wrote:\n> Hi,\n> \n> Brandon Williams wrote:\n> \n> > When using the 'ssh' transport, the '-o' option is used to specify an\n> > environment variable which should be set on the remote end. This allows\n> > git to send additional information when contacting the server,\n> > requesting the use of a different protocol version via the\n> > 'GIT_PROTOCOL' environment variable like so: \"-o SendEnv=GIT_PROTOCOL\"\n> >\n> > Unfortunately not all ssh variants support the sending of environment\n> > variables to the remote end. To account for this, only use the '-o'\n> > option for ssh variants which are OpenSSH compliant. This is done by\n> > checking that the basename of the ssh command is 'ssh' or the ssh\n> > variant is overridden to be 'ssh' (via the ssh.variant config).\n> \n> This also affects -p (port), right?\n\nYeah I'll add a comment in the commit msg indicating that options like\n-p and -4 -6 are are only supported by some variants.\n\n> \n> What happens if I specify a ssh://host:port/path URL and the SSH\n> implementation is of 'simple' type?\n\nThe port would only be sent if your ssh command supported it.\n\n> \n> > Previously if an ssh command's basename wasn't 'plink' or\n> \n> Git's commit messages use the present tense to describe the current\n> state of the code, so this is \"Currently\". :)\n\nI'll fix this :)\n\n> \n> > 'tortoiseplink' git assumed that the command was an OpenSSH variant.\n> > Since user configured ssh commands may not be OpenSSH compliant, tighten\n> > this constraint and assume a variant of 'simple' if the basename of the\n> > command doesn't match the variants known to git. The new ssh variant\n> > 'simple' will only have the host and command to execute ([username@]host\n> > command) passed as parameters to the ssh command.\n> >\n> > Update the Documentation to better reflect the command-line options sent\n> > to ssh commands based on their variant.\n> >\n> > Reported-by: Jeffrey Yasskin \n> > Signed-off-by: Brandon Williams \n> \n> Thanks for working on this.\n> \n> For background, the GIT_SSH implementation that motivated this is\n> https://github.com/travis-ci/dpl/blob/6c3fddfda1f2a85944c544446b068bac0a77c049/lib/dpl/provider.rb#L215,\n> which does not support -p or -4/-6, either.\n> \n> > ---\n> > Documentation/config.txt | 27 ++++++++++--\n> > Documentation/git.txt | 9 ++--\n> > connect.c | 107 ++++++++++++++++++++++++++---------------------\n> > t/t5601-clone.sh | 9 ++--\n> > t/t5700-protocol-v1.sh | 2 +\n> > 5 files changed, 95 insertions(+), 59 deletions(-)\n> [...]\n> > --- a/connect.c\n> > +++ b/connect.c\n> > @@ -776,37 +776,44 @@ static const char *get_ssh_command(void)\n> [...]\n> > +static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n> > +\t\t\t\t\t int is_cmdline)\n> [...]\n> > -\tif (!strcasecmp(variant, \"plink\") ||\n> > -\t !strcasecmp(variant, \"plink.exe\"))\n> > -\t\t*port_option = 'P';\n> > +\tif (!strcasecmp(variant, \"ssh\"))\n> > +\t\tssh_variant = VARIANT_SSH;\n> \n> Could this handle ssh.exe, too?\n\nYeah I'll add the additional comparison.\n\n> \n> [...]\n> > --- a/t/t5601-clone.sh\n> > +++ b/t/t5601-clone.sh\n> \n> Can this get tests for the new defaulting behavior? E.g.\n> \n> - default is \"simple\"\n> - how \"simple\" treats an ssh://host:port/path URL\n> - how \"simple\" treats ipv4/ipv6 switching\n> - ssh defaults to \"ssh\"\n> - if GIT_SSH=ssh, can set ssh.variant to \"simple\" to force the \"simple\"\n> mode\n\nI'll look to adding a few more tests.\n\n> \n> One other worry: this (intentionally) changes the behavior of a\n> previously-working GIT_SSH=ssh-wrapper that wants to support\n> OpenSSH-style options but does not declare ssh.variant=ssh. When\n> discovering this change, what should the author of such an ssh-wrapper\n> do?\n> \n> They could instruct their users to set ssh.variant or GIT_SSH_VARIANT\n> to \"ssh\", but then they are at the mercy of future additional options\n> supported by OpenSSH we may want to start using in the future (e.g.,\n> maybe we will start passing \"--\" to separate options from the\n> hostname). So this is not a futureproof option for them.\n> \n> They could take the new default behavior or instruct their users to\n> set ssh.variant or GIT_SSH_VARIANT to \"simple\", but then they lose\n> support for handling alternate ports, ipv4/ipv6, and specifying -o\n> SendEnv to propagate GIT_PROTOCOL or other envvars. They can handle\n> GIT_PROTOCOL propagation manually, but losing port support seems like\n> a heavy cost.\n> \n> They could send a patch to define yet another variant that is\n> forward-compatible, for example using an interface similar to what\n> git-credential(1) uses. Then they can set GIT_SSH to their\n> OpenSSH-style helper and GIT_FANCY_NEW_SSH to their more modern\n> helper, so that old Git versions could use GIT_SSH and new Git\n> versions could use GIT_FANCY_NEW_SSH. This might be their best\n> option. It feels odd to say that their only good way forward is to\n> send a patch, but on the other hand someone with such an itch is\n> likely to be in the best position to define an appropriate interface.\n> \n> They could send a documentation patch to make more promises about the\n> commandline used in OpenSSH mode: e.g. setting a rule in advance about\n> which options can take an argument so that they can properly parse an\n> OpenSSH command line in a future-compatible way.\n> \n> Or they could send a patch to allow passing the port in \"simple\"\n> mode, for example using an environment variable.\n> \n> Am I missing another option? What advice do we give to this person?\n> \n> Thanks,\n> Jonathan\n\n-- \nBrandon Williams\n"},{"id":"330463","messageId":"20171016175532.73459-1-bmwill@google.com","threadId":"46746","inReplyTo":"20171003201507.3589-1-bmwill@google.com","subject":"[PATCH v4 00/11] protocol transition","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:21Z","receivedAt":"2017-10-16T17:55:49Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Changes in v4:\n * Added more tests for the new handeling of ssh variants.\n * Removed the 'default' case in upload_pack and receive_pack and instead\n ensured that all enum values were accounted for. This way when a new\n protocol version is introduced the compiler will throw an error if the new\n protocol version isn't accounted for in these switch statements.\n * Added Jonathan's Documentation patch ontop of the series (with the small\n change I pointed out in reply to the patch itself)\n * A few other small changes due to reviewer comments.\n\n\nBrandon Williams (9):\n pkt-line: add packet_write function\n protocol: introduce protocol extension mechanisms\n daemon: recognize hidden request arguments\n upload-pack, receive-pack: introduce protocol version 1\n connect: teach client to recognize v1 server response\n connect: tell server that the client understands v1\n http: tell server that the client understands v1\n i5700: add interop test for protocol transition\n ssh: introduce a 'simple' ssh variant\n\nJonathan Tan (2):\n connect: in ref advertisement, shallows are last\n Documentation: document Extra Parameters\n\n Documentation/config.txt | 44 +++-\n Documentation/git.txt | 15 +-\n Documentation/technical/http-protocol.txt | 8 +\n Documentation/technical/pack-protocol.txt | 43 +++-\n Makefile | 1 +\n builtin/receive-pack.c | 17 ++\n cache.h | 10 +\n connect.c | 354 ++++++++++++++++++++----------\n daemon.c | 71 +++++-\n http.c | 18 ++\n pkt-line.c | 6 +\n pkt-line.h | 1 +\n protocol.c | 79 +++++++\n protocol.h | 33 +++\n t/interop/i5700-protocol-transition.sh | 68 ++++++\n t/lib-httpd/apache.conf | 7 +\n t/t5601-clone.sh | 26 ++-\n t/t5700-protocol-v1.sh | 294 +++++++++++++++++++++++++\n upload-pack.c | 20 +-\n 19 files changed, 967 insertions(+), 148 deletions(-)\n create mode 100644 protocol.c\n create mode 100644 protocol.h\n create mode 100755 t/interop/i5700-protocol-transition.sh\n create mode 100755 t/t5700-protocol-v1.sh\n\n--- interdiff with 'origin/bw/protocol-v1'\n\n\ndiff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt\nindex 1c561bdd9..a0e45f288 100644\n--- a/Documentation/technical/http-protocol.txt\n+++ b/Documentation/technical/http-protocol.txt\n@@ -219,6 +219,10 @@ smart server reply:\n S: 003c2cb58b79488a98d2721cea644875a8dd0026b115 refs/tags/v1.0\\n\n S: 003fa3c2e2402b99163d1d59756e5f207ae21cccba4c refs/tags/v1.0^{}\\n\n \n+The client may send Extra Parameters (see\n+Documentation/technical/pack-protocol.txt) as a colon-separated string\n+in the Git-Protocol HTTP header.\n+\n Dumb Server Response\n ^^^^^^^^^^^^^^^^^^^^\n Dumb servers MUST respond with the dumb server reply format.\n@@ -269,7 +273,11 @@ the C locale ordering. The stream SHOULD include the default ref\n named `HEAD` as the first ref. The stream MUST include capability\n declarations behind a NUL on the first ref.\n \n+The returned response contains \"version 1\" if \"version=1\" was sent as an\n+Extra Parameter.\n+\n smart_reply = PKT-LINE(\"# service=$servicename\" LF)\n+\t\t *1(\"version 1\")\n \t\t ref_list\n \t\t \"0000\"\n ref_list = empty_list / non_empty_list\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex ed1eae8b8..cd31edc91 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -39,6 +39,19 @@ communicates with that invoked process over the SSH connection.\n The file:// transport runs the 'upload-pack' or 'receive-pack'\n process locally and communicates with it over a pipe.\n \n+Extra Parameters\n+----------------\n+\n+The protocol provides a mechanism in which clients can send additional\n+information in its first message to the server. These are called \"Extra\n+Parameters\", and are supported by the Git, SSH, and HTTP protocols.\n+\n+Each Extra Parameter takes the form of `=` or ``.\n+\n+Servers that receive any such Extra Parameters MUST ignore all\n+unrecognized keys. Currently, the only Extra Parameter recognized is\n+\"version=1\".\n+\n Git Transport\n -------------\n \n@@ -46,18 +59,25 @@ The Git transport starts off by sending the command and repository\n on the wire using the pkt-line format, followed by a NUL byte and a\n hostname parameter, terminated by a NUL byte.\n \n- 0032git-upload-pack /project.git\\0host=myserver.com\\0\n+ 0033git-upload-pack /project.git\\0host=myserver.com\\0\n+\n+The transport may send Extra Parameters by adding an additional NUL\n+byte, and then adding one or more NUL-terminated strings:\n+\n+ 003egit-upload-pack /project.git\\0host=myserver.com\\0\\0version=1\\0\n \n --\n- git-proto-request = request-command SP pathname NUL [ host-parameter NUL ]\n+ git-proto-request = request-command SP pathname NUL\n+\t\t [ host-parameter NUL ] [ NUL extra-parameters ]\n request-command = \"git-upload-pack\" / \"git-receive-pack\" /\n \t\t \"git-upload-archive\" ; case sensitive\n pathname = *( %x01-ff ) ; exclude NUL\n host-parameter = \"host=\" hostname [ \":\" port ]\n+ extra-parameters = 1*extra-parameter\n+ extra-parameter = 1*( %x01-ff ) NUL\n --\n \n-Only host-parameter is allowed in the git-proto-request. Clients\n-MUST NOT attempt to send additional parameters. It is used for the\n+host-parameter is used for the\n git-daemon name based virtual hosting. See --interpolated-path\n option to git daemon, with the %H/%CH format characters.\n \n@@ -117,6 +137,12 @@ we execute it without the leading '/'.\n \t\t v\n ssh user@example.com \"git-upload-pack '~alice/project.git'\"\n \n+Depending on the value of the `protocol.version` configuration variable,\n+Git may attempt to send Extra Parameters as a colon-separated string in\n+the GIT_PROTOCOL environment variable. This is done only if\n+the `ssh.variant` configuration variable indicates that the ssh command\n+supports passing environment variables as an argument.\n+\n A few things to remember here:\n \n - The \"command name\" is spelled with dash (e.g. git-upload-pack), but\n@@ -137,11 +163,13 @@ Reference Discovery\n -------------------\n \n When the client initially connects the server will immediately respond\n-with a listing of each reference it has (all branches and tags) along\n+with a version number (if \"version=1\" is sent as an Extra Parameter),\n+and a listing of each reference it has (all branches and tags) along\n with the object name that each reference currently points to.\n \n- $ echo -e -n \"0039git-upload-pack /schacon/gitbook.git\\0host=example.com\\0\" |\n+ $ echo -e -n \"0044git-upload-pack /schacon/gitbook.git\\0host=example.com\\0\\0version=1\\0\" |\n nc -v example.com 9418\n+ 000aversion 1\n 00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\\0multi_ack thin-pack\n \t\tside-band side-band-64k ofs-delta shallow no-progress include-tag\n 00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration\n@@ -165,7 +193,8 @@ immediately after the ref itself, if presented. A conforming server\n MUST peel the ref if it's an annotated tag.\n \n ----\n- advertised-refs = (no-refs / list-of-refs)\n+ advertised-refs = *1(\"version 1\")\n+\t\t (no-refs / list-of-refs)\n \t\t *shallow\n \t\t flush-pkt\n \ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex 94b7d29ea..839c1462d 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -1966,15 +1966,17 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \n \tswitch (determine_protocol_version_server()) {\n \tcase protocol_v1:\n-\t\tif (advertise_refs || !stateless_rpc)\n-\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\n \t\t * so just fall through after writing the version string.\n \t\t */\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\n+\t\t/* fallthrough */\n \tcase protocol_v0:\n \t\tbreak;\n-\tdefault:\n+\tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n \t}\n \ndiff --git a/connect.c b/connect.c\nindex 65cee49b6..7fbd396b3 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -836,7 +836,8 @@ static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n \t\t}\n \t}\n \n-\tif (!strcasecmp(variant, \"ssh\"))\n+\tif (!strcasecmp(variant, \"ssh\") ||\n+\t !strcasecmp(variant, \"ssh.exe\"))\n \t\tssh_variant = VARIANT_SSH;\n \telse if (!strcasecmp(variant, \"plink\") ||\n \t\t !strcasecmp(variant, \"plink.exe\"))\ndiff --git a/daemon.c b/daemon.c\nindex 36cc794c9..e37e343d0 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -582,6 +582,9 @@ static void canonicalize_client(struct strbuf *out, const char *in)\n \n /*\n * Read the host as supplied by the client connection.\n+ *\n+ * Returns a pointer to the character after the NUL byte terminating the host\n+ * arguemnt, or 'extra_args' if there is no host arguemnt.\n */\n static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n {\ndiff --git a/pkt-line.c b/pkt-line.c\nindex c025d0332..7006b3587 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -188,7 +188,7 @@ static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n \treturn 0;\n }\n \n-void packet_write(const int fd_out, const char *buf, size_t size)\n+void packet_write(int fd_out, const char *buf, size_t size)\n {\n \tif (packet_write_gently(fd_out, buf, size))\n \t\tdie_errno(\"packet write failed\");\ndiff --git a/pkt-line.h b/pkt-line.h\nindex d9e9783b1..3dad583e2 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -22,7 +22,7 @@\n void packet_flush(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n-void packet_write(const int fd_out, const char *buf, size_t size);\n+void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex ee1a24c5b..86811a0c3 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -309,21 +309,18 @@ test_expect_success 'clone checking out a tag' '\n setup_ssh_wrapper () {\n \ttest_expect_success 'setup ssh wrapper' '\n \t\tcp \"$GIT_BUILD_DIR/t/helper/test-fake-ssh$X\" \\\n-\t\t\t\"$TRASH_DIRECTORY/ssh-wrapper$X\" &&\n-\t\tGIT_SSH=\"$TRASH_DIRECTORY/ssh-wrapper$X\" &&\n+\t\t\t\"$TRASH_DIRECTORY/ssh$X\" &&\n+\t\tGIT_SSH=\"$TRASH_DIRECTORY/ssh$X\" &&\n \t\texport GIT_SSH &&\n-\t\tGIT_SSH_VARIANT=ssh &&\n-\t\texport GIT_SSH_VARIANT &&\n \t\texport TRASH_DIRECTORY &&\n \t\t>\"$TRASH_DIRECTORY\"/ssh-output\n \t'\n }\n \n copy_ssh_wrapper_as () {\n-\tcp \"$TRASH_DIRECTORY/ssh-wrapper$X\" \"${1%$X}$X\" &&\n+\tcp \"$TRASH_DIRECTORY/ssh$X\" \"${1%$X}$X\" &&\n \tGIT_SSH=\"${1%$X}$X\" &&\n-\texport GIT_SSH &&\n-\tunset GIT_SSH_VARIANT\n+\texport GIT_SSH\n }\n \n expect_ssh () {\n@@ -365,6 +362,22 @@ test_expect_success 'bracketed hostnames are still ssh' '\n \texpect_ssh \"-p 123\" myhost src\n '\n \n+test_expect_success 'OpenSSH variant passes -4' '\n+\tgit clone -4 \"[myhost:123]:src\" ssh-ipv4-clone &&\n+\texpect_ssh \"-4 -p 123\" myhost src\n+'\n+\n+test_expect_success 'variant can be overriden' '\n+\tgit -c ssh.variant=simple clone -4 \"[myhost:123]:src\" ssh-simple-clone &&\n+\texpect_ssh myhost src\n+'\n+\n+test_expect_success 'simple is treated as simple' '\n+\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n+\tgit clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple &&\n+\texpect_ssh myhost src\n+'\n+\n test_expect_success 'uplink is treated as simple' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/uplink\" &&\n \tgit clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\ndiff --git a/upload-pack.c b/upload-pack.c\nindex ef438e9c2..ef99a029c 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -1071,16 +1071,18 @@ int cmd_main(int argc, const char **argv)\n \n \tswitch (determine_protocol_version_server()) {\n \tcase protocol_v1:\n-\t\tif (advertise_refs || !stateless_rpc)\n-\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n \t\t/*\n \t\t * v1 is just the original protocol with a version string,\n \t\t * so just fall through after writing the version string.\n \t\t */\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\n+\t\t/* fallthrough */\n \tcase protocol_v0:\n \t\tupload_pack();\n \t\tbreak;\n-\tdefault:\n+\tcase protocol_unknown_version:\n \t\tBUG(\"unknown protocol version\");\n \t}\n \n\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330464","messageId":"20171016175532.73459-2-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 01/11] connect: in ref advertisement, shallows are last","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:22Z","receivedAt":"2017-10-16T17:55:52Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"From: Jonathan Tan \n\nCurrently, get_remote_heads() parses the ref advertisement in one loop,\nallowing refs and shallow lines to intersperse, despite this not being\nallowed by the specification. Refactor get_remote_heads() to use two\nloops instead, enforcing that refs come first, and then shallows.\n\nThis also makes it easier to teach get_remote_heads() to interpret other\nlines in the ref advertisement, which will be done in a subsequent\npatch.\n\nAs part of this change, this patch interprets capabilities only on the\nfirst line in the ref advertisement, printing a warning message when\nencountering capabilities on other lines.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Brandon Williams \n---\n connect.c | 189 ++++++++++++++++++++++++++++++++++++++++----------------------\n 1 file changed, 123 insertions(+), 66 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex df56c0cbf..8e2e276b6 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -11,6 +11,7 @@\n #include \"string-list.h\"\n #include \"sha1-array.h\"\n #include \"transport.h\"\n+#include \"strbuf.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -107,6 +108,104 @@ static void annotate_refs_with_symref_info(struct ref *ref)\n \tstring_list_clear(&symref, 0);\n }\n \n+/*\n+ * Read one line of a server's ref advertisement into packet_buffer.\n+ */\n+static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n+\t\t\t int *responded)\n+{\n+\tint len = packet_read(in, src_buf, src_len,\n+\t\t\t packet_buffer, sizeof(packet_buffer),\n+\t\t\t PACKET_READ_GENTLE_ON_EOF |\n+\t\t\t PACKET_READ_CHOMP_NEWLINE);\n+\tconst char *arg;\n+\tif (len < 0)\n+\t\tdie_initial_contact(*responded);\n+\tif (len > 4 && skip_prefix(packet_buffer, \"ERR \", &arg))\n+\t\tdie(\"remote error: %s\", arg);\n+\n+\t*responded = 1;\n+\n+\treturn len;\n+}\n+\n+#define EXPECTING_FIRST_REF 0\n+#define EXPECTING_REF 1\n+#define EXPECTING_SHALLOW 2\n+\n+static void process_capabilities(int *len)\n+{\n+\tint nul_location = strlen(packet_buffer);\n+\tif (nul_location == *len)\n+\t\treturn;\n+\tserver_capabilities = xstrdup(packet_buffer + nul_location + 1);\n+\t*len = nul_location;\n+}\n+\n+static int process_dummy_ref(void)\n+{\n+\tstruct object_id oid;\n+\tconst char *name;\n+\n+\tif (parse_oid_hex(packet_buffer, &oid, &name))\n+\t\treturn 0;\n+\tif (*name != ' ')\n+\t\treturn 0;\n+\tname++;\n+\n+\treturn !oidcmp(&null_oid, &oid) && !strcmp(name, \"capabilities^{}\");\n+}\n+\n+static void check_no_capabilities(int len)\n+{\n+\tif (strlen(packet_buffer) != len)\n+\t\twarning(\"Ignoring capabilities after first line '%s'\",\n+\t\t\tpacket_buffer + strlen(packet_buffer));\n+}\n+\n+static int process_ref(int len, struct ref ***list, unsigned int flags,\n+\t\t struct oid_array *extra_have)\n+{\n+\tstruct object_id old_oid;\n+\tconst char *name;\n+\n+\tif (parse_oid_hex(packet_buffer, &old_oid, &name))\n+\t\treturn 0;\n+\tif (*name != ' ')\n+\t\treturn 0;\n+\tname++;\n+\n+\tif (extra_have && !strcmp(name, \".have\")) {\n+\t\toid_array_append(extra_have, &old_oid);\n+\t} else if (!strcmp(name, \"capabilities^{}\")) {\n+\t\tdie(\"protocol error: unexpected capabilities^{}\");\n+\t} else if (check_ref(name, flags)) {\n+\t\tstruct ref *ref = alloc_ref(name);\n+\t\toidcpy(&ref->old_oid, &old_oid);\n+\t\t**list = ref;\n+\t\t*list = &ref->next;\n+\t}\n+\tcheck_no_capabilities(len);\n+\treturn 1;\n+}\n+\n+static int process_shallow(int len, struct oid_array *shallow_points)\n+{\n+\tconst char *arg;\n+\tstruct object_id old_oid;\n+\n+\tif (!skip_prefix(packet_buffer, \"shallow \", &arg))\n+\t\treturn 0;\n+\n+\tif (get_oid_hex(arg, &old_oid))\n+\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n+\tif (!shallow_points)\n+\t\tdie(\"repository on the other end cannot be shallow\");\n+\toid_array_append(shallow_points, &old_oid);\n+\tcheck_no_capabilities(len);\n+\treturn 1;\n+}\n+\n /*\n * Read all the refs from the other end\n */\n@@ -123,76 +222,34 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t * willing to talk to us. A hang-up before seeing any\n \t * response does not necessarily mean an ACL problem, though.\n \t */\n-\tint saw_response;\n-\tint got_dummy_ref_with_capabilities_declaration = 0;\n+\tint responded = 0;\n+\tint len;\n+\tint state = EXPECTING_FIRST_REF;\n \n \t*list = NULL;\n-\tfor (saw_response = 0; ; saw_response = 1) {\n-\t\tstruct ref *ref;\n-\t\tstruct object_id old_oid;\n-\t\tchar *name;\n-\t\tint len, name_len;\n-\t\tchar *buffer = packet_buffer;\n-\t\tconst char *arg;\n-\n-\t\tlen = packet_read(in, &src_buf, &src_len,\n-\t\t\t\t packet_buffer, sizeof(packet_buffer),\n-\t\t\t\t PACKET_READ_GENTLE_ON_EOF |\n-\t\t\t\t PACKET_READ_CHOMP_NEWLINE);\n-\t\tif (len < 0)\n-\t\t\tdie_initial_contact(saw_response);\n-\n-\t\tif (!len)\n-\t\t\tbreak;\n-\n-\t\tif (len > 4 && skip_prefix(buffer, \"ERR \", &arg))\n-\t\t\tdie(\"remote error: %s\", arg);\n-\n-\t\tif (len == GIT_SHA1_HEXSZ + strlen(\"shallow \") &&\n-\t\t\tskip_prefix(buffer, \"shallow \", &arg)) {\n-\t\t\tif (get_oid_hex(arg, &old_oid))\n-\t\t\t\tdie(\"protocol error: expected shallow sha-1, got '%s'\", arg);\n-\t\t\tif (!shallow_points)\n-\t\t\t\tdie(\"repository on the other end cannot be shallow\");\n-\t\t\toid_array_append(shallow_points, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n \n-\t\tif (len < GIT_SHA1_HEXSZ + 2 || get_oid_hex(buffer, &old_oid) ||\n-\t\t\tbuffer[GIT_SHA1_HEXSZ] != ' ')\n-\t\t\tdie(\"protocol error: expected sha/ref, got '%s'\", buffer);\n-\t\tname = buffer + GIT_SHA1_HEXSZ + 1;\n-\n-\t\tname_len = strlen(name);\n-\t\tif (len != name_len + GIT_SHA1_HEXSZ + 1) {\n-\t\t\tfree(server_capabilities);\n-\t\t\tserver_capabilities = xstrdup(name + name_len + 1);\n-\t\t}\n-\n-\t\tif (extra_have && !strcmp(name, \".have\")) {\n-\t\t\toid_array_append(extra_have, &old_oid);\n-\t\t\tcontinue;\n-\t\t}\n-\n-\t\tif (!strcmp(name, \"capabilities^{}\")) {\n-\t\t\tif (saw_response)\n-\t\t\t\tdie(\"protocol error: unexpected capabilities^{}\");\n-\t\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\t\tdie(\"protocol error: multiple capabilities^{}\");\n-\t\t\tgot_dummy_ref_with_capabilities_declaration = 1;\n-\t\t\tcontinue;\n+\twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n+\t\tswitch (state) {\n+\t\tcase EXPECTING_FIRST_REF:\n+\t\t\tprocess_capabilities(&len);\n+\t\t\tif (process_dummy_ref()) {\n+\t\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tstate = EXPECTING_REF;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_REF:\n+\t\t\tif (process_ref(len, &list, flags, extra_have))\n+\t\t\t\tbreak;\n+\t\t\tstate = EXPECTING_SHALLOW;\n+\t\t\t/* fallthrough */\n+\t\tcase EXPECTING_SHALLOW:\n+\t\t\tif (process_shallow(len, shallow_points))\n+\t\t\t\tbreak;\n+\t\t\tdie(\"protocol error: unexpected '%s'\", packet_buffer);\n+\t\tdefault:\n+\t\t\tdie(\"unexpected state %d\", state);\n \t\t}\n-\n-\t\tif (!check_ref(name, flags))\n-\t\t\tcontinue;\n-\n-\t\tif (got_dummy_ref_with_capabilities_declaration)\n-\t\t\tdie(\"protocol error: unexpected ref after capabilities^{}\");\n-\n-\t\tref = alloc_ref(buffer + GIT_SHA1_HEXSZ + 1);\n-\t\toidcpy(&ref->old_oid, &old_oid);\n-\t\t*list = ref;\n-\t\tlist = &ref->next;\n \t}\n \n \tannotate_refs_with_symref_info(*orig_list);\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330465","messageId":"20171016175532.73459-3-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 02/11] pkt-line: add packet_write function","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:23Z","receivedAt":"2017-10-16T17:55:54Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Add a function which can be used to write the contents of an arbitrary\nbuffer. This makes it easy to build up data in a buffer before writing\nthe packet instead of formatting the entire contents of the packet using\n'packet_write_fmt()'.\n\nSigned-off-by: Brandon Williams \n---\n pkt-line.c | 6 ++++++\n pkt-line.h | 1 +\n 2 files changed, 7 insertions(+)\n\ndiff --git a/pkt-line.c b/pkt-line.c\nindex 647bbd3bc..7006b3587 100644\n--- a/pkt-line.c\n+++ b/pkt-line.c\n@@ -188,6 +188,12 @@ static int packet_write_gently(const int fd_out, const char *buf, size_t size)\n \treturn 0;\n }\n \n+void packet_write(int fd_out, const char *buf, size_t size)\n+{\n+\tif (packet_write_gently(fd_out, buf, size))\n+\t\tdie_errno(\"packet write failed\");\n+}\n+\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...)\n {\n \tva_list args;\ndiff --git a/pkt-line.h b/pkt-line.h\nindex 66ef610fc..3dad583e2 100644\n--- a/pkt-line.h\n+++ b/pkt-line.h\n@@ -22,6 +22,7 @@\n void packet_flush(int fd);\n void packet_write_fmt(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n void packet_buf_flush(struct strbuf *buf);\n+void packet_write(int fd_out, const char *buf, size_t size);\n void packet_buf_write(struct strbuf *buf, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n int packet_flush_gently(int fd);\n int packet_write_fmt_gently(int fd, const char *fmt, ...) __attribute__((format (printf, 2, 3)));\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330466","messageId":"20171016175532.73459-4-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 03/11] protocol: introduce protocol extension mechanisms","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:24Z","receivedAt":"2017-10-16T17:55:56Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Create protocol.{c,h} and provide functions which future servers and\nclients can use to determine which protocol to use or is being used.\n\nAlso introduce the 'GIT_PROTOCOL' environment variable which will be\nused to communicate a colon separated list of keys with optional values\nto a server. Unknown keys and values must be tolerated. This mechanism\nis used to communicate which version of the wire protocol a client would\nlike to use with a server.\n\nSigned-off-by: Brandon Williams \n---\n Documentation/config.txt | 17 +++++++++++\n Documentation/git.txt | 6 ++++\n Makefile | 1 +\n cache.h | 8 +++++\n protocol.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++++\n protocol.h | 33 ++++++++++++++++++++\n 6 files changed, 144 insertions(+)\n create mode 100644 protocol.c\n create mode 100644 protocol.h\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex dc4e3f58a..b78747abc 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2517,6 +2517,23 @@ The protocol names currently used by git are:\n `hg` to allow the `git-remote-hg` helper)\n --\n \n+protocol.version::\n+\tExperimental. If set, clients will attempt to communicate with a\n+\tserver using the specified protocol version. If unset, no\n+\tattempt will be made by the client to communicate using a\n+\tparticular protocol version, this results in protocol version 0\n+\tbeing used.\n+\tSupported versions:\n++\n+--\n+\n+* `0` - the original wire protocol.\n+\n+* `1` - the original wire protocol with the addition of a version string\n+ in the initial response from the server.\n+\n+--\n+\n pull.ff::\n \tBy default, Git does not create an extra merge commit when merging\n \ta commit that is a descendant of the current commit. Instead, the\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 6e3a6767e..7518ea3af 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -697,6 +697,12 @@ of clones and fetches.\n \twhich feed potentially-untrusted URLS to git commands. See\n \tlinkgit:git-config[1] for more details.\n \n+`GIT_PROTOCOL`::\n+\tFor internal use only. Used in handshaking the wire protocol.\n+\tContains a colon ':' separated list of keys with optional values\n+\t'key[=value]'. Presence of unknown keys and values must be\n+\tignored.\n+\n Discussion[[Discussion]]\n ------------------------\n \ndiff --git a/Makefile b/Makefile\nindex ed4ca438b..9ce68cded 100644\n--- a/Makefile\n+++ b/Makefile\n@@ -842,6 +842,7 @@ LIB_OBJS += pretty.o\n LIB_OBJS += prio-queue.o\n LIB_OBJS += progress.o\n LIB_OBJS += prompt.o\n+LIB_OBJS += protocol.o\n LIB_OBJS += quote.o\n LIB_OBJS += reachable.o\n LIB_OBJS += read-cache.o\ndiff --git a/cache.h b/cache.h\nindex 49b083ee0..c74b73671 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -445,6 +445,14 @@ static inline enum object_type object_type(unsigned int mode)\n #define GIT_ICASE_PATHSPECS_ENVIRONMENT \"GIT_ICASE_PATHSPECS\"\n #define GIT_QUARANTINE_ENVIRONMENT \"GIT_QUARANTINE_PATH\"\n \n+/*\n+ * Environment variable used in handshaking the wire protocol.\n+ * Contains a colon ':' separated list of keys with optional values\n+ * 'key[=value]'. Presence of unknown keys and values must be\n+ * ignored.\n+ */\n+#define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n+\n /*\n * This environment variable is expected to contain a boolean indicating\n * whether we should or should not treat:\ndiff --git a/protocol.c b/protocol.c\nnew file mode 100644\nindex 000000000..43012b7eb\n--- /dev/null\n+++ b/protocol.c\n@@ -0,0 +1,79 @@\n+#include \"cache.h\"\n+#include \"config.h\"\n+#include \"protocol.h\"\n+\n+static enum protocol_version parse_protocol_version(const char *value)\n+{\n+\tif (!strcmp(value, \"0\"))\n+\t\treturn protocol_v0;\n+\telse if (!strcmp(value, \"1\"))\n+\t\treturn protocol_v1;\n+\telse\n+\t\treturn protocol_unknown_version;\n+}\n+\n+enum protocol_version get_protocol_version_config(void)\n+{\n+\tconst char *value;\n+\tif (!git_config_get_string_const(\"protocol.version\", &value)) {\n+\t\tenum protocol_version version = parse_protocol_version(value);\n+\n+\t\tif (version == protocol_unknown_version)\n+\t\t\tdie(\"unknown value for config 'protocol.version': %s\",\n+\t\t\t value);\n+\n+\t\treturn version;\n+\t}\n+\n+\treturn protocol_v0;\n+}\n+\n+enum protocol_version determine_protocol_version_server(void)\n+{\n+\tconst char *git_protocol = getenv(GIT_PROTOCOL_ENVIRONMENT);\n+\tenum protocol_version version = protocol_v0;\n+\n+\t/*\n+\t * Determine which protocol version the client has requested. Since\n+\t * multiple 'version' keys can be sent by the client, indicating that\n+\t * the client is okay to speak any of them, select the greatest version\n+\t * that the client has requested. This is due to the assumption that\n+\t * the most recent protocol version will be the most state-of-the-art.\n+\t */\n+\tif (git_protocol) {\n+\t\tstruct string_list list = STRING_LIST_INIT_DUP;\n+\t\tconst struct string_list_item *item;\n+\t\tstring_list_split(&list, git_protocol, ':', -1);\n+\n+\t\tfor_each_string_list_item(item, &list) {\n+\t\t\tconst char *value;\n+\t\t\tenum protocol_version v;\n+\n+\t\t\tif (skip_prefix(item->string, \"version=\", &value)) {\n+\t\t\t\tv = parse_protocol_version(value);\n+\t\t\t\tif (v > version)\n+\t\t\t\t\tversion = v;\n+\t\t\t}\n+\t\t}\n+\n+\t\tstring_list_clear(&list, 0);\n+\t}\n+\n+\treturn version;\n+}\n+\n+enum protocol_version determine_protocol_version_client(const char *server_response)\n+{\n+\tenum protocol_version version = protocol_v0;\n+\n+\tif (skip_prefix(server_response, \"version \", &server_response)) {\n+\t\tversion = parse_protocol_version(server_response);\n+\n+\t\tif (version == protocol_unknown_version)\n+\t\t\tdie(\"server is speaking an unknown protocol\");\n+\t\tif (version == protocol_v0)\n+\t\t\tdie(\"protocol error: server explicitly said version 0\");\n+\t}\n+\n+\treturn version;\n+}\ndiff --git a/protocol.h b/protocol.h\nnew file mode 100644\nindex 000000000..1b2bc94a8\n--- /dev/null\n+++ b/protocol.h\n@@ -0,0 +1,33 @@\n+#ifndef PROTOCOL_H\n+#define PROTOCOL_H\n+\n+enum protocol_version {\n+\tprotocol_unknown_version = -1,\n+\tprotocol_v0 = 0,\n+\tprotocol_v1 = 1,\n+};\n+\n+/*\n+ * Used by a client to determine which protocol version to request be used when\n+ * communicating with a server, reflecting the configured value of the\n+ * 'protocol.version' config. If unconfigured, a value of 'protocol_v0' is\n+ * returned.\n+ */\n+extern enum protocol_version get_protocol_version_config(void);\n+\n+/*\n+ * Used by a server to determine which protocol version should be used based on\n+ * a client's request, communicated via the 'GIT_PROTOCOL' environment variable\n+ * by setting appropriate values for the key 'version'. If a client doesn't\n+ * request a particular protocol version, a default of 'protocol_v0' will be\n+ * used.\n+ */\n+extern enum protocol_version determine_protocol_version_server(void);\n+\n+/*\n+ * Used by a client to determine which protocol version the server is speaking\n+ * based on the server's initial response.\n+ */\n+extern enum protocol_version determine_protocol_version_client(const char *server_response);\n+\n+#endif /* PROTOCOL_H */\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330467","messageId":"20171016175532.73459-7-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 06/11] connect: teach client to recognize v1 server response","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:27Z","receivedAt":"2017-10-16T17:56:03Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach a client to recognize that a server understands protocol v1 by\nlooking at the first pkt-line the server sends in response. This is\ndone by looking for the response \"version 1\" send by upload-pack or\nreceive-pack.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 30 ++++++++++++++++++++++++++----\n 1 file changed, 26 insertions(+), 4 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 8e2e276b6..a5e708a61 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -12,6 +12,7 @@\n #include \"sha1-array.h\"\n #include \"transport.h\"\n #include \"strbuf.h\"\n+#include \"protocol.h\"\n \n static char *server_capabilities;\n static const char *parse_feature_value(const char *, const char *, int *);\n@@ -129,9 +130,23 @@ static int read_remote_ref(int in, char **src_buf, size_t *src_len,\n \treturn len;\n }\n \n-#define EXPECTING_FIRST_REF 0\n-#define EXPECTING_REF 1\n-#define EXPECTING_SHALLOW 2\n+#define EXPECTING_PROTOCOL_VERSION 0\n+#define EXPECTING_FIRST_REF 1\n+#define EXPECTING_REF 2\n+#define EXPECTING_SHALLOW 3\n+\n+/* Returns 1 if packet_buffer is a protocol version pkt-line, 0 otherwise. */\n+static int process_protocol_version(void)\n+{\n+\tswitch (determine_protocol_version_client(packet_buffer)) {\n+\tcase protocol_v1:\n+\t\treturn 1;\n+\tcase protocol_v0:\n+\t\treturn 0;\n+\tdefault:\n+\t\tdie(\"server is speaking an unknown protocol\");\n+\t}\n+}\n \n static void process_capabilities(int *len)\n {\n@@ -224,12 +239,19 @@ struct ref **get_remote_heads(int in, char *src_buf, size_t src_len,\n \t */\n \tint responded = 0;\n \tint len;\n-\tint state = EXPECTING_FIRST_REF;\n+\tint state = EXPECTING_PROTOCOL_VERSION;\n \n \t*list = NULL;\n \n \twhile ((len = read_remote_ref(in, &src_buf, &src_len, &responded))) {\n \t\tswitch (state) {\n+\t\tcase EXPECTING_PROTOCOL_VERSION:\n+\t\t\tif (process_protocol_version()) {\n+\t\t\t\tstate = EXPECTING_FIRST_REF;\n+\t\t\t\tbreak;\n+\t\t\t}\n+\t\t\tstate = EXPECTING_FIRST_REF;\n+\t\t\t/* fallthrough */\n \t\tcase EXPECTING_FIRST_REF:\n \t\t\tprocess_capabilities(&len);\n \t\t\tif (process_dummy_ref()) {\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330468","messageId":"20171016175532.73459-5-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 04/11] daemon: recognize hidden request arguments","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:25Z","receivedAt":"2017-10-16T17:56:05Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"A normal request to git-daemon is structured as\n\"command path/to/repo\\0host=..\\0\" and due to a bug introduced in\n49ba83fb6 (Add virtualization support to git-daemon, 2006-09-19) we\naren't able to place any extra arguments (separated by NULs) besides the\nhost otherwise the parsing of those arguments would enter an infinite\nloop. This bug was fixed in 73bb33a94 (daemon: Strictly parse the\n\"extra arg\" part of the command, 2009-06-04) but a check was put in\nplace to disallow extra arguments so that new clients wouldn't trigger\nthis bug in older servers.\n\nIn order to get around this limitation teach git-daemon to recognize\nadditional request arguments hidden behind a second NUL byte. Requests\ncan then be structured like:\n\"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\". git-daemon\ncan then parse out the extra arguments and set 'GIT_PROTOCOL'\naccordingly.\n\nBy placing these extra arguments behind a second NUL byte we can skirt\naround both the infinite loop bug in 49ba83fb6 (Add virtualization\nsupport to git-daemon, 2006-09-19) as well as the explicit disallowing\nof extra arguments introduced in 73bb33a94 (daemon: Strictly parse the\n\"extra arg\" part of the command, 2009-06-04) because both of these\nversions of git-daemon check for a single NUL byte after the host\nargument before terminating the argument parsing.\n\nSigned-off-by: Brandon Williams \n---\n daemon.c | 71 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--------\n 1 file changed, 62 insertions(+), 9 deletions(-)\n\ndiff --git a/daemon.c b/daemon.c\nindex 30747075f..e37e343d0 100644\n--- a/daemon.c\n+++ b/daemon.c\n@@ -282,7 +282,7 @@ static const char *path_ok(const char *directory, struct hostinfo *hi)\n \treturn NULL;\t\t/* Fallthrough. Deny by default */\n }\n \n-typedef int (*daemon_service_fn)(void);\n+typedef int (*daemon_service_fn)(const struct argv_array *env);\n struct daemon_service {\n \tconst char *name;\n \tconst char *config_name;\n@@ -363,7 +363,7 @@ static int run_access_hook(struct daemon_service *service, const char *dir,\n }\n \n static int run_service(const char *dir, struct daemon_service *service,\n-\t\t struct hostinfo *hi)\n+\t\t struct hostinfo *hi, const struct argv_array *env)\n {\n \tconst char *path;\n \tint enabled = service->enabled;\n@@ -422,7 +422,7 @@ static int run_service(const char *dir, struct daemon_service *service,\n \t */\n \tsignal(SIGTERM, SIG_IGN);\n \n-\treturn service->fn();\n+\treturn service->fn(env);\n }\n \n static void copy_to_log(int fd)\n@@ -462,25 +462,34 @@ static int run_service_command(struct child_process *cld)\n \treturn finish_command(cld);\n }\n \n-static int upload_pack(void)\n+static int upload_pack(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_pushl(&cld.args, \"upload-pack\", \"--strict\", NULL);\n \targv_array_pushf(&cld.args, \"--timeout=%u\", timeout);\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n-static int upload_archive(void)\n+static int upload_archive(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_push(&cld.args, \"upload-archive\");\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n-static int receive_pack(void)\n+static int receive_pack(const struct argv_array *env)\n {\n \tstruct child_process cld = CHILD_PROCESS_INIT;\n \targv_array_push(&cld.args, \"receive-pack\");\n+\n+\targv_array_pushv(&cld.env_array, env->argv);\n+\n \treturn run_service_command(&cld);\n }\n \n@@ -573,8 +582,11 @@ static void canonicalize_client(struct strbuf *out, const char *in)\n \n /*\n * Read the host as supplied by the client connection.\n+ *\n+ * Returns a pointer to the character after the NUL byte terminating the host\n+ * arguemnt, or 'extra_args' if there is no host arguemnt.\n */\n-static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n+static char *parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n {\n \tchar *val;\n \tint vallen;\n@@ -602,6 +614,43 @@ static void parse_host_arg(struct hostinfo *hi, char *extra_args, int buflen)\n \t\tif (extra_args < end && *extra_args)\n \t\t\tdie(\"Invalid request\");\n \t}\n+\n+\treturn extra_args;\n+}\n+\n+static void parse_extra_args(struct hostinfo *hi, struct argv_array *env,\n+\t\t\t char *extra_args, int buflen)\n+{\n+\tconst char *end = extra_args + buflen;\n+\tstruct strbuf git_protocol = STRBUF_INIT;\n+\n+\t/* First look for the host argument */\n+\textra_args = parse_host_arg(hi, extra_args, buflen);\n+\n+\t/* Look for additional arguments places after a second NUL byte */\n+\tfor (; extra_args < end; extra_args += strlen(extra_args) + 1) {\n+\t\tconst char *arg = extra_args;\n+\n+\t\t/*\n+\t\t * Parse the extra arguments, adding most to 'git_protocol'\n+\t\t * which will be used to set the 'GIT_PROTOCOL' envvar in the\n+\t\t * service that will be run.\n+\t\t *\n+\t\t * If there ends up being a particular arg in the future that\n+\t\t * git-daemon needs to parse specificly (like the 'host' arg)\n+\t\t * then it can be parsed here and not added to 'git_protocol'.\n+\t\t */\n+\t\tif (*arg) {\n+\t\t\tif (git_protocol.len > 0)\n+\t\t\t\tstrbuf_addch(&git_protocol, ':');\n+\t\t\tstrbuf_addstr(&git_protocol, arg);\n+\t\t}\n+\t}\n+\n+\tif (git_protocol.len > 0)\n+\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=%s\",\n+\t\t\t\t git_protocol.buf);\n+\tstrbuf_release(&git_protocol);\n }\n \n /*\n@@ -695,6 +744,7 @@ static int execute(void)\n \tint pktlen, len, i;\n \tchar *addr = getenv(\"REMOTE_ADDR\"), *port = getenv(\"REMOTE_PORT\");\n \tstruct hostinfo hi;\n+\tstruct argv_array env = ARGV_ARRAY_INIT;\n \n \thostinfo_init(&hi);\n \n@@ -716,8 +766,9 @@ static int execute(void)\n \t\tpktlen--;\n \t}\n \n+\t/* parse additional args hidden behind a NUL byte */\n \tif (len != pktlen)\n-\t\tparse_host_arg(&hi, line + len + 1, pktlen - len - 1);\n+\t\tparse_extra_args(&hi, &env, line + len + 1, pktlen - len - 1);\n \n \tfor (i = 0; i < ARRAY_SIZE(daemon_service); i++) {\n \t\tstruct daemon_service *s = &(daemon_service[i]);\n@@ -730,13 +781,15 @@ static int execute(void)\n \t\t\t * Note: The directory here is probably context sensitive,\n \t\t\t * and might depend on the actual service being performed.\n \t\t\t */\n-\t\t\tint rc = run_service(arg, s, &hi);\n+\t\t\tint rc = run_service(arg, s, &hi, &env);\n \t\t\thostinfo_clear(&hi);\n+\t\t\targv_array_clear(&env);\n \t\t\treturn rc;\n \t\t}\n \t}\n \n \thostinfo_clear(&hi);\n+\targv_array_clear(&env);\n \tlogerror(\"Protocol error: '%s'\", line);\n \treturn -1;\n }\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330469","messageId":"20171016175532.73459-10-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 09/11] i5700: add interop test for protocol transition","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:30Z","receivedAt":"2017-10-16T17:56:06Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Signed-off-by: Brandon Williams \n---\n t/interop/i5700-protocol-transition.sh | 68 ++++++++++++++++++++++++++++++++++\n 1 file changed, 68 insertions(+)\n create mode 100755 t/interop/i5700-protocol-transition.sh\n\ndiff --git a/t/interop/i5700-protocol-transition.sh b/t/interop/i5700-protocol-transition.sh\nnew file mode 100755\nindex 000000000..97e8e580e\n--- /dev/null\n+++ b/t/interop/i5700-protocol-transition.sh\n@@ -0,0 +1,68 @@\n+#!/bin/sh\n+\n+VERSION_A=.\n+VERSION_B=v2.0.0\n+\n+: ${LIB_GIT_DAEMON_PORT:=5700}\n+LIB_GIT_DAEMON_COMMAND='git.b daemon'\n+\n+test_description='clone and fetch by client who is trying to use a new protocol'\n+. ./interop-lib.sh\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+\n+start_git_daemon --export-all\n+\n+repo=$GIT_DAEMON_DOCUMENT_ROOT_PATH/repo\n+\n+test_expect_success \"create repo served by $VERSION_B\" '\n+\tgit.b init \"$repo\" &&\n+\tgit.b -C \"$repo\" commit --allow-empty -m one\n+'\n+\n+test_expect_success \"git:// clone with $VERSION_A and protocol v1\" '\n+\tGIT_TRACE_PACKET=1 git.a -c protocol.version=1 clone \"$GIT_DAEMON_URL/repo\" child 2>log &&\n+\tgit.a -C child log -1 --format=%s >actual &&\n+\tgit.b -C \"$repo\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\tgrep \"version=1\" log\n+'\n+\n+test_expect_success \"git:// fetch with $VERSION_A and protocol v1\" '\n+\tgit.b -C \"$repo\" commit --allow-empty -m two &&\n+\tgit.b -C \"$repo\" log -1 --format=%s >expect &&\n+\n+\tGIT_TRACE_PACKET=1 git.a -C child -c protocol.version=1 fetch 2>log &&\n+\tgit.a -C child log -1 --format=%s FETCH_HEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\tgrep \"version=1\" log &&\n+\t! grep \"version 1\" log\n+'\n+\n+stop_git_daemon\n+\n+test_expect_success \"create repo served by $VERSION_B\" '\n+\tgit.b init parent &&\n+\tgit.b -C parent commit --allow-empty -m one\n+'\n+\n+test_expect_success \"file:// clone with $VERSION_A and protocol v1\" '\n+\tGIT_TRACE_PACKET=1 git.a -c protocol.version=1 clone --upload-pack=\"git.b upload-pack\" parent child2 2>log &&\n+\tgit.a -C child2 log -1 --format=%s >actual &&\n+\tgit.b -C parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\t! grep \"version 1\" log\n+'\n+\n+test_expect_success \"file:// fetch with $VERSION_A and protocol v1\" '\n+\tgit.b -C parent commit --allow-empty -m two &&\n+\tgit.b -C parent log -1 --format=%s >expect &&\n+\n+\tGIT_TRACE_PACKET=1 git.a -C child2 -c protocol.version=1 fetch --upload-pack=\"git.b upload-pack\" 2>log &&\n+\tgit.a -C child2 log -1 --format=%s FETCH_HEAD >actual &&\n+\n+\ttest_cmp expect actual &&\n+\t! grep \"version 1\" log\n+'\n+\n+test_done\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330470","messageId":"20171016175532.73459-9-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 08/11] http: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:29Z","receivedAt":"2017-10-16T17:56:08Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Tell a server that protocol v1 can be used by sending the http header\n'Git-Protocol' with 'version=1' indicating this.\n\nAlso teach the apache http server to pass through the 'Git-Protocol'\nheader as an environment variable 'GIT_PROTOCOL'.\n\nSigned-off-by: Brandon Williams \n---\n cache.h | 2 ++\n http.c | 18 +++++++++++++\n t/lib-httpd/apache.conf | 7 +++++\n t/t5700-protocol-v1.sh | 69 +++++++++++++++++++++++++++++++++++++++++++++++++\n 4 files changed, 96 insertions(+)\n\ndiff --git a/cache.h b/cache.h\nindex c74b73671..3a6b869c2 100644\n--- a/cache.h\n+++ b/cache.h\n@@ -452,6 +452,8 @@ static inline enum object_type object_type(unsigned int mode)\n * ignored.\n */\n #define GIT_PROTOCOL_ENVIRONMENT \"GIT_PROTOCOL\"\n+/* HTTP header used to handshake the wire protocol */\n+#define GIT_PROTOCOL_HEADER \"Git-Protocol\"\n \n /*\n * This environment variable is expected to contain a boolean indicating\ndiff --git a/http.c b/http.c\nindex 9e40a465f..ffb719216 100644\n--- a/http.c\n+++ b/http.c\n@@ -12,6 +12,7 @@\n #include \"gettext.h\"\n #include \"transport.h\"\n #include \"packfile.h\"\n+#include \"protocol.h\"\n \n static struct trace_key trace_curl = TRACE_KEY_INIT(CURL);\n #if LIBCURL_VERSION_NUM >= 0x070a08\n@@ -897,6 +898,21 @@ static void set_from_env(const char **var, const char *envname)\n \t\t*var = val;\n }\n \n+static void protocol_http_header(void)\n+{\n+\tif (get_protocol_version_config() > 0) {\n+\t\tstruct strbuf protocol_header = STRBUF_INIT;\n+\n+\t\tstrbuf_addf(&protocol_header, GIT_PROTOCOL_HEADER \": version=%d\",\n+\t\t\t get_protocol_version_config());\n+\n+\n+\t\textra_http_headers = curl_slist_append(extra_http_headers,\n+\t\t\t\t\t\t protocol_header.buf);\n+\t\tstrbuf_release(&protocol_header);\n+\t}\n+}\n+\n void http_init(struct remote *remote, const char *url, int proactive_auth)\n {\n \tchar *low_speed_limit;\n@@ -927,6 +943,8 @@ void http_init(struct remote *remote, const char *url, int proactive_auth)\n \tif (remote)\n \t\tvar_override(&http_proxy_authmethod, remote->http_proxy_authmethod);\n \n+\tprotocol_http_header();\n+\n \tpragma_header = curl_slist_append(http_copy_default_headers(),\n \t\t\"Pragma: no-cache\");\n \tno_pragma_header = curl_slist_append(http_copy_default_headers(),\ndiff --git a/t/lib-httpd/apache.conf b/t/lib-httpd/apache.conf\nindex 0642ae7e6..df1943631 100644\n--- a/t/lib-httpd/apache.conf\n+++ b/t/lib-httpd/apache.conf\n@@ -67,6 +67,9 @@ LockFile accept.lock\n \n \tLoadModule unixd_module modules/mod_unixd.so\n \n+\n+\tLoadModule setenvif_module modules/mod_setenvif.so\n+\n \n \n PassEnv GIT_VALGRIND\n@@ -76,6 +79,10 @@ PassEnv ASAN_OPTIONS\n PassEnv GIT_TRACE\n PassEnv GIT_CONFIG_NOSYSTEM\n \n+= 2.4>\n+\tSetEnvIf Git-Protocol \".*\" GIT_PROTOCOL=$0\n+\n+\n Alias /dumb/ www/\n Alias /auth/dumb/ www/auth/dumb/\n \ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nindex 6551932da..b0779d362 100755\n--- a/t/t5700-protocol-v1.sh\n+++ b/t/t5700-protocol-v1.sh\n@@ -220,4 +220,73 @@ test_expect_success 'push with ssh:// using protocol v1' '\n \tgrep \"push< version 1\" log\n '\n \n+# Test protocol v1 with 'http://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-httpd.sh\n+start_httpd\n+\n+test_expect_success 'create repo to be served by http:// transport' '\n+\tgit init \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" config http.receivepack true &&\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" one\n+'\n+\n+test_expect_success 'clone with http:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 GIT_TRACE_CURL=1 git -c protocol.version=1 \\\n+\t\tclone \"$HTTPD_URL/smart/http_parent\" http_child 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"Git-Protocol: version=1\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'fetch with http:// using protocol v1' '\n+\ttest_commit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'pull with http:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+test_expect_success 'push with http:// using protocol v1' '\n+\ttest_commit -C http_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C http_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch && #2>log &&\n+\n+\tgit -C http_child log -1 --format=%s >actual &&\n+\tgit -C \"$HTTPD_DOCUMENT_ROOT_PATH/http_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"git< version 1\" log\n+'\n+\n+stop_httpd\n+\n test_done\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330471","messageId":"20171016175532.73459-6-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 05/11] upload-pack, receive-pack: introduce protocol version 1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:26Z","receivedAt":"2017-10-16T17:56:11Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach upload-pack and receive-pack to understand and respond using\nprotocol version 1, if requested.\n\nProtocol version 1 is simply the original and current protocol (what I'm\ncalling version 0) with the addition of a single packet line, which\nprecedes the ref advertisement, indicating the protocol version being\nspoken.\n\nSigned-off-by: Brandon Williams \n---\n builtin/receive-pack.c | 17 +++++++++++++++++\n upload-pack.c | 20 +++++++++++++++++++-\n 2 files changed, 36 insertions(+), 1 deletion(-)\n\ndiff --git a/builtin/receive-pack.c b/builtin/receive-pack.c\nindex dd06b3fb4..839c1462d 100644\n--- a/builtin/receive-pack.c\n+++ b/builtin/receive-pack.c\n@@ -24,6 +24,7 @@\n #include \"tmp-objdir.h\"\n #include \"oidset.h\"\n #include \"packfile.h\"\n+#include \"protocol.h\"\n \n static const char * const receive_pack_usage[] = {\n \tN_(\"git receive-pack \"),\n@@ -1963,6 +1964,22 @@ int cmd_receive_pack(int argc, const char **argv, const char *prefix)\n \telse if (0 <= receive_unpack_limit)\n \t\tunpack_limit = receive_unpack_limit;\n \n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\n+\t\t/* fallthrough */\n+\tcase protocol_v0:\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \tif (advertise_refs || !stateless_rpc) {\n \t\twrite_head_info();\n \t}\ndiff --git a/upload-pack.c b/upload-pack.c\nindex 7efff2fbf..ef99a029c 100644\n--- a/upload-pack.c\n+++ b/upload-pack.c\n@@ -18,6 +18,7 @@\n #include \"parse-options.h\"\n #include \"argv-array.h\"\n #include \"prio-queue.h\"\n+#include \"protocol.h\"\n \n static const char * const upload_pack_usage[] = {\n \tN_(\"git upload-pack [] \"),\n@@ -1067,6 +1068,23 @@ int cmd_main(int argc, const char **argv)\n \t\tdie(\"'%s' does not appear to be a git repository\", dir);\n \n \tgit_config(upload_pack_config, NULL);\n-\tupload_pack();\n+\n+\tswitch (determine_protocol_version_server()) {\n+\tcase protocol_v1:\n+\t\t/*\n+\t\t * v1 is just the original protocol with a version string,\n+\t\t * so just fall through after writing the version string.\n+\t\t */\n+\t\tif (advertise_refs || !stateless_rpc)\n+\t\t\tpacket_write_fmt(1, \"version 1\\n\");\n+\n+\t\t/* fallthrough */\n+\tcase protocol_v0:\n+\t\tupload_pack();\n+\t\tbreak;\n+\tcase protocol_unknown_version:\n+\t\tBUG(\"unknown protocol version\");\n+\t}\n+\n \treturn 0;\n }\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330472","messageId":"20171016175532.73459-8-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 07/11] connect: tell server that the client understands v1","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:28Z","receivedAt":"2017-10-16T17:56:16Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"Teach the connection logic to tell a serve that it understands protocol\nv1. This is done in 2 different ways for the builtin transports, both\nof which ultimately set 'GIT_PROTOCOL' to 'version=1' on the server.\n\n1. git://\n A normal request to git-daemon is structured as\n \"command path/to/repo\\0host=..\\0\" and due to a bug introduced in\n 49ba83fb6 (Add virtualization support to git-daemon, 2006-09-19) we\n aren't able to place any extra arguments (separated by NULs) besides\n the host otherwise the parsing of those arguments would enter an\n infinite loop. This bug was fixed in 73bb33a94 (daemon: Strictly\n parse the \"extra arg\" part of the command, 2009-06-04) but a check\n was put in place to disallow extra arguments so that new clients\n wouldn't trigger this bug in older servers.\n\n In order to get around this limitation git-daemon was taught to\n recognize additional request arguments hidden behind a second\n NUL byte. Requests can then be structured like:\n \"command path/to/repo\\0host=..\\0\\0version=1\\0key=value\\0\".\n git-daemon can then parse out the extra arguments and set\n 'GIT_PROTOCOL' accordingly.\n\n By placing these extra arguments behind a second NUL byte we can\n skirt around both the infinite loop bug in 49ba83fb6 (Add\n virtualization support to git-daemon, 2006-09-19) as well as the\n explicit disallowing of extra arguments introduced in 73bb33a94\n (daemon: Strictly parse the \"extra arg\" part of the command,\n 2009-06-04) because both of these versions of git-daemon check for a\n single NUL byte after the host argument before terminating the\n argument parsing.\n\n2. ssh://, file://\n Set 'GIT_PROTOCOL' environment variable with the desired protocol\n version. With the file:// transport, 'GIT_PROTOCOL' can be set\n explicitly in the locally running git-upload-pack or git-receive-pack\n processes. With the ssh:// transport and OpenSSH compliant ssh\n programs, 'GIT_PROTOCOL' can be sent across ssh by using '-o\n SendEnv=GIT_PROTOCOL' and having the server whitelist this\n environment variable.\n\nSigned-off-by: Brandon Williams \n---\n connect.c | 37 ++++++--\n t/t5700-protocol-v1.sh | 223 +++++++++++++++++++++++++++++++++++++++++++++++++\n 2 files changed, 255 insertions(+), 5 deletions(-)\n create mode 100755 t/t5700-protocol-v1.sh\n\ndiff --git a/connect.c b/connect.c\nindex a5e708a61..b8695a2fa 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -871,6 +871,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tprintf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n \t\tconn = NULL;\n \t} else if (protocol == PROTO_GIT) {\n+\t\tstruct strbuf request = STRBUF_INIT;\n \t\t/*\n \t\t * Set up virtual host information based on where we will\n \t\t * connect, unless the user has overridden us in\n@@ -898,13 +899,25 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t * Note: Do not add any other headers here! Doing so\n \t\t * will cause older git-daemon servers to crash.\n \t\t */\n-\t\tpacket_write_fmt(fd[1],\n-\t\t\t \"%s %s%chost=%s%c\",\n-\t\t\t prog, path, 0,\n-\t\t\t target_host, 0);\n+\t\tstrbuf_addf(&request,\n+\t\t\t \"%s %s%chost=%s%c\",\n+\t\t\t prog, path, 0,\n+\t\t\t target_host, 0);\n+\n+\t\t/* If using a new version put that stuff here after a second null byte */\n+\t\tif (get_protocol_version_config() > 0) {\n+\t\t\tstrbuf_addch(&request, '\\0');\n+\t\t\tstrbuf_addf(&request, \"version=%d%c\",\n+\t\t\t\t get_protocol_version_config(), '\\0');\n+\t\t}\n+\n+\t\tpacket_write(fd[1], request.buf, request.len);\n+\n \t\tfree(target_host);\n+\t\tstrbuf_release(&request);\n \t} else {\n \t\tstruct strbuf cmd = STRBUF_INIT;\n+\t\tconst char *const *var;\n \n \t\tconn = xmalloc(sizeof(*conn));\n \t\tchild_process_init(conn);\n@@ -917,7 +930,9 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tsq_quote_buf(&cmd, path);\n \n \t\t/* remove repo-local variables from the environment */\n-\t\tconn->env = local_repo_env;\n+\t\tfor (var = local_repo_env; *var; var++)\n+\t\t\targv_array_push(&conn->env_array, *var);\n+\n \t\tconn->use_shell = 1;\n \t\tconn->in = conn->out = -1;\n \t\tif (protocol == PROTO_SSH) {\n@@ -971,6 +986,14 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t}\n \n \t\t\targv_array_push(&conn->args, ssh);\n+\n+\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\t\targv_array_push(&conn->args, \"-o\");\n+\t\t\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n+\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t}\n+\n \t\t\tif (flags & CONNECT_IPV4)\n \t\t\t\targv_array_push(&conn->args, \"-4\");\n \t\t\telse if (flags & CONNECT_IPV6)\n@@ -985,6 +1008,10 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\targv_array_push(&conn->args, ssh_host);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\n+\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t\t\t get_protocol_version_config());\n+\t\t\t}\n \t\t}\n \t\targv_array_push(&conn->args, cmd.buf);\n \ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nnew file mode 100755\nindex 000000000..6551932da\n--- /dev/null\n+++ b/t/t5700-protocol-v1.sh\n@@ -0,0 +1,223 @@\n+#!/bin/sh\n+\n+test_description='test git wire-protocol transition'\n+\n+TEST_NO_CREATE_REPO=1\n+\n+. ./test-lib.sh\n+\n+# Test protocol v1 with 'git://' transport\n+#\n+. \"$TEST_DIRECTORY\"/lib-git-daemon.sh\n+start_git_daemon --export-all --enable=receive-pack\n+daemon_parent=$GIT_DAEMON_DOCUMENT_ROOT_PATH/parent\n+\n+test_expect_success 'create repo to be served by git-daemon' '\n+\tgit init \"$daemon_parent\" &&\n+\ttest_commit -C \"$daemon_parent\" one\n+'\n+\n+test_expect_success 'clone with git:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"$GIT_DAEMON_URL/parent\" daemon_child 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"clone> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with git:// using protocol v1' '\n+\ttest_commit -C \"$daemon_parent\" two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s origin/master >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with git:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"fetch> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with git:// using protocol v1' '\n+\ttest_commit -C daemon_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C daemon_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\n+\tgit -C daemon_child log -1 --format=%s >actual &&\n+\tgit -C \"$daemon_parent\" log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Client requested to use protocol v1\n+\tgrep \"push> .*\\\\\\0\\\\\\0version=1\\\\\\0$\" log &&\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+stop_git_daemon\n+\n+# Test protocol v1 with 'file://' transport\n+#\n+test_expect_success 'create repo to be served by file:// transport' '\n+\tgit init file_parent &&\n+\ttest_commit -C file_parent one\n+'\n+\n+test_expect_success 'clone with file:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"file://$(pwd)/file_parent\" file_child 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with file:// using protocol v1' '\n+\ttest_commit -C file_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s origin/master >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with file:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with file:// using protocol v1' '\n+\ttest_commit -C file_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C file_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\n+\tgit -C file_child log -1 --format=%s >actual &&\n+\tgit -C file_parent log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+# Test protocol v1 with 'ssh://' transport\n+#\n+test_expect_success 'setup ssh wrapper' '\n+\tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh\" &&\n+\texport GIT_SSH &&\n+\texport TRASH_DIRECTORY &&\n+\t>\"$TRASH_DIRECTORY\"/ssh-output\n+'\n+\n+expect_ssh () {\n+\ttest_when_finished '(cd \"$TRASH_DIRECTORY\" && rm -f ssh-expect && >ssh-output)' &&\n+\techo \"ssh: -o SendEnv=GIT_PROTOCOL myhost $1 '$PWD/ssh_parent'\" >\"$TRASH_DIRECTORY/ssh-expect\" &&\n+\t(cd \"$TRASH_DIRECTORY\" && test_cmp ssh-expect ssh-output)\n+}\n+\n+test_expect_success 'create repo to be served by ssh:// transport' '\n+\tgit init ssh_parent &&\n+\ttest_commit -C ssh_parent one\n+'\n+\n+test_expect_success 'clone with ssh:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -c protocol.version=1 \\\n+\t\tclone \"ssh://myhost:$(pwd)/ssh_parent\" ssh_child 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"clone< version 1\" log\n+'\n+\n+test_expect_success 'fetch with ssh:// using protocol v1' '\n+\ttest_commit -C ssh_parent two &&\n+\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tfetch 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s origin/master >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'pull with ssh:// using protocol v1' '\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tpull 2>log &&\n+\texpect_ssh git-upload-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"fetch< version 1\" log\n+'\n+\n+test_expect_success 'push with ssh:// using protocol v1' '\n+\ttest_commit -C ssh_child three &&\n+\n+\t# Push to another branch, as the target repository has the\n+\t# master branch checked out and we cannot push into it.\n+\tGIT_TRACE_PACKET=1 git -C ssh_child -c protocol.version=1 \\\n+\t\tpush origin HEAD:client_branch 2>log &&\n+\texpect_ssh git-receive-pack &&\n+\n+\tgit -C ssh_child log -1 --format=%s >actual &&\n+\tgit -C ssh_parent log -1 --format=%s client_branch >expect &&\n+\ttest_cmp expect actual &&\n+\n+\t# Server responded using protocol v1\n+\tgrep \"push< version 1\" log\n+'\n+\n+test_done\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330473","messageId":"20171016175532.73459-11-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 10/11] ssh: introduce a 'simple' ssh variant","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:31Z","receivedAt":"2017-10-16T17:56:36Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"When using the 'ssh' transport, the '-o' option is used to specify an\nenvironment variable which should be set on the remote end. This allows\ngit to send additional information when contacting the server,\nrequesting the use of a different protocol version via the\n'GIT_PROTOCOL' environment variable like so: \"-o SendEnv=GIT_PROTOCOL\".\n\nUnfortunately not all ssh variants support the sending of environment\nvariables to the remote end. To account for this, only use the '-o'\noption for ssh variants which are OpenSSH compliant. This is done by\nchecking that the basename of the ssh command is 'ssh' or the ssh\nvariant is overridden to be 'ssh' (via the ssh.variant config).\n\nOther options like '-p' and '-P', which are used to specify a specific\nport to use, or '-4' and '-6', which are used to indicate that IPV4 or\nIPV6 addresses should be used, may also not be supported by all ssh\nvariants.\n\nCurrently if an ssh command's basename wasn't 'plink' or\n'tortoiseplink' git assumes that the command is an OpenSSH variant.\nSince user configured ssh commands may not be OpenSSH compliant, tighten\nthis constraint and assume a variant of 'simple' if the basename of the\ncommand doesn't match the variants known to git. The new ssh variant\n'simple' will only have the host and command to execute ([username@]host\ncommand) passed as parameters to the ssh command.\n\nUpdate the Documentation to better reflect the command-line options sent\nto ssh commands based on their variant.\n\nReported-by: Jeffrey Yasskin \nSigned-off-by: Brandon Williams \n---\n Documentation/config.txt | 27 ++++++++++--\n Documentation/git.txt | 9 ++--\n connect.c | 108 ++++++++++++++++++++++++++---------------------\n t/t5601-clone.sh | 26 +++++++++---\n t/t5700-protocol-v1.sh | 2 +\n 5 files changed, 111 insertions(+), 61 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex b78747abc..0460af37e 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2084,12 +2084,31 @@ ssh.variant::\n \tDepending on the value of the environment variables `GIT_SSH` or\n \t`GIT_SSH_COMMAND`, or the config setting `core.sshCommand`, Git\n \tauto-detects whether to adjust its command-line parameters for use\n-\twith plink or tortoiseplink, as opposed to the default (OpenSSH).\n+\twith ssh (OpenSSH), plink or tortoiseplink, as opposed to the default\n+\t(simple).\n +\n The config variable `ssh.variant` can be set to override this auto-detection;\n-valid values are `ssh`, `plink`, `putty` or `tortoiseplink`. Any other value\n-will be treated as normal ssh. This setting can be overridden via the\n-environment variable `GIT_SSH_VARIANT`.\n+valid values are `ssh`, `simple`, `plink`, `putty` or `tortoiseplink`. Any\n+other value will be treated as normal ssh. This setting can be overridden via\n+the environment variable `GIT_SSH_VARIANT`.\n++\n+The current command-line parameters used for each variant are as\n+follows:\n++\n+--\n+\n+* `ssh` - [-p port] [-4] [-6] [-o option] [username@]host command\n+\n+* `simple` - [username@]host command\n+\n+* `plink` or `putty` - [-P port] [-4] [-6] [username@]host command\n+\n+* `tortoiseplink` - [-P port] [-4] [-6] -batch [username@]host command\n+\n+--\n++\n+Except for the `simple` variant, command-line parameters are likely to\n+change as git gains new features.\n \n i18n.commitEncoding::\n \tCharacter encoding the commit messages are stored in; Git itself\ndiff --git a/Documentation/git.txt b/Documentation/git.txt\nindex 7518ea3af..8bc3f2147 100644\n--- a/Documentation/git.txt\n+++ b/Documentation/git.txt\n@@ -518,11 +518,10 @@ other\n \tIf either of these environment variables is set then 'git fetch'\n \tand 'git push' will use the specified command instead of 'ssh'\n \twhen they need to connect to a remote system.\n-\tThe command will be given exactly two or four arguments: the\n-\t'username@host' (or just 'host') from the URL and the shell\n-\tcommand to execute on that remote system, optionally preceded by\n-\t`-p` (literally) and the 'port' from the URL when it specifies\n-\tsomething other than the default SSH port.\n+\tThe command-line parameters passed to the configured command are\n+\tdetermined by the ssh variant. See `ssh.variant` option in\n+\tlinkgit:git-config[1] for details.\n+\n +\n `$GIT_SSH_COMMAND` takes precedence over `$GIT_SSH`, and is interpreted\n by the shell, which allows additional arguments to be included.\ndiff --git a/connect.c b/connect.c\nindex b8695a2fa..7fbd396b3 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -776,37 +776,44 @@ static const char *get_ssh_command(void)\n \treturn NULL;\n }\n \n-static int override_ssh_variant(int *port_option, int *needs_batch)\n+enum ssh_variant {\n+\tVARIANT_SIMPLE,\n+\tVARIANT_SSH,\n+\tVARIANT_PLINK,\n+\tVARIANT_PUTTY,\n+\tVARIANT_TORTOISEPLINK,\n+};\n+\n+static int override_ssh_variant(enum ssh_variant *ssh_variant)\n {\n-\tchar *variant;\n+\tconst char *variant = getenv(\"GIT_SSH_VARIANT\");\n \n-\tvariant = xstrdup_or_null(getenv(\"GIT_SSH_VARIANT\"));\n-\tif (!variant &&\n-\t git_config_get_string(\"ssh.variant\", &variant))\n+\tif (!variant && git_config_get_string_const(\"ssh.variant\", &variant))\n \t\treturn 0;\n \n-\tif (!strcmp(variant, \"plink\") || !strcmp(variant, \"putty\")) {\n-\t\t*port_option = 'P';\n-\t\t*needs_batch = 0;\n-\t} else if (!strcmp(variant, \"tortoiseplink\")) {\n-\t\t*port_option = 'P';\n-\t\t*needs_batch = 1;\n-\t} else {\n-\t\t*port_option = 'p';\n-\t\t*needs_batch = 0;\n-\t}\n-\tfree(variant);\n+\tif (!strcmp(variant, \"plink\"))\n+\t\t*ssh_variant = VARIANT_PLINK;\n+\telse if (!strcmp(variant, \"putty\"))\n+\t\t*ssh_variant = VARIANT_PUTTY;\n+\telse if (!strcmp(variant, \"tortoiseplink\"))\n+\t\t*ssh_variant = VARIANT_TORTOISEPLINK;\n+\telse if (!strcmp(variant, \"simple\"))\n+\t\t*ssh_variant = VARIANT_SIMPLE;\n+\telse\n+\t\t*ssh_variant = VARIANT_SSH;\n+\n \treturn 1;\n }\n \n-static void handle_ssh_variant(const char *ssh_command, int is_cmdline,\n-\t\t\t int *port_option, int *needs_batch)\n+static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n+\t\t\t\t\t int is_cmdline)\n {\n+\tenum ssh_variant ssh_variant = VARIANT_SIMPLE;\n \tconst char *variant;\n \tchar *p = NULL;\n \n-\tif (override_ssh_variant(port_option, needs_batch))\n-\t\treturn;\n+\tif (override_ssh_variant(&ssh_variant))\n+\t\treturn ssh_variant;\n \n \tif (!is_cmdline) {\n \t\tp = xstrdup(ssh_command);\n@@ -825,19 +832,22 @@ static void handle_ssh_variant(const char *ssh_command, int is_cmdline,\n \t\t\tfree(ssh_argv);\n \t\t} else {\n \t\t\tfree(p);\n-\t\t\treturn;\n+\t\t\treturn ssh_variant;\n \t\t}\n \t}\n \n-\tif (!strcasecmp(variant, \"plink\") ||\n-\t !strcasecmp(variant, \"plink.exe\"))\n-\t\t*port_option = 'P';\n+\tif (!strcasecmp(variant, \"ssh\") ||\n+\t !strcasecmp(variant, \"ssh.exe\"))\n+\t\tssh_variant = VARIANT_SSH;\n+\telse if (!strcasecmp(variant, \"plink\") ||\n+\t\t !strcasecmp(variant, \"plink.exe\"))\n+\t\tssh_variant = VARIANT_PLINK;\n \telse if (!strcasecmp(variant, \"tortoiseplink\") ||\n-\t\t !strcasecmp(variant, \"tortoiseplink.exe\")) {\n-\t\t*port_option = 'P';\n-\t\t*needs_batch = 1;\n-\t}\n+\t\t !strcasecmp(variant, \"tortoiseplink.exe\"))\n+\t\tssh_variant = VARIANT_TORTOISEPLINK;\n+\n \tfree(p);\n+\treturn ssh_variant;\n }\n \n /*\n@@ -937,8 +947,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tconn->in = conn->out = -1;\n \t\tif (protocol == PROTO_SSH) {\n \t\t\tconst char *ssh;\n-\t\t\tint needs_batch = 0;\n-\t\t\tint port_option = 'p';\n+\t\t\tenum ssh_variant variant;\n \t\t\tchar *ssh_host = hostandport;\n \t\t\tconst char *port = NULL;\n \t\t\ttransport_check_allowed(\"ssh\");\n@@ -965,10 +974,9 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tdie(\"strange hostname '%s' blocked\", ssh_host);\n \n \t\t\tssh = get_ssh_command();\n-\t\t\tif (ssh)\n-\t\t\t\thandle_ssh_variant(ssh, 1, &port_option,\n-\t\t\t\t\t\t &needs_batch);\n-\t\t\telse {\n+\t\t\tif (ssh) {\n+\t\t\t\tvariant = determine_ssh_variant(ssh, 1);\n+\t\t\t} else {\n \t\t\t\t/*\n \t\t\t\t * GIT_SSH is the no-shell version of\n \t\t\t\t * GIT_SSH_COMMAND (and must remain so for\n@@ -979,32 +987,38 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tssh = getenv(\"GIT_SSH\");\n \t\t\t\tif (!ssh)\n \t\t\t\t\tssh = \"ssh\";\n-\t\t\t\telse\n-\t\t\t\t\thandle_ssh_variant(ssh, 0,\n-\t\t\t\t\t\t\t &port_option,\n-\t\t\t\t\t\t\t &needs_batch);\n+\t\t\t\tvariant = determine_ssh_variant(ssh, 0);\n \t\t\t}\n \n \t\t\targv_array_push(&conn->args, ssh);\n \n-\t\t\tif (get_protocol_version_config() > 0) {\n+\t\t\tif (variant == VARIANT_SSH &&\n+\t\t\t get_protocol_version_config() > 0) {\n \t\t\t\targv_array_push(&conn->args, \"-o\");\n \t\t\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n \t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n \t\t\t\t\t\t get_protocol_version_config());\n \t\t\t}\n \n-\t\t\tif (flags & CONNECT_IPV4)\n-\t\t\t\targv_array_push(&conn->args, \"-4\");\n-\t\t\telse if (flags & CONNECT_IPV6)\n-\t\t\t\targv_array_push(&conn->args, \"-6\");\n-\t\t\tif (needs_batch)\n+\t\t\tif (variant != VARIANT_SIMPLE) {\n+\t\t\t\tif (flags & CONNECT_IPV4)\n+\t\t\t\t\targv_array_push(&conn->args, \"-4\");\n+\t\t\t\telse if (flags & CONNECT_IPV6)\n+\t\t\t\t\targv_array_push(&conn->args, \"-6\");\n+\t\t\t}\n+\n+\t\t\tif (variant == VARIANT_TORTOISEPLINK)\n \t\t\t\targv_array_push(&conn->args, \"-batch\");\n-\t\t\tif (port) {\n-\t\t\t\targv_array_pushf(&conn->args,\n-\t\t\t\t\t\t \"-%c\", port_option);\n+\n+\t\t\tif (port && variant != VARIANT_SIMPLE) {\n+\t\t\t\tif (variant == VARIANT_SSH)\n+\t\t\t\t\targv_array_push(&conn->args, \"-p\");\n+\t\t\t\telse\n+\t\t\t\t\targv_array_push(&conn->args, \"-P\");\n+\n \t\t\t\targv_array_push(&conn->args, port);\n \t\t\t}\n+\n \t\t\targv_array_push(&conn->args, ssh_host);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 9c56f771b..86811a0c3 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -309,8 +309,8 @@ test_expect_success 'clone checking out a tag' '\n setup_ssh_wrapper () {\n \ttest_expect_success 'setup ssh wrapper' '\n \t\tcp \"$GIT_BUILD_DIR/t/helper/test-fake-ssh$X\" \\\n-\t\t\t\"$TRASH_DIRECTORY/ssh-wrapper$X\" &&\n-\t\tGIT_SSH=\"$TRASH_DIRECTORY/ssh-wrapper$X\" &&\n+\t\t\t\"$TRASH_DIRECTORY/ssh$X\" &&\n+\t\tGIT_SSH=\"$TRASH_DIRECTORY/ssh$X\" &&\n \t\texport GIT_SSH &&\n \t\texport TRASH_DIRECTORY &&\n \t\t>\"$TRASH_DIRECTORY\"/ssh-output\n@@ -318,7 +318,7 @@ setup_ssh_wrapper () {\n }\n \n copy_ssh_wrapper_as () {\n-\tcp \"$TRASH_DIRECTORY/ssh-wrapper$X\" \"${1%$X}$X\" &&\n+\tcp \"$TRASH_DIRECTORY/ssh$X\" \"${1%$X}$X\" &&\n \tGIT_SSH=\"${1%$X}$X\" &&\n \texport GIT_SSH\n }\n@@ -362,10 +362,26 @@ test_expect_success 'bracketed hostnames are still ssh' '\n \texpect_ssh \"-p 123\" myhost src\n '\n \n-test_expect_success 'uplink is not treated as putty' '\n+test_expect_success 'OpenSSH variant passes -4' '\n+\tgit clone -4 \"[myhost:123]:src\" ssh-ipv4-clone &&\n+\texpect_ssh \"-4 -p 123\" myhost src\n+'\n+\n+test_expect_success 'variant can be overriden' '\n+\tgit -c ssh.variant=simple clone -4 \"[myhost:123]:src\" ssh-simple-clone &&\n+\texpect_ssh myhost src\n+'\n+\n+test_expect_success 'simple is treated as simple' '\n+\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n+\tgit clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple &&\n+\texpect_ssh myhost src\n+'\n+\n+test_expect_success 'uplink is treated as simple' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/uplink\" &&\n \tgit clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n-\texpect_ssh \"-p 123\" myhost src\n+\texpect_ssh myhost src\n '\n \n test_expect_success 'plink is treated specially (as putty)' '\ndiff --git a/t/t5700-protocol-v1.sh b/t/t5700-protocol-v1.sh\nindex b0779d362..ba86a44eb 100755\n--- a/t/t5700-protocol-v1.sh\n+++ b/t/t5700-protocol-v1.sh\n@@ -147,6 +147,8 @@ test_expect_success 'push with file:// using protocol v1' '\n test_expect_success 'setup ssh wrapper' '\n \tGIT_SSH=\"$GIT_BUILD_DIR/t/helper/test-fake-ssh\" &&\n \texport GIT_SSH &&\n+\tGIT_SSH_VARIANT=ssh &&\n+\texport GIT_SSH_VARIANT &&\n \texport TRASH_DIRECTORY &&\n \t>\"$TRASH_DIRECTORY\"/ssh-output\n '\n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330474","messageId":"20171016175532.73459-12-bmwill@google.com","threadId":"46746","inReplyTo":"20171016175532.73459-1-bmwill@google.com","subject":"[PATCH v4 11/11] Documentation: document Extra Parameters","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-16T17:55:32Z","receivedAt":"2017-10-16T17:56:39Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"From: Jonathan Tan \n\nDocument the server support for Extra Parameters, additional information\nthat the client can send in its first message to the server during a\nGit client-server interaction.\n\nSigned-off-by: Jonathan Tan \nSigned-off-by: Brandon Williams \n---\n Documentation/technical/http-protocol.txt | 8 ++++++\n Documentation/technical/pack-protocol.txt | 43 ++++++++++++++++++++++++++-----\n 2 files changed, 44 insertions(+), 7 deletions(-)\n\ndiff --git a/Documentation/technical/http-protocol.txt b/Documentation/technical/http-protocol.txt\nindex 1c561bdd9..a0e45f288 100644\n--- a/Documentation/technical/http-protocol.txt\n+++ b/Documentation/technical/http-protocol.txt\n@@ -219,6 +219,10 @@ smart server reply:\n S: 003c2cb58b79488a98d2721cea644875a8dd0026b115 refs/tags/v1.0\\n\n S: 003fa3c2e2402b99163d1d59756e5f207ae21cccba4c refs/tags/v1.0^{}\\n\n \n+The client may send Extra Parameters (see\n+Documentation/technical/pack-protocol.txt) as a colon-separated string\n+in the Git-Protocol HTTP header.\n+\n Dumb Server Response\n ^^^^^^^^^^^^^^^^^^^^\n Dumb servers MUST respond with the dumb server reply format.\n@@ -269,7 +273,11 @@ the C locale ordering. The stream SHOULD include the default ref\n named `HEAD` as the first ref. The stream MUST include capability\n declarations behind a NUL on the first ref.\n \n+The returned response contains \"version 1\" if \"version=1\" was sent as an\n+Extra Parameter.\n+\n smart_reply = PKT-LINE(\"# service=$servicename\" LF)\n+\t\t *1(\"version 1\")\n \t\t ref_list\n \t\t \"0000\"\n ref_list = empty_list / non_empty_list\ndiff --git a/Documentation/technical/pack-protocol.txt b/Documentation/technical/pack-protocol.txt\nindex ed1eae8b8..cd31edc91 100644\n--- a/Documentation/technical/pack-protocol.txt\n+++ b/Documentation/technical/pack-protocol.txt\n@@ -39,6 +39,19 @@ communicates with that invoked process over the SSH connection.\n The file:// transport runs the 'upload-pack' or 'receive-pack'\n process locally and communicates with it over a pipe.\n \n+Extra Parameters\n+----------------\n+\n+The protocol provides a mechanism in which clients can send additional\n+information in its first message to the server. These are called \"Extra\n+Parameters\", and are supported by the Git, SSH, and HTTP protocols.\n+\n+Each Extra Parameter takes the form of `=` or ``.\n+\n+Servers that receive any such Extra Parameters MUST ignore all\n+unrecognized keys. Currently, the only Extra Parameter recognized is\n+\"version=1\".\n+\n Git Transport\n -------------\n \n@@ -46,18 +59,25 @@ The Git transport starts off by sending the command and repository\n on the wire using the pkt-line format, followed by a NUL byte and a\n hostname parameter, terminated by a NUL byte.\n \n- 0032git-upload-pack /project.git\\0host=myserver.com\\0\n+ 0033git-upload-pack /project.git\\0host=myserver.com\\0\n+\n+The transport may send Extra Parameters by adding an additional NUL\n+byte, and then adding one or more NUL-terminated strings:\n+\n+ 003egit-upload-pack /project.git\\0host=myserver.com\\0\\0version=1\\0\n \n --\n- git-proto-request = request-command SP pathname NUL [ host-parameter NUL ]\n+ git-proto-request = request-command SP pathname NUL\n+\t\t [ host-parameter NUL ] [ NUL extra-parameters ]\n request-command = \"git-upload-pack\" / \"git-receive-pack\" /\n \t\t \"git-upload-archive\" ; case sensitive\n pathname = *( %x01-ff ) ; exclude NUL\n host-parameter = \"host=\" hostname [ \":\" port ]\n+ extra-parameters = 1*extra-parameter\n+ extra-parameter = 1*( %x01-ff ) NUL\n --\n \n-Only host-parameter is allowed in the git-proto-request. Clients\n-MUST NOT attempt to send additional parameters. It is used for the\n+host-parameter is used for the\n git-daemon name based virtual hosting. See --interpolated-path\n option to git daemon, with the %H/%CH format characters.\n \n@@ -117,6 +137,12 @@ we execute it without the leading '/'.\n \t\t v\n ssh user@example.com \"git-upload-pack '~alice/project.git'\"\n \n+Depending on the value of the `protocol.version` configuration variable,\n+Git may attempt to send Extra Parameters as a colon-separated string in\n+the GIT_PROTOCOL environment variable. This is done only if\n+the `ssh.variant` configuration variable indicates that the ssh command\n+supports passing environment variables as an argument.\n+\n A few things to remember here:\n \n - The \"command name\" is spelled with dash (e.g. git-upload-pack), but\n@@ -137,11 +163,13 @@ Reference Discovery\n -------------------\n \n When the client initially connects the server will immediately respond\n-with a listing of each reference it has (all branches and tags) along\n+with a version number (if \"version=1\" is sent as an Extra Parameter),\n+and a listing of each reference it has (all branches and tags) along\n with the object name that each reference currently points to.\n \n- $ echo -e -n \"0039git-upload-pack /schacon/gitbook.git\\0host=example.com\\0\" |\n+ $ echo -e -n \"0044git-upload-pack /schacon/gitbook.git\\0host=example.com\\0\\0version=1\\0\" |\n nc -v example.com 9418\n+ 000aversion 1\n 00887217a7c7e582c46cec22a130adf4b9d7d950fba0 HEAD\\0multi_ack thin-pack\n \t\tside-band side-band-64k ofs-delta shallow no-progress include-tag\n 00441d3fcd5ced445d1abc402225c0b8a1299641f497 refs/heads/integration\n@@ -165,7 +193,8 @@ immediately after the ref itself, if presented. A conforming server\n MUST peel the ref if it's an annotated tag.\n \n ----\n- advertised-refs = (no-refs / list-of-refs)\n+ advertised-refs = *1(\"version 1\")\n+\t\t (no-refs / list-of-refs)\n \t\t *shallow\n \t\t flush-pkt\n \n-- \n2.15.0.rc0.271.g36b669edcc-goog\n\n"},{"id":"330480","messageId":"20171016212538.GA19335@alpha.vpn.ikke.info","threadId":"46746","inReplyTo":"20171016175532.73459-4-bmwill@google.com","subject":"Re: [PATCH v4 03/11] protocol: introduce protocol extension mechanisms","fromName":"Kevin Daudt","fromEmail":"me@ikke.info","sentAt":"2017-10-16T21:25:38Z","receivedAt":"2017-10-16T21:25:45Z","isPatch":true,"sender":{"key":"me@ikke.info","avatar":"https://avatars.githubusercontent.com/u/135698?v=4"},"body":"On Mon, Oct 16, 2017 at 10:55:24AM -0700, Brandon Williams wrote:\n> Create protocol.{c,h} and provide functions which future servers and\n> clients can use to determine which protocol to use or is being used.\n> \n> Also introduce the 'GIT_PROTOCOL' environment variable which will be\n> used to communicate a colon separated list of keys with optional values\n> to a server. Unknown keys and values must be tolerated. This mechanism\n> is used to communicate which version of the wire protocol a client would\n> like to use with a server.\n> \n> Signed-off-by: Brandon Williams \n> ---\n> Documentation/config.txt | 17 +++++++++++\n> Documentation/git.txt | 6 ++++\n> Makefile | 1 +\n> cache.h | 8 +++++\n> protocol.c | 79 ++++++++++++++++++++++++++++++++++++++++++++++++\n> protocol.h | 33 ++++++++++++++++++++\n> 6 files changed, 144 insertions(+)\n> create mode 100644 protocol.c\n> create mode 100644 protocol.h\n>\n> [...]\n> \n> diff --git a/protocol.h b/protocol.h\n> new file mode 100644\n> index 000000000..1b2bc94a8\n> --- /dev/null\n> +++ b/protocol.h\n> @@ -0,0 +1,33 @@\n> +#ifndef PROTOCOL_H\n> +#define PROTOCOL_H\n> +\n> +enum protocol_version {\n> +\tprotocol_unknown_version = -1,\n> +\tprotocol_v0 = 0,\n> +\tprotocol_v1 = 1,\n> +};\n> +\n> +/*\n> + * Used by a client to determine which protocol version to request be used when\n> + * communicating with a server, reflecting the configured value of the\n> + * 'protocol.version' config. If unconfigured, a value of 'protocol_v0' is\n> + * returned.\n> + */\n\nThe first sentence reads a little weird to me around 'which version to\nrequest be used'. \n"},{"id":"330850","messageId":"20171023212740.qodxzsq5w7rn2r6y@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171016171812.GA4487@google.com","subject":"[PATCH 0/5] Coping with unrecognized ssh wrapper scripts in GIT_SSH","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T21:28:21Z","receivedAt":"2017-10-23T21:28:38Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nBrandon Williams wrote:\n> On 10/03, Jonathan Nieder wrote:\n\n>> What happens if I specify a ssh://host:port/path URL and the SSH\n>> implementation is of 'simple' type?\n>\n> The port would only be sent if your ssh command supported it.\n\nThanks again for this patch. William Yan (cc-ed) ran into this case,\nand based on his experience here are some patches to handle it.\n\nSee patches 3 and 5 for more details.\n\nLonger term, I hope that wrapper script authors start setting\nGIT_SSH_VARIANT based on the behavior they want (e.g. like (*))\ninstead of making us autodetect, but this should be useful as a\nstopgap in the meantime.\n\nThese patches are against bw/protocol-v1, which is in \"next\".\n\nThoughts of all kinds welcome, as always.\n\nSincerely,\nJonathan Nieder (5):\n connect: split git:// setup into a separate function\n connect: split ssh command line options into separate function\n ssh: 'auto' variant to select between 'ssh' and 'simple'\n ssh: 'simple' variant does not support -4/-6\n ssh: 'simple' variant does not support --port\n\n Documentation/config.txt | 17 +--\n connect.c | 275 ++++++++++++++++++++++++++++++-----------------\n t/t5601-clone.sh | 34 ++++--\n t/t5603-clone-dirname.sh | 2 +\n 4 files changed, 214 insertions(+), 114 deletions(-)\n"},{"id":"330851","messageId":"20171023212916.4aarismli6io4ro6@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023212740.qodxzsq5w7rn2r6y@aiede.mtv.corp.google.com","subject":"[PATCH 1/5] connect: split git:// setup into a separate function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T21:29:16Z","receivedAt":"2017-10-23T21:29:25Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"The git_connect function is growing long. Split the\nPROTO_GIT-specific portion to a separate function to make it easier to\nread.\n\nNo functional change intended.\n\nSigned-off-by: Jonathan Nieder \n---\n connect.c | 103 +++++++++++++++++++++++++++++++++++---------------------------\n 1 file changed, 59 insertions(+), 44 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 7fbd396b35..068e70caad 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -850,6 +850,64 @@ static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n \treturn ssh_variant;\n }\n \n+/*\n+ * Open a connection using Git's native protocol.\n+ *\n+ * The caller is responsible for freeing hostandport, but this function may\n+ * modify it (for example, to truncate it to remove the port part).\n+ */\n+static struct child_process *git_connect_git(int fd[2], char *hostandport,\n+\t\t\t\t\t const char *path, const char *prog,\n+\t\t\t\t\t int flags)\n+{\n+\tstruct child_process *conn = &no_fork;\n+\tstruct strbuf request = STRBUF_INIT;\n+\t/*\n+\t * Set up virtual host information based on where we will\n+\t * connect, unless the user has overridden us in\n+\t * the environment.\n+\t */\n+\tchar *target_host = getenv(\"GIT_OVERRIDE_VIRTUAL_HOST\");\n+\tif (target_host)\n+\t\ttarget_host = xstrdup(target_host);\n+\telse\n+\t\ttarget_host = xstrdup(hostandport);\n+\n+\ttransport_check_allowed(\"git\");\n+\n+\t/* These underlying connection commands die() if they\n+\t * cannot connect.\n+\t */\n+\tif (git_use_proxy(hostandport))\n+\t\tconn = git_proxy_connect(fd, hostandport);\n+\telse\n+\t\tgit_tcp_connect(fd, hostandport, flags);\n+\t/*\n+\t * Separate original protocol components prog and path\n+\t * from extended host header with a NUL byte.\n+\t *\n+\t * Note: Do not add any other headers here! Doing so\n+\t * will cause older git-daemon servers to crash.\n+\t */\n+\tstrbuf_addf(&request,\n+\t\t \"%s %s%chost=%s%c\",\n+\t\t prog, path, 0,\n+\t\t target_host, 0);\n+\n+\t/* If using a new version put that stuff here after a second null byte */\n+\tif (get_protocol_version_config() > 0) {\n+\t\tstrbuf_addch(&request, '\\0');\n+\t\tstrbuf_addf(&request, \"version=%d%c\",\n+\t\t\t get_protocol_version_config(), '\\0');\n+\t}\n+\n+\tpacket_write(fd[1], request.buf, request.len);\n+\n+\tfree(target_host);\n+\tstrbuf_release(&request);\n+\treturn conn;\n+}\n+\n /*\n * This returns a dummy child_process if the transport protocol does not\n * need fork(2), or a struct child_process object if it does. Once done,\n@@ -881,50 +939,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tprintf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n \t\tconn = NULL;\n \t} else if (protocol == PROTO_GIT) {\n-\t\tstruct strbuf request = STRBUF_INIT;\n-\t\t/*\n-\t\t * Set up virtual host information based on where we will\n-\t\t * connect, unless the user has overridden us in\n-\t\t * the environment.\n-\t\t */\n-\t\tchar *target_host = getenv(\"GIT_OVERRIDE_VIRTUAL_HOST\");\n-\t\tif (target_host)\n-\t\t\ttarget_host = xstrdup(target_host);\n-\t\telse\n-\t\t\ttarget_host = xstrdup(hostandport);\n-\n-\t\ttransport_check_allowed(\"git\");\n-\n-\t\t/* These underlying connection commands die() if they\n-\t\t * cannot connect.\n-\t\t */\n-\t\tif (git_use_proxy(hostandport))\n-\t\t\tconn = git_proxy_connect(fd, hostandport);\n-\t\telse\n-\t\t\tgit_tcp_connect(fd, hostandport, flags);\n-\t\t/*\n-\t\t * Separate original protocol components prog and path\n-\t\t * from extended host header with a NUL byte.\n-\t\t *\n-\t\t * Note: Do not add any other headers here! Doing so\n-\t\t * will cause older git-daemon servers to crash.\n-\t\t */\n-\t\tstrbuf_addf(&request,\n-\t\t\t \"%s %s%chost=%s%c\",\n-\t\t\t prog, path, 0,\n-\t\t\t target_host, 0);\n-\n-\t\t/* If using a new version put that stuff here after a second null byte */\n-\t\tif (get_protocol_version_config() > 0) {\n-\t\t\tstrbuf_addch(&request, '\\0');\n-\t\t\tstrbuf_addf(&request, \"version=%d%c\",\n-\t\t\t\t get_protocol_version_config(), '\\0');\n-\t\t}\n-\n-\t\tpacket_write(fd[1], request.buf, request.len);\n-\n-\t\tfree(target_host);\n-\t\tstrbuf_release(&request);\n+\t\tconn = git_connect_git(fd, hostandport, path, prog, flags);\n \t} else {\n \t\tstruct strbuf cmd = STRBUF_INIT;\n \t\tconst char *const *var;\n-- \n2.15.0.rc1.287.g2b38de12cc\n\n"},{"id":"330852","messageId":"20171023213016.4eakv4fok2mfjt6e@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023212740.qodxzsq5w7rn2r6y@aiede.mtv.corp.google.com","subject":"[PATCH 2/5] connect: split ssh command line options into separate function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T21:30:16Z","receivedAt":"2017-10-23T21:30:24Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"The git_connect function is growing long. Split the portion that\ndiscovers an ssh command and options it accepts before the service\nname and path to a separate function to make it easier to read.\n\nNo functional change intended.\n\nSigned-off-by: Jonathan Nieder \n---\n connect.c | 116 +++++++++++++++++++++++++++++++++-----------------------------\n 1 file changed, 61 insertions(+), 55 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 068e70caad..77ab6db3bb 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -908,6 +908,65 @@ static struct child_process *git_connect_git(int fd[2], char *hostandport,\n \treturn conn;\n }\n \n+/* Prepare a child_process for use by Git's SSH-tunneled transport. */\n+static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n+\t\t\t const char *port, int flags)\n+{\n+\tconst char *ssh;\n+\tenum ssh_variant variant;\n+\n+\tif (looks_like_command_line_option(ssh_host))\n+\t\tdie(\"strange hostname '%s' blocked\", ssh_host);\n+\n+\tssh = get_ssh_command();\n+\tif (ssh) {\n+\t\tvariant = determine_ssh_variant(ssh, 1);\n+\t} else {\n+\t\t/*\n+\t\t * GIT_SSH is the no-shell version of\n+\t\t * GIT_SSH_COMMAND (and must remain so for\n+\t\t * historical compatibility).\n+\t\t */\n+\t\tconn->use_shell = 0;\n+\n+\t\tssh = getenv(\"GIT_SSH\");\n+\t\tif (!ssh)\n+\t\t\tssh = \"ssh\";\n+\t\tvariant = determine_ssh_variant(ssh, 0);\n+\t}\n+\n+\targv_array_push(&conn->args, ssh);\n+\n+\tif (variant == VARIANT_SSH &&\n+\t get_protocol_version_config() > 0) {\n+\t\targv_array_push(&conn->args, \"-o\");\n+\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n+\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t get_protocol_version_config());\n+\t}\n+\n+\tif (variant != VARIANT_SIMPLE) {\n+\t\tif (flags & CONNECT_IPV4)\n+\t\t\targv_array_push(&conn->args, \"-4\");\n+\t\telse if (flags & CONNECT_IPV6)\n+\t\t\targv_array_push(&conn->args, \"-6\");\n+\t}\n+\n+\tif (variant == VARIANT_TORTOISEPLINK)\n+\t\targv_array_push(&conn->args, \"-batch\");\n+\n+\tif (port && variant != VARIANT_SIMPLE) {\n+\t\tif (variant == VARIANT_SSH)\n+\t\t\targv_array_push(&conn->args, \"-p\");\n+\t\telse\n+\t\t\targv_array_push(&conn->args, \"-P\");\n+\n+\t\targv_array_push(&conn->args, port);\n+\t}\n+\n+\targv_array_push(&conn->args, ssh_host);\n+}\n+\n /*\n * This returns a dummy child_process if the transport protocol does not\n * need fork(2), or a struct child_process object if it does. Once done,\n@@ -961,16 +1020,13 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tconn->use_shell = 1;\n \t\tconn->in = conn->out = -1;\n \t\tif (protocol == PROTO_SSH) {\n-\t\t\tconst char *ssh;\n-\t\t\tenum ssh_variant variant;\n \t\t\tchar *ssh_host = hostandport;\n \t\t\tconst char *port = NULL;\n+\n \t\t\ttransport_check_allowed(\"ssh\");\n \t\t\tget_host_and_port(&ssh_host, &port);\n-\n \t\t\tif (!port)\n \t\t\t\tport = get_port(ssh_host);\n-\n \t\t\tif (flags & CONNECT_DIAG_URL) {\n \t\t\t\tprintf(\"Diag: url=%s\\n\", url ? url : \"NULL\");\n \t\t\t\tprintf(\"Diag: protocol=%s\\n\", prot_name(protocol));\n@@ -984,57 +1040,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tstrbuf_release(&cmd);\n \t\t\t\treturn NULL;\n \t\t\t}\n-\n-\t\t\tif (looks_like_command_line_option(ssh_host))\n-\t\t\t\tdie(\"strange hostname '%s' blocked\", ssh_host);\n-\n-\t\t\tssh = get_ssh_command();\n-\t\t\tif (ssh) {\n-\t\t\t\tvariant = determine_ssh_variant(ssh, 1);\n-\t\t\t} else {\n-\t\t\t\t/*\n-\t\t\t\t * GIT_SSH is the no-shell version of\n-\t\t\t\t * GIT_SSH_COMMAND (and must remain so for\n-\t\t\t\t * historical compatibility).\n-\t\t\t\t */\n-\t\t\t\tconn->use_shell = 0;\n-\n-\t\t\t\tssh = getenv(\"GIT_SSH\");\n-\t\t\t\tif (!ssh)\n-\t\t\t\t\tssh = \"ssh\";\n-\t\t\t\tvariant = determine_ssh_variant(ssh, 0);\n-\t\t\t}\n-\n-\t\t\targv_array_push(&conn->args, ssh);\n-\n-\t\t\tif (variant == VARIANT_SSH &&\n-\t\t\t get_protocol_version_config() > 0) {\n-\t\t\t\targv_array_push(&conn->args, \"-o\");\n-\t\t\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n-\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n-\t\t\t\t\t\t get_protocol_version_config());\n-\t\t\t}\n-\n-\t\t\tif (variant != VARIANT_SIMPLE) {\n-\t\t\t\tif (flags & CONNECT_IPV4)\n-\t\t\t\t\targv_array_push(&conn->args, \"-4\");\n-\t\t\t\telse if (flags & CONNECT_IPV6)\n-\t\t\t\t\targv_array_push(&conn->args, \"-6\");\n-\t\t\t}\n-\n-\t\t\tif (variant == VARIANT_TORTOISEPLINK)\n-\t\t\t\targv_array_push(&conn->args, \"-batch\");\n-\n-\t\t\tif (port && variant != VARIANT_SIMPLE) {\n-\t\t\t\tif (variant == VARIANT_SSH)\n-\t\t\t\t\targv_array_push(&conn->args, \"-p\");\n-\t\t\t\telse\n-\t\t\t\t\targv_array_push(&conn->args, \"-P\");\n-\n-\t\t\t\targv_array_push(&conn->args, port);\n-\t\t\t}\n-\n-\t\t\targv_array_push(&conn->args, ssh_host);\n+\t\t\tfill_ssh_args(conn, ssh_host, port, flags);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\n \t\t\tif (get_protocol_version_config() > 0) {\n-- \n2.15.0.rc1.287.g2b38de12cc\n\n"},{"id":"330853","messageId":"20171023213159.eitrjrqrh277advm@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023212740.qodxzsq5w7rn2r6y@aiede.mtv.corp.google.com","subject":"[PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T21:31:59Z","receivedAt":"2017-10-23T21:32:07Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Android's \"repo\" tool is a tool for managing a large codebase\nconsisting of multiple smaller repositories, similar to Git's\nsubmodule feature. Starting with Git 94b8ae5a (ssh: introduce a\n'simple' ssh variant, 2017-10-16), users noticed that it stopped\nhandling the port in ssh:// URLs.\n\nThe cause: when it encounters ssh:// URLs, repo pre-connects to the\nserver and sets GIT_SSH to a helper \".repo/repo/git_ssh\" that reuses\nthat connection. Before 94b8ae5a, the helper was assumed to support\nOpenSSH options for lack of a better guess and got passed a -p option\nto set the port. After that patch, it uses the new default of a\nsimple helper that does not accept an option to set the port.\n\nThe next release of \"repo\" will set GIT_SSH_VARIANT to \"ssh\" to avoid\nthat. But users of old versions and of other similar GIT_SSH\nimplementations would not get the benefit of that fix.\n\nSo update the default to use OpenSSH options again, with a twist. As\nobserved in 94b8ae5a, we cannot assume that $GIT_SSH always handles\nOpenSSH options: common helpers such as travis-ci's dpl[*] are\nconfigured using GIT_SSH and do not accept OpenSSH options. So make\nthe default a new variant \"auto\", with the following behavior:\n\n 1. First, check whether $GIT_SSH supports OpenSSH options by running\n\n\t$GIT_SSH -G \n\n This returns status 0 and prints configuration in OpenSSH if it\n recognizes all and returns status 255 if it encounters\n an unrecognized option. A wrapper script like\n\n\texec ssh -- \"$@\"\n\n would fail with\n\n\tssh: Could not resolve hostname -g: Name or service not known\n\n , correctly reflecting that it does not support OpenSSH options.\n\n 2. Based on the result from step (1), behave like \"ssh\" (if it\n succeeded) or \"simple\" (if it failed).\n\nThis way, the default ssh variant for unrecognized commands can handle\nboth the repo and dpl cases as intended.\n\nIf the GIT_SSH command name is recognized (e.g., \"ssh\") then continue\nto use that variant directly without the autodetection step (1), as\nbefore.\n\n[*] https://github.com/travis-ci/dpl/blob/6c3fddfda1f2a85944c544446b068bac0a77c049/lib/dpl/provider.rb#L215\n\nReported-by: William Yan \nSigned-off-by: Jonathan Nieder \n---\nrepo fix: https://gerrit-review.googlesource.com/c/git-repo/+/134950\n\n Documentation/config.txt | 17 ++++++++----\n connect.c | 72 ++++++++++++++++++++++++++++++++----------------\n t/t5601-clone.sh | 14 ++++++++++\n 3 files changed, 73 insertions(+), 30 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 0460af37e2..4a16b324f0 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2083,14 +2083,19 @@ visited as a result of a redirection do not participate in matching.\n ssh.variant::\n \tDepending on the value of the environment variables `GIT_SSH` or\n \t`GIT_SSH_COMMAND`, or the config setting `core.sshCommand`, Git\n-\tauto-detects whether to adjust its command-line parameters for use\n-\twith ssh (OpenSSH), plink or tortoiseplink, as opposed to the default\n-\t(simple).\n+\tauto-detects whether to pass command-line parameters for use\n+\twith a simple wrapper script (simple), OpenSSH (ssh), plink, or\n+\ttortoiseplink.\n++\n+The default is `auto`, which means to auto-detect whether the ssh command\n+implements OpenSSH options using the `-G` (print configuration) option.\n+If the ssh command supports OpenSSH options, it then behaves like `ssh`;\n+otherwise, it behaves like `simple`.\n +\n The config variable `ssh.variant` can be set to override this auto-detection;\n-valid values are `ssh`, `simple`, `plink`, `putty` or `tortoiseplink`. Any\n-other value will be treated as normal ssh. This setting can be overridden via\n-the environment variable `GIT_SSH_VARIANT`.\n+valid values are `ssh`, `simple`, `plink`, `putty`, `tortoiseplink`, and\n+`auto`. Any other value will be treated as normal ssh. This setting can be\n+overridden via the environment variable `GIT_SSH_VARIANT`.\n +\n The current command-line parameters used for each variant are as\n follows:\ndiff --git a/connect.c b/connect.c\nindex 77ab6db3bb..2dc9554b30 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -777,6 +777,7 @@ static const char *get_ssh_command(void)\n }\n \n enum ssh_variant {\n+\tVARIANT_AUTO,\n \tVARIANT_SIMPLE,\n \tVARIANT_SSH,\n \tVARIANT_PLINK,\n@@ -791,7 +792,9 @@ static int override_ssh_variant(enum ssh_variant *ssh_variant)\n \tif (!variant && git_config_get_string_const(\"ssh.variant\", &variant))\n \t\treturn 0;\n \n-\tif (!strcmp(variant, \"plink\"))\n+\tif (!strcmp(variant, \"auto\"))\n+\t\t*ssh_variant = VARIANT_AUTO;\n+\telse if (!strcmp(variant, \"plink\"))\n \t\t*ssh_variant = VARIANT_PLINK;\n \telse if (!strcmp(variant, \"putty\"))\n \t\t*ssh_variant = VARIANT_PUTTY;\n@@ -808,7 +811,7 @@ static int override_ssh_variant(enum ssh_variant *ssh_variant)\n static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n \t\t\t\t\t int is_cmdline)\n {\n-\tenum ssh_variant ssh_variant = VARIANT_SIMPLE;\n+\tenum ssh_variant ssh_variant = VARIANT_AUTO;\n \tconst char *variant;\n \tchar *p = NULL;\n \n@@ -908,6 +911,38 @@ static struct child_process *git_connect_git(int fd[2], char *hostandport,\n \treturn conn;\n }\n \n+static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n+\t\t\t enum ssh_variant variant, const char *port,\n+\t\t\t int flags)\n+{\n+\tif (variant == VARIANT_SSH &&\n+\t get_protocol_version_config() > 0) {\n+\t\targv_array_push(args, \"-o\");\n+\t\targv_array_push(args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n+\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t get_protocol_version_config());\n+\t}\n+\n+\tif (variant != VARIANT_SIMPLE) {\n+\t\tif (flags & CONNECT_IPV4)\n+\t\t\targv_array_push(args, \"-4\");\n+\t\telse if (flags & CONNECT_IPV6)\n+\t\t\targv_array_push(args, \"-6\");\n+\t}\n+\n+\tif (variant == VARIANT_TORTOISEPLINK)\n+\t\targv_array_push(args, \"-batch\");\n+\n+\tif (port && variant != VARIANT_SIMPLE) {\n+\t\tif (variant == VARIANT_SSH)\n+\t\t\targv_array_push(args, \"-p\");\n+\t\telse\n+\t\t\targv_array_push(args, \"-P\");\n+\n+\t\targv_array_push(args, port);\n+\t}\n+}\n+\n /* Prepare a child_process for use by Git's SSH-tunneled transport. */\n static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n \t\t\t const char *port, int flags)\n@@ -937,33 +972,22 @@ static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n \n \targv_array_push(&conn->args, ssh);\n \n-\tif (variant == VARIANT_SSH &&\n-\t get_protocol_version_config() > 0) {\n-\t\targv_array_push(&conn->args, \"-o\");\n-\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n-\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n-\t\t\t\t get_protocol_version_config());\n-\t}\n-\n-\tif (variant != VARIANT_SIMPLE) {\n-\t\tif (flags & CONNECT_IPV4)\n-\t\t\targv_array_push(&conn->args, \"-4\");\n-\t\telse if (flags & CONNECT_IPV6)\n-\t\t\targv_array_push(&conn->args, \"-6\");\n-\t}\n+\tif (variant == VARIANT_AUTO) {\n+\t\tstruct child_process detect = CHILD_PROCESS_INIT;\n \n-\tif (variant == VARIANT_TORTOISEPLINK)\n-\t\targv_array_push(&conn->args, \"-batch\");\n+\t\tdetect.use_shell = conn->use_shell;\n+\t\tdetect.no_stdin = detect.no_stdout = detect.no_stderr = 1;\n \n-\tif (port && variant != VARIANT_SIMPLE) {\n-\t\tif (variant == VARIANT_SSH)\n-\t\t\targv_array_push(&conn->args, \"-p\");\n-\t\telse\n-\t\t\targv_array_push(&conn->args, \"-P\");\n+\t\targv_array_push(&detect.args, ssh);\n+\t\targv_array_push(&detect.args, \"-G\");\n+\t\tpush_ssh_options(&detect.args, &detect.env_array,\n+\t\t\t\t VARIANT_SSH, port, flags);\n+\t\targv_array_push(&detect.args, ssh_host);\n \n-\t\targv_array_push(&conn->args, port);\n+\t\tvariant = run_command(&detect) ? VARIANT_SIMPLE : VARIANT_SSH;\n \t}\n \n+\tpush_ssh_options(&conn->args, &conn->env_array, variant, port, flags);\n \targv_array_push(&conn->args, ssh_host);\n }\n \ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 86811a0c35..fd94dd40d2 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -384,6 +384,20 @@ test_expect_success 'uplink is treated as simple' '\n \texpect_ssh myhost src\n '\n \n+test_expect_success 'OpenSSH-like uplink is treated as ssh' '\n+\twrite_script \"$TRASH_DIRECTORY/uplink\" <<-EOF &&\n+\tif test \"\\$1\" = \"-G\"\n+\tthen\n+\t\texit 0\n+\tfi &&\n+\texec \"\\$TRASH_DIRECTORY/ssh$X\" \"\\$@\"\n+\tEOF\n+\tGIT_SSH=\"$TRASH_DIRECTORY/uplink\" &&\n+\texport GIT_SSH &&\n+\tgit clone \"[myhost:123]:src\" ssh-bracket-clone-sshlike-uplink &&\n+\texpect_ssh \"-p 123\" myhost src\n+'\n+\n test_expect_success 'plink is treated specially (as putty)' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/plink\" &&\n \tgit clone \"[myhost:123]:src\" ssh-bracket-clone-plink-0 &&\n-- \n2.15.0.rc1.287.g2b38de12cc\n\n"},{"id":"330854","messageId":"20171023213235.btjysimp6a6hdzql@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023212740.qodxzsq5w7rn2r6y@aiede.mtv.corp.google.com","subject":"[PATCH 4/5] ssh: 'simple' variant does not support -4/-6","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T21:32:35Z","receivedAt":"2017-10-23T21:32:42Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"If the user passes -4/--ipv4 or -6/--ipv6 to \"git fetch\" or \"git push\"\nand the ssh command configured with GIT_SSH does not support such a\nsetting, error out instead of ignoring the option and continuing.\n\nSigned-off-by: Jonathan Nieder \n---\n connect.c | 25 ++++++++++++++++++++++---\n t/t5601-clone.sh | 12 ++++++------\n 2 files changed, 28 insertions(+), 9 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 2dc9554b30..a1c0ba1b3a 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -923,11 +923,30 @@ static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n \t\t\t\t get_protocol_version_config());\n \t}\n \n-\tif (variant != VARIANT_SIMPLE) {\n-\t\tif (flags & CONNECT_IPV4)\n+\tif (flags & CONNECT_IPV4) {\n+\t\tswitch (variant) {\n+\t\tcase VARIANT_AUTO:\n+\t\t\tBUG(\"VARIANT_AUTO passed to push_ssh_options\");\n+\t\tcase VARIANT_SIMPLE:\n+\t\t\tdie(\"ssh variant 'simple' does not support -4\");\n+\t\tcase VARIANT_SSH:\n+\t\tcase VARIANT_PLINK:\n+\t\tcase VARIANT_PUTTY:\n+\t\tcase VARIANT_TORTOISEPLINK:\n \t\t\targv_array_push(args, \"-4\");\n-\t\telse if (flags & CONNECT_IPV6)\n+\t\t}\n+\t} else if (flags & CONNECT_IPV6) {\n+\t\tswitch (variant) {\n+\t\tcase VARIANT_AUTO:\n+\t\t\tBUG(\"VARIANT_AUTO passed to push_ssh_options\");\n+\t\tcase VARIANT_SIMPLE:\n+\t\t\tdie(\"ssh variant 'simple' does not support -6\");\n+\t\tcase VARIANT_SSH:\n+\t\tcase VARIANT_PLINK:\n+\t\tcase VARIANT_PUTTY:\n+\t\tcase VARIANT_TORTOISEPLINK:\n \t\t\targv_array_push(args, \"-6\");\n+\t\t}\n \t}\n \n \tif (variant == VARIANT_TORTOISEPLINK)\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex fd94dd40d2..a672fc1e08 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -367,15 +367,15 @@ test_expect_success 'OpenSSH variant passes -4' '\n \texpect_ssh \"-4 -p 123\" myhost src\n '\n \n-test_expect_success 'variant can be overriden' '\n-\tgit -c ssh.variant=simple clone -4 \"[myhost:123]:src\" ssh-simple-clone &&\n-\texpect_ssh myhost src\n+test_expect_success 'variant can be overridden' '\n+\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/putty\" &&\n+\tgit -c ssh.variant=putty clone -4 \"[myhost:123]:src\" ssh-putty-clone &&\n+\texpect_ssh \"-4 -P 123\" myhost src\n '\n \n-test_expect_success 'simple is treated as simple' '\n+test_expect_success 'simple does not support -4/-6' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n-\tgit clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple &&\n-\texpect_ssh myhost src\n+\ttest_must_fail git clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple\n '\n \n test_expect_success 'uplink is treated as simple' '\n-- \n2.15.0.rc1.287.g2b38de12cc\n\n"},{"id":"330855","messageId":"20171023213324.v3tm5eiir4gmkqbz@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023212740.qodxzsq5w7rn2r6y@aiede.mtv.corp.google.com","subject":"[PATCH 5/5] ssh: 'simple' variant does not support --port","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T21:33:24Z","receivedAt":"2017-10-23T21:33:34Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"When trying to connect to an ssh:// URL with port explicitly specified\nand the ssh command configured with GIT_SSH does not support such a\nsetting, it is less confusing to error out than to silently suppress\nthe port setting and continue.\n\nThis requires updating the GIT_SSH setting in t5603-clone-dirname.sh.\nThat test is about the directory name produced when cloning various\nURLs. It uses an ssh wrapper that ignores all its arguments but does\nnot declare that it supports a port argument; update it to set\nGIT_SSH_VARIANT=ssh to do so. (Real-life ssh wrappers that pass a\nport argument to OpenSSH would also support -G and would not require\nsuch an update.)\n\nReported-by: William Yan \nSigned-off-by: Jonathan Nieder \n---\nThat's the end of the series. Thanks for reading.\n\n connect.c | 15 ++++++++++++---\n t/t5601-clone.sh | 10 ++++++++--\n t/t5603-clone-dirname.sh | 2 ++\n 3 files changed, 22 insertions(+), 5 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex a1c0ba1b3a..98f2d9ce57 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -952,11 +952,20 @@ static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n \tif (variant == VARIANT_TORTOISEPLINK)\n \t\targv_array_push(args, \"-batch\");\n \n-\tif (port && variant != VARIANT_SIMPLE) {\n-\t\tif (variant == VARIANT_SSH)\n+\tif (port) {\n+\t\tswitch (variant) {\n+\t\tcase VARIANT_AUTO:\n+\t\t\tBUG(\"VARIANT_AUTO passed to push_ssh_options\");\n+\t\tcase VARIANT_SIMPLE:\n+\t\t\tdie(\"ssh variant 'simple' does not support setting port\");\n+\t\tcase VARIANT_SSH:\n \t\t\targv_array_push(args, \"-p\");\n-\t\telse\n+\t\t\tbreak;\n+\t\tcase VARIANT_PLINK:\n+\t\tcase VARIANT_PUTTY:\n+\t\tcase VARIANT_TORTOISEPLINK:\n \t\t\targv_array_push(args, \"-P\");\n+\t\t}\n \n \t\targv_array_push(args, port);\n \t}\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex a672fc1e08..11fa516997 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -375,12 +375,18 @@ test_expect_success 'variant can be overridden' '\n \n test_expect_success 'simple does not support -4/-6' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n-\ttest_must_fail git clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple\n+\ttest_must_fail git clone -4 \"myhost:src\" ssh-4-clone-simple\n+'\n+\n+test_expect_success 'simple does not support port' '\n+\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n+\ttest_must_fail git clone \"[myhost:123]:src\" ssh-bracket-clone-simple\n '\n \n test_expect_success 'uplink is treated as simple' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/uplink\" &&\n-\tgit clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n+\ttest_must_fail git clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n+\tgit clone \"myhost:src\" ssh-clone-uplink &&\n \texpect_ssh myhost src\n '\n \ndiff --git a/t/t5603-clone-dirname.sh b/t/t5603-clone-dirname.sh\nindex d5af758129..13b5e5eb9b 100755\n--- a/t/t5603-clone-dirname.sh\n+++ b/t/t5603-clone-dirname.sh\n@@ -11,7 +11,9 @@ test_expect_success 'setup ssh wrapper' '\n \tgit upload-pack \"$TRASH_DIRECTORY\"\n \tEOF\n \tGIT_SSH=\"$TRASH_DIRECTORY/ssh-wrapper\" &&\n+\tGIT_SSH_VARIANT=ssh &&\n \texport GIT_SSH &&\n+\texport GIT_SSH_VARIANT &&\n \texport TRASH_DIRECTORY\n '\n \n-- \n2.15.0.rc1.287.g2b38de12cc\n\n"},{"id":"330856","messageId":"CAGZ79kYTbWNw04n7Dmv5_1gPR51jgUFgXQn9KYQajG4WEsyfMQ@mail.gmail.com","threadId":"46746","inReplyTo":"20171023213016.4eakv4fok2mfjt6e@aiede.mtv.corp.google.com","subject":"Re: [PATCH 2/5] connect: split ssh command line options into separate function","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-23T21:48:21Z","receivedAt":"2017-10-23T21:48:28Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 23, 2017 at 2:30 PM, Jonathan Nieder wrote:\n> The git_connect function is growing long. Split the portion that\n> discovers an ssh command and options it accepts before the service\n> name and path to a separate function to make it easier to read.\n>\n> No functional change intended.\n>\n> Signed-off-by: Jonathan Nieder \n\nand\nReviewed-by: Stefan Beller \n\nThanks,\nStefan\n"},{"id":"330858","messageId":"CAGZ79kYwARXNWRS4AgwTP7peZiWiwCBvWFiEr9TbpbWjgysfZA@mail.gmail.com","threadId":"46746","inReplyTo":"20171023212916.4aarismli6io4ro6@aiede.mtv.corp.google.com","subject":"Re: [PATCH 1/5] connect: split git:// setup into a separate function","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-23T22:16:13Z","receivedAt":"2017-10-23T22:16:19Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 23, 2017 at 2:29 PM, Jonathan Nieder wrote:\n> The git_connect function is growing long. Split the\n> PROTO_GIT-specific portion to a separate function to make it easier to\n> read.\n>\n> No functional change intended.\n>\n> Signed-off-by: Jonathan Nieder \n\nThis also looks good to me.\n\nunrelated:\nPatch 2 was very easy to review using \"log -p -w --color-moved\",\nthis one however was not. This is because -w caused the diff machinery\nto generate a completely different diff. (Not showing the new function\ncompletely but some weird function header trickery. The white space\nmangled output is below; most of it was colored \"moved\")\n\nI had to have -w as otherwise --color-moved would not work,\nso maybe we want to have an option to ignore white space for the\nsake of move detection only, not affecting the diff in general;\nmaybe '--ignore-white-space-in-move-detection'?\n\nI think once this option is given, all we have to do is pay attention to\nthis option in diff.c#moved_entry_cmp/next_byte, which is best built\non top of Peffs recent fixes origin/jk/diff-color-moved-fix.\nWould that be of interest for people?\n\nThanks,\nStefan\n\ndiff --git a/connect.c b/connect.c\nindex 7fbd396b35..068e70caad 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -851,36 +851,16 @@ static enum ssh_variant\ndetermine_ssh_variant(const char *ssh_command,\n }\n\n /*\n- * This returns a dummy child_process if the transport protocol does not\n- * need fork(2), or a struct child_process object if it does. Once done,\n- * finish the connection with finish_connect() with the value returned from\n- * this function (it is safe to call finish_connect() with NULL to support\n- * the former case).\n+ * Open a connection using Git's native protocol.\n *\n- * If it returns, the connect is successful; it just dies on errors (this\n- * will hopefully be changed in a libification effort, to return NULL when\n- * the connection failed).\n+ * The caller is responsible for freeing hostandport, but this function may\n+ * modify it (for example, to truncate it to remove the port part).\n */\n-struct child_process *git_connect(int fd[2], const char *url,\n- const char *prog, int flags)\n+static struct child_process *git_connect_git(int fd[2], char *hostandport,\n+ const char *path, const\nchar *prog,\n+ int flags)\n {\n- char *hostandport, *path;\n struct child_process *conn = &no_fork;\n- enum protocol protocol;\n-\n- /* Without this we cannot rely on waitpid() to tell\n- * what happened to our children.\n- */\n- signal(SIGCHLD, SIG_DFL);\n-\n- protocol = parse_connect_url(url, &hostandport, &path);\n- if ((flags & CONNECT_DIAG_URL) && (protocol != PROTO_SSH)) {\n- printf(\"Diag: url=%s\\n\", url ? url : \"NULL\");\n- printf(\"Diag: protocol=%s\\n\", prot_name(protocol));\n- printf(\"Diag: hostandport=%s\\n\", hostandport ?\nhostandport : \"NULL\");\n- printf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n- conn = NULL;\n- } else if (protocol == PROTO_GIT) {\n struct strbuf request = STRBUF_INIT;\n /*\n * Set up virtual host information based on where we will\n@@ -925,6 +905,41 @@ struct child_process *git_connect(int fd[2],\nconst char *url,\n\n free(target_host);\n strbuf_release(&request);\n+ return conn;\n+}\n+\n+/*\n+ * This returns a dummy child_process if the transport protocol does not\n+ * need fork(2), or a struct child_process object if it does. Once done,\n+ * finish the connection with finish_connect() with the value returned from\n+ * this function (it is safe to call finish_connect() with NULL to support\n+ * the former case).\n+ *\n+ * If it returns, the connect is successful; it just dies on errors (this\n+ * will hopefully be changed in a libification effort, to return NULL when\n+ * the connection failed).\n+ */\n+struct child_process *git_connect(int fd[2], const char *url,\n+ const char *prog, int flags)\n+{\n+ char *hostandport, *path;\n+ struct child_process *conn = &no_fork;\n+ enum protocol protocol;\n+\n+ /* Without this we cannot rely on waitpid() to tell\n+ * what happened to our children.\n+ */\n+ signal(SIGCHLD, SIG_DFL);\n+\n+ protocol = parse_connect_url(url, &hostandport, &path);\n+ if ((flags & CONNECT_DIAG_URL) && (protocol != PROTO_SSH)) {\n+ printf(\"Diag: url=%s\\n\", url ? url : \"NULL\");\n+ printf(\"Diag: protocol=%s\\n\", prot_name(protocol));\n+ printf(\"Diag: hostandport=%s\\n\", hostandport ?\nhostandport : \"NULL\");\n+ printf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n+ conn = NULL;\n+ } else if (protocol == PROTO_GIT) {\n+ conn = git_connect_git(fd, hostandport, path, prog, flags);\n } else {\n struct strbuf cmd = STRBUF_INIT;\n const char *const *var;\n"},{"id":"330859","messageId":"20171023151929.67165aea67353e5c24a15229@google.com","threadId":"46746","inReplyTo":"20171023213159.eitrjrqrh277advm@aiede.mtv.corp.google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-23T22:19:29Z","receivedAt":"2017-10-23T22:19:36Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, 23 Oct 2017 14:31:59 -0700\nJonathan Nieder wrote:\n\n> @@ -2083,14 +2083,19 @@ visited as a result of a redirection do not participate in matching.\n> ssh.variant::\n> \tDepending on the value of the environment variables `GIT_SSH` or\n> \t`GIT_SSH_COMMAND`, or the config setting `core.sshCommand`, Git\n> -\tauto-detects whether to adjust its command-line parameters for use\n> -\twith ssh (OpenSSH), plink or tortoiseplink, as opposed to the default\n> -\t(simple).\n> +\tauto-detects whether to pass command-line parameters for use\n> +\twith a simple wrapper script (simple), OpenSSH (ssh), plink, or\n> +\ttortoiseplink.\n> ++\n> +The default is `auto`, which means to auto-detect whether the ssh command\n> +implements OpenSSH options using the `-G` (print configuration) option.\n> +If the ssh command supports OpenSSH options, it then behaves like `ssh`;\n> +otherwise, it behaves like `simple`.\n> +\n> The config variable `ssh.variant` can be set to override this auto-detection;\n> -valid values are `ssh`, `simple`, `plink`, `putty` or `tortoiseplink`. Any\n> -other value will be treated as normal ssh. This setting can be overridden via\n> -the environment variable `GIT_SSH_VARIANT`.\n> +valid values are `ssh`, `simple`, `plink`, `putty`, `tortoiseplink`, and\n> +`auto`. Any other value will be treated as normal ssh. This setting can be\n> +overridden via the environment variable `GIT_SSH_VARIANT`.\n\nThe new documentation seems to imply that setting ssh.variant (or\nGIT_SSH_VARIANT) to \"auto\" is equivalent to not setting it at all, but\nlooking at the code, it doesn't seem to be the case (not setting it at\nall invokes checking the first word of core.sshCommand, and only uses\nVARIANT_AUTO if that check is inconclusive, whereas setting\nssh.variant=auto skips the core.sshCommand check entirely).\n\nMaybe document ssh.variant as follows:\n\n If unset, Git will determine the command-line arguments to use based\n on the basename of the configured SSH command (through the\n environment variable `GIT_SSH` or `GIT_SSH_COMMAND`, or the config\n setting `core.sshCommand`). If the basename is unrecognized, Git\n will attempt to detect support of OpenSSH options by first invoking\n the configured SSH command with the `-G` (print configuration) flag,\n and will subsequently use OpenSSH options (upon success) or no\n options besides the host (upon failure).\n\n If set, Git will not do any auto-detection based on the basename of\n the configured SSH command. This can be set to `ssh` (OpenSSH\n options), `plink`, `putty`, `tortoiseplink`, `simple` (no options\n besides the host), or `auto` (the detection with `-G` as described\n above). If set to any other value, Git behaves as if this is set to\n `ssh`.\n\n(Patches 1, 2, 4, and 5 seem fine to me.)\n"},{"id":"330860","messageId":"CAGZ79kZTjUvcq_hKHCqTDoaBxt2x+9XcqYc6ao1bhcET2SM-PQ@mail.gmail.com","threadId":"46746","inReplyTo":"20171023213159.eitrjrqrh277advm@aiede.mtv.corp.google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-23T22:33:11Z","receivedAt":"2017-10-23T22:33:17Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 23, 2017 at 2:31 PM, Jonathan Nieder wrote:\n\n> 1. First, check whether $GIT_SSH supports OpenSSH options by running\n>\n> $GIT_SSH -G \n>\n> This returns status 0 and prints configuration in OpenSSH if it\n> recognizes all and returns status 255 if it encounters\n> an unrecognized option. A wrapper script like\n>\n> exec ssh -- \"$@\"\n>\n> would fail with\n>\n> ssh: Could not resolve hostname -g: Name or service not known\n\ncapital -G?\n\n\n> - if (variant == VARIANT_TORTOISEPLINK)\n> - argv_array_push(&conn->args, \"-batch\");\n> + detect.use_shell = conn->use_shell;\n\nWhy do we have to use a shell for evaluation of this\ntest balloon?\n\n> + detect.no_stdin = detect.no_stdout = detect.no_stderr = 1;\n\nokay.\n\n...\n> + argv_array_push(&detect.args, \"-G\");\n...\n> + variant = run_command(&detect) ? VARIANT_SIMPLE : VARIANT_SSH;\n\nWhat if we have a VARIANT_SIMPLE, that doesn't care about -G\nbut just connects to the remote host (keeping the session open), do we need\nto kill it after some time to have run_command return eventually?\n\nOr can we give a command to be executed remotely (e.g. 'false') that\nwe know returns quickly?\n\n> '\n>\n> +test_expect_success 'OpenSSH-like uplink is treated as ssh' '\n> + write_script \"$TRASH_DIRECTORY/uplink\" <<-EOF &&\n> + if test \"\\$1\" = \"-G\"\n\nReading this test (and the commit message) I realize, we do care\nabout the order of options, so this is fine.\n"},{"id":"330861","messageId":"CAGZ79kYzaTDaLszx0pGK+dOG3ni7NpeWOf7sbo37X+pE59F41Q@mail.gmail.com","threadId":"46746","inReplyTo":"20171023213324.v3tm5eiir4gmkqbz@aiede.mtv.corp.google.com","subject":"Re: [PATCH 5/5] ssh: 'simple' variant does not support --port","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-23T22:37:05Z","receivedAt":"2017-10-23T22:37:11Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 23, 2017 at 2:33 PM, Jonathan Nieder wrote:\n> When trying to connect to an ssh:// URL with port explicitly specified\n> and the ssh command configured with GIT_SSH does not support such a\n> setting, it is less confusing to error out than to silently suppress\n> the port setting and continue.\n>\n> This requires updating the GIT_SSH setting in t5603-clone-dirname.sh.\n> That test is about the directory name produced when cloning various\n> URLs. It uses an ssh wrapper that ignores all its arguments but does\n> not declare that it supports a port argument; update it to set\n> GIT_SSH_VARIANT=ssh to do so. (Real-life ssh wrappers that pass a\n> port argument to OpenSSH would also support -G and would not require\n> such an update.)\n>\n> Reported-by: William Yan \n> Signed-off-by: Jonathan Nieder \n> ---\n> That's the end of the series. Thanks for reading.\n\nPatches 4 & 5 look good to me,\n\nThanks,\nStefan\n"},{"id":"330862","messageId":"20171023224310.o7ftwmbr7n74vvc6@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023151929.67165aea67353e5c24a15229@google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T22:43:10Z","receivedAt":"2017-10-23T22:43:19Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nJonathan Tan wrote:\n\n> The new documentation seems to imply that setting ssh.variant (or\n> GIT_SSH_VARIANT) to \"auto\" is equivalent to not setting it at all, but\n> looking at the code, it doesn't seem to be the case (not setting it at\n> all invokes checking the first word of core.sshCommand, and only uses\n> VARIANT_AUTO if that check is inconclusive, whereas setting\n> ssh.variant=auto skips the core.sshCommand check entirely).\n>\n> Maybe document ssh.variant as follows:\n>\n> If unset, Git will determine the command-line arguments to use based\n> on the basename of the configured SSH command (through the\n> environment variable `GIT_SSH` or `GIT_SSH_COMMAND`, or the config\n> setting `core.sshCommand`). If the basename is unrecognized, Git\n> will attempt to detect support of OpenSSH options by first invoking\n> the configured SSH command with the `-G` (print configuration) flag,\n> and will subsequently use OpenSSH options (upon success) or no\n> options besides the host (upon failure).\n>\n> If set, Git will not do any auto-detection based on the basename of\n> the configured SSH command. This can be set to `ssh` (OpenSSH\n> options), `plink`, `putty`, `tortoiseplink`, `simple` (no options\n> besides the host), or `auto` (the detection with `-G` as described\n> above). If set to any other value, Git behaves as if this is set to\n> `ssh`.\n\nGood point. Brandon noticed something similar as well.\n\nSeparately from how to document it, what do you think a good behavior\nwould be? Should the \"auto\" configuration trigger command line based\ndetection just like no configuration at all? Should the \"auto\" value\nfor configuration be removed and that behavior restricted to the\nno-configuration case?\n\nI'm tempted to go with the former, which would look like the following.\nWhat do you think?\n\nIf this looks good, I can reroll in a moment.\n\ndiff --git i/Documentation/config.txt w/Documentation/config.txt\nindex 4a16b324f0..6dffa4aa3d 100644\n--- i/Documentation/config.txt\n+++ w/Documentation/config.txt\n@@ -2081,20 +2081,21 @@ matched against are those given directly to Git commands. This means any URLs\n visited as a result of a redirection do not participate in matching.\n \n ssh.variant::\n-\tDepending on the value of the environment variables `GIT_SSH` or\n-\t`GIT_SSH_COMMAND`, or the config setting `core.sshCommand`, Git\n-\tauto-detects whether to pass command-line parameters for use\n-\twith a simple wrapper script (simple), OpenSSH (ssh), plink, or\n-\ttortoiseplink.\n-+\n-The default is `auto`, which means to auto-detect whether the ssh command\n-implements OpenSSH options using the `-G` (print configuration) option.\n-If the ssh command supports OpenSSH options, it then behaves like `ssh`;\n-otherwise, it behaves like `simple`.\n-+\n-The config variable `ssh.variant` can be set to override this auto-detection;\n-valid values are `ssh`, `simple`, `plink`, `putty`, `tortoiseplink`, and\n-`auto`. Any other value will be treated as normal ssh. This setting can be\n+\tBy default, Git determines the command line arguments to use\n+\tbased on the basename of the configured SSH command (configured\n+\tusing the environment variable `GIT_SSH` or `GIT_SSH_COMMAND` or\n+\tthe config setting `core.sshCommand`). If the basename is\n+\tunrecognized, Git will attempt to detect support of OpenSSH\n+\toptions by first invoking the configured SSH command with the\n+\t`-G` (print configuration) option and will subsequently use\n+\tOpenSSH options (if that is successful) or no options besides\n+\tthe host and remote command (if it fails).\n++\n+The config variable `ssh.variant` can be set to override this detection:\n+valid values are `ssh` (to use OpenSSH options), `plink`, `putty`,\n+`tortoiseplink`, `simple` (no options except the host and remote command).\n+The default auto-detection can be explicitly requested using the value\n+`auto`. Any other value is treated as `ssh`. This setting can also be\n overridden via the environment variable `GIT_SSH_VARIANT`.\n +\n The current command-line parameters used for each variant are as\ndiff --git i/connect.c w/connect.c\nindex 98f2d9ce57..06bcd3981e 100644\n--- i/connect.c\n+++ w/connect.c\n@@ -785,12 +785,12 @@ enum ssh_variant {\n \tVARIANT_TORTOISEPLINK,\n };\n \n-static int override_ssh_variant(enum ssh_variant *ssh_variant)\n+static void override_ssh_variant(enum ssh_variant *ssh_variant)\n {\n \tconst char *variant = getenv(\"GIT_SSH_VARIANT\");\n \n \tif (!variant && git_config_get_string_const(\"ssh.variant\", &variant))\n-\t\treturn 0;\n+\t\treturn;\n \n \tif (!strcmp(variant, \"auto\"))\n \t\t*ssh_variant = VARIANT_AUTO;\n@@ -804,8 +804,6 @@ static int override_ssh_variant(enum ssh_variant *ssh_variant)\n \t\t*ssh_variant = VARIANT_SIMPLE;\n \telse\n \t\t*ssh_variant = VARIANT_SSH;\n-\n-\treturn 1;\n }\n \n static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n@@ -815,7 +813,9 @@ static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n \tconst char *variant;\n \tchar *p = NULL;\n \n-\tif (override_ssh_variant(&ssh_variant))\n+\toverride_ssh_variant(&ssh_variant);\n+\n+\tif (ssh_variant != VARIANT_AUTO)\n \t\treturn ssh_variant;\n \n \tif (!is_cmdline) {\ndiff --git i/t/t5601-clone.sh w/t/t5601-clone.sh\nindex 11fa516997..f9a2ae84c7 100755\n--- i/t/t5601-clone.sh\n+++ w/t/t5601-clone.sh\n@@ -373,6 +373,12 @@ test_expect_success 'variant can be overridden' '\n \texpect_ssh \"-4 -P 123\" myhost src\n '\n \n+test_expect_success 'variant=auto picks based on basename' '\n+\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/plink\" &&\n+\tgit -c ssh.variant=auto clone -4 \"[myhost:123]:src\" ssh-auto-clone &&\n+\texpect_ssh \"-4 -P 123\" myhost src\n+'\n+\n test_expect_success 'simple does not support -4/-6' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n \ttest_must_fail git clone -4 \"myhost:src\" ssh-4-clone-simple\n"},{"id":"330863","messageId":"20171023225106.GA73667@google.com","threadId":"46746","inReplyTo":"20171023224310.o7ftwmbr7n74vvc6@aiede.mtv.corp.google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-23T22:51:06Z","receivedAt":"2017-10-23T22:51:14Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/23, Jonathan Nieder wrote:\n> Hi,\n> \n> Jonathan Tan wrote:\n> \n> > The new documentation seems to imply that setting ssh.variant (or\n> > GIT_SSH_VARIANT) to \"auto\" is equivalent to not setting it at all, but\n> > looking at the code, it doesn't seem to be the case (not setting it at\n> > all invokes checking the first word of core.sshCommand, and only uses\n> > VARIANT_AUTO if that check is inconclusive, whereas setting\n> > ssh.variant=auto skips the core.sshCommand check entirely).\n> >\n> > Maybe document ssh.variant as follows:\n> >\n> > If unset, Git will determine the command-line arguments to use based\n> > on the basename of the configured SSH command (through the\n> > environment variable `GIT_SSH` or `GIT_SSH_COMMAND`, or the config\n> > setting `core.sshCommand`). If the basename is unrecognized, Git\n> > will attempt to detect support of OpenSSH options by first invoking\n> > the configured SSH command with the `-G` (print configuration) flag,\n> > and will subsequently use OpenSSH options (upon success) or no\n> > options besides the host (upon failure).\n> >\n> > If set, Git will not do any auto-detection based on the basename of\n> > the configured SSH command. This can be set to `ssh` (OpenSSH\n> > options), `plink`, `putty`, `tortoiseplink`, `simple` (no options\n> > besides the host), or `auto` (the detection with `-G` as described\n> > above). If set to any other value, Git behaves as if this is set to\n> > `ssh`.\n> \n> Good point. Brandon noticed something similar as well.\n> \n> Separately from how to document it, what do you think a good behavior\n> would be? Should the \"auto\" configuration trigger command line based\n> detection just like no configuration at all? Should the \"auto\" value\n> for configuration be removed and that behavior restricted to the\n> no-configuration case?\n> \n> I'm tempted to go with the former, which would look like the following.\n> What do you think?\n\nAs a user having some variant as 'auto' doesn't make much sense, i mean\nisn't that exactly what the default behavior is? Check if my ssh\ncommand matches existing variants and go with that. What you are\nproposing is the make the existing auto detection better (yay!) though I\ndon't know if it warrants adding a new variant all together.\n\nInstead it may be better to stick this new improved detection at the end\nof the existing variant discovery function 'determine_ssh_variant()' as\na last ditch effort to figure out the variant. That way we don't have\nan extra variant type that can be configured and eliminates some of the\nadditional code in the switch statements to handle that enum value\n(though that isn't really that big of a deal).\n\n> \n> If this looks good, I can reroll in a moment.\n> \n> diff --git i/Documentation/config.txt w/Documentation/config.txt\n> index 4a16b324f0..6dffa4aa3d 100644\n> --- i/Documentation/config.txt\n> +++ w/Documentation/config.txt\n> @@ -2081,20 +2081,21 @@ matched against are those given directly to Git commands. This means any URLs\n> visited as a result of a redirection do not participate in matching.\n> \n> ssh.variant::\n> -\tDepending on the value of the environment variables `GIT_SSH` or\n> -\t`GIT_SSH_COMMAND`, or the config setting `core.sshCommand`, Git\n> -\tauto-detects whether to pass command-line parameters for use\n> -\twith a simple wrapper script (simple), OpenSSH (ssh), plink, or\n> -\ttortoiseplink.\n> -+\n> -The default is `auto`, which means to auto-detect whether the ssh command\n> -implements OpenSSH options using the `-G` (print configuration) option.\n> -If the ssh command supports OpenSSH options, it then behaves like `ssh`;\n> -otherwise, it behaves like `simple`.\n> -+\n> -The config variable `ssh.variant` can be set to override this auto-detection;\n> -valid values are `ssh`, `simple`, `plink`, `putty`, `tortoiseplink`, and\n> -`auto`. Any other value will be treated as normal ssh. This setting can be\n> +\tBy default, Git determines the command line arguments to use\n> +\tbased on the basename of the configured SSH command (configured\n> +\tusing the environment variable `GIT_SSH` or `GIT_SSH_COMMAND` or\n> +\tthe config setting `core.sshCommand`). If the basename is\n> +\tunrecognized, Git will attempt to detect support of OpenSSH\n> +\toptions by first invoking the configured SSH command with the\n> +\t`-G` (print configuration) option and will subsequently use\n> +\tOpenSSH options (if that is successful) or no options besides\n> +\tthe host and remote command (if it fails).\n> ++\n> +The config variable `ssh.variant` can be set to override this detection:\n> +valid values are `ssh` (to use OpenSSH options), `plink`, `putty`,\n> +`tortoiseplink`, `simple` (no options except the host and remote command).\n> +The default auto-detection can be explicitly requested using the value\n> +`auto`. Any other value is treated as `ssh`. This setting can also be\n> overridden via the environment variable `GIT_SSH_VARIANT`.\n> +\n> The current command-line parameters used for each variant are as\n> diff --git i/connect.c w/connect.c\n> index 98f2d9ce57..06bcd3981e 100644\n> --- i/connect.c\n> +++ w/connect.c\n> @@ -785,12 +785,12 @@ enum ssh_variant {\n> \tVARIANT_TORTOISEPLINK,\n> };\n> \n> -static int override_ssh_variant(enum ssh_variant *ssh_variant)\n> +static void override_ssh_variant(enum ssh_variant *ssh_variant)\n> {\n> \tconst char *variant = getenv(\"GIT_SSH_VARIANT\");\n> \n> \tif (!variant && git_config_get_string_const(\"ssh.variant\", &variant))\n> -\t\treturn 0;\n> +\t\treturn;\n> \n> \tif (!strcmp(variant, \"auto\"))\n> \t\t*ssh_variant = VARIANT_AUTO;\n> @@ -804,8 +804,6 @@ static int override_ssh_variant(enum ssh_variant *ssh_variant)\n> \t\t*ssh_variant = VARIANT_SIMPLE;\n> \telse\n> \t\t*ssh_variant = VARIANT_SSH;\n> -\n> -\treturn 1;\n> }\n> \n> static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n> @@ -815,7 +813,9 @@ static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n> \tconst char *variant;\n> \tchar *p = NULL;\n> \n> -\tif (override_ssh_variant(&ssh_variant))\n> +\toverride_ssh_variant(&ssh_variant);\n> +\n> +\tif (ssh_variant != VARIANT_AUTO)\n> \t\treturn ssh_variant;\n> \n> \tif (!is_cmdline) {\n> diff --git i/t/t5601-clone.sh w/t/t5601-clone.sh\n> index 11fa516997..f9a2ae84c7 100755\n> --- i/t/t5601-clone.sh\n> +++ w/t/t5601-clone.sh\n> @@ -373,6 +373,12 @@ test_expect_success 'variant can be overridden' '\n> \texpect_ssh \"-4 -P 123\" myhost src\n> '\n> \n> +test_expect_success 'variant=auto picks based on basename' '\n> +\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/plink\" &&\n> +\tgit -c ssh.variant=auto clone -4 \"[myhost:123]:src\" ssh-auto-clone &&\n> +\texpect_ssh \"-4 -P 123\" myhost src\n> +'\n> +\n> test_expect_success 'simple does not support -4/-6' '\n> \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n> \ttest_must_fail git clone -4 \"myhost:src\" ssh-4-clone-simple\n\n-- \nBrandon Williams\n"},{"id":"330864","messageId":"20171023225432.2lj4gaybkvpyhcgm@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"CAGZ79kZTjUvcq_hKHCqTDoaBxt2x+9XcqYc6ao1bhcET2SM-PQ@mail.gmail.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T22:54:32Z","receivedAt":"2017-10-23T22:54:41Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Stefan Beller wrote:\n> On Mon, Oct 23, 2017 at 2:31 PM, Jonathan Nieder wrote:\n\n>> 1. First, check whether $GIT_SSH supports OpenSSH options by running\n>>\n>> $GIT_SSH -G \n>>\n>> This returns status 0 and prints configuration in OpenSSH if it\n>> recognizes all and returns status 255 if it encounters\n>> an unrecognized option. A wrapper script like\n>>\n>> exec ssh -- \"$@\"\n>>\n>> would fail with\n>>\n>> ssh: Could not resolve hostname -g: Name or service not known\n>\n> capital -G?\n\nThe actual error message uses lowercase (presumably they use tolower on\nthe hostname).\n\n>> - if (variant == VARIANT_TORTOISEPLINK)\n>> - argv_array_push(&conn->args, \"-batch\");\n>> + detect.use_shell = conn->use_shell;\n>\n> Why do we have to use a shell for evaluation of this\n> test balloon?\n\nIf the user set GIT_SSH_COMMAND instead of GIT_SSH then we need to run\nit using a shell. The above line inherits the use_shell setting so it\nends up doing whatever conn would do.\n\n[...]\n>> + argv_array_push(&detect.args, \"-G\");\n> ...\n>> + variant = run_command(&detect) ? VARIANT_SIMPLE : VARIANT_SSH;\n>\n> What if we have a VARIANT_SIMPLE, that doesn't care about -G\n> but just connects to the remote host (keeping the session open), do we need\n> to kill it after some time to have run_command return eventually?\n>\n> Or can we give a command to be executed remotely (e.g. 'false') that\n> we know returns quickly?\n\nSince stdin is /dev/null, it would presumably return quickly. But I\nalso don't expect this kind of behavior from GIT_SSH commands. The\nkinds I'd expect are\n\n A. The repo case, which forwards to 'ssh' and supports all 'ssh'\n flags\n\n B. The travis-ci case, which expects a host and command and does\n not accept any flags\n\n C. More sophisticated wrappers that parse flags but still do not\n handle -G\n\nFor case (A), the detection run would figure out that this accepts\nOpenSSH options. Good.\n\nFor case (B), the detection run would figure out that this does not\naccept OpenSSH options. Good.\n\nFor case (C), the detection run would think that this does not accept\nOpenSSH options, when it accepts some. But I think that's the best we\ncan do for now. Longer term, we need to work with the author of such a\nscript to find out what kind of interface they want.\n\nThanks and hope that helps,\nJonathan\n"},{"id":"330865","messageId":"20171023155713.5055125d7467d8daaee42e32@google.com","threadId":"46746","inReplyTo":"20171023225106.GA73667@google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Jonathan Tan","fromEmail":"jonathantanmy@google.com","sentAt":"2017-10-23T22:57:13Z","receivedAt":"2017-10-23T22:57:20Z","isPatch":true,"sender":{"key":"jonathantanmy@fastmail.com","avatar":null},"body":"On Mon, 23 Oct 2017 15:51:06 -0700\nBrandon Williams wrote:\n\n> On 10/23, Jonathan Nieder wrote:\n> > Separately from how to document it, what do you think a good behavior\n> > would be? Should the \"auto\" configuration trigger command line based\n> > detection just like no configuration at all? Should the \"auto\" value\n> > for configuration be removed and that behavior restricted to the\n> > no-configuration case?\n> > \n> > I'm tempted to go with the former, which would look like the following.\n> > What do you think?\n> \n> As a user having some variant as 'auto' doesn't make much sense, i mean\n> isn't that exactly what the default behavior is?\n\nSo you're suggesting the second option (\"that behavior restricted to the\nno-configuration case\")?\n\nI'm leaning towards supporting \"auto\", actually. At the very least, it\ngives the user a clear way to override an existing config.\n\n> Check if my ssh\n> command matches existing variants and go with that. What you are\n> proposing is the make the existing auto detection better (yay!) though I\n> don't know if it warrants adding a new variant all together.\n> \n> Instead it may be better to stick this new improved detection at the end\n> of the existing variant discovery function 'determine_ssh_variant()' as\n> a last ditch effort to figure out the variant. That way we don't have\n> an extra variant type that can be configured and eliminates some of the\n> additional code in the switch statements to handle that enum value\n> (though that isn't really that big of a deal).\n\nThis sounds like what is already being done in the code.\n\n> > If this looks good, I can reroll in a moment.\n\nYes, this looks good.\n"},{"id":"330866","messageId":"20171023231259.ohrpcnxpaznpgkbl@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023225106.GA73667@google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T23:12:59Z","receivedAt":"2017-10-23T23:13:07Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Brandon Williams wrote:\n> On 10/23, Jonathan Nieder wrote:\n\n>> Separately from how to document it, what do you think a good behavior\n>> would be? Should the \"auto\" configuration trigger command line based\n>> detection just like no configuration at all? Should the \"auto\" value\n>> for configuration be removed and that behavior restricted to the\n>> no-configuration case?\n>>\n>> I'm tempted to go with the former, which would look like the following.\n>> What do you think?\n>\n> As a user having some variant as 'auto' doesn't make much sense, i mean\n> isn't that exactly what the default behavior is? Check if my ssh\n> command matches existing variants and go with that. What you are\n> proposing is the make the existing auto detection better (yay!) though I\n> don't know if it warrants adding a new variant all together.\n>\n> Instead it may be better to stick this new improved detection at the end\n> of the existing variant discovery function 'determine_ssh_variant()' as\n> a last ditch effort to figure out the variant. That way we don't have\n> an extra variant type that can be configured and eliminates some of the\n> additional code in the switch statements to handle that enum value\n> (though that isn't really that big of a deal).\n\nYes, if git config allowed e.g. \".git/config\" to unset a setting from\ne.g. \"/etc/gitconfig\", then I wouldn't want the 'auto' value to exist\nin configuration at all. But because git's config language doesn't\nhave a way to unset a variable, this series provided \"auto\" as a way\nto explicitly request the same behavior as unset.\n\nIn that spirit, the patch I sent was broken (\"auto\" meant something\ndifferent from unset), so thanks for pointing the issue out.\n\nSensible?\nJonathan\n"},{"id":"330867","messageId":"20171023231625.6mhcyqti7vdg6yot@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023155713.5055125d7467d8daaee42e32@google.com","subject":"[PATCH v2 0/5] Coping with unrecognized ssh wrapper scripts in GIT_SSH","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T23:16:25Z","receivedAt":"2017-10-23T23:16:33Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Jonathan Tan wrote:\n>> On 10/23, Jonathan Nieder wrote:\n\n>>> If this looks good, I can reroll in a moment.\n>\n> Yes, this looks good.\n\nThanks. Here goes.\n\nThe interdiff is upthread. Thanks, all, for the quick review.\n\nJonathan Nieder (5):\n connect: split git:// setup into a separate function\n connect: split ssh command line options into separate function\n ssh: 'auto' variant to select between 'ssh' and 'simple'\n ssh: 'simple' variant does not support -4/-6\n ssh: 'simple' variant does not support --port\n\n Documentation/config.txt | 24 ++--\n connect.c | 285 +++++++++++++++++++++++++++++------------------\n t/t5601-clone.sh | 40 +++++--\n t/t5603-clone-dirname.sh | 2 +\n 4 files changed, 229 insertions(+), 122 deletions(-)\n"},{"id":"330868","messageId":"20171023231703.f457jp43dh7fddpv@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023231625.6mhcyqti7vdg6yot@aiede.mtv.corp.google.com","subject":"[PATCH 1/5] connect: split git:// setup into a separate function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T23:17:03Z","receivedAt":"2017-10-23T23:17:10Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"The git_connect function is growing long. Split the\nPROTO_GIT-specific portion to a separate function to make it easier to\nread.\n\nNo functional change intended.\n\nSigned-off-by: Jonathan Nieder \nReviewed-by: Stefan Beller \n---\nAs before, except with sbeller's Reviewed-by.\n\n connect.c | 103 +++++++++++++++++++++++++++++++++++---------------------------\n 1 file changed, 59 insertions(+), 44 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 7fbd396b35..068e70caad 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -850,6 +850,64 @@ static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n \treturn ssh_variant;\n }\n \n+/*\n+ * Open a connection using Git's native protocol.\n+ *\n+ * The caller is responsible for freeing hostandport, but this function may\n+ * modify it (for example, to truncate it to remove the port part).\n+ */\n+static struct child_process *git_connect_git(int fd[2], char *hostandport,\n+\t\t\t\t\t const char *path, const char *prog,\n+\t\t\t\t\t int flags)\n+{\n+\tstruct child_process *conn = &no_fork;\n+\tstruct strbuf request = STRBUF_INIT;\n+\t/*\n+\t * Set up virtual host information based on where we will\n+\t * connect, unless the user has overridden us in\n+\t * the environment.\n+\t */\n+\tchar *target_host = getenv(\"GIT_OVERRIDE_VIRTUAL_HOST\");\n+\tif (target_host)\n+\t\ttarget_host = xstrdup(target_host);\n+\telse\n+\t\ttarget_host = xstrdup(hostandport);\n+\n+\ttransport_check_allowed(\"git\");\n+\n+\t/* These underlying connection commands die() if they\n+\t * cannot connect.\n+\t */\n+\tif (git_use_proxy(hostandport))\n+\t\tconn = git_proxy_connect(fd, hostandport);\n+\telse\n+\t\tgit_tcp_connect(fd, hostandport, flags);\n+\t/*\n+\t * Separate original protocol components prog and path\n+\t * from extended host header with a NUL byte.\n+\t *\n+\t * Note: Do not add any other headers here! Doing so\n+\t * will cause older git-daemon servers to crash.\n+\t */\n+\tstrbuf_addf(&request,\n+\t\t \"%s %s%chost=%s%c\",\n+\t\t prog, path, 0,\n+\t\t target_host, 0);\n+\n+\t/* If using a new version put that stuff here after a second null byte */\n+\tif (get_protocol_version_config() > 0) {\n+\t\tstrbuf_addch(&request, '\\0');\n+\t\tstrbuf_addf(&request, \"version=%d%c\",\n+\t\t\t get_protocol_version_config(), '\\0');\n+\t}\n+\n+\tpacket_write(fd[1], request.buf, request.len);\n+\n+\tfree(target_host);\n+\tstrbuf_release(&request);\n+\treturn conn;\n+}\n+\n /*\n * This returns a dummy child_process if the transport protocol does not\n * need fork(2), or a struct child_process object if it does. Once done,\n@@ -881,50 +939,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tprintf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n \t\tconn = NULL;\n \t} else if (protocol == PROTO_GIT) {\n-\t\tstruct strbuf request = STRBUF_INIT;\n-\t\t/*\n-\t\t * Set up virtual host information based on where we will\n-\t\t * connect, unless the user has overridden us in\n-\t\t * the environment.\n-\t\t */\n-\t\tchar *target_host = getenv(\"GIT_OVERRIDE_VIRTUAL_HOST\");\n-\t\tif (target_host)\n-\t\t\ttarget_host = xstrdup(target_host);\n-\t\telse\n-\t\t\ttarget_host = xstrdup(hostandport);\n-\n-\t\ttransport_check_allowed(\"git\");\n-\n-\t\t/* These underlying connection commands die() if they\n-\t\t * cannot connect.\n-\t\t */\n-\t\tif (git_use_proxy(hostandport))\n-\t\t\tconn = git_proxy_connect(fd, hostandport);\n-\t\telse\n-\t\t\tgit_tcp_connect(fd, hostandport, flags);\n-\t\t/*\n-\t\t * Separate original protocol components prog and path\n-\t\t * from extended host header with a NUL byte.\n-\t\t *\n-\t\t * Note: Do not add any other headers here! Doing so\n-\t\t * will cause older git-daemon servers to crash.\n-\t\t */\n-\t\tstrbuf_addf(&request,\n-\t\t\t \"%s %s%chost=%s%c\",\n-\t\t\t prog, path, 0,\n-\t\t\t target_host, 0);\n-\n-\t\t/* If using a new version put that stuff here after a second null byte */\n-\t\tif (get_protocol_version_config() > 0) {\n-\t\t\tstrbuf_addch(&request, '\\0');\n-\t\t\tstrbuf_addf(&request, \"version=%d%c\",\n-\t\t\t\t get_protocol_version_config(), '\\0');\n-\t\t}\n-\n-\t\tpacket_write(fd[1], request.buf, request.len);\n-\n-\t\tfree(target_host);\n-\t\tstrbuf_release(&request);\n+\t\tconn = git_connect_git(fd, hostandport, path, prog, flags);\n \t} else {\n \t\tstruct strbuf cmd = STRBUF_INIT;\n \t\tconst char *const *var;\n-- \n2.15.0.rc1.287.g2b38de12cc\n\n"},{"id":"330869","messageId":"20171023231733.tattcu2aws7kgous@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023231625.6mhcyqti7vdg6yot@aiede.mtv.corp.google.com","subject":"[PATCH 2/5] connect: split ssh command line options into separate function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T23:17:33Z","receivedAt":"2017-10-23T23:17:41Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"The git_connect function is growing long. Split the portion that\ndiscovers an ssh command and options it accepts before the service\nname and path to a separate function to make it easier to read.\n\nNo functional change intended.\n\nSigned-off-by: Jonathan Nieder \nReviewed-by: Stefan Beller \n---\nAs before, except for the Reviewed-by.\n\n connect.c | 116 +++++++++++++++++++++++++++++++++-----------------------------\n 1 file changed, 61 insertions(+), 55 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 068e70caad..77ab6db3bb 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -908,6 +908,65 @@ static struct child_process *git_connect_git(int fd[2], char *hostandport,\n \treturn conn;\n }\n \n+/* Prepare a child_process for use by Git's SSH-tunneled transport. */\n+static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n+\t\t\t const char *port, int flags)\n+{\n+\tconst char *ssh;\n+\tenum ssh_variant variant;\n+\n+\tif (looks_like_command_line_option(ssh_host))\n+\t\tdie(\"strange hostname '%s' blocked\", ssh_host);\n+\n+\tssh = get_ssh_command();\n+\tif (ssh) {\n+\t\tvariant = determine_ssh_variant(ssh, 1);\n+\t} else {\n+\t\t/*\n+\t\t * GIT_SSH is the no-shell version of\n+\t\t * GIT_SSH_COMMAND (and must remain so for\n+\t\t * historical compatibility).\n+\t\t */\n+\t\tconn->use_shell = 0;\n+\n+\t\tssh = getenv(\"GIT_SSH\");\n+\t\tif (!ssh)\n+\t\t\tssh = \"ssh\";\n+\t\tvariant = determine_ssh_variant(ssh, 0);\n+\t}\n+\n+\targv_array_push(&conn->args, ssh);\n+\n+\tif (variant == VARIANT_SSH &&\n+\t get_protocol_version_config() > 0) {\n+\t\targv_array_push(&conn->args, \"-o\");\n+\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n+\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t get_protocol_version_config());\n+\t}\n+\n+\tif (variant != VARIANT_SIMPLE) {\n+\t\tif (flags & CONNECT_IPV4)\n+\t\t\targv_array_push(&conn->args, \"-4\");\n+\t\telse if (flags & CONNECT_IPV6)\n+\t\t\targv_array_push(&conn->args, \"-6\");\n+\t}\n+\n+\tif (variant == VARIANT_TORTOISEPLINK)\n+\t\targv_array_push(&conn->args, \"-batch\");\n+\n+\tif (port && variant != VARIANT_SIMPLE) {\n+\t\tif (variant == VARIANT_SSH)\n+\t\t\targv_array_push(&conn->args, \"-p\");\n+\t\telse\n+\t\t\targv_array_push(&conn->args, \"-P\");\n+\n+\t\targv_array_push(&conn->args, port);\n+\t}\n+\n+\targv_array_push(&conn->args, ssh_host);\n+}\n+\n /*\n * This returns a dummy child_process if the transport protocol does not\n * need fork(2), or a struct child_process object if it does. Once done,\n@@ -961,16 +1020,13 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tconn->use_shell = 1;\n \t\tconn->in = conn->out = -1;\n \t\tif (protocol == PROTO_SSH) {\n-\t\t\tconst char *ssh;\n-\t\t\tenum ssh_variant variant;\n \t\t\tchar *ssh_host = hostandport;\n \t\t\tconst char *port = NULL;\n+\n \t\t\ttransport_check_allowed(\"ssh\");\n \t\t\tget_host_and_port(&ssh_host, &port);\n-\n \t\t\tif (!port)\n \t\t\t\tport = get_port(ssh_host);\n-\n \t\t\tif (flags & CONNECT_DIAG_URL) {\n \t\t\t\tprintf(\"Diag: url=%s\\n\", url ? url : \"NULL\");\n \t\t\t\tprintf(\"Diag: protocol=%s\\n\", prot_name(protocol));\n@@ -984,57 +1040,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\tstrbuf_release(&cmd);\n \t\t\t\treturn NULL;\n \t\t\t}\n-\n-\t\t\tif (looks_like_command_line_option(ssh_host))\n-\t\t\t\tdie(\"strange hostname '%s' blocked\", ssh_host);\n-\n-\t\t\tssh = get_ssh_command();\n-\t\t\tif (ssh) {\n-\t\t\t\tvariant = determine_ssh_variant(ssh, 1);\n-\t\t\t} else {\n-\t\t\t\t/*\n-\t\t\t\t * GIT_SSH is the no-shell version of\n-\t\t\t\t * GIT_SSH_COMMAND (and must remain so for\n-\t\t\t\t * historical compatibility).\n-\t\t\t\t */\n-\t\t\t\tconn->use_shell = 0;\n-\n-\t\t\t\tssh = getenv(\"GIT_SSH\");\n-\t\t\t\tif (!ssh)\n-\t\t\t\t\tssh = \"ssh\";\n-\t\t\t\tvariant = determine_ssh_variant(ssh, 0);\n-\t\t\t}\n-\n-\t\t\targv_array_push(&conn->args, ssh);\n-\n-\t\t\tif (variant == VARIANT_SSH &&\n-\t\t\t get_protocol_version_config() > 0) {\n-\t\t\t\targv_array_push(&conn->args, \"-o\");\n-\t\t\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n-\t\t\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n-\t\t\t\t\t\t get_protocol_version_config());\n-\t\t\t}\n-\n-\t\t\tif (variant != VARIANT_SIMPLE) {\n-\t\t\t\tif (flags & CONNECT_IPV4)\n-\t\t\t\t\targv_array_push(&conn->args, \"-4\");\n-\t\t\t\telse if (flags & CONNECT_IPV6)\n-\t\t\t\t\targv_array_push(&conn->args, \"-6\");\n-\t\t\t}\n-\n-\t\t\tif (variant == VARIANT_TORTOISEPLINK)\n-\t\t\t\targv_array_push(&conn->args, \"-batch\");\n-\n-\t\t\tif (port && variant != VARIANT_SIMPLE) {\n-\t\t\t\tif (variant == VARIANT_SSH)\n-\t\t\t\t\targv_array_push(&conn->args, \"-p\");\n-\t\t\t\telse\n-\t\t\t\t\targv_array_push(&conn->args, \"-P\");\n-\n-\t\t\t\targv_array_push(&conn->args, port);\n-\t\t\t}\n-\n-\t\t\targv_array_push(&conn->args, ssh_host);\n+\t\t\tfill_ssh_args(conn, ssh_host, port, flags);\n \t\t} else {\n \t\t\ttransport_check_allowed(\"file\");\n \t\t\tif (get_protocol_version_config() > 0) {\n-- \n2.15.0.rc1.287.g2b38de12cc-goog\n\n"},{"id":"330870","messageId":"20171023231839.2g5wudvmoj3hphua@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023231625.6mhcyqti7vdg6yot@aiede.mtv.corp.google.com","subject":"[PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T23:18:39Z","receivedAt":"2017-10-23T23:18:50Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Android's \"repo\" tool is a tool for managing a large codebase\nconsisting of multiple smaller repositories, similar to Git's\nsubmodule feature. Starting with Git 94b8ae5a (ssh: introduce a\n'simple' ssh variant, 2017-10-16), users noticed that it stopped\nhandling the port in ssh:// URLs.\n\nThe cause: when it encounters ssh:// URLs, repo pre-connects to the\nserver and sets GIT_SSH to a helper \".repo/repo/git_ssh\" that reuses\nthat connection. Before 94b8ae5a, the helper was assumed to support\nOpenSSH options for lack of a better guess and got passed a -p option\nto set the port. After that patch, it uses the new default of a\nsimple helper that does not accept an option to set the port.\n\nThe next release of \"repo\" will set GIT_SSH_VARIANT to \"ssh\" to avoid\nthat. But users of old versions and of other similar GIT_SSH\nimplementations would not get the benefit of that fix.\n\nSo update the default to use OpenSSH options again, with a twist. As\nobserved in 94b8ae5a, we cannot assume that $GIT_SSH always handles\nOpenSSH options: common helpers such as travis-ci's dpl[*] are\nconfigured using GIT_SSH and do not accept OpenSSH options. So make\nthe default a new variant \"auto\", with the following behavior:\n\n 1. First, check for a recognized basename, like today.\n\n 2. If the basename is not recognized, check whether $GIT_SSH supports\n OpenSSH options by running\n\n\t$GIT_SSH -G \n\n This returns status 0 and prints configuration in OpenSSH if it\n recognizes all and returns status 255 if it encounters\n an unrecognized option. A wrapper script like\n\n\texec ssh -- \"$@\"\n\n would fail with\n\n\tssh: Could not resolve hostname -g: Name or service not known\n\n , correctly reflecting that it does not support OpenSSH options.\n\n 3. Based on the result from step (2), behave like \"ssh\" (if it\n succeeded) or \"simple\" (if it failed).\n\nThis way, the default ssh variant for unrecognized commands can handle\nboth the repo and dpl cases as intended.\n\n[*] https://github.com/travis-ci/dpl/blob/6c3fddfda1f2a85944c544446b068bac0a77c049/lib/dpl/provider.rb#L215\n\nReported-by: William Yan \nImproved-by: Jonathan Tan \nSigned-off-by: Jonathan Nieder \n---\nThis is the main one. Simplified by making \"auto\" behave the same as\nunset.\n\n Documentation/config.txt | 24 ++++++++------\n connect.c | 82 +++++++++++++++++++++++++++++++-----------------\n t/t5601-clone.sh | 20 ++++++++++++\n 3 files changed, 88 insertions(+), 38 deletions(-)\n\ndiff --git a/Documentation/config.txt b/Documentation/config.txt\nindex 0460af37e2..6dffa4aa3d 100644\n--- a/Documentation/config.txt\n+++ b/Documentation/config.txt\n@@ -2081,16 +2081,22 @@ matched against are those given directly to Git commands. This means any URLs\n visited as a result of a redirection do not participate in matching.\n \n ssh.variant::\n-\tDepending on the value of the environment variables `GIT_SSH` or\n-\t`GIT_SSH_COMMAND`, or the config setting `core.sshCommand`, Git\n-\tauto-detects whether to adjust its command-line parameters for use\n-\twith ssh (OpenSSH), plink or tortoiseplink, as opposed to the default\n-\t(simple).\n+\tBy default, Git determines the command line arguments to use\n+\tbased on the basename of the configured SSH command (configured\n+\tusing the environment variable `GIT_SSH` or `GIT_SSH_COMMAND` or\n+\tthe config setting `core.sshCommand`). If the basename is\n+\tunrecognized, Git will attempt to detect support of OpenSSH\n+\toptions by first invoking the configured SSH command with the\n+\t`-G` (print configuration) option and will subsequently use\n+\tOpenSSH options (if that is successful) or no options besides\n+\tthe host and remote command (if it fails).\n +\n-The config variable `ssh.variant` can be set to override this auto-detection;\n-valid values are `ssh`, `simple`, `plink`, `putty` or `tortoiseplink`. Any\n-other value will be treated as normal ssh. This setting can be overridden via\n-the environment variable `GIT_SSH_VARIANT`.\n+The config variable `ssh.variant` can be set to override this detection:\n+valid values are `ssh` (to use OpenSSH options), `plink`, `putty`,\n+`tortoiseplink`, `simple` (no options except the host and remote command).\n+The default auto-detection can be explicitly requested using the value\n+`auto`. Any other value is treated as `ssh`. This setting can also be\n+overridden via the environment variable `GIT_SSH_VARIANT`.\n +\n The current command-line parameters used for each variant are as\n follows:\ndiff --git a/connect.c b/connect.c\nindex 77ab6db3bb..0441dcbacf 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -777,6 +777,7 @@ static const char *get_ssh_command(void)\n }\n \n enum ssh_variant {\n+\tVARIANT_AUTO,\n \tVARIANT_SIMPLE,\n \tVARIANT_SSH,\n \tVARIANT_PLINK,\n@@ -784,14 +785,16 @@ enum ssh_variant {\n \tVARIANT_TORTOISEPLINK,\n };\n \n-static int override_ssh_variant(enum ssh_variant *ssh_variant)\n+static void override_ssh_variant(enum ssh_variant *ssh_variant)\n {\n \tconst char *variant = getenv(\"GIT_SSH_VARIANT\");\n \n \tif (!variant && git_config_get_string_const(\"ssh.variant\", &variant))\n-\t\treturn 0;\n+\t\treturn;\n \n-\tif (!strcmp(variant, \"plink\"))\n+\tif (!strcmp(variant, \"auto\"))\n+\t\t*ssh_variant = VARIANT_AUTO;\n+\telse if (!strcmp(variant, \"plink\"))\n \t\t*ssh_variant = VARIANT_PLINK;\n \telse if (!strcmp(variant, \"putty\"))\n \t\t*ssh_variant = VARIANT_PUTTY;\n@@ -801,18 +804,18 @@ static int override_ssh_variant(enum ssh_variant *ssh_variant)\n \t\t*ssh_variant = VARIANT_SIMPLE;\n \telse\n \t\t*ssh_variant = VARIANT_SSH;\n-\n-\treturn 1;\n }\n \n static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n \t\t\t\t\t int is_cmdline)\n {\n-\tenum ssh_variant ssh_variant = VARIANT_SIMPLE;\n+\tenum ssh_variant ssh_variant = VARIANT_AUTO;\n \tconst char *variant;\n \tchar *p = NULL;\n \n-\tif (override_ssh_variant(&ssh_variant))\n+\toverride_ssh_variant(&ssh_variant);\n+\n+\tif (ssh_variant != VARIANT_AUTO)\n \t\treturn ssh_variant;\n \n \tif (!is_cmdline) {\n@@ -908,6 +911,38 @@ static struct child_process *git_connect_git(int fd[2], char *hostandport,\n \treturn conn;\n }\n \n+static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n+\t\t\t enum ssh_variant variant, const char *port,\n+\t\t\t int flags)\n+{\n+\tif (variant == VARIANT_SSH &&\n+\t get_protocol_version_config() > 0) {\n+\t\targv_array_push(args, \"-o\");\n+\t\targv_array_push(args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n+\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n+\t\t\t\t get_protocol_version_config());\n+\t}\n+\n+\tif (variant != VARIANT_SIMPLE) {\n+\t\tif (flags & CONNECT_IPV4)\n+\t\t\targv_array_push(args, \"-4\");\n+\t\telse if (flags & CONNECT_IPV6)\n+\t\t\targv_array_push(args, \"-6\");\n+\t}\n+\n+\tif (variant == VARIANT_TORTOISEPLINK)\n+\t\targv_array_push(args, \"-batch\");\n+\n+\tif (port && variant != VARIANT_SIMPLE) {\n+\t\tif (variant == VARIANT_SSH)\n+\t\t\targv_array_push(args, \"-p\");\n+\t\telse\n+\t\t\targv_array_push(args, \"-P\");\n+\n+\t\targv_array_push(args, port);\n+\t}\n+}\n+\n /* Prepare a child_process for use by Git's SSH-tunneled transport. */\n static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n \t\t\t const char *port, int flags)\n@@ -937,33 +972,22 @@ static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n \n \targv_array_push(&conn->args, ssh);\n \n-\tif (variant == VARIANT_SSH &&\n-\t get_protocol_version_config() > 0) {\n-\t\targv_array_push(&conn->args, \"-o\");\n-\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n-\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n-\t\t\t\t get_protocol_version_config());\n-\t}\n-\n-\tif (variant != VARIANT_SIMPLE) {\n-\t\tif (flags & CONNECT_IPV4)\n-\t\t\targv_array_push(&conn->args, \"-4\");\n-\t\telse if (flags & CONNECT_IPV6)\n-\t\t\targv_array_push(&conn->args, \"-6\");\n-\t}\n+\tif (variant == VARIANT_AUTO) {\n+\t\tstruct child_process detect = CHILD_PROCESS_INIT;\n \n-\tif (variant == VARIANT_TORTOISEPLINK)\n-\t\targv_array_push(&conn->args, \"-batch\");\n+\t\tdetect.use_shell = conn->use_shell;\n+\t\tdetect.no_stdin = detect.no_stdout = detect.no_stderr = 1;\n \n-\tif (port && variant != VARIANT_SIMPLE) {\n-\t\tif (variant == VARIANT_SSH)\n-\t\t\targv_array_push(&conn->args, \"-p\");\n-\t\telse\n-\t\t\targv_array_push(&conn->args, \"-P\");\n+\t\targv_array_push(&detect.args, ssh);\n+\t\targv_array_push(&detect.args, \"-G\");\n+\t\tpush_ssh_options(&detect.args, &detect.env_array,\n+\t\t\t\t VARIANT_SSH, port, flags);\n+\t\targv_array_push(&detect.args, ssh_host);\n \n-\t\targv_array_push(&conn->args, port);\n+\t\tvariant = run_command(&detect) ? VARIANT_SIMPLE : VARIANT_SSH;\n \t}\n \n+\tpush_ssh_options(&conn->args, &conn->env_array, variant, port, flags);\n \targv_array_push(&conn->args, ssh_host);\n }\n \ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex 86811a0c35..df9dfafdd8 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -372,6 +372,12 @@ test_expect_success 'variant can be overriden' '\n \texpect_ssh myhost src\n '\n \n+test_expect_success 'variant=auto picks based on basename' '\n+\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/plink\" &&\n+\tgit -c ssh.variant=auto clone -4 \"[myhost:123]:src\" ssh-auto-clone &&\n+\texpect_ssh \"-4 -P 123\" myhost src\n+'\n+\n test_expect_success 'simple is treated as simple' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n \tgit clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple &&\n@@ -384,6 +390,20 @@ test_expect_success 'uplink is treated as simple' '\n \texpect_ssh myhost src\n '\n \n+test_expect_success 'OpenSSH-like uplink is treated as ssh' '\n+\twrite_script \"$TRASH_DIRECTORY/uplink\" <<-EOF &&\n+\tif test \"\\$1\" = \"-G\"\n+\tthen\n+\t\texit 0\n+\tfi &&\n+\texec \"\\$TRASH_DIRECTORY/ssh$X\" \"\\$@\"\n+\tEOF\n+\tGIT_SSH=\"$TRASH_DIRECTORY/uplink\" &&\n+\texport GIT_SSH &&\n+\tgit clone \"[myhost:123]:src\" ssh-bracket-clone-sshlike-uplink &&\n+\texpect_ssh \"-p 123\" myhost src\n+'\n+\n test_expect_success 'plink is treated specially (as putty)' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/plink\" &&\n \tgit clone \"[myhost:123]:src\" ssh-bracket-clone-plink-0 &&\n-- \n2.15.0.rc1.287.g2b38de12cc\n\n"},{"id":"330871","messageId":"20171023231914.azmri2cjahmc7obe@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023231625.6mhcyqti7vdg6yot@aiede.mtv.corp.google.com","subject":"[PATCH 4/5] ssh: 'simple' variant does not support -4/-6","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T23:19:14Z","receivedAt":"2017-10-23T23:19:21Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"If the user passes -4/--ipv4 or -6/--ipv6 to \"git fetch\" or \"git push\"\nand the ssh command configured with GIT_SSH does not support such a\nsetting, error out instead of ignoring the option and continuing.\n\nSigned-off-by: Jonathan Nieder \nAcked-by: Stefan Beller \n---\nAs before, just rebased.\n\n connect.c | 25 ++++++++++++++++++++++---\n t/t5601-clone.sh | 12 ++++++------\n 2 files changed, 28 insertions(+), 9 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 0441dcbacf..9c64e8155a 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -923,11 +923,30 @@ static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n \t\t\t\t get_protocol_version_config());\n \t}\n \n-\tif (variant != VARIANT_SIMPLE) {\n-\t\tif (flags & CONNECT_IPV4)\n+\tif (flags & CONNECT_IPV4) {\n+\t\tswitch (variant) {\n+\t\tcase VARIANT_AUTO:\n+\t\t\tBUG(\"VARIANT_AUTO passed to push_ssh_options\");\n+\t\tcase VARIANT_SIMPLE:\n+\t\t\tdie(\"ssh variant 'simple' does not support -4\");\n+\t\tcase VARIANT_SSH:\n+\t\tcase VARIANT_PLINK:\n+\t\tcase VARIANT_PUTTY:\n+\t\tcase VARIANT_TORTOISEPLINK:\n \t\t\targv_array_push(args, \"-4\");\n-\t\telse if (flags & CONNECT_IPV6)\n+\t\t}\n+\t} else if (flags & CONNECT_IPV6) {\n+\t\tswitch (variant) {\n+\t\tcase VARIANT_AUTO:\n+\t\t\tBUG(\"VARIANT_AUTO passed to push_ssh_options\");\n+\t\tcase VARIANT_SIMPLE:\n+\t\t\tdie(\"ssh variant 'simple' does not support -6\");\n+\t\tcase VARIANT_SSH:\n+\t\tcase VARIANT_PLINK:\n+\t\tcase VARIANT_PUTTY:\n+\t\tcase VARIANT_TORTOISEPLINK:\n \t\t\targv_array_push(args, \"-6\");\n+\t\t}\n \t}\n \n \tif (variant == VARIANT_TORTOISEPLINK)\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex df9dfafdd8..ea401cec1f 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -367,9 +367,10 @@ test_expect_success 'OpenSSH variant passes -4' '\n \texpect_ssh \"-4 -p 123\" myhost src\n '\n \n-test_expect_success 'variant can be overriden' '\n-\tgit -c ssh.variant=simple clone -4 \"[myhost:123]:src\" ssh-simple-clone &&\n-\texpect_ssh myhost src\n+test_expect_success 'variant can be overridden' '\n+\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/putty\" &&\n+\tgit -c ssh.variant=putty clone -4 \"[myhost:123]:src\" ssh-putty-clone &&\n+\texpect_ssh \"-4 -P 123\" myhost src\n '\n \n test_expect_success 'variant=auto picks based on basename' '\n@@ -378,10 +379,9 @@ test_expect_success 'variant=auto picks based on basename' '\n \texpect_ssh \"-4 -P 123\" myhost src\n '\n \n-test_expect_success 'simple is treated as simple' '\n+test_expect_success 'simple does not support -4/-6' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n-\tgit clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple &&\n-\texpect_ssh myhost src\n+\ttest_must_fail git clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple\n '\n \n test_expect_success 'uplink is treated as simple' '\n-- \n2.15.0.rc1.287.g2b38de12cc\n\n"},{"id":"330872","messageId":"20171023231948.5jltnftktu5bmny3@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023231625.6mhcyqti7vdg6yot@aiede.mtv.corp.google.com","subject":"[PATCH 5/5] ssh: 'simple' variant does not support --port","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-23T23:19:48Z","receivedAt":"2017-10-23T23:19:55Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"When trying to connect to an ssh:// URL with port explicitly specified\nand the ssh command configured with GIT_SSH does not support such a\nsetting, it is less confusing to error out than to silently suppress\nthe port setting and continue.\n\nThis requires updating the GIT_SSH setting in t5603-clone-dirname.sh.\nThat test is about the directory name produced when cloning various\nURLs. It uses an ssh wrapper that ignores all its arguments but does\nnot declare that it supports a port argument; update it to set\nGIT_SSH_VARIANT=ssh to do so. (Real-life ssh wrappers that pass a\nport argument to OpenSSH would also support -G and would not require\nsuch an update.)\n\nReported-by: William Yan \nSigned-off-by: Jonathan Nieder \nAcked-by: Stefan Beller \n---\nThat's the end of the series. Thanks for reading.\n\n connect.c | 15 ++++++++++++---\n t/t5601-clone.sh | 10 ++++++++--\n t/t5603-clone-dirname.sh | 2 ++\n 3 files changed, 22 insertions(+), 5 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 9c64e8155a..06bcd3981e 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -952,11 +952,20 @@ static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n \tif (variant == VARIANT_TORTOISEPLINK)\n \t\targv_array_push(args, \"-batch\");\n \n-\tif (port && variant != VARIANT_SIMPLE) {\n-\t\tif (variant == VARIANT_SSH)\n+\tif (port) {\n+\t\tswitch (variant) {\n+\t\tcase VARIANT_AUTO:\n+\t\t\tBUG(\"VARIANT_AUTO passed to push_ssh_options\");\n+\t\tcase VARIANT_SIMPLE:\n+\t\t\tdie(\"ssh variant 'simple' does not support setting port\");\n+\t\tcase VARIANT_SSH:\n \t\t\targv_array_push(args, \"-p\");\n-\t\telse\n+\t\t\tbreak;\n+\t\tcase VARIANT_PLINK:\n+\t\tcase VARIANT_PUTTY:\n+\t\tcase VARIANT_TORTOISEPLINK:\n \t\t\targv_array_push(args, \"-P\");\n+\t\t}\n \n \t\targv_array_push(args, port);\n \t}\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex ea401cec1f..f9a2ae84c7 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -381,12 +381,18 @@ test_expect_success 'variant=auto picks based on basename' '\n \n test_expect_success 'simple does not support -4/-6' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n-\ttest_must_fail git clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple\n+\ttest_must_fail git clone -4 \"myhost:src\" ssh-4-clone-simple\n+'\n+\n+test_expect_success 'simple does not support port' '\n+\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n+\ttest_must_fail git clone \"[myhost:123]:src\" ssh-bracket-clone-simple\n '\n \n test_expect_success 'uplink is treated as simple' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/uplink\" &&\n-\tgit clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n+\ttest_must_fail git clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n+\tgit clone \"myhost:src\" ssh-clone-uplink &&\n \texpect_ssh myhost src\n '\n \ndiff --git a/t/t5603-clone-dirname.sh b/t/t5603-clone-dirname.sh\nindex d5af758129..13b5e5eb9b 100755\n--- a/t/t5603-clone-dirname.sh\n+++ b/t/t5603-clone-dirname.sh\n@@ -11,7 +11,9 @@ test_expect_success 'setup ssh wrapper' '\n \tgit upload-pack \"$TRASH_DIRECTORY\"\n \tEOF\n \tGIT_SSH=\"$TRASH_DIRECTORY/ssh-wrapper\" &&\n+\tGIT_SSH_VARIANT=ssh &&\n \texport GIT_SSH &&\n+\texport GIT_SSH_VARIANT &&\n \texport TRASH_DIRECTORY\n '\n \n-- \n2.15.0.rc1.287.g2b38de12cc\n\n"},{"id":"330873","messageId":"20171023232713.GA79163@google.com","threadId":"46746","inReplyTo":"20171023231839.2g5wudvmoj3hphua@aiede.mtv.corp.google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-23T23:27:13Z","receivedAt":"2017-10-23T23:27:26Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/23, Jonathan Nieder wrote:\n> Android's \"repo\" tool is a tool for managing a large codebase\n> consisting of multiple smaller repositories, similar to Git's\n> submodule feature. Starting with Git 94b8ae5a (ssh: introduce a\n> 'simple' ssh variant, 2017-10-16), users noticed that it stopped\n> handling the port in ssh:// URLs.\n> \n> The cause: when it encounters ssh:// URLs, repo pre-connects to the\n> server and sets GIT_SSH to a helper \".repo/repo/git_ssh\" that reuses\n> that connection. Before 94b8ae5a, the helper was assumed to support\n> OpenSSH options for lack of a better guess and got passed a -p option\n> to set the port. After that patch, it uses the new default of a\n> simple helper that does not accept an option to set the port.\n> \n> The next release of \"repo\" will set GIT_SSH_VARIANT to \"ssh\" to avoid\n> that. But users of old versions and of other similar GIT_SSH\n> implementations would not get the benefit of that fix.\n> \n> So update the default to use OpenSSH options again, with a twist. As\n> observed in 94b8ae5a, we cannot assume that $GIT_SSH always handles\n> OpenSSH options: common helpers such as travis-ci's dpl[*] are\n> configured using GIT_SSH and do not accept OpenSSH options. So make\n> the default a new variant \"auto\", with the following behavior:\n> \n> 1. First, check for a recognized basename, like today.\n> \n> 2. If the basename is not recognized, check whether $GIT_SSH supports\n> OpenSSH options by running\n> \n> \t$GIT_SSH -G \n> \n> This returns status 0 and prints configuration in OpenSSH if it\n> recognizes all and returns status 255 if it encounters\n> an unrecognized option. A wrapper script like\n> \n> \texec ssh -- \"$@\"\n> \n> would fail with\n> \n> \tssh: Could not resolve hostname -g: Name or service not known\n> \n> , correctly reflecting that it does not support OpenSSH options.\n> \n> 3. Based on the result from step (2), behave like \"ssh\" (if it\n> succeeded) or \"simple\" (if it failed).\n> \n> This way, the default ssh variant for unrecognized commands can handle\n> both the repo and dpl cases as intended.\n> \n> [*] https://github.com/travis-ci/dpl/blob/6c3fddfda1f2a85944c544446b068bac0a77c049/lib/dpl/provider.rb#L215\n> \n> Reported-by: William Yan \n> Improved-by: Jonathan Tan \n> Signed-off-by: Jonathan Nieder \n> ---\n> This is the main one. Simplified by making \"auto\" behave the same as\n> unset.\n\nI still don't see the benefit of allowing a user to explicitly set\n'auto' then, if setting it to 'auto' is effectively a noop. But maybe\nthere's something I'm not seeing.\n\n> \n> Documentation/config.txt | 24 ++++++++------\n> connect.c | 82 +++++++++++++++++++++++++++++++-----------------\n> t/t5601-clone.sh | 20 ++++++++++++\n> 3 files changed, 88 insertions(+), 38 deletions(-)\n> \n> diff --git a/Documentation/config.txt b/Documentation/config.txt\n> index 0460af37e2..6dffa4aa3d 100644\n> --- a/Documentation/config.txt\n> +++ b/Documentation/config.txt\n> @@ -2081,16 +2081,22 @@ matched against are those given directly to Git commands. This means any URLs\n> visited as a result of a redirection do not participate in matching.\n> \n> ssh.variant::\n> -\tDepending on the value of the environment variables `GIT_SSH` or\n> -\t`GIT_SSH_COMMAND`, or the config setting `core.sshCommand`, Git\n> -\tauto-detects whether to adjust its command-line parameters for use\n> -\twith ssh (OpenSSH), plink or tortoiseplink, as opposed to the default\n> -\t(simple).\n> +\tBy default, Git determines the command line arguments to use\n> +\tbased on the basename of the configured SSH command (configured\n> +\tusing the environment variable `GIT_SSH` or `GIT_SSH_COMMAND` or\n> +\tthe config setting `core.sshCommand`). If the basename is\n> +\tunrecognized, Git will attempt to detect support of OpenSSH\n> +\toptions by first invoking the configured SSH command with the\n> +\t`-G` (print configuration) option and will subsequently use\n> +\tOpenSSH options (if that is successful) or no options besides\n> +\tthe host and remote command (if it fails).\n> +\n> -The config variable `ssh.variant` can be set to override this auto-detection;\n> -valid values are `ssh`, `simple`, `plink`, `putty` or `tortoiseplink`. Any\n> -other value will be treated as normal ssh. This setting can be overridden via\n> -the environment variable `GIT_SSH_VARIANT`.\n> +The config variable `ssh.variant` can be set to override this detection:\n> +valid values are `ssh` (to use OpenSSH options), `plink`, `putty`,\n> +`tortoiseplink`, `simple` (no options except the host and remote command).\n> +The default auto-detection can be explicitly requested using the value\n> +`auto`. Any other value is treated as `ssh`. This setting can also be\n> +overridden via the environment variable `GIT_SSH_VARIANT`.\n> +\n> The current command-line parameters used for each variant are as\n> follows:\n> diff --git a/connect.c b/connect.c\n> index 77ab6db3bb..0441dcbacf 100644\n> --- a/connect.c\n> +++ b/connect.c\n> @@ -777,6 +777,7 @@ static const char *get_ssh_command(void)\n> }\n> \n> enum ssh_variant {\n> +\tVARIANT_AUTO,\n> \tVARIANT_SIMPLE,\n> \tVARIANT_SSH,\n> \tVARIANT_PLINK,\n> @@ -784,14 +785,16 @@ enum ssh_variant {\n> \tVARIANT_TORTOISEPLINK,\n> };\n> \n> -static int override_ssh_variant(enum ssh_variant *ssh_variant)\n> +static void override_ssh_variant(enum ssh_variant *ssh_variant)\n> {\n> \tconst char *variant = getenv(\"GIT_SSH_VARIANT\");\n> \n> \tif (!variant && git_config_get_string_const(\"ssh.variant\", &variant))\n> -\t\treturn 0;\n> +\t\treturn;\n> \n> -\tif (!strcmp(variant, \"plink\"))\n> +\tif (!strcmp(variant, \"auto\"))\n> +\t\t*ssh_variant = VARIANT_AUTO;\n> +\telse if (!strcmp(variant, \"plink\"))\n> \t\t*ssh_variant = VARIANT_PLINK;\n> \telse if (!strcmp(variant, \"putty\"))\n> \t\t*ssh_variant = VARIANT_PUTTY;\n> @@ -801,18 +804,18 @@ static int override_ssh_variant(enum ssh_variant *ssh_variant)\n> \t\t*ssh_variant = VARIANT_SIMPLE;\n> \telse\n> \t\t*ssh_variant = VARIANT_SSH;\n> -\n> -\treturn 1;\n> }\n> \n> static enum ssh_variant determine_ssh_variant(const char *ssh_command,\n> \t\t\t\t\t int is_cmdline)\n> {\n> -\tenum ssh_variant ssh_variant = VARIANT_SIMPLE;\n> +\tenum ssh_variant ssh_variant = VARIANT_AUTO;\n> \tconst char *variant;\n> \tchar *p = NULL;\n> \n> -\tif (override_ssh_variant(&ssh_variant))\n> +\toverride_ssh_variant(&ssh_variant);\n> +\n> +\tif (ssh_variant != VARIANT_AUTO)\n> \t\treturn ssh_variant;\n> \n> \tif (!is_cmdline) {\n> @@ -908,6 +911,38 @@ static struct child_process *git_connect_git(int fd[2], char *hostandport,\n> \treturn conn;\n> }\n> \n> +static void push_ssh_options(struct argv_array *args, struct argv_array *env,\n> +\t\t\t enum ssh_variant variant, const char *port,\n> +\t\t\t int flags)\n> +{\n> +\tif (variant == VARIANT_SSH &&\n> +\t get_protocol_version_config() > 0) {\n> +\t\targv_array_push(args, \"-o\");\n> +\t\targv_array_push(args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n> +\t\targv_array_pushf(env, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n> +\t\t\t\t get_protocol_version_config());\n> +\t}\n> +\n> +\tif (variant != VARIANT_SIMPLE) {\n> +\t\tif (flags & CONNECT_IPV4)\n> +\t\t\targv_array_push(args, \"-4\");\n> +\t\telse if (flags & CONNECT_IPV6)\n> +\t\t\targv_array_push(args, \"-6\");\n> +\t}\n> +\n> +\tif (variant == VARIANT_TORTOISEPLINK)\n> +\t\targv_array_push(args, \"-batch\");\n> +\n> +\tif (port && variant != VARIANT_SIMPLE) {\n> +\t\tif (variant == VARIANT_SSH)\n> +\t\t\targv_array_push(args, \"-p\");\n> +\t\telse\n> +\t\t\targv_array_push(args, \"-P\");\n> +\n> +\t\targv_array_push(args, port);\n> +\t}\n> +}\n> +\n> /* Prepare a child_process for use by Git's SSH-tunneled transport. */\n> static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n> \t\t\t const char *port, int flags)\n> @@ -937,33 +972,22 @@ static void fill_ssh_args(struct child_process *conn, const char *ssh_host,\n> \n> \targv_array_push(&conn->args, ssh);\n> \n> -\tif (variant == VARIANT_SSH &&\n> -\t get_protocol_version_config() > 0) {\n> -\t\targv_array_push(&conn->args, \"-o\");\n> -\t\targv_array_push(&conn->args, \"SendEnv=\" GIT_PROTOCOL_ENVIRONMENT);\n> -\t\targv_array_pushf(&conn->env_array, GIT_PROTOCOL_ENVIRONMENT \"=version=%d\",\n> -\t\t\t\t get_protocol_version_config());\n> -\t}\n> -\n> -\tif (variant != VARIANT_SIMPLE) {\n> -\t\tif (flags & CONNECT_IPV4)\n> -\t\t\targv_array_push(&conn->args, \"-4\");\n> -\t\telse if (flags & CONNECT_IPV6)\n> -\t\t\targv_array_push(&conn->args, \"-6\");\n> -\t}\n> +\tif (variant == VARIANT_AUTO) {\n> +\t\tstruct child_process detect = CHILD_PROCESS_INIT;\n> \n> -\tif (variant == VARIANT_TORTOISEPLINK)\n> -\t\targv_array_push(&conn->args, \"-batch\");\n> +\t\tdetect.use_shell = conn->use_shell;\n> +\t\tdetect.no_stdin = detect.no_stdout = detect.no_stderr = 1;\n> \n> -\tif (port && variant != VARIANT_SIMPLE) {\n> -\t\tif (variant == VARIANT_SSH)\n> -\t\t\targv_array_push(&conn->args, \"-p\");\n> -\t\telse\n> -\t\t\targv_array_push(&conn->args, \"-P\");\n> +\t\targv_array_push(&detect.args, ssh);\n> +\t\targv_array_push(&detect.args, \"-G\");\n> +\t\tpush_ssh_options(&detect.args, &detect.env_array,\n> +\t\t\t\t VARIANT_SSH, port, flags);\n> +\t\targv_array_push(&detect.args, ssh_host);\n> \n> -\t\targv_array_push(&conn->args, port);\n> +\t\tvariant = run_command(&detect) ? VARIANT_SIMPLE : VARIANT_SSH;\n> \t}\n> \n> +\tpush_ssh_options(&conn->args, &conn->env_array, variant, port, flags);\n> \targv_array_push(&conn->args, ssh_host);\n> }\n> \n> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\n> index 86811a0c35..df9dfafdd8 100755\n> --- a/t/t5601-clone.sh\n> +++ b/t/t5601-clone.sh\n> @@ -372,6 +372,12 @@ test_expect_success 'variant can be overriden' '\n> \texpect_ssh myhost src\n> '\n> \n> +test_expect_success 'variant=auto picks based on basename' '\n> +\tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/plink\" &&\n> +\tgit -c ssh.variant=auto clone -4 \"[myhost:123]:src\" ssh-auto-clone &&\n> +\texpect_ssh \"-4 -P 123\" myhost src\n> +'\n> +\n> test_expect_success 'simple is treated as simple' '\n> \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/simple\" &&\n> \tgit clone -4 \"[myhost:123]:src\" ssh-bracket-clone-simple &&\n> @@ -384,6 +390,20 @@ test_expect_success 'uplink is treated as simple' '\n> \texpect_ssh myhost src\n> '\n> \n> +test_expect_success 'OpenSSH-like uplink is treated as ssh' '\n> +\twrite_script \"$TRASH_DIRECTORY/uplink\" <<-EOF &&\n> +\tif test \"\\$1\" = \"-G\"\n> +\tthen\n> +\t\texit 0\n> +\tfi &&\n> +\texec \"\\$TRASH_DIRECTORY/ssh$X\" \"\\$@\"\n> +\tEOF\n> +\tGIT_SSH=\"$TRASH_DIRECTORY/uplink\" &&\n> +\texport GIT_SSH &&\n> +\tgit clone \"[myhost:123]:src\" ssh-bracket-clone-sshlike-uplink &&\n> +\texpect_ssh \"-p 123\" myhost src\n> +'\n> +\n> test_expect_success 'plink is treated specially (as putty)' '\n> \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/plink\" &&\n> \tgit clone \"[myhost:123]:src\" ssh-bracket-clone-plink-0 &&\n> -- \n> 2.15.0.rc1.287.g2b38de12cc\n> \n\n-- \nBrandon Williams\n"},{"id":"330874","messageId":"CAGZ79kaC1-YR7zoboU4pAk0MHeYJcPv-34CiwpRY1Ng9WBaSLQ@mail.gmail.com","threadId":"46746","inReplyTo":"20171023232713.GA79163@google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-23T23:33:17Z","receivedAt":"2017-10-23T23:33:23Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":">> This is the main one. Simplified by making \"auto\" behave the same as\n>> unset.\n>\n> I still don't see the benefit of allowing a user to explicitly set\n> 'auto' then, if setting it to 'auto' is effectively a noop. But maybe\n> there's something I'm not seeing.\n>\n\nIf /etc/gitconfig says SSH, and you have different configs in your\ndifferent repos,\nthe easiest way out is setting AUTO in ~/.gitconfig.\n"},{"id":"330875","messageId":"20171024000931.14814-1-sbeller@google.com","threadId":"46746","inReplyTo":"CAGZ79kYwARXNWRS4AgwTP7peZiWiwCBvWFiEr9TbpbWjgysfZA@mail.gmail.com","subject":"[WIP PATCH] diff: add option to ignore whitespaces for move detection only","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-24T00:09:31Z","receivedAt":"2017-10-24T00:09:40Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"Signed-off-by: Stefan Beller \n---\n\n diff.c | 10 ++--\n diff.h | 1 +\n t/t4015-diff-whitespace.sh | 114 ++++++++++++++++++++++++++++++++++++++++++++-\n \nSee, only 10 lines of code! (and a few more for tests)\n\nWe have run out of space in diff_options.flags,touched_flags.\nas we 1<color_moved_ignore_space) {\n \t\t\twhile (*cp < *endp && isspace(**cp))\n \t\t\t\t(*cp)++;\n \t\t\t/*\n@@ -751,7 +752,8 @@ static int moved_entry_cmp(const struct diff_options *diffopt,\n \tconst char *ap = a->es->line, *ae = a->es->line + a->es->len;\n \tconst char *bp = b->es->line, *be = b->es->line + b->es->len;\n \n-\tif (!(diffopt->xdl_opts & XDF_WHITESPACE_FLAGS))\n+\tif (!(diffopt->xdl_opts & XDF_WHITESPACE_FLAGS) &&\n+\t !diffopt->color_moved_ignore_space)\n \t\treturn a->es->len != b->es->len || memcmp(ap, bp, a->es->len);\n \n \tif (DIFF_XDL_TST(diffopt, IGNORE_WHITESPACE_AT_EOL)) {\n@@ -774,7 +776,7 @@ static int moved_entry_cmp(const struct diff_options *diffopt,\n \n static unsigned get_string_hash(struct emitted_diff_symbol *es, struct diff_options *o)\n {\n-\tif (o->xdl_opts & XDF_WHITESPACE_FLAGS) {\n+\tif ((o->xdl_opts & XDF_WHITESPACE_FLAGS) || o->color_moved_ignore_space) {\n \t\tstatic struct strbuf sb = STRBUF_INIT;\n \t\tconst char *ap = es->line, *ae = es->line + es->len;\n \t\tint c;\n@@ -4660,6 +4662,8 @@ int diff_opt_parse(struct diff_options *options,\n \t\tDIFF_XDL_CLR(options, NEED_MINIMAL);\n \telse if (!strcmp(arg, \"-w\") || !strcmp(arg, \"--ignore-all-space\"))\n \t\tDIFF_XDL_SET(options, IGNORE_WHITESPACE);\n+\telse if (!strcmp(arg, \"--ignore-all-space-in-move-detection\"))\n+\t\toptions->color_moved_ignore_space = 1;\n \telse if (!strcmp(arg, \"-b\") || !strcmp(arg, \"--ignore-space-change\"))\n \t\tDIFF_XDL_SET(options, IGNORE_WHITESPACE_CHANGE);\n \telse if (!strcmp(arg, \"--ignore-space-at-eol\"))\ndiff --git a/diff.h b/diff.h\nindex aca150ba2e..6ba3f53bbd 100644\n--- a/diff.h\n+++ b/diff.h\n@@ -196,6 +196,7 @@ struct diff_options {\n \t} color_moved;\n \t#define COLOR_MOVED_DEFAULT COLOR_MOVED_ZEBRA\n \t#define COLOR_MOVED_MIN_ALNUM_COUNT 20\n+\tint color_moved_ignore_space;\n };\n \n void diff_emit_submodule_del(struct diff_options *o, const char *line);\ndiff --git a/t/t4015-diff-whitespace.sh b/t/t4015-diff-whitespace.sh\nindex 6c9a93b734..d7ee3aabf2 100755\n--- a/t/t4015-diff-whitespace.sh\n+++ b/t/t4015-diff-whitespace.sh\n@@ -1677,7 +1677,119 @@ test_expect_success 'move detection with submodules' '\n \n \t# nor did we mess with it another way\n \tgit diff --submodule=diff --color | test_decode_color >expect &&\n-\ttest_cmp expect decoded_actual\n+\ttest_cmp expect decoded_actual &&\n+\trm -rf bananas &&\n+\tgit submodule deinit bananas\n+'\n+\n+test_expect_success 'move detection only ignores white spaces' '\n+\tgit reset --hard &&\n+\tq_to_tab <<-\\EOF >function.c &&\n+\tint func()\n+\t{\n+\tQif (foo) {\n+\tQQ// this part of the function\n+\tQQ// function will be very long\n+\tQQ// indeed. We must exceed both\n+\tQQ// per-line and number of line\n+\tQQ// minimums\n+\tQQ;\n+\tQ}\n+\tQbaz();\n+\tQbar();\n+\tQ// more unrelated stuff\n+\t}\n+\tEOF\n+\tgit add function.c &&\n+\tgit commit -m \"add function.c\" &&\n+\tq_to_tab <<-\\EOF >function.c &&\n+\tint do_foo()\n+\t{\n+\tQ// this part of the function\n+\tQ// function will be very long\n+\tQ// indeed. We must exceed both\n+\tQ// per-line and number of line\n+\tQ// minimums\n+\tQ;\n+\t}\n+\n+\tint func()\n+\t{\n+\tQif (foo)\n+\tQQdo_foo();\n+\tQbaz();\n+\tQbar();\n+\tQ// more unrelated stuff\n+\t}\n+\tEOF\n+\n+\t# Make sure we get a different diff using -w (\"moved function header\")\n+\tgit diff --color --color-moved -w |\n+\t\tgrep -v \"index\" |\n+\t\ttest_decode_color >actual &&\n+\tq_to_tab <<-\\EOF >expected &&\n+\tdiff --git a/function.c b/function.c\n+\t--- a/function.c\n+\t+++ b/function.c\n+\t@@ -1,6 +1,5 @@\n+\t-int func()\n+\t+int do_foo()\n+\t {\n+\t-\tif (foo) {\n+\t Q// this part of the function\n+\t Q// function will be very long\n+\t Q// indeed. We must exceed both\n+\t@@ -8,6 +7,11 @@ int func()\n+\t Q// minimums\n+\t Q;\n+\t }\n+\t+\n+\t+int func()\n+\t+{\n+\t+Qif (foo)\n+\t+QQdo_foo();\n+\t Qbaz();\n+\t Qbar();\n+\t Q// more unrelated stuff\n+\tEOF\n+\ttest_cmp expected actual &&\n+\n+\t# And now ignoring white space only in the move detection\n+\tgit diff --color --color-moved --ignore-all-space-in-move-detection |\n+\t\tgrep -v \"index\" |\n+\t\ttest_decode_color >actual &&\n+\tq_to_tab <<-\\EOF >expected &&\n+\tdiff --git a/function.c b/function.c\n+\t--- a/function.c\n+\t+++ b/function.c\n+\t@@ -1,13 +1,17 @@\n+\t+int do_foo()\n+\t+{\n+\t+Q// this part of the function\n+\t+Q// function will be very long\n+\t+Q// indeed. We must exceed both\n+\t+Q// per-line and number of line\n+\t+Q// minimums\n+\t+Q;\n+\t+}\n+\t+\n+\t int func()\n+\t {\n+\t-Qif (foo) {\n+\t-QQ// this part of the function\n+\t-QQ// function will be very long\n+\t-QQ// indeed. We must exceed both\n+\t-QQ// per-line and number of line\n+\t-QQ// minimums\n+\t-QQ;\n+\t-Q}\n+\t+Qif (foo)\n+\t+QQdo_foo();\n+\t Qbaz();\n+\t Qbar();\n+\t Q// more unrelated stuff\n+\tEOF\n+\ttest_cmp expected actual\n '\n \n test_done\n-- \n2.15.0.rc2.6.g953226eb5f\n\n"},{"id":"330884","messageId":"xmqqpo9d9v5y.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023231703.f457jp43dh7fddpv@aiede.mtv.corp.google.com","subject":"Re: [PATCH 1/5] connect: split git:// setup into a separate function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-24T01:44:25Z","receivedAt":"2017-10-24T01:44:35Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> The git_connect function is growing long. Split the\n> PROTO_GIT-specific portion to a separate function to make it easier to\n> read.\n>\n> No functional change intended.\n>\n> Signed-off-by: Jonathan Nieder \n> Reviewed-by: Stefan Beller \n> ---\n> As before, except with sbeller's Reviewed-by.\n\nI found this quite nice, except for one thing.\n\n> +/*\n> + * Open a connection using Git's native protocol.\n> + *\n> + * The caller is responsible for freeing hostandport, but this function may\n> + * modify it (for example, to truncate it to remove the port part).\n> + */\n> +static struct child_process *git_connect_git(int fd[2], char *hostandport,\n> +\t\t\t\t\t const char *path, const char *prog,\n> +\t\t\t\t\t int flags)\n> +{\n> +\tstruct child_process *conn = &no_fork;\n> +\tstruct strbuf request = STRBUF_INIT;\n\nAs this one decides what \"conn\" to return, including the fallback\n&no_fork instance,...\n\n> +\t...\n> +\treturn conn;\n> +}\n> +\n> /*\n> * This returns a dummy child_process if the transport protocol does not\n> * need fork(2), or a struct child_process object if it does. Once done,\n> @@ -881,50 +939,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n\nEach of the if/elseif/ cascade, one of which calls the new helper,\nnow makes an explicit assignment to \"conn\" declared in\ngit_connect().\n\nWhich means the defaulting of git_connect::conn to &no_fork is now\nunneeded. One of the things that made the original cascade a bit\nharder to follow than necessary, aside from the physical length of\nthe PROTO_GIT part, was that the case where conn remains to point at\nno_fork looked very special and it was buried in that long PROTO_GIT\npart. Now the main source of that issue is fixed, it would make it\nclear to leave conn uninitialized (or initialize to NULL---but leaving\nit uninitialized would make the intention of the code more clear, I\nwould think, that each of the if/elseif/ cascade must assign to it).\n\n> \t\tprintf(\"Diag: path=%s\\n\", path ? path : \"NULL\");\n> \t\tconn = NULL;\n> \t} else if (protocol == PROTO_GIT) {\n> -\t\tstruct strbuf request = STRBUF_INIT;\n> -...\n> +\t\tconn = git_connect_git(fd, hostandport, path, prog, flags);\n> \t} else {\n> \t\tstruct strbuf cmd = STRBUF_INIT;\n> \t\tconst char *const *var;\n"},{"id":"330886","messageId":"xmqqlgk19upq.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"CAGZ79kYwARXNWRS4AgwTP7peZiWiwCBvWFiEr9TbpbWjgysfZA@mail.gmail.com","subject":"Re: [PATCH 1/5] connect: split git:// setup into a separate function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-24T01:54:09Z","receivedAt":"2017-10-24T01:54:16Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Stefan Beller writes:\n\n> I think once this option is given, all we have to do is pay attention to\n> this option in diff.c#moved_entry_cmp/next_byte, which is best built\n> on top of Peffs recent fixes origin/jk/diff-color-moved-fix.\n> Would that be of interest for people?\n\nTwo things and a half.\n\n * I was hoping that the next_byte() and string_hash() thing, once\n they are cleaned up, will eventually be shared with the xdiff/\n code at the lower layer, which needs to do pretty much the same\n in order to implement various whitespace ignoring options. I am\n not sure how well the approach taken by the WIP patch meshes with\n the needs of the lower layer.\n\n * I agree that -w that applies only one or the other and not both\n may sometimes produce a better/readable result, but the more\n important part is how the user can tell when to exercise the\n option. Would it be realistic to expect them to try -w in\n different combinations and see which looks the best? What if we\n have a patch that touch two files, one looks better with -w only\n for coloring moved and the other looks better with -w for both?\n\n * As moved-lines display is mostly a presentation thing, I wonder\n if it makes sense to always match loosely wrt whitespace\n differences. It is tempting because if it is true, we do not\n have to worry about the second issue above.\n\nThanks.\n"},{"id":"330887","messageId":"xmqqh8up9udp.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023231733.tattcu2aws7kgous@aiede.mtv.corp.google.com","subject":"Re: [PATCH 2/5] connect: split ssh command line options into separate function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-24T02:01:22Z","receivedAt":"2017-10-24T02:01:29Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> The git_connect function is growing long. Split the portion that\n> discovers an ssh command and options it accepts before the service\n> name and path to a separate function to make it easier to read.\n>\n> No functional change intended.\n>\n> Signed-off-by: Jonathan Nieder \n> Reviewed-by: Stefan Beller \n> ---\n> As before, except for the Reviewed-by.\n>\n> connect.c | 116 +++++++++++++++++++++++++++++++++-----------------------------\n> 1 file changed, 61 insertions(+), 55 deletions(-)\n\nLooks like a straight-forwrd split. Makes sense.\n\nThanks.\n"},{"id":"330888","messageId":"xmqq60b59toe.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023213159.eitrjrqrh277advm@aiede.mtv.corp.google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-24T02:16:33Z","receivedAt":"2017-10-24T02:16:41Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> 1. First, check whether $GIT_SSH supports OpenSSH options by running\n>\n> \t$GIT_SSH -G \n>\n> This returns status 0 and prints configuration in OpenSSH if it\n> recognizes all and returns status 255 if it encounters\n> an unrecognized option. A wrapper script like\n>\n> \texec ssh -- \"$@\"\n>\n> would fail with\n>\n> \tssh: Could not resolve hostname -g: Name or service not known\n>\n> , correctly reflecting that it does not support OpenSSH options.\n\nTwo comments.\n\n * It would have been really nicer if the push_ssh_options() got\n split from its caller in a separate preparatory [PATCH 2.5/5].\n\n * Use of -G for auto-detection is clever and cute, but do we know\n how safe it would be in the real world? What worries me the most\n is \"myssh -G localhost\" (no extra options) that does not fit our\n expectation on \"-G\" to either exit immediately with an error when\n it is not understood, or to exit like OpenSSH does, and instead\n successfully makes a connection and gets stuck.\n"},{"id":"330889","messageId":"xmqq1slt9tdt.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171023231625.6mhcyqti7vdg6yot@aiede.mtv.corp.google.com","subject":"Re: [PATCH v2 0/5] Coping with unrecognized ssh wrapper scripts in GIT_SSH","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-24T02:22:54Z","receivedAt":"2017-10-24T02:23:01Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> Jonathan Tan wrote:\n>>> On 10/23, Jonathan Nieder wrote:\n>\n>>>> If this looks good, I can reroll in a moment.\n>>\n>> Yes, this looks good.\n>\n> Thanks. Here goes.\n>\n> The interdiff is upthread. Thanks, all, for the quick review.\n>\n> Jonathan Nieder (5):\n> connect: split git:// setup into a separate function\n> connect: split ssh command line options into separate function\n> ssh: 'auto' variant to select between 'ssh' and 'simple'\n> ssh: 'simple' variant does not support -4/-6\n> ssh: 'simple' variant does not support --port\n\nThese looked mostly good. I left some nitpicks on individual\npatches.\n\n"},{"id":"330892","messageId":"CAGZ79kbba9KuDX7HsxhW3jXs7ocWfZg=KSHE-GsXjntnT4PWdg@mail.gmail.com","threadId":"46746","inReplyTo":"xmqqlgk19upq.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/5] connect: split git:// setup into a separate function","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-24T02:52:22Z","receivedAt":"2017-10-24T02:52:28Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Mon, Oct 23, 2017 at 6:54 PM, Junio C Hamano wrote:\n> Stefan Beller writes:\n>\n>> I think once this option is given, all we have to do is pay attention to\n>> this option in diff.c#moved_entry_cmp/next_byte, which is best built\n>> on top of Peffs recent fixes origin/jk/diff-color-moved-fix.\n>> Would that be of interest for people?\n>\n> Two things and a half.\n>\n> * I was hoping that the next_byte() and string_hash() thing, once\n> they are cleaned up, will eventually be shared with the xdiff/\n> code at the lower layer, which needs to do pretty much the same\n> in order to implement various whitespace ignoring options. I am\n> not sure how well the approach taken by the WIP patch meshes with\n> the needs of the lower layer.\n\nGood point. I'll keep that in mind when redoing that patch.\n(I might even try to clean up the xdiff stuff and reuse the code\nhere)\n\n> * I agree that -w that applies only one or the other and not both\n> may sometimes produce a better/readable result, but the more\n> important part is how the user can tell when to exercise the\n> option. Would it be realistic to expect them to try -w in\n> different combinations and see which looks the best? What if we\n> have a patch that touch two files, one looks better with -w only\n> for coloring moved and the other looks better with -w for both?\n>\n> * As moved-lines display is mostly a presentation thing, I wonder\n> if it makes sense to always match loosely wrt whitespace\n> differences. It is tempting because if it is true, we do not\n> have to worry about the second issue above.\n\nWell, sometimes the user wants to know if it is byte-for-byte identical\n(unlikely to be code, but maybe column oriented data for input;\nthink of all our FORTRAN users. ;)\nand at other times the user just wants to approximately know\nif it is the same (i.e. code ignoring white space changes).\nIf Git was *really* smart w.r.t. languages it might even ignore\nidentifier renames in the programming language.\n\nSo I think an \"unconditionally ignore all white space in\nmove detection\" is not the best (it could be a good default though!)\nbut the user wants to have a possibility to byte-for-byte comparision\n(maybe even including CRLFs/LFs)\n\nThanks,\nStefan\n\n> Thanks.\n"},{"id":"330933","messageId":"20171024184809.GC79163@google.com","threadId":"46746","inReplyTo":"20171024000931.14814-1-sbeller@google.com","subject":"Re: [WIP PATCH] diff: add option to ignore whitespaces for move detection only","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-24T18:48:09Z","receivedAt":"2017-10-24T18:48:17Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/23, Stefan Beller wrote:\n> Signed-off-by: Stefan Beller \n> ---\n> \n> diff.c | 10 ++--\n> diff.h | 1 +\n> t/t4015-diff-whitespace.sh | 114 ++++++++++++++++++++++++++++++++++++++++++++-\n> \n> See, only 10 lines of code! (and a few more for tests)\n> \n> We have run out of space in diff_options.flags,touched_flags.\n> as we 1< unused (removed in 882749a04f (diff: add --word-diff option that\n> generalizes --color-words, 2010-04-14)). But that postpones the\n> real fix for only a short amount of time.\n> \n> Ideas welcome how to extend the flag space. (We cannot just make it\n> a long either, as some arcane architecures have 32 bit longs.)\n\nOne simple idea would be to convert the single 'flag' into various bit\nfields themselves, that way if you need to add a new flag you would just\nmake a new bit field. I'm unaware of any downsides of doing so (though\ni may be missing something) but doing so would probably cause a bit of\ncode churn.\n\n\n-- \nBrandon Williams\n"},{"id":"330976","messageId":"xmqqshe8j9we.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171024184809.GC79163@google.com","subject":"Re: [WIP PATCH] diff: add option to ignore whitespaces for move detection only","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-25T01:25:53Z","receivedAt":"2017-10-25T01:26:00Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Brandon Williams writes:\n\n> One simple idea would be to convert the single 'flag' into various bit\n> fields themselves, that way if you need to add a new flag you would just\n> make a new bit field. I'm unaware of any downsides of doing so (though\n> i may be missing something) but doing so would probably cause a bit of\n> code churn.\n\nThe reason why people want to have their own bit in the flags word\nis because they want to use DIFF_OPT_{SET,CLR,TST,TOUCHED} but they\ndo not want to do the work to extend them beyond a single word. \n\nI think it is doable by making everything a 1-bit wide bitfield\nwithout affecting existing users.\n"},{"id":"330977","messageId":"xmqqo9owj9uz.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"xmqqshe8j9we.fsf@gitster.mtv.corp.google.com","subject":"Re: [WIP PATCH] diff: add option to ignore whitespaces for move detection only","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-25T01:26:44Z","receivedAt":"2017-10-25T01:26:51Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Junio C Hamano writes:\n\n> Brandon Williams writes:\n>\n>> One simple idea would be to convert the single 'flag' into various bit\n>> fields themselves, that way if you need to add a new flag you would just\n>> make a new bit field. I'm unaware of any downsides of doing so (though\n>> i may be missing something) but doing so would probably cause a bit of\n>> code churn.\n>\n> The reason why people want to have their own bit in the flags word\n> is because they want to use DIFF_OPT_{SET,CLR,TST,TOUCHED} but they\n> do not want to do the work to extend them beyond a single word. \n>\n> I think it is doable by making everything a 1-bit wide bitfield\n> without affecting existing users.\n\n... but the \"touched\" thing may be harder---I haven't thought it\nthrough.\n"},{"id":"331015","messageId":"alpine.DEB.2.21.1.1710251437090.6482@virtualbox","threadId":"46746","inReplyTo":"20171023213159.eitrjrqrh277advm@aiede.mtv.corp.google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-10-25T12:51:38Z","receivedAt":"2017-10-25T12:52:08Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Jonathan,\n\n[I only saw that you replied to 3/5 with v2 after writing this reply, but\nit would apply to v2's 3/5, too]\n\nOn Mon, 23 Oct 2017, Jonathan Nieder wrote:\n\n> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\n> index 86811a0c35..fd94dd40d2 100755\n> --- a/t/t5601-clone.sh\n> +++ b/t/t5601-clone.sh\n> @@ -384,6 +384,20 @@ test_expect_success 'uplink is treated as simple' '\n> \texpect_ssh myhost src\n> '\n> \n> +test_expect_success 'OpenSSH-like uplink is treated as ssh' '\n> +\twrite_script \"$TRASH_DIRECTORY/uplink\" <<-EOF &&\n> +\tif test \"\\$1\" = \"-G\"\n> +\tthen\n> +\t\texit 0\n> +\tfi &&\n> +\texec \"\\$TRASH_DIRECTORY/ssh$X\" \"\\$@\"\n> +\tEOF\n> +\tGIT_SSH=\"$TRASH_DIRECTORY/uplink\" &&\n> +\texport GIT_SSH &&\n> +\tgit clone \"[myhost:123]:src\" ssh-bracket-clone-sshlike-uplink &&\n> +\texpect_ssh \"-p 123\" myhost src\n> +'\n> +\n> test_expect_success 'plink is treated specially (as putty)' '\n> \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/plink\" &&\n> \tgit clone \"[myhost:123]:src\" ssh-bracket-clone-plink-0 &&\n\nThis breaks on Windows. And it is not immediately obvious how so, so let\nme explain:\n\nAs you know, on Windows there is no executable flag. There is the .exe\nfile extension to indicate an executable (and .com and .bat and .cmd are\nalso handled as executable, at least as executable script).\n\nNow, what happens if you call \"abc\" in the MSYS2 Bash and there is no\nscript called \"abc\" but an executable called \"abc.exe\" in the PATH? Why,\nof course we execute that executable. It has to, because if \"abc.exe\"\nwould be renamed into \"abc\", it would not work any longer.\n\nThat is also the reason why that \"copy_ssh_wrapper\" helper function\nautomatically appends that .exe file suffix on Windows: it has to.\n\nEvery workaround breaks down at some point, and this workaround is no\nexception. What should the MSYS2 Bash do if asked to overwrite \"abc\" and\nthere is only an \"abc.exe\"? It actually overwrites \"abc.exe\" and moves on.\n\nAnd this is where we are here: the previous test case copied the ssh\nwrapper as \"uplink\". Except on Windows, it is \"uplink.exe\". And your newly\nintroduced test case overwrites it. And then it tells Git specifically to\nlook for \"uplink\", and Git does *not* append that .exe suffix\nautomatically as the MSYS2 Bash would do, because git.exe is not intended\nto work MSYS2-like.\n\nAs a consequence, the test fails. Could you please squash in this, to fix\nthe test on Windows?\n\n-- snipsnap --\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex ec4b17bca62..1afcbd00617 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -393,6 +393,7 @@ test_expect_success 'simple does not support port' '\n \n test_expect_success 'uplink is treated as simple' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/uplink\" &&\n+\ttest_when_finished \"rm \\\"$TRASH_DIRECTORY/uplink$X\\\"\" &&\n \ttest_must_fail git clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n \tgit clone \"myhost:src\" ssh-clone-uplink &&\n \texpect_ssh myhost src\n\n"},{"id":"331022","messageId":"CAGZ79kYTYg09A7UbhvyiT0QmRQG5ZJV6v1iaQBNCZY8RTj300A@mail.gmail.com","threadId":"46746","inReplyTo":"alpine.DEB.2.21.1.1710251437090.6482@virtualbox","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Stefan Beller","fromEmail":"sbeller@google.com","sentAt":"2017-10-25T16:18:18Z","receivedAt":"2017-10-25T16:18:27Z","isPatch":true,"sender":{"key":"stefanbeller@gmail.com","avatar":"https://avatars.githubusercontent.com/u/455868?v=4"},"body":"On Wed, Oct 25, 2017 at 5:51 AM, Johannes Schindelin\n wrote:\n> Hi Jonathan,\n>\n> [I only saw that you replied to 3/5 with v2 after writing this reply, but\n> it would apply to v2's 3/5, too]\n>\n> On Mon, 23 Oct 2017, Jonathan Nieder wrote:\n>\n>> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\n>> index 86811a0c35..fd94dd40d2 100755\n>> --- a/t/t5601-clone.sh\n>> +++ b/t/t5601-clone.sh\n>> @@ -384,6 +384,20 @@ test_expect_success 'uplink is treated as simple' '\n>> expect_ssh myhost src\n>> '\n>>\n>> +test_expect_success 'OpenSSH-like uplink is treated as ssh' '\n>> + write_script \"$TRASH_DIRECTORY/uplink\" <<-EOF &&\n>> + if test \"\\$1\" = \"-G\"\n>> + then\n>> + exit 0\n>> + fi &&\n>> + exec \"\\$TRASH_DIRECTORY/ssh$X\" \"\\$@\"\n>> + EOF\n>> + GIT_SSH=\"$TRASH_DIRECTORY/uplink\" &&\n>> + export GIT_SSH &&\n>> + git clone \"[myhost:123]:src\" ssh-bracket-clone-sshlike-uplink &&\n>> + expect_ssh \"-p 123\" myhost src\n>> +'\n>> +\n>> test_expect_success 'plink is treated specially (as putty)' '\n>> copy_ssh_wrapper_as \"$TRASH_DIRECTORY/plink\" &&\n>> git clone \"[myhost:123]:src\" ssh-bracket-clone-plink-0 &&\n>\n> This breaks on Windows. And it is not immediately obvious how so, so let\n> me explain:\n>\n> As you know, on Windows there is no executable flag. There is the .exe\n> file extension to indicate an executable (and .com and .bat and .cmd are\n> also handled as executable, at least as executable script).\n>\n> Now, what happens if you call \"abc\" in the MSYS2 Bash and there is no\n> script called \"abc\" but an executable called \"abc.exe\" in the PATH? Why,\n> of course we execute that executable. It has to, because if \"abc.exe\"\n> would be renamed into \"abc\", it would not work any longer.\n>\n> That is also the reason why that \"copy_ssh_wrapper\" helper function\n> automatically appends that .exe file suffix on Windows: it has to.\n>\n> Every workaround breaks down at some point, and this workaround is no\n> exception. What should the MSYS2 Bash do if asked to overwrite \"abc\" and\n> there is only an \"abc.exe\"? It actually overwrites \"abc.exe\" and moves on.\n>\n> And this is where we are here: the previous test case copied the ssh\n> wrapper as \"uplink\". Except on Windows, it is \"uplink.exe\". And your newly\n> introduced test case overwrites it. And then it tells Git specifically to\n> look for \"uplink\", and Git does *not* append that .exe suffix\n> automatically as the MSYS2 Bash would do, because git.exe is not intended\n> to work MSYS2-like.\n>\n> As a consequence, the test fails. Could you please squash in this, to fix\n> the test on Windows?\n\nThis explanation is in detail and would even make a good commit message\nfor a follow up commit. (Squashing just that line would loose the explanation\nas I suspect the original commit will not dedicate so much text to\nthis single line)\n\n>\n> -- snipsnap --\n> diff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\n> index ec4b17bca62..1afcbd00617 100755\n> --- a/t/t5601-clone.sh\n> +++ b/t/t5601-clone.sh\n> @@ -393,6 +393,7 @@ test_expect_success 'simple does not support port' '\n>\n> test_expect_success 'uplink is treated as simple' '\n> copy_ssh_wrapper_as \"$TRASH_DIRECTORY/uplink\" &&\n> + test_when_finished \"rm \\\"$TRASH_DIRECTORY/uplink$X\\\"\" &&\n> test_must_fail git clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n> git clone \"myhost:src\" ssh-clone-uplink &&\n> expect_ssh myhost src\n>\n"},{"id":"331023","messageId":"20171025163243.xmm7szrkwgblpgcc@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"CAGZ79kYTYg09A7UbhvyiT0QmRQG5ZJV6v1iaQBNCZY8RTj300A@mail.gmail.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-10-25T16:32:43Z","receivedAt":"2017-10-25T16:33:05Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nStefan Beller wrote:\n> On Wed, Oct 25, 2017 at 5:51 AM, Johannes Schindelin\n> wrote:\n\n>> This breaks on Windows. And it is not immediately obvious how so, so let\n>> me explain:\n[nice explanation snipped]\n>> As a consequence, the test fails. Could you please squash in this, to fix\n>> the test on Windows?\n>\n> This explanation is in detail and would even make a good commit message\n> for a follow up commit. (Squashing just that line would loose the explanation\n> as I suspect the original commit will not dedicate so much text to\n> this single line)\n\nI have other changes to make when rerolling anyway (from Junio's\nreview), so no need for a followup patch. Will fix this in the\nreroll today.\n\nThanks for catching and diagnosing this, Dscho!\n\nJonathan\n"},{"id":"331032","messageId":"20171025185812.GB18778@google.com","threadId":"46746","inReplyTo":"xmqqo9owj9uz.fsf@gitster.mtv.corp.google.com","subject":"Re: [WIP PATCH] diff: add option to ignore whitespaces for move detection only","fromName":"Brandon Williams","fromEmail":"bmwill@google.com","sentAt":"2017-10-25T18:58:12Z","receivedAt":"2017-10-25T18:58:20Z","isPatch":true,"sender":{"key":"bwilliams.eng@gmail.com","avatar":null},"body":"On 10/25, Junio C Hamano wrote:\n> Junio C Hamano writes:\n> \n> > Brandon Williams writes:\n> >\n> >> One simple idea would be to convert the single 'flag' into various bit\n> >> fields themselves, that way if you need to add a new flag you would just\n> >> make a new bit field. I'm unaware of any downsides of doing so (though\n> >> i may be missing something) but doing so would probably cause a bit of\n> >> code churn.\n> >\n> > The reason why people want to have their own bit in the flags word\n> > is because they want to use DIFF_OPT_{SET,CLR,TST,TOUCHED} but they\n> > do not want to do the work to extend them beyond a single word. \n> >\n> > I think it is doable by making everything a 1-bit wide bitfield\n> > without affecting existing users.\n> \n> ... but the \"touched\" thing may be harder---I haven't thought it\n> through.\n\nFrom what I can tell the 'touched' thing is implemented as a parallel\nflag field so we would just need to have each flag use 2-bits, one\nfor the flag itself and one for the 'touched' field. Then when using\nthose macros it would just need to update the corresponding 'touched'\nfield as well as what ever happens with the flag itself. It may be a\nlittle more involved than the current scheme but it should be doable if\nwe need to extend the flag space past 32 bits.\n\n-- \nBrandon Williams\n"},{"id":"331301","messageId":"xmqqo9op8o35.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171025163243.xmm7szrkwgblpgcc@aiede.mtv.corp.google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-10-30T00:40:46Z","receivedAt":"2017-10-30T00:40:53Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n> I have other changes to make when rerolling anyway (from Junio's\n> review), so no need for a followup patch. Will fix this in the\n> reroll today.\n>\n> Thanks for catching and diagnosing this, Dscho!\n\nIn the meantime, I've queued this from Dscho; please take it into\nconsideration when you reroll.\n\nThanks.\n\n-- >8 --\nFrom: Johannes Schindelin \nDate: Sun, 29 Oct 2017 16:12:46 +0100\nSubject: [PATCH] fixup! ssh: 'auto' variant to select between 'ssh' and 'simple'\n\nThis is needed because on Windows, if `uplink.exe` exists, the MSYS2\nBash will overwrite that when redirecting via `>uplink`.\n\nSigned-off-by: Johannes Schindelin \n---\n t/t5601-clone.sh | 1 +\n 1 file changed, 1 insertion(+)\n\ndiff --git a/t/t5601-clone.sh b/t/t5601-clone.sh\nindex f9a2ae84c7..534eb21915 100755\n--- a/t/t5601-clone.sh\n+++ b/t/t5601-clone.sh\n@@ -391,6 +391,7 @@ test_expect_success 'simple does not support port' '\n \n test_expect_success 'uplink is treated as simple' '\n \tcopy_ssh_wrapper_as \"$TRASH_DIRECTORY/uplink\" &&\n+\ttest_when_finished \"rm \\\"$TRASH_DIRECTORY/uplink$X\\\"\" &&\n \ttest_must_fail git clone \"[myhost:123]:src\" ssh-bracket-clone-uplink &&\n \tgit clone \"myhost:src\" ssh-clone-uplink &&\n \texpect_ssh myhost src\n-- \n2.15.0-rc2-267-g7d3ed0014a\n\n"},{"id":"331348","messageId":"alpine.DEB.2.21.1.1710301337430.6482@virtualbox","threadId":"46746","inReplyTo":"xmqqo9op8o35.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 3/5] ssh: 'auto' variant to select between 'ssh' and 'simple'","fromName":"Johannes Schindelin","fromEmail":"johannes.schindelin@gmx.de","sentAt":"2017-10-30T12:37:58Z","receivedAt":"2017-10-30T12:38:44Z","isPatch":true,"sender":{"key":"johannes.schindelin@gmx.de","avatar":"https://avatars.githubusercontent.com/u/127790?v=4"},"body":"Hi Junio,\n\nOn Mon, 30 Oct 2017, Junio C Hamano wrote:\n\n> Jonathan Nieder writes:\n> \n> > I have other changes to make when rerolling anyway (from Junio's\n> > review), so no need for a followup patch. Will fix this in the\n> > reroll today.\n> >\n> > Thanks for catching and diagnosing this, Dscho!\n> \n> In the meantime, I've queued this from Dscho; please take it into\n> consideration when you reroll.\n> \n> Thanks.\n\nThanks for sending it out as an email.\n\nCiao,\nDscho\n"},{"id":"332618","messageId":"20171115202516.hduhzsgeoff5a22b@aiede.mtv.corp.google.com","threadId":"46746","inReplyTo":"xmqqpo9d9v5y.fsf@gitster.mtv.corp.google.com","subject":"Re: [PATCH 1/5] connect: split git:// setup into a separate function","fromName":"Jonathan Nieder","fromEmail":"jrnieder@gmail.com","sentAt":"2017-11-15T20:25:16Z","receivedAt":"2017-11-15T20:26:12Z","isPatch":true,"sender":{"key":"jrnieder@gmail.com","avatar":"https://avatars.githubusercontent.com/u/281595?v=4"},"body":"Hi,\n\nOn Oct 24, 2017, Junio C Hamano wrote:\n> Jonathan Nieder writes:\n\n>> +static struct child_process *git_connect_git(int fd[2], char *hostandport,\n>> +\t\t\t\t\t const char *path, const char *prog,\n>> +\t\t\t\t\t int flags)\n>> +{\n>> +\tstruct child_process *conn = &no_fork;\n>> +\tstruct strbuf request = STRBUF_INIT;\n>\n> As this one decides what \"conn\" to return, including the fallback\n> &no_fork instance,...\n>\n>> +\t...\n>> +\treturn conn;\n>> +}\n>> +\n>> /*\n>> * This returns a dummy child_process if the transport protocol does not\n>> * need fork(2), or a struct child_process object if it does. Once done,\n>> @@ -881,50 +939,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n>\n> Each of the if/elseif/ cascade, one of which calls the new helper,\n> now makes an explicit assignment to \"conn\" declared in\n> git_connect().\n>\n> Which means the defaulting of git_connect::conn to &no_fork is now\n> unneeded. One of the things that made the original cascade a bit\n> harder to follow than necessary, aside from the physical length of\n> the PROTO_GIT part, was that the case where conn remains to point at\n> no_fork looked very special and it was buried in that long PROTO_GIT\n> part.\n\nGood idea. Here's what I'll include in the reroll.\n\n-- >8 --\nSubject: connect: move no_fork fallback to git_tcp_connect\n\ngit_connect has the structure\n\n\tstruct child_process *conn = &no_fork;\n\n\t...\n\tswitch (protocol) {\n\tcase PROTO_GIT:\n\t\tif (git_use_proxy(hostandport))\n\t\t\tconn = git_proxy_connect(fd, hostandport);\n\t\telse\n\t\t\tgit_tcp_connect(fd, hostandport, flags);\n\t\t...\n\t\tbreak;\n\tcase PROTO_SSH:\n\t\tconn = xmalloc(sizeof(*conn));\n\t\tchild_process_init(conn);\n\t\targv_array_push(&conn->args, ssh);\n\t\t...\n\t\tbreak;\n\t...\n\treturn conn;\n\nIn all cases except the git_tcp_connect case, conn is explicitly\nassigned a value. Make the code clearer by explicitly assigning\n'conn = &no_fork' in the tcp case and eliminating the default so the\ncompiler can ensure conn is always correctly assigned.\n\nNoticed-by: Junio C Hamano \nSigned-off-by: Jonathan Nieder \n---\n connect.c | 22 ++++++++++++----------\n 1 file changed, 12 insertions(+), 10 deletions(-)\n\ndiff --git a/connect.c b/connect.c\nindex 7fbd396b35..b6accf71cb 100644\n--- a/connect.c\n+++ b/connect.c\n@@ -582,12 +582,21 @@ static int git_tcp_connect_sock(char *host, int flags)\n #endif /* NO_IPV6 */\n \n \n-static void git_tcp_connect(int fd[2], char *host, int flags)\n+static struct child_process no_fork = CHILD_PROCESS_INIT;\n+\n+int git_connection_is_socket(struct child_process *conn)\n+{\n+\treturn conn == &no_fork;\n+}\n+\n+static struct child_process *git_tcp_connect(int fd[2], char *host, int flags)\n {\n \tint sockfd = git_tcp_connect_sock(host, flags);\n \n \tfd[0] = sockfd;\n \tfd[1] = dup(sockfd);\n+\n+\treturn &no_fork;\n }\n \n \n@@ -761,8 +770,6 @@ static enum protocol parse_connect_url(const char *url_orig, char **ret_host,\n \treturn protocol;\n }\n \n-static struct child_process no_fork = CHILD_PROCESS_INIT;\n-\n static const char *get_ssh_command(void)\n {\n \tconst char *ssh;\n@@ -865,7 +872,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\t\t\t const char *prog, int flags)\n {\n \tchar *hostandport, *path;\n-\tstruct child_process *conn = &no_fork;\n+\tstruct child_process *conn;\n \tenum protocol protocol;\n \n \t/* Without this we cannot rely on waitpid() to tell\n@@ -901,7 +908,7 @@ struct child_process *git_connect(int fd[2], const char *url,\n \t\tif (git_use_proxy(hostandport))\n \t\t\tconn = git_proxy_connect(fd, hostandport);\n \t\telse\n-\t\t\tgit_tcp_connect(fd, hostandport, flags);\n+\t\t\tconn = git_tcp_connect(fd, hostandport, flags);\n \t\t/*\n \t\t * Separate original protocol components prog and path\n \t\t * from extended host header with a NUL byte.\n@@ -1041,11 +1048,6 @@ struct child_process *git_connect(int fd[2], const char *url,\n \treturn conn;\n }\n \n-int git_connection_is_socket(struct child_process *conn)\n-{\n-\treturn conn == &no_fork;\n-}\n-\n int finish_connect(struct child_process *conn)\n {\n \tint code;\n-- \n2.15.0.448.gf294e3d99a\n\n"},{"id":"332731","messageId":"xmqq4lptvhch.fsf@gitster.mtv.corp.google.com","threadId":"46746","inReplyTo":"20171115202516.hduhzsgeoff5a22b@aiede.mtv.corp.google.com","subject":"Re: [PATCH 1/5] connect: split git:// setup into a separate function","fromName":"Junio C Hamano","fromEmail":"gitster@pobox.com","sentAt":"2017-11-17T01:12:14Z","receivedAt":"2017-11-17T01:12:21Z","isPatch":true,"sender":{"key":"gitster@pobox.com","avatar":"https://avatars.githubusercontent.com/u/54884?v=4"},"body":"Jonathan Nieder writes:\n\n>> Which means the defaulting of git_connect::conn to &no_fork is now\n>> unneeded. One of the things that made the original cascade a bit\n>> harder to follow than necessary, aside from the physical length of\n>> the PROTO_GIT part, was that the case where conn remains to point at\n>> no_fork looked very special and it was buried in that long PROTO_GIT\n>> part.\n>\n> Good idea. Here's what I'll include in the reroll.\n\nSounds good.\n"}]}